Sometimes even movement so slight that you can't even see it will cause the whole works to go out of adjustment. A roll pin that has sheared but not completely may allow only a degree or two of rotation on the shaft but that is enough, when multiplied by the moment arm, to make a large difference inside the clutch. One way to detect the sheared roll pin is to put your finger over the end of the shaft and press lightly so you can feel the movement – if you feel any shifting between the shaft and the lever, then you can be the roll pin needs to be replaced.
Since replacing the exterior roll pins isn't really that difficult, and be done in the field if necessary, I would advise against trying to beef up that connection much at all. Let it be the weak link as Bob recommended. Better to replace it three or four times than to split the tractor even once.