I think the problems from HSE's viewpoint would be that there's nothing stopping them falling out of the bucket, and the hydraulics on the loader won't have been fitted with the kind of failsafe systems that a machine designed to carry people would have, so they could have been tipped out or the bucket drop to the ground.
The farmer says he did a risk assessment, but I wonder if he wrote down the result. HSE's expectation is that a risk assessment will be done, and the risks reduced to ALARP. While they don't say it has to be written, they do expect proof that it was done. I'm guessing that a risk assessment involving the workers, who would therefore be witnesses to it, might be acceptable.