As far as I can tell, and from talking with urbex'ers and protestors, there are two types of trespass:
Trespass to Land
This is a civil offence and this is what Bob is talking about. If you wander on to someone's land, or dump your broken washing machine in their field, then you can be guilty of trespass to land. The police have no rights to get involved and you certainly shouldn't be arrested for this. The landowner may decide to sue you for damages or loss, but these will be hard to prove if you just wandered onto someone's land via an established path.
Aggravated Trespass (AT)
This is a criminal offence and many police officers will relish arresting you if they have the slightest suspicion that you've committed this. This is one of the justifications our public servants use to arrest you, briefly detain you, and then dearrest you once the occasion to exercise your lawful rights has passed. To be actually guilty of aggravated trespass you need to be, a) Trespassing and b) "Intentionally obstructing, disrupting, or intimidating others from carrying out ?lawful activities?". Essentially, if you are asked to leave and don't immediately leave then you may be guilty of AT and face criminal prosecution.
https://www.justanswer.com/uk-law/8l3c3-sue-trespass-land-damages-done-land.html
https://www.lawteacher.net/lecture-notes/tort-law/trespass-to-land.php
https://greenandblackcross.org/guides/laws/5-trespass-aggravated-trespass/
http://www.cps.gov.uk/legal/s_to_u/trespass_and_nuisance_on_land/
Of course, for caving this raises some interesting questions.
First, assuming you were caught on CROW land, you could take it to court and launch a legal defence that way. However, this could be expensive and certain portions of the caving community would probably put a fatwa on your head.
Second, you can legally walk to the entrance of a cave on CROW land and at least as far as the daylight extends to. Therefore, you could not be accused of trespass if you were caught after exiting the cave, assuming you left as soon as possible and didn't interfere with anything. Nor could you be accused of trespass if you were caught walking to the cave unless the landowner was willing to walk to the cave with you and witness you enter.
Thirdly, to be charged with aggravated trespass on CROW land, the landowner would somehow have to prove that you were interfering with their lawful activities while you were underground. If they had some cheese maturation facility in the cave, then fair enough, you might get in the way. Perhaps if they wanted to go caving themselves? Maybe they want to avoid paying for disposal fees and are planning to dump some dead sheep down the exact pothole that you're planning to visit? Generally though, landowners aren't going to be doing anything lawful in a cave and so it would be impossible for you to interfere with it.
TLDR; As far as I can see, there's essentially no way to receive a criminal charge for caving on CROW land. Provided you don't damage anything, there's also no way to end up being forced to pay more than a small sum in a civil case. Furthermore, I think, in the highly unlikely event that someone was sued for caving on CROW land, many cavers (including myself) would be happy to contribute to any legal costs or judgements against them.