Of interest may be the following quote, from a document called Commons Toolkit Fact Sheet 13, published by Natural England in 2010:
13.4.2 The access afforded by CRoW
CROW provides the public with a right of access on foot only. This includes running, climbing,
photography, having a picnic, and bird watching. Wheelchairs are also allowed.
It does not include cycling, horse riding, camping, or rock climbing.
Have you spotted the oddity here?
A report commissioned by Natural England and first published in June 2009:
Natural England Commissioned Report NECR012
Countryside and Rights of Way Act, 2000 Part I: Access to the Countryside
actually has a recommendation that caving should be one of the activities allowed but as this was only a "recommendation" and it wasn't followed up at the time nothing changed.
The matter has been taken up by BCA and a CRoW Working Party has been set up to look into this and report back. (Details are in the Minutes of the 2013 BCA AGM.)
It does seem likely that a change in the law (or a change in the interpretation of the law) would be required to alter the situation. In order to do this, it requires that BCA itself, as the "Representative body for British Caving", agrees that a change is necessary. This is why the Working Party is looking at a number of related issues, including conservation as well as access issues.