I have checked the situation in Derbyshire and there are surprisingly few caves affected - most of the CRoW land is on the sides of the steep limestone dales but there are some areas near Castleton which are CRoW land: Winnats Pass, Eldon Hill (including Eldon Hole which has no restrictions), Conies Dale and Ox Low. In Winnats Pass Old Tor Mine is owned by the National Trust but kept locked to prevent mineral collectors removing the few remains of Blue John still there.
So most of the favourites, such as P8, Dr. Jackson's, Perryfoot, Gautries, Giants, Oxlow, Maskhill, Rowter, Nettle, JH, Titan, etc. are not on CRoW land and would not be affected - most require that you call on the farmer and pay a trespass fee. Areas around Monyash: Knotlow, Hillocks, Whalf Pipe, Water Icicle, etc. are not on CRoW land but the shafts are protected by lids accessible with a spanner to undo the bolt.
In Lathkill Dale, which is CRoW land, there are no restrictions on entry to Lathkill Head Cave, whilst Boulder Pot, Ricklow the Cales Dale caves and Cascade have gates accessible with a spanner. Lathkill Head Upper Entrance, on private land, has no restrictions but cavers should have a BCA insurance. Lathkill Head Garden Path entrance was excavated on private land and is kept locked at the request of the owner and NE and accessible for scientific purposes only.
The situation in Dovedale and the Manifold Valley is complicated by being on National Trust Land and also nature reserves but, on the whole, where there is a gate to safeguard children or stock, the site is accessible with a spanner.
It is possible to start a dig in an SSSI and there is a helpful leaflet "Cave Digging on SSSIs, a Guide to Good Practice" issued jointly by DCA and Natural England, available f.o.c. from DCA and also on the DCA website for a free download.
What is evident from this is that the situation is by no means as cut and dried as some people have been given to understand - in some cases there are tight restrictions or no access at all on CRoW land, in other cases it's a free for all. This applies even in the Dales: for example, there seems to be no restriction affecting caves on the west and southwest sides of Ingleborough/Simons Fell, whereas on the east and south you are expected to ask permission from the farmer or from Ingleborough Estates.
CNCC's website has a useful list of caves and their access requirements so it is relatively easy to check the situation and relate it to the CRoW areas on the maps. Similarly, DCA has issued the "Peak District access and Rigging Guide" (also downloadable from the website) so, again, it is easy to check.
I don't know the situation in other caving areas of England (Shropshire, Forest of Dean, Devon & Cornwall, Dorset), which would still come under Natural England. I am also not sure how the law applies to mines which are effectively treated as caves as opposed to natural caves - something which affects the Peak District in particular.
Wales is probably now in a different situation because of the removal of the single statutory body, formerly the Countryside Council for Wales, and its replacement by an amalgamation of 3(?) other bodies which may not yet be clear as to how they are to work.
Clearly the law relating to access to caves on CRoW land in England is not being applied consistently and there should be grounds for challenging the interpretation by some landowners. It might be sensible to start by querying the situation in England, (leaving out Wales for the moment) simply because there is an existing statutory body with which some of the regional councils already have a useful working relationship.