This has happened in the Peak District where mineral collectors have gone after mines and also caves in search of specimens.
It is not unknown to find a mine with lots of newspaper parcels strewn around the floor and, when these are unwrapped, they turn out to be unwanted mineral specimens - the collectors hacked out everything they thought might be worth something but then sorted through and discarded items thought to be of lesser value. So all the damage had been done but they didn't actually want all that they had hacked off.
It also happened in Giants Hole a few years ago: a large curtain was smashed relatively close to the entrance but in a side passage which was not believed to well known or easily accessibly. Some of us took the remaining broken pieces out of the cave with the thought that we might be able to repair the formation and replace it in situ (as was done with the Lancaster Columns) - but we then realised that most of the pieces were missing altogether; collectors had smashed it and taken the best and largest chunks and left the rest lying around. Sadly the cave is the property of an "absentee landowner" who collects money from people who visit the cave but does not restrict entry to "cavers" - it's well known, close to the road and is known to be very easy going for several hundred meters so that it is visited by many non-cavers (stag party groups have been met there) and also by badly led and inappropriate "adventure groups".
Giants can't be gated or protected in any way now as the difficult sections of the entrance series were blasted away by the then owner in 1967 in an ill-advised attempt to turn it into a show cave. It was done without the knowledge of cavers while there was a foot-and-mouth epidemic on and cavers were obeying requests to stay away, there was no statutory protection we could have called on to protect the site and, in the event, the County Surveyor refused permission for the show cave on the grounds of poor access to any car park proposed.
So it's not just "cavers" who can be vandals - some cave owners can be responsible for encouraging damage by their actions and don't seem to care about the consequences.
I don't know how you get the conservation message across in this situation.