Way back when pre 2000, two exercises were carried out to obtain thougths of cavers by distributing questionnaires and having free mail post back. These showed a fair percentage of responders who were members of more than one club. Up until 2004 BCRA ran its insurance on the basis of just asking each club how many members it had without requesting the names. This gave values which I recall topped 10,000, but I am not overly confident about that recollection. So this means the number includes those who were members of more than one club. BCRA were able to take this relaxed approach because the individual cost of insurance to them was around ?1 to ?2, so every one was relaxed about paying twice. Unfortunately when BCRA's insurer stopped covering them, the premium of the cover obtained by BCA was significantly more. (It came at a time when insurance companies suddenly realised they were covering a sizeable amount of higher than thought of risks and so started to charge risk related premiums.) So BCA had to work on individuals and remove duplication from the list of cavers it was to cover under its insurance policy. (There were some other very important factors but I won't digress.) Membership of BCA has climbed since it started in 2004, but I am of the opinion that this is more due to clubs and cavers joining in, rather than new cavers arriving on the scene. Membership of BCRA did drop at the same time but I think that to not be relevant to the question.
The other reasonably well proven fact is that the number of university caving clubs has dropped. But as we are only talking of a few tens of clubs in total, I am not sure this makes a lot of difference to overall numbers, though it is thought to have make an impact on the influx of highly active cavers onto the caving scene. Likewise, the number of clubs around is not overly helpful since although there are some very large clubs (300 plus membership), the bulk of clubs are very small and the figure pre 2004 used to bounce around a lot, I suspect more due to Treasurers deciding not / forgetting to pay subs to NCA. The numbers of clubs in regional caving councils also varied and grossly exceeded the number in NCA for a range of reasons. That also has changed with BCA, with virtually all clubs who are members of regional caving council also being members of BCA. (I can't cite figures as I have just done a Data Protection Act assessment of my information holdings and deleted a whole lot of stuff since I am no longer involved in the finances of BCA. It would also take a lot of effort to remake the data from that which I think is now held by others.)
I will say I have not looked into potential other sources of numbers like number of people entering some caves; but often the caves which have such data were / are already linked into the BCRA / BCA insurance schemes, so I am not sure they would be of much extra value. So it is very difficult to get a handle on the ratio of cavers who are members of clubs / BCA to those who are not members.
Thus although cavers will say there were more around way back when, I believe the numbers lost are probably much smaller than many think. My guess for what it is worth is that the peak number was around 7,000. We can say thanks to Cookie, that there are at least 5300 cavers (albeit some are non active) last year and that there must be some others who are not members of BCA.