The BCRA insurance policy which covered cavers up until 2003 only asked clubs for numbers of members not individual names and addresses. In the run up to setting up the BCA policy it became clear this approach would not work because the cost was going to be much higher. We knew there was considerable double counting in the BCRA list and asking an individual to pay for separate membership of BCA via 5 separate clubs would provoke a major reaction. So we chose to obtain names and addresses and count people once. (In the first couple of years of BCA we had a mammoth headache driving out duplication. I was surprised at the number of persons who were members of different clubs with same address but slightly different name, cf Bob v Robert.) I can't recall how many people were covered under the BCRA scheme but I suspect it was of the order of 12,000 which is perhaps what mrodoc recalls. I am not sure about 20,000.
I had gained the impression the three year figure was derived in the 70s from the then typical University degree period. The claim at that time was that university clubs were the major source of cavers who kept on caving (like myself). 3 years was sufficient time for an individual to develop enough to have done a reasonable range of caves and decide whether to continue caving or give up on getting their degree.
I do not have direct access to BCA's membership data base but I guess that those BCA member clubs who do not obtain insurance via BCA are fairly small in size and thus the extra cavers is a small number compared to 6000, well under 1000. I suggest the only way you can get an estimate of cavers who have no link to BCA is to go and sit at some cave entrances and ask those who descend. It could well be a moderate sized figure given the way access to kit is now almost all in the hands of shops and knowledge of caves fairly free on the internet. Back in the 60's there were few ladder suppliers and you had to find the name of a guide book before you could buy it. So people then tended to join clubs. Not so now.
BCA membership numbers are given each AGM, see page 14 at
http://british-caving.org.uk/wiki3/lib/exe/fetch.php?media=about:documents:general_meetings:agm_minutes_2016.pdf. The first few years were not typical (DCA supplied insurance for a couple of years). But I would suggest the number of cavers has been slightly increasing since 2007 (4965) up to 2014 (5564). (Not sure where I found 6099 I reported earlier :-[)
I agree with Badlad that the foot and mouth ban in 2001 did cause a number of cavers to give up (as my daughter did). And soon after then, health and safety demands closed a number of university clubs. Fortunately CHECC has been set up since then.
More interestingly, membership numbers are not reused, so the AGM reported that just under 16,000 had been issued since 2004. I reckon that indicates around 500 persons leave BCA and 500 newly join each year in addition to the small yearly increase of some 50 or so over the pasty 10 plus years.
BCA does distinguish between active and inactive cavers. The AGM indicates some 900 non / inactive caving. But as argued earlier is there really a difference between the two?