Good info from Antwan, there.
Antwan said:
joining a club is one of the best ways to make friends and get introduced to the sport proper.
More correctly, joining a good club ... etc..
Another alternative, especially if you want to do caving to suit your own schedule, on a good caver:leader ratio and therefore managing to do lots of underground stuff (rather than faff, cook breakfast, faff, drink tea, faff and possibly not actually go caving whatsoever) is to engage the services of a qualified leader who will provide all the equipment (often of a very high quality), take you on a tailored trip suitable for your ability/requirements/aspirations and make you feel like you are king for the day, rather than prospective tea-boy.
http://www.caveinstructor.org.uk/instructors
Clearly, being based in Suffolk means you're going to be looking at spending some moolah travelling/staying in a caving region so at the very least you'll want to be guaranteed that you'll get some excellent introductory caving achieved while there so this approach could well be a wise starting point - you can save heaps off your budget by staying at a caving club hut, usually around ?5/night (move around to check out the facilities available at a variety of them to start off with as you may well end up joining one or more of them in the longer run).
Some clubs might even be welcoming towards you so you could mix-and-match, doing a club-run trip with borrowed kit on one day of the weekend and a professional trip on the other.
If you hire an instructor you'll also be able to get two trips back-to-back in a single day; this isn't the cheapest way of getting underground but is the most guaranteed. If you inform an instructor that you intend to take up caving as a hobby your introductory trips could also serve a dual purpose insofar as they can be tailored to introduce a training element, providing underpinning knowledge and covering a range of useful topics/techniques as a primer.
With this in mind you could be doing yourself a huge favour if you check out this thread:
http://ukcaving.com/board/index.php?topic=13026.msg168027#new
NB: Hiring good quality caving gear for a day is usually around ?15 so a weekend of caving including all gear at ?60/day represents excellent value as it effectively equates to only ?45 for the instruction/day. I think you'd be hard pressed to get better value for money than this present offer so my recommendation would be to book a place.