The furthest place in England and Wales from a known cave in limestone/chalk is probably Happisburgh on the north Norfolk coast (nearest known cave is I think Water End swallow holes in Hertfordshire which were dug open many years ago, but now infilled). The are natural caves in the Chalk near Dover and Chatham. That said, there are likely to be other as yet undiscovered small caves in the Chalk of East Anglia. I'm excluding sediment filled dissolution pipes and deneholes. The other contender is the Isles of Scilly, which is quite a way from the nearest limestone caves around Plymouth. In Scotland, then the Shetland Isles is the clear winner. On the mainland it is probably somewhere between Fraserburgh & Peterhead.