We were there in October for a relaxing week.
There is one cave we found down south near Gardiki, very small, a bit of old flowstone inside, more of a rock shelter. Easy walk up from the road.
There are some caves north of Perithia.
One was marked on the map we had but we did not go there.
There are a number of rock shelter type caves turned into Shrines.
One of which can be found past a very nice isthmus beach on a headland south of Afionas, accessible by walking the path from Afionas or by boat from Ag Georgios or Arillas. Nice place for a swim.
Another shrine/rock shelter/cistern can be found at the castle of Angelokastro.
We did not do any of the boat trips which visit the sea caves up north west.