Climbers talk about roofs, is an overhanging cliff indoors?
This is something of a silly bit of semantics, it is perfectly possible for a term like "outdoors" to have a meaning other than the literal meaning of the constituent words. Caving is an adventurous activity with all the trappings and requirements of other outdoors pursuits. Outdoors is a term used to distinguish those activities from activities done in sports halls etc., or other things (off-roading, dogging) which are carried on in vehicles and the like.
Is football or Wimbledon an outdoors pursuit? I think we use the term in the sense of "The Great Outdoors", i.e. natural, not man-made, as opposed to sports conducted on a specially created surface.