I'd like to second Peter's words of caution in relation to old mines and give an example why people need to be sure what they are proposing to remove is definitely litter. Following a PDMHS trip into Hanging Flat mine earlier this year, comments were made about there being a lot of rubbish in the place and the desirability of removing this. This was the subject of some discussion between John Barnatt and myself as we both recalled that a lot of what was in there had been left behind by the spar miners working it in the 1970s, but a date was arranged for a party to go and inspect what was there with a view to removing any rubbish. I couldn't make this myself but had arranged to call in later that evening and pick up any surplus bags of rubbish for a tip run, but there were none at the entrance. Carrying on up to the Miners to meet with John, I found that very little material had in the end been removed; other than some obvious rubbish near the entrance and fragments of the infamous Peak Practice polystyrene boulders that had been taken out, most of the contents of the place were indeed mining-related and so had been left where they were.
It's easy to imagine what could have happened here if a less knowledgeable group had done a cleanup, namely the archaeological interest of the site would have ended up damaged. My advice is simply to be mindful of what you are doing, and stop if you're not sure - you can always come back and finish a job off later having sought advice/clarification.