Making survey data available

Rhys

Moderator
I'm keen to make any survey data that I control freely available for other people to use. Anyone working in caves nearby now or in the future can benefit by being able to add it to their own models without having to chase me down or even re-survey the cave. To this end, with resepct to Cabsav, an ongoing project in the Forest of Dean, I've recently put all my scanned survey notes, data, 3D model files and the survey itself up on the wiki as part of the cave page: http://www.ukcaving.com/wiki/index.php/Cabsav

I've noticed some other club and expedition websites doing similar things. The Garimpeiros club for some sites on the Black Mountain ( http://www.mlaverty.plus.com/gog/survey.htm ), Oxford UCC for parts of Ogof Draenen ( http://www.oucc.org.uk/draenen/survey/draenen_survey_main.htm ) for example.

What do other people do and think about this? It's all quite ad-hoc and lots of different formats are being used. I believe this sort of issue has been talked about in the past within the Surveying group of BCRA, though I've not heard any thing for ages. Is any thing happening on this at the moment?

Anyway, the UKCaving wiki is available to store data if people wish to use it. I've set up a page where this information can be collated: http://www.ukcaving.com/wiki/index.php/Survey_data_repository . At the moment I think the wiki is only configured to handle Survex files (.svx and .3d) and possibly Google Earth files, but if there was a demand, other formats could be added. Common formats like JPEGs and PDFs can also be used, of course.

I'm aware that people may be uneasy about sharing sensitive data when projects are ongoing, but once things settle down, surely it's a good idea to get this stuff out there and saved before the key players, retire, die, emigrate, disappear...

Any thoughts?
Rhys
 

TheBitterEnd

Well-known member
I know this doesn't immediately address the original point but you did ask ? I can't help but think that what is needed is an extensible open standard for cave data. Not only do we have Survey data, we have conservation data (e.g. various cave monitoring projects), archaeological data, flow & dye trace data, CO2/bad air, ecological/biological data etc. As we learn more we will record more but it would be good to have all this data available in a way that relates it together ? a Cave Information System (similar to GIS).

For a variety of reasons some people just do not want to share data and that is fine if they record in and store in a standards compliant way. When they do feel able to share their data it can be made publically available and would just integrate into the bigger picture.  Also as many people as wanted to could hold data, and where appropriate share via the web or other means. Trusted third parties could even hold data privately, guaranteeing it would not be shared until permission is granted.

It's ambitious but doesn't have to be completed in a single step, it could be built up in stages.
 

Juan

Active member
"For a variety of reasons some people just do not want to share data"
Can someone list any good reasons why people would not want to share data?
 
Juan said:
"For a variety of reasons some people just do not want to share data"
Can someone list any good reasons why people would not want to share data?
Quite a number of clubs (and individuals) want to retain the survey data so that they can publish and sell surveys, bringing in a much needed revenue stream. If the data is available free online, who's going to fork out ?5 for a paper survey?

 

SamT

Moderator
JessopSmythe said:
Quite a number of clubs (and individuals) want to retain the survey data so that they can publish and sell surveys, bringing in a much needed revenue stream. If the data is available free online, who's going to fork out ?5 for a paper survey?

old_hat-p148275001235351232z74s0_152.jpg
 

Rhys

Moderator
Old hat! Yes, it has been talked about before...

Jessop: The raw survey data itself is a very different thing to a saleable 3 or 5 sheet completed survey you can hang on the wall. I don't believe releasing the raw data affects the sale of finished paper surveys. There's a hell of a lot of time and effort involved converting one to the other.

I can totally respect withholding the data while a cave is being pushed to prevent piracy of leads.

Rhys
 

bubba

Administrator
JessopSmythe said:
True, but people still buy them.

Rhys said:
The raw survey data itself is a very different thing to a saleable 3 or 5 sheet completed survey you can hang on the wall. I don't believe releasing the raw data affects the sale of finished paper surveys. There's a hell of a lot of time and effort involved converting one to the other.

I'm with Sam and Rhys here - making survey data available for people to see/use is unlikely to affect sales of paper surveys...maybe a few people would go to the trouble of getting a A0 copy printed but I reckon most people would just find it easier (and probably just as cheap) to shell out the required fiver.

I suppose somebody unscrupulous could try to set up a small enterprise that used the collective data to sell surveys on ebay or something but I reckon the caving world is small enough that word would soon get round and they'd be ostracised. Most people are aware that buying a survey direct from a club provides it with much needed revenue and would buy accordingly.


PS
JessopSmythe said:
Sent from my HTC Desire S using Tapatalk
Did you know you can turn this automatic signature message off?  Go to settings/tapatalk signature if you so wish.
 

Rhys

Moderator
To be clear, generally I'm talking about the release of raw survey data and notes, not a file with the finished and drawn up survey that could be directly printed and hung on the wall in a conventional fashion (though if people are happy to release that, great!).

Sent from my computer.
 

andrew

Member
I have been trying to set something like this up for years, decades even. At last the technology is here to do it.

An extensible open standard for cave data has been looked at before, and never seems to happen, so in the mean time we need to do something about it.

Although a little dated I think this still captures my feeling (last piece, sorry it is in a rubbish format)

http://www.chaos.org.uk/survex/cp/CP25/CP25.htm

I suggest, and working with MCRA and BCA to achieve this that you let people store it in whatever format they like so long as there is a way to translate it at a later date if the program used goes out of use. It could be stored in a repository, I am working on SVN which is what I use for the Cheddar Catchment data and also contains County Clare and other areas like Fairy cave quarries. (please don't start a debate on the best repository) Using a Change control repoitory gives other advantages as well. Such as a backup and many people being able to work on the data, without a complex exchange system.

Anyway BCA have kindly given me the rights and space on there server to set these up, I just have not pulled my fingure out yet (caving getting in the way of computing) but now the nights are drawing in I will get on with it. BCA seems a good place to hold the data, and increases the chance of it not been lost, a good function for a national body.
The alternative would be to use a third party like KnowledgeForge. A cave data repository is already set up

http://www.knowledgeforge.net/project/sesame/

which granted has been up for a long time and working, but I personally would feel better if it was under the cavers national body, KnowledgeForge may just stop (one would hope not tho')
Kforge is possibly the solution to use on the BCA site, in terms of future administration.

Now the real data that gets lost is the bits people keep secret for various reasons for years, which inevitably turns to ever. Well a repository could help here as well, the data is stored on the BCA server and gets all the backups etc, but does not get published. If the site is not used for say 5 years the data becomes public, so it is up the the secret holders to use it. Okay long term, but at least the data is saved. I believe this is possible with Kforge (not the 5 year bit). A variation on the CP idea

As for selling surveys for ?5.
Now another solution to bring in profit is if Post print on Demand was available somewhere, then most people would just click print, pay the money and wait a couple of days for it to arrive. For each print the club/individual would get a percentage, it would also reduce the risk as the club would not have piles of them it had to buy in the first place, and increase the availability so increase the sales, winners all round, and the data is also there for everyone.
 

caving_fox

Active member
Now another solution to bring in profit is if Post print on Demand was available somewhere, then most people would just click print, pay the money and wait a couple of days for it to arrive

Would love this!
 

Rhys

Moderator
TheBitterEnd said:
Is raw survey data ever in any other form than bearing/distance?

"Standard" centreline data would be: distance, bearing, inclination. But, apparently cave diving surveys use a depth gauge rather than a clinometer, so the data would be: distance, bearing, depth. Clearly different calculations are required to plots these. I suppose what we're really talking about when we say "format" though is "software format". That is what software package are you using to plot the data. Each package will use different notation for various things, recognise different commands etc.


andrew said:
I have been trying to set something like this up for years, decades even...

Thanks for the Compass Points link Andrew. (A more readable format is in Microsoft Word here: http://www.chaos.org.uk/survex/cp/CP25/CP25.doc ). I think I must have read it before some time ago. Food for thought. It's obviously quite a bit out of date now given the way technology has moved on. What happened to those microfilms, can anyone read them now?!

I like the idea of the stuff becoming public x years into the future if no objection is registered from the surveyors at that time.

As Andrew says, I don't think a standard format is necessarily required so long as we can actually hack into the raw data at some point in the future if the software disappears.

BCA would be an obvious body to hold data and I would see it as extremely useful way of spending some of the Association's money.

I looked at the Knowledge Forge page, but couldn't really understand how it was meant to work or what it was meant to do. Do I need to sign up some how to see what data is in there?
 

robjones

New member
Rhys said:
TheBitterEnd said:
Is raw survey data ever in any other form than bearing/distance?

"Standard" centreline data would be: distance, bearing, inclination. But, apparently cave diving surveys use a depth gauge rather than a clinometer, so the data would be: distance, bearing, depth. Clearly different calculations are required to plots these. I suppose what we're really talking about when we say "format" though is "software format". That is what software package are you using to plot the data. Each package will use different notation for various things, recognise different commands etc.

How about raw data in the form of rectangular coordinates?  i.e. 'x' and 'y' being relative to the National Grid (or any other grid) and 'z' being relative to Ordnance Datum (or any other specified datum).
 

Rhys

Moderator
robjones said:
How about raw data in the form of rectangular coordinates?  i.e. 'x' and 'y' being relative to the National Grid (or any other grid) and 'z' being relative to Ordnance Datum (or any other specified datum).

Entrances and surface points are specified in raw data by such cartesian coordinates. For underground survey stations, personally, I wouldn't call cartesian coordinates "raw" data because they need to be calculated using the tape/comapss/clino measurements from a known point on the surface. Other people may view it differently...

What I've earlier termed a "3D model" is effectively all the point coordinates and how the points are connected together to make a centreline. These should ideally be made public too!

Rhys
 

graham

New member
Rhys said:
For underground survey stations, personally, I wouldn't call cartesian coordinates "raw" data because they need to be calculated using the tape/comapss/clino measurements from a known point on the surface. Other people may view it differently...

You are quite correct. "Raw data" is the stuff actually collected in the cave, in whatever form. In the UBSS Library we still have boxes of "muddy books" which were the ones taken down caves in the 1950s. I haven't had to look at them, but if we had found problems with the neat data when we computerised it in recent years, then we could have gone back and discovered whether there had been simple transcription errors.

That sort of stuff simply does not exist, however, in our latest survey projects, carried out using DistoX & PDA. However the repository does contain the 'raw' files downloaded from the PDA as well as  the Therion files into which the data has been imported.

Once the data has been crunched and verified, the Therion files are, really, all that is needed as the data can be read from them with a text editor even if the relevant program is not available. What is important here is to ensure that the exact form of the data is documented - preferably within the file - Not everyone does that, if they use the default settings, but who will remember what the default was fifty years from now.

We try - and usually succeed - to work according to two principles: document everything and never discard data.
 
Top