Paper survey to digital tool

[marsrat]

Morning all.
I am currently developing a bit of software which will take a paper survey (.pdf of one) and generate survex/therion data from it.
Effectively recovers data from old surveys.

What features/extensions would you like from this? (I have already thought of creating an intermediate representation language for surveys alongside transpilers for Therion/Survex/QGIS)

Yours aye,
Marsrat.

 

[Chris Curry]

This sounds like an excellent tool Marsrat, thanks for working on this. I'm currently georeferencing survey rasters in QGIS to generate line shapefiles. Not worked out translating back to .svx yet, but I understand it can be done with a bit of faff. Off the top of my head, presumably you'd have a tool to click at either end of a scalebar and enter the length between them? And to rotate about a north arrow? Declination adjustment would be a handy feature, for older surveys which just have a magnetic north arrow. Would your software actually interpret the image, or require the user to trace over, clicking at points along a passage?

 

[andrew]

That would be really excellent.

I did try, nine years ago, it turns out, and was thinking technology had moved on and would be worth another go. However someone else doing it would be even better.
What I learnt, in the hope it will give you a start.
The tracing was not good enough by any means, walls went round rocks and across pitch lines, even when guided, however that seems to have become far better.
Even with vector pdfs it was not that easy. The challenges as I found them

Walls and other symbols in Therion have a direction, from that inside and outside are worked out. In long thin passages Therion pdf output does not care that much, but the 3d model does (No idea if that has changed in the last nine years)
Tunnel does not care about the direction, but does need the line to be linked to the centreline.

I think the key features I would be after are

Centreline
We still use this for length and is often the best visualization for sets of caves, although this is likely to change. It is also the basis for most (all) surveys so future discoveries could then be added. For most caves. doing this in 2d is relatively easy, but adding the z dimension can be very time consuming. Automatically picking up altitude points or taking from the elevation would be ideal, but not all surveys have these. The next approach would be intelligent guessing from total depth taking into account pitch boundaries, drops etc.

Therion (or Tunnel) graphic files
I'm a Therion user so will stick to that. As mentioned direction for lines is important for walls, and critical for things like pitch boundaries. Therion works on none overlapping scraps, so grouping into scraps would be brilliant. Once in Therion, getting it into QGIS is relatively easy and reasonably supported..

I'll keep thinking, hope it goes well

Andrew

 

[Relict]

Personally battling through Qgis to try to make sense of layering old surveys on maps and new surveys. A number of us up north would really appreciate any input on this. Any chance of a course sponsered by CNCC to spread the knowledge?

 

[andrew]

Any chance of a course sponsered by CNCC to spread the knowledge?

Not sure if this was meant for me or the OP.
I don't believe the CSG has the skill set needed to deliver GIS at the moment.
The field meet at the end of the month is likely to spend time on it, however most of us are still at the learning stage of having cave data displayed in GIS.
Personally I'm sure this is the way it will go, just not there yet. Happy to discuss more on a separate thread, keeping this one for the specific question asked.

 

[marsrat]

This sounds like an excellent tool Marsrat, thanks for working on this. I'm currently georeferencing survey rasters in QGIS to generate line shapefiles. Not worked out translating back to .svx yet, but I understand it can be done with a bit of faff. Off the top of my head, presumably you'd have a tool to click at either end of a scalebar and enter the length between them? And to rotate about a north arrow? Declination adjustment would be a handy feature, for older surveys which just have a magnetic north arrow. Would your software actually interpret the image, or require the user to trace over, clicking at points along a passage?

The current plan is to have the software automatically interpet the image to produce passage details alongside a centreline.
The only input the user would give is a .png of the survey, a tracing of the scale bar and the direction of north. The user would also need to select where they want their stations to be (random stations in a passage make no sense) and it'll produce a .svx and .th2.

 

[Relict]

After a lot of faff been able to display a 1966 survey onto modern mapping . Its definitely worthwhile .

 

[complex]

Something similar to this? https://github.com/patrickbwarren/inkscape-survex-export (this is a python script that exports inkscape data to a survex file, so you can import an old survey as an image, trace over it in inkscape and generate a survex data file from the trace).

 

[marsrat]

Something similar to this? https://github.com/patrickbwarren/inkscape-survex-export (this is a python script that exports inkscape data to a survex file, so you can import an old survey as an image, trace over it in inkscape and generate a survex data file from the trace).

Similar, but the aim is to automate the process so the user does not have to trace over passage details + centreline. Thank you for the link however, even for curiosity/research purposes it will be useful

 

[Relict]

Hi Marsrat,
Given a long complex survey in plan only, where the data is lost, there is very little reliable elevation data and few GCP's, I came up with a proposal to resurvey the whole thing Centreline only, a project more achievable in our time than a full resurvey.
We were thinking then how could we get the pdf or .PNG of the old survey to display in Therion as a .th2 allowing the survey to be re drawn and compiled in Therion. I was wrestling with the challenges of scale and orientation but you seem to have nailed that. I'm guessing that your software would output a .th file, although if the pdf is without any reliable elevation data it sounds like a difficulty unless I'm misunderstanding. It's a great idea btw.
Would your software output a .th2 that could be merged with new Centreline data in the .th file or am I out with the fairies here?
Otherwise perhaps we could cut and paste the new Centreline .th data into the latter file, would that be feasible? I guess I'm risking being sent to the bottom of the class but hey ho.

 

[Space Doubt Caver]

Morning all.
I am currently developing a bit of software which will take a paper survey (.pdf of one) and generate survex/therion data from it.
Effectively recovers data from old surveys.

What features/extensions would you like from this? (I have already thought of creating an intermediate representation language for surveys alongside transpilers for Therion/Survex/QGIS)

Yours aye,
Marsrat.

If you need any help coding such a program I'm a full stack developer and would be happy to help.

 

[Steve Clark]

XTherion editor is already designed for the task you describe Relict. You can re-survey the cave with fresh centreline data, ensuring that you put stations at common locations that are identifiable on the existing paper survey. This may require splay shots to identifiable features.

You can then import the existing survey as a background image in Xterion Editor, identify the common stations with correctly labelled station points and then directly trace the walls using the walls line command. Therion will then automatically morph and scale the result for you. This can be quite quick if you do it as a single scrap.

If you can't re-survey, then the import can be scaled quite easily by referencing the scale bar or grid on the existing survey if present, our manually drawing on a, say 50 or 100mm scale bar yourself at the appropriate scale before you scan it.

 

[Relict]

XTherion editor is already designed for the task you describe Relict. You can re-survey the cave with fresh centreline data, ensuring that you put stations at common locations that are identifiable on the existing paper survey. This may require splay shots to identifiable features.

You can then import the existing survey as a background image in Xterion Editor, identify the common stations with correctly labelled station points and then directly trace the walls using the walls line command. Therion will then automatically morph and scale the result for you. This can be quite quick if you do it as a single scrap.

If you can't re-survey, then the import can be scaled quite easily by referencing the scale bar or grid on the existing survey if present, our manually drawing on a, say 50 or 100mm scale bar yourself at the appropriate scale before you scan it.

Thanks Steve, as you rightly indicate reference station locations , i had anticipated that choosing these would need planning to ensure locating them accurately. I hadn't realised it was possible to import a scan of the survey into Therion but had experimented with georeferencing it in Qgis.

 

[Steve Clark]

Thanks Steve, as you rightly indicate reference station locations , i had anticipated that choosing these would need planning to ensure locating them accurately. I hadn't realised it was possible to import a scan of the survey into Therion but had experimented with georeferencing it in Qgis.

The image morphing in therion is an exceptionally powerful tool if you can figure out how to use it! It allows even poor quality sketches to be used effectively whilst maintaining dimensional accuracy of the core survey.

I struggled to get it to work initially. You need to first import a 'background image' using the command in the edit top menu, then create a scrap, then use the left hand side tab menu / scraps / 'background sketches' / Insert button to select the image now you have imported it. This will associate it with the scrap. You can then draw on the relevant stations within that scrap etc.

This is a completely different methodology to drawing over a centreline xvi and a grid.

Some, but not lots of info in the therion book. See page 38 & 98-100 for an example. https://therion.speleo.sk/downloads/thbook.pdf

 

[Steve Clark]

Oh and if you are scanning an existing survey without centreline data and it's not aligned to paper north = grid north, then it's worth drawing on your scale bar (with a pencil) in the direction of north / grid north before you scan it. Then when you use the scale feature, it will align the model properly in the internal maths.

You can then choose how to align it in the final output later to suit the paper size and it will work correctly with the therion north arrow and any superimposed grid etc.

The 'gamma' tool is very useful too. Like a brightness/contrast button and makes it much easier to trace even really messy raw survey sketches or poor quality scans.

 

[Relict]

Many thanks Steve, as ever in Therion the devil's are in the details. I shall experiment!

 

[Relict]

Great post Steve, made progress and I have the survey in the T map editor. Is there a tip to using the tab menu / scraps / 'background sketches' / Insert button? I found it unresponsive but then somehow the background survey title appeared in it and I'm still not sure how. I have the latest Therion version. Gamma tool great btw.

 

[andrewmcleod]

Worth remembering that Therion predates 'digital surveying' methods (i.e. DistoX and PDA) and would presumably have been originally intended for exactly this workflow (entering manually calculated centerline data and drawing onto scanned paper drawings).

 

[alastairgott]

Morning all.
I am currently developing a bit of software which will take a paper survey (.pdf of one) and generate survex/therion data from it.
Effectively recovers data from old surveys.

What features/extensions would you like from this? (I have already thought of creating an intermediate representation language for surveys alongside transpilers for Therion/Survex/QGIS)

Yours aye,
Marsrat.

Ability to create prime points, ie points which you wish to join on another cave survey which the original was missing.

I’ll have a think for more, but am mindful that my own extended elevation surveys would be difficult to back engineer to ’original’ data.

Close as damn it would be great!!! (Mindful of time, drunkenness and rambling of post)

 

[Rob]

Yer another reason why people should be discouraged from using extended surveys. Projected wherever possible 👍