Survey Digitisation Software

[Topimo]

Hello hivemind ,

Background
Last night I began the process of digitising (Survex) a paper survey for our upcoming SUSS Tresviso expedition.
This is a paper survey for which the original data had been lost.

There are no doubt countless other caves with paper surveys but lacking the original data where it is impractical to resurvey them in the short term - i.e. large quantities of caves abroad.

Does anybody know of a piece of software that could assist with such a process?

Current Lines of Enquiry
There are numerous xy plot digitisers online, but nothing immediately visible to me for the digitisation of xyz data from plan and elevation views.
I have found a mine-survey vectorising company that appear to also generate 3d models - I will contact them tonight.
I have sent a Facebook message to the "Abandoned Mine Models" page: https://www.facebook.com/Abandoned-Mine-Models-407369656071364/

Any other suggestions would be appreciated.

Likewise if there is a keen coder out there willing to work for peanuts for the good of the caving community, I'm more than happy to work with you and write up a thorough and clear specification. 

In the meantime, back to the protractor and ruler this evening.

Tommy

[2xw]

scan at 600dpi, use ImageJ or whatever to measure your xy points (speedier than using a protractor and ruler)
Plot your points in a proper survex, process them as therion files, put into therion
Import aforementioned scan into Therion and draw over it in both plan and elevation
output as 3d or .kml or whatever you want

That's about as close you're going to get to heads up digitisation for free, if you want automated raster to vector digitisation. R2V does this but its $795 per license

Any identify any problems with this process?

[alastairgott]

I watched an interesting presentation last night, where the speaker talked about Random Forest models. (ask a PHD mathematician cos I don't know).

But essentially I think you might be able to do it by picking a few choice points in the Plan and Elevation, ie start and end, and perhaps some side passages.
 Then if you feed it into a machine learning thing, or ask a mathematician. They might be able to cobble some "best guess" data together for you.

I've attached a link for an extract of a journal which put me onto Random Forests just now. https://3d.bk.tudelft.nl/news/2017/01/17/inferring-heights-ceus-paper.html

[Topimo]

You've misunderstood mate, vectorisation of paper surveys I assume is fairly simple with Inkscape or some other such package.
Limited to "neatening up" of a yellow, ripped old scan. See attached.
We are aiming to create "fake data".

One of the main challenges is having station names defined and controllable on both the plan and profile views. This facilitates the 3d trigonometry to get accurate distance measurements between stations - by hand you have to keep ensure your station locations are indeed the same - tough on complex systems, aim to keep legs at 90 degrees Hoz/Vert (ish depending on your error tolerance) or it becomes complex and fafftacular.

Taking 2D points off and manipulating them is trivial with existing software, it's the 3rd dimension that adds difficulty.

The survey I am working with is a scanned drawing, that had been overlayed with a vector path and the background made plain white and general neatening/legibility.
No need for Therion, this isn't about producing a final vector drawing, it's about going from 2D paper survey to 3D data for viewing widespread subsurface geological features, i.e. beddings and dips, rifts, whatever.

[alastairgott]

I guess what I meant, Is try to simplify the data.

So what I think you're trying to achieve with Layby pot is great, but the transferable skills across to the surveys you'll have for Tresviso maybe limited.

I've only seen surveys of Potholes from tresviso (and only a handful), they would be easier to model with computer software than a Spanish version of layby pot on steroids.

The "fake data" would be far easier to mass produce. If then when you output it to survex or Therion, it looks wrong, then you could manipulate the machine learning model.

The key thing being that you will measure x,y and Z for each of the points which you think are critical to creating a model. If that means passages developed on beddings or passages developed on faults. Then build them into the model.

I'll keep thinking though, as I do get what you're thinking of, like playing snap with cave passages by dotting the plan and dotting the elevation... Hmmm.

[alastairgott]

If you can do 2D easily using inkscape or some other such package, then is this page any use?

https://help.rockware.com/rockworks17/WebHelp/utilities_grafix_summary.htm

They seem to be tools to help people create a 3d fault plane model based on 2d files, but you tell me?

[Bob Mehew]

If you have a digital image giving both plan & elevation which are in correspondence, then many image manipulation programs will identify the pixel location with top left hand corner being (0,0) [format x,y where x is the horizontal and y the vertical].  Then say with you Lay-By Pot example, you can take a spot for say the entrance on the plan part whose location in pixels is say (x1, y1).  If you then drop down to the elevation part, you can locate the same x2 pixel value and then read off the new y value, say y2 which is equivalent to the height / z value.  That means you can translate this point as (0, 0, 0) [that is x, y, z] by simply subtracting the x1, y1, y2 values.  Thus you can get the coordinates for evey other point you locate and measure in pixel units by subtraction. 

Even if the plan & elevation are not in correspondence, the the same process remains applicable provided you select the same location by eye ball.

Using protractor and ruler means you are extracting polar coordinates which are fearsome to transform compared to Cartesian coordinates. 

I (poorly) use GIMP which is freeware and can do this.  However, I have not found a means by which the pixel location can be automatically recorded.  I have a vague recollection that Photoshop / MatLab combination could do this.  Possibly some one has done / could do similar for GIMP.  But it sounds as if Inkscape might do this, though I have never used it.   

If you don't have an image, then use graph tracing paper over the survey on a light box.

[andrew]

Patrick Warren has created a script that takes inkscape points and produces fake tape compass clino data.

https://github.com/patrickbwarren/inkscape-survex-export

I have not used it in anger, and it appears to (well when I played with it) just do it for 2d with the clino set at zero, which is not what you want.

However, I mention it as I suspect it has lots of the coding that is needed to do what I have done manually in the past.

In inkscape (or I guess any other vector program) plot some stations on the plan where you want your fake points to be. Save these points as svg. These points and names can be viewed in a table and with appropriate scaling can be turned into x,y coordinated, usually in metres. (Survex can do the scaling as there is a function to calibrate the unit, however I think that this makes the next bit harder) For each x,y point the z point has to be added manually.  (There might be a way to points on the elevation and then just combine them, but I have never found this worth the effort of trying)

[NB there is/was an extension for inkscape called exportXY, no idea if it works]

Once you have a table of data you can use the nosurvey function of Survex to join it all together, there is a description of how to do this for surface data here, but it is the same for underground. Warning fix without a sd specified are treated as perfect and will not have errors distributed to them so if connecting to other data this can cause weird results.

see - https://survex.com/docs/manual/genhowto.htm - Add surface topography

The other option is to translate the absolute coordinates to relative x,y,z and input the data as Cartesian. This is why the data Cartesian is there and probably the preferred method, but one more step

see - https://survex.com/docs/manual/datafile.htm - DATA - CARTESIAN

[Topimo]

Thanks Andrew, that's along the lines I'm thinking, just with the 3rd dimension! Nice work Patrick

I wonder if anyone here is able to contribute to his Python code?

[alastairgott]

If you can Export the data to excel, then you can work out where the major changes in height occur and therefore you can work out the clino.

Once you know where the major changes in height are, there will be some manual inputting. Add a column for "z", z being the change in height in metres which you have.

before you get the clino, you will probably need to change the xy tape reading given by the GitHub software into an xyz tape reading.
=SQRT(POWER(H4,2)+POWER(I4,2))
where H4 is the change in height which you've worked out.
 and
I4 is the xy tape reading.

To work out the clino reading
=DEGREES(ASIN(H4/J4))
where H4 is the difference in height from point to point
 and
J4 is your xyz tape reading.

not ideal for you, I know, but will get you so far.

[alastairgott]

Had a go at manipulating Patrick's loneranger data by adding some dummy height differences. and in no way shape or form are these accurate whatsoever.


Exported the data from the text file into excel and applied the dummy manipulations and came out with the following survex file. It doesn't seem to have broken the plan view, but has added pretty colours for the heights and made it 3d.

; survex file autogenerated from caveymccavefaceloneranger_cpcj6-2.svg
; embedded image file name caveymccavefaceloneranger_cpcj6-2.png
; generated 06/19/18 22:53:16
; SVG orientation: (0, -1) is 000
; SVG orientation: (1, 0) is 090
; SVG scale: 201.429 is 100 m, scale factor = 0.496454
; SVG contained 7 traverses and 40 stations
; tolerance for identifying equates = 0.2 m

*begin caveymccavefaceloneranger_cpcj6-2
*equate path4288.3 path4298.0 ; separation 0.08 m
*equate path4288.6 path4325.0 ; separation 0.04 m
*equate path4288.8 path4290.0 ; separation 0.08 m
*equate path4290.12 path4292.0 ; separation 0.08 m
*equate path4290.12 path4294.0 ; separation 0.07 m
*equate path4290.17 path4296.0 ; separation 0.04 m
*equate path4292.0 path4294.0 ; separation 0.15 m
*data normal from to tape compass clino
*begin path4288   
*export 3 6 8 
 0 1 6.023586971 107 9.556147091
 1 2 19.28399336 214 -5.953019654
 2 3 12.42753395 114 13.96915242
 3 4 47.83372869 54 9.627734365
 4 5 48.39339211 70 -7.122079309
 5 6 9.592267719 48 -5.98399416
 6 7 24.89347103 110 9.246641056
 7 8 78.39835776 91 11.03044308
*end path4288   
     
*begin path4298   
*export 0   
 0 1 21.29413065 212 5.38931176
*end path4298   
     
*begin path4325   
*export 0   
 0 1 25.36801332 268 6.791644511
*end path4325   
     
*begin path4290   
*export 0 12 17 
 0 1 35.13608402 107 9.832278585
 1 2 54.70914 77 1.047338168
 2 3 41.86626804 55 -31.70074476
 3 4 18.0411779 67 9.57196408
 4 5 32.65866041 69 17.83022257
 5 6 26.5549694 45 10.85293004
 6 7 46.40038901 354 -12.44576846
 7 8 32.4766393 18 42.64158079
 8 9 16.0954186 61 43.11181199
 9 10 6.863242674 70 8.378032266
 10 11 14.33910736 25 -33.91171354
 11 12 15.41557654 3 -22.90576966
 12 13 31.09463137 50 -9.253302897
 13 14 53.69918528 349 4.271863517
 14 15 24.67306426 236 6.983879836
 15 16 25.38724286 266 -11.35861949
 16 17 29.93864559 224 -15.49850051
 17 18 16.4769536 264 -17.6651862
 18 19 15.92206331 257 38.9070862
*end path4290   
     
*begin path4292   
*export 0   
 0 1 27.33354167 259 33.28310283
 1 2 42.12776875 355 -6.816306356
*end path4292   
     
*begin path4294   
*export 0   
 0 1 135.2710054 110 -8.074394564
*end path4294   
     
*begin path4296   
*export 0   
 0 1 23.05862312 208 12.52341745
*end path4296   

*end caveymccavefaceloneranger_cpcj6-2
; end of file

[andrew]

I've been reminded that Compass has a built in tool to do some of this, again, never used it myself. Seems to be aiming to get to tape, compass, clino data which seems a little pointless. On the principle that survey data should always be input in a closest format to the way it was derived Cartesian should be used and less prone to error.

http://cavingnews.com/20130903-new-compass-update-adds-cave-survey-reconstruction-tool

[Badlad]

Hey Topimo.  Only just looked at this.  I recreated 'fake data' for over 100km of very complex cave in Mulu.  I'm afraid the process was lengthy and accuracy very reliant on the quality of the original survey (very good).  Results are very useful to add to known data as I'm sure you realise.  We won the Arthur Butcher survey award for it.  Anyway, Wookey guided us and gave us the correct Survex software but you still have to come up with the xyz by hand - well I did.  The procedure was written up in a Mulu expedition report.  I'll dig it out later and post if you are interested.

[andrew]

Thought that I would have a quick play with this, so did a bit of passage for you.

CentrelinePlain_withguides.svg is a plain svg (as opposed to inkscape svg) where I have added a centreline to the plan, then added guidlines and used this to add the same points to the elevation.
The 2 centrelines from the svg can then be pasted into a spreadsheet Extracted_Data.ods (More care could be taken with the positions and using better snap to get more exact positions.) These lines are a check to make sure there are the same number of points on each line.
There is also a line along the scale bar, which will be used later.
In this survey there is a slight problem that the plan is rotated, to solve this we need to rotate the plan and the plan centreline. Copying the svg numbers from this file the spread sheet can be completed. CentrelinePlain_Rotated.svg

Copy and pasting the correct columns to a survex file, the functions in survex can be used to fix the scale and that drawing programs tend to have y with positive down. This is done by adding

*calibrate altitude 0 -0.567375886524823
*calibrate northing 0 -0.567375886524823
*calibrate easting 0 0.567375886524823

The scale factor is from 40/70.5 and the - is to reverse the upside downiness of the y scale.

Took about half an hour, including trying the extension export XY which seems to give the wrong answers, which is a shame as it would have made the extraction easier.

Hope this helps

Code: [Select]

*begin LayByPot_Right

*calibrate altitude 0 -0.567375886524823
*calibrate northing 0 -0.567375886524823
*calibrate easting 0 0.567375886524823

*data cartesian from to easting northing altitude 
0 1 1.714387 -5.14437 24.72243
1 2 1.845022 1.32043 -0.26727
2 3 4.787179 9.56747 -0.27957
3 4 1.932998 0.92362 0.0123
4 5 2.310694 3.38144 0.0668
5 6 11.797908 5.04603 0.0668
6 7 9.486398 -2.44235 0.0668
7 8 5.721076 1.80287 -0.0472
8 9 3.138022 -1.79646 -0.28683
9 10 3.030036 -0.90341 0.0668
10 11 5.843686 8.91284 0.0945
11 12 4.448697 2.0102 0.0391
12 13 5.492198 1.56649 0.0391
13 14 2.793647 -0.67454 0.17104
14 15 1.259471 -2.22984 0.0847
15 16 4.884715 -3.00128 -0.22812
16 17 4.669743 -1.575 0.14174
17 18 4.41884 -0.59551 -0.14174
18 19 4.08887 2.0092 0.16227
19 20 3.61325 -0.92038 -0.0367
20 21 5.36522 2.59596 0.11405
21 22 2.23288 0.18002 0.0391
22 23 7.40041 -1.91471 0.008
23 24 5.19252 -2.15669 -0.0668
24 25 12.38691 -1.41673 -0.16129
25 26 2.2339 -0.1798 1.56586
26 27 4.96456 -0.51951 2.00788
27 28 3.39342 -2.16177 -0.12218
28 29 1.23631 1.57927 -0.51972
29 30 7.27037 0.19424 2.22257

*end LayByPot_Right
NB the extensions of the files have been changed to get past the restrictions, change them to the correct extensions to get them to work