I created a set of tools for mining database visualisation and spatial analysis

Development
,
1

Hello everyone,

I work in the mining industry. Im based in France, near Grenoble, but work mostly with environmental and geochemical data of mining projects from South America. At the start of each project, we have to intersect 3 important datasets: the 3D Block Model of the mine (a 3D mesh of cubes with the geo-statistical data of the mine like Lithology, Alteration, Mineral Zone, Ore, metals, other…), the borehole database with all the measured info that builds the bock model, and finally the mining phases of the pit (natural, current and future topography), to get the kind of materials that will be left in the waste deposits or exposed in the pit. This was mostly done by engineers from the mining area of the consulting firm I work for. Problem is we were never the priority so, many times, we were delayed or blocked because we couldnt process our data on time or reprocess it when we got updated data. I got tired of waiting and started searching alternative open source software that could work to processes this kind of data and I found none. Most projects are outdated and abandoned. All paid software is VERY overpriced, like, 50K USD a license (Maptek Vulkan, GEMs, Leapfrog). So, since i know some python, I programmed a series of tools myself, in python using the VTK library. Then, since some less tech savvy colleagues wanted to use my tools, i made a (very ugly) GUI using tkinter.

Now, im not saying my rudimentary set of tools with an ugly GUI is comparable to industry standards like Vulkan or GEMs, but they are a lot cheaper (50K USD cheaper) and all the geostatitical analyses could be integrated later if they are used for this purpose. For some time now I’ve been wondering if I could/should:
a) open the tools for free (as in speech, as freedom, but I’m not sure how)
b) try to port the tools as actual paraview or VTK tools (I just know python and R)
So my questions are: would anyone be interested in this tools? is it possible to integrate python scripts as tools in paraview? is there something like this mining geoprocessing tools already somewhere in paraview I didn’t find?

What are your thoughts?

PD. What the tools do:
convert 3D surfaces from DXF (industry standard) to VTP
Find the XYZ middle point of a borehole and all its info in a typical Collar/Survey/Litho/Alt/MZone borehole database
Find the intersection between boreholes, block models and topographies (between each of them, for example the intersection between a 3D topography and a 3D block model to get the BM info in the surface and/or the distance between each block and the surface)
Visualization of topographies, block models and borehole databases
Get the volume between two topographies with different triangulations/vertex, get 3D surface, get info from Google Earth Engine, make filters on the block model, create new columns with simple arithmetic or logic operators, and many other small data processing tools, like clipping and cleaning data Etc.

Cheers

2

This is hard to answer without knowing what you want, and how committed you are to a long-term project. Other factors include whether you like working in a community environment, and whether financial gain is important.

Probably the easiest thing to do is establish a github/gitlab repository, making sure you pay attention to the documentation and lots of examples / tests. Then depending on the response, this may lead to the formation of a community. Finally, put a lot of thought into the license. I prefer permissive-style open source licenses like Apache 2.0. If you are interested in monetizing the work, such a permissive license will likely lead to consulting/support/services revenues.

Finally, since it seems that you are knowledgeable in Python, I suggest that you look into trame. The reason for this: 1) you can create a scalable web application (which of course will also work fine on a desktop); 2) you can create nice GUIs; 3) trame integrates access to many other Python libraries and tools of you want to extend what you have. Other trame resources include:

This is potentially a very long discussion so ask additional questions.

1 Like
3

Hi @Altertango

Great project from the sound of it!

For some time now I’ve been wondering if I could/should:
a) open the tools for free (as in speech, as freedom, but I’m not sure how)

Definitely, you should choose a license adapted to what you want to do but that makes complete sense to me.

b) try to port the tools as actual paraview or VTK tools (I just know python and R)

ParaView supports python plugins, I could image such a tool as a python plugin for ParaView

FYI @Francois_Mazen

4

Hi @will.schroeder, thanks for the prompt reply. I would love to get some work in the future related to this project, without the need to hoard to the code and be able to share it for tyhose who need it. Also, I support and like open source models of software development so I’d love to contribute. My biggest concerns are that I have limited knowledge of GIT (mostly use it to manage editing scripts with collages) and have never in my life contributed to a big, ongoing, open source project, so I don’t have the know how of managing something like that. About the GUI, I think the best would be to integrate it to paraview, since it’s already a great GUI for most VTK tools, the python GUI was mostly a way to get things done fast without the need to learn new things, as it is generally done in consulting contexts. Anyway, trame looks awesome and will definitively give it a look. Thanks!

5

Hi @mwestphal,
Thanks for your swift answer. At the beginning, I tried to use the python scripting tools inside paraview but was having trouble getting new libraries installed in the internal python. I need to install libraries like google earth engine, scipy, matplotlib, xml readers, pyproj among others. I might have not done enough research and jumped directly to python scripting because im more used to it and the gui was made in a hurry. Anyways I will check again if i manage to add a custom scripts in paraview using external libraries, this looks to me like the most efficient way to share the tools.

6

I guess, I see different level of integration (money aside which you need to figure out) and based on your background and what you are comfortable with, you may stop anywhere along the way. I’m just capturing what was already said by Will and Mathieu while adding my point of view regarding a possible GUI.

  1. It seems that what you’ve created is a set of tools/filters that are core for many mining needs. In that sense, capturing it with a license/copyright into a git repo (github) would be great start and maybe help you get traction around it. (From Will)
  2. Cleaning the code to make it a Python library that can be pip/conda installable and leveraging the CI to automate that process as much as possible. Add doc, examples so other people can use it. (From Will)
  3. Expose some pieces as ParaView Python plugins so they can be use in ParaView as-is. (From Mathieu)
  4. Make some GUI to streamline actual workflow/usage
    a. trame is definitely a nice way to go about that as it should enable you to produce bespoke solution faster. Trame can rely on ParaView and the plugin you’ve built initially but skip the generic/complex UI of ParaView.
    b. ParaView GUI will be nice but definitely way more involved in software process and dev time. But maybe, the auto-generated panel editor for the filters exposed could be enough?

HTH

7

Hello,

That’s a wonderful initiative, really. But, as others already stated, it is not possible to know beforehand whether it’ll develop a community around it. In 2014, 10 years ago, during my post-grad years, I put up a software dubbed GammaRay (GitHub - PauloCarvalhoRJ/gammaray: GammaRay: a graphical interface to GSLib and other geomodeling algorithms. *NEW* in Apr, 22nd: MCRFSim for Bayesian approach.) and it still doesn’t really have a community. Maybe because geostatistics is a quite small world or it is not that good at all or the main users in the mining/oil industry simply use what those billionaire companies can buy. The main drive behind it is to avoid usage of commercial bundles and to serve as a friendly interface to the sole open-source free geoestatistical software in that time: GSLib. Maybe students in universities in developing contries are using it… who knows. Later, I added non-GSLib algorithms to it and I still use it for data preparation and run experiments on it without spending thousands of dollars in the process.

If you plan to develop a serious Leapfrog-like application, please consider migrating to C++, C# or Java (for GUI applications). Python is all-well but designing complex user interfaces with it is a troublesome endeavour. If you still want to stick to Python, then you might be interested in these two projects: GitHub - GeostatsGuy/GeostatsPy: GeostatsPy Python package for spatial data analytics and geostatistics. Mostly a reimplementation of GSLIB, Geostatistical Library (Deutsch and Journel, 1992) in Python. Geostatistics in a Python package. I hope this resources is helpful, Prof. Michael Pyrcz and Welcome to pyLPM’s documentation! — pyLPM 1 documentation . The latter is pip-installable and is from the mining engineering laboratory where I did my post-graduations.

best,

PC

8

Hello Paulo,
Thanks a lot for your replay. And thanks for your links to your amazing work and those libraries. I totally agree with your analysis about the reason there is no interest in open source mining database software. I will have a look at GammaRay, congrats for your amazing contribution, does it has the capability to handle drillhole databases, block models and topographies? I don’t think I run into it when searching for a free software that did what I wanted to do. I think I did check the GSLIB libreary for python in my initial search but found it too complicated and run into compatibility problems with the libraries or something like that (it was 3 years ago). My main concern was the intersection between all data, so I don’t really have any needs for geostatistical tools.

About the GUI, Once i’ve got a lower work load I will try to make some paraview scripts since it’s on my opinion the best way to create something myself sans the hassle of making a GUI from scratch. I will try to see if adding GeostatsPy or other GSLIB capabilities is possible, but first just the basic database handling I need would be great. I’ll be sure to update this thread if I ever get to do it.

9

Hi,

Depends on you what you mean by “handle” and “databases”. If you have the data as Geo-EAS/GSLib plain text files and want to perform some exploratory data analyses, run some estimation/simulation on grids and display them in 3D, then, yes. GammaRay supports data like drillhole/well (either as point sets or segment/polyline sets), regular grids (2D and 3D), stratigraphic grids (good for orebodies and reservoirs) and something dubbed “geologic sections” used to represent, for example, outcrops:
image

Indeed. GammaRay evolved better suited for petroleum geostatistics/geologic modeling. This means it still lacks some essential features for mining application such as contact analysis and grade-tonnage curves. Anyway, it’s all opensource, you can fork it and modify it any way you want. Although it requires getting acquainted with modern C++.

10

@Altertango I am in a very similar industry and situation, in having many autochthonous Python snippets performing workflows in place of getting the license for the expensive software. Working with drilling databases, geostatistics (largely thanks to integrating R, or controlling SGeMS), block models, working with meshes, etc. Visualization is a missing piece of what I have done for myself. I am not very familiar with VTK itself yet, but would like to know if you have put your code up anywhere? There is demand to have something community-maintained that is more minerals-specific i.e. as part of PyVista/PVGeo, or adjacent to projects in the Software Underground (geophysics-driven) community. I think we just have to put it all together.

11

Hello,

I think you could take a look at the VTK primer (VTK Textbook | VTK) so you can have a glimpse of how a pipeline-oriented graphics engine works as well as to know its several data set types (and their algorithms) and to identify the ones likely used in a mining application (e.g. the vtkUnstructuredGrid class is useful to model orebodies, the vtkPolyData can be used to model faults and horizons, vtkImage can be used to display a seismic survey, etc.).

Here is a VTK Python guide: https://examples.vtk.org/site/Python/ . There you can learn how to setup your environment and find some examples with increasing complexity to help you get started.

best,

PC