Funny thing, I was thinking of the best way to raise this today!
I think it is a resource that should be maintained and kept up to date. You will see that since December, I have committed changes to the master (but not updated the web pages) to ensure that it builds under VTK 9.0. Indeed just before the VTK 9 release, building the master revealed some issues with the VTK CMake files.
In looking at your multiple solutions:
From my point of view I have the authority update and merge commits to the master. However I cannot update the web pages. So whilst the master is up-to-date there is a divergence between the master and the web pages. In fact I would be very reluctant to update the web pages as there is a significant risk that the web pages could be rendered useless. Bill used to occasionally update these pages from his own machine.
The problem here is that this has grown into a fairly large and complex project for example, in my builds, I keep to the minimal VTK build so these features are not present in my builds:
Personally I would like to see this integrated into Kitware as a VTK Project. My feeling is that the original site should remain or be archived in memory of Bill’s incredible work in this area. Just look at the web pages!!!
If it integrated into Kitware these are the directions I feel we should take:
The examples should track the VTK master closely - this would simplify a lot of the work involved in maintaining it. In this way, we showcase all the best features of VTK. This will mean that some of the code will become redundant however it will remain on the original site for those with older versions of VTK. Also, more importantly, we can get rid of all the CMake code necessary to support older versions of VTK.
I like the idea of the web pages and to this end I have converted Bill’s SyncSiteWithRepo.sh and associated Python files from Python 2 to Python 3. They are untested but they are in the master. These new files just have *Py3 appended to their name to distinguish them from the originals.
The process of generating html files needs to simplified. It is quite time-consuming and complex, although the new code I have written may be faster. I would like to see a nightly generation.
Regular testing - in the past week I have got test failures down to one or two for both Windows and Linux.
Welcoming new contributions - I think this is really important. The process is a bit arcane for new users at present so guidance is needed. A lot of the work I have done in the past few years has involved this.
After the initial setup the workload should just drop significantly and mainly consist of following the VTK changes and welcoming new contributions.
I would dearly like to see this become a Kitware project and I will devote as much time as you want to achieving this.
Updating to use/showcase the new features of VTK will be fun!
If this approach is taken it will be more like what the VTK examples were in the “old days”, when they were part of VTK.
I would also like to emphasize the usefulness of this project. I use it for:
proposals: for instance, what is available in VTK for mesh and point processing? The VTK Examples uses a filter categorization not available anywhere else as far as I know.
coding: in the past I used to use tests to see how to use various classes, now I often use the VTK Examples.
I also think VTKExamples is a great advertising tool for VTK.
I completely agree with the usefulness of these exercises for VTK developers as learning materials. I think it is important to keep them on web pages as well and not only in a repository as this really help with referencing. A lot of time when I look for a generic request about VTK I still get on the VTK example website as search engines can index their content with keywords more cleverly than pure repositories. Maybe a question may be how to keep these web pages up to date with low maintaining costs. @amaclean You mention a SyncSiteWithRepo.sh script, is it robust a easy to use ? Do we need another solution like hugo to help contributors integrate their examples on the website easily ?
I haven’t look at how the VTK/C++ example pages and so on are built or maintained, but in general, I agree that examples are important for documentation and community usage.
Typically that was key for me when I built the vtk.js web site and I think it is as important for VTK/C++ as well. Even better if the VTK/C++ documentation get reworked to provide a uniform web site with proper sub-sections and integration. Since we also have several version of VTK out there, we may also need to version such website and allow user to find example for VTK 8 vs VTK 9…
Unfortunately, I don’t necessary have much more to offer in term of solution, but I think something that go beyond just keeping the examples alive is important too.
I’m fine with a gitlab-ci job that runs to ensure they continue to work, but I’d like that to happen after VTK has migrated at least a few of its regular builds. I don’t have any thoughts on how to maintain the repo/site itself at the moment.
The master is being maintained by myself, so it is tracking the VTK master nicely for Python and C++. However this means that there is a disconnect between the master and the web pages. Most of the C++ tests based on my VTK master build are passing Ok for Linux and Windows 10. Except for the fact that I do have an ensight reader fail for a particular dataset that fails in the latest Visual Studio but not in Linux, I can explain the other test fails, currently only 2 or 3 failures.
Also note that most examples will run on older versions of VTK anyway with the obvious exception of the newer API’s. Where deprecations have occurred I have used the VTK versioning to trap these. If an example used a new API there are functions testing for this. This should alert the user. E.g see: PhysicallyBasedRendering, the master version also checks what function should be used:
renderer.UseImageBasedLightingOn()
if vtk.VTK_VERSION_NUMBER >= 90000000000:
renderer.SetEnvironmentTexture(cubemap)
else:
renderer.SetEnvironmentCubeMap(cubemap)
renderer.SetBackground(colors.GetColor3d("BkgColor"))
renderer.AddActor(actor)
Also in the master Cxx\Utilities\CheckVTKVersion.cxx is vastly different from the web version.
@Charles_Gueunet the SyncSiteWithRepo.sh calls Python 2 scripts and uses an old version of mkdocs.yml. I have been updating these to Python 3. Hopefully I can get this working.
VTK examples are awesome but it is certainly a lot of work to create and maintain them. One way to address this is to make it easier for community members to try, modify, and test the examples and simplify make automatic building and website infrastructure.
This could be all achieved by converting each example to a Jupyter notebook:
simple for users:
run and modify and run VTK examples, including an interactive VTK render window, without building or installing anything; just from a web browser
VTK API documentation is built in, auto-complete works nicely
easy to contribute - just send a pull request to the github repository
simple for maintainers:
no need to build and run the examples (the notebook kernel takes care of that)
same infrastructure can be used for both Python and C++ - yes, C++ code is interpreted, you don’t need to build the entire file but you can run the code line-by-line - click here to try a C++ notebook
trivial hosting: each example is an notebook file in a github repository, you can use branches/tags that match VTK repository’s
free binder runs the notebooks (using a chosen docker image)
I think the only part that is not readily available is indexing and searching. Hopefully the existing infrastructure could be adapted to this.
I’ve created a simple VTK notebook now, which shows a live VTK renderer and you can run any VTK example code in it (if you are not familiar with Jupyter notebooks yet, just click the link below and hit Shift+Enter to run a code cell):
We developed this infrastructure to allow running 3D Slicer application on the web and allow using it from Jupyter notebooks, but it can be used for any other VTK-based projects, too. It could make VTK available conveniently for the machine learning and scientific Python community. Here is a demo video of what 3D Slicer can do using VTK rendering and Jupyter notebooks:
@lassoan Looks interesting however there are also CSharp and Java examples that need to be considered. Also indexing and searching may be problematic. One of the nice features about the existing VTKExamples page is that you have a picture of the result, the code and the necessary CMakeLists.txt file all available on the one page. Also being able to use the multiple indexing to find examples is really useful. This is probably one of Bill Lorensen’s great achievements.
Jupyter kernels are available for all major langauages, including C# and Java.
Creating the index page and search index requires some effort but should not be hard. For example, the view display command could switch between “live view” / “save output to file” with an environment variable. A script could run all the notebooks with “save output to file” to generate the index file with images. I guess the current mechanism is something similar, so that could be adapted to the new environment.
The ability to edit/re-run VTK examples in a web browser without downloading, installing/building anything is not just interesting but it is a game changer. It makes VTK very accessible for potential new users. Have you tried the notebook link that I’ve posted above?
Yes, this free server takes some time to launch a new environment. Normally it takes 15 seconds for the docker container to start, but it may take up to 1-2 minutes if the docker image is not found in the cache of the server that serves your request. This is very short time compared to installing or building VTK. Also, the image that you have tried contains 3D Slicer, which includes not just VTK, but also Qt, ITK, CTK, DICOM toolkits, etc. A much smaller image will be enough for a pure VTK test environment, which could start quickly, even if the image is not cached.
After this initial wait, startup of notebooks and running of cells is fast. Certainly much quicker than rebuilding and running an examples.
If you just want to view a page (not edit it interactively) then you can use nbview - see this example. Notebook content can be displayed as quickly as a static html page, because all displayable content is saved in the file as simple static markdown or html. Notebook files can also be natively viewed/edited/executed locally in Visual Studio Code.
def vtk_version_ok(major, minor, build):
"""
Check the VTK version.
:param major: Major version.
:param minor: Minor version.
:param build: Build version.
:return: True if the requested VTK version is greater or equal to the actual VTK version.
"""
needed_version = 10000000000 * int(major) + 100000000 * int(minor) + int(build)
try:
vtk_version_number = vtk.VTK_VERSION_NUMBER
except AttributeError: # as error:
ver = vtk.vtkVersion()
vtk_version_number = 10000000000 * ver.GetVTKMajorVersion() + 100000000 * ver.GetVTKMinorVersion() \
+ ver.GetVTKBuildVersion()
if vtk_version_number >= needed_version:
return True
else:
return False
In C++
/**
* Check the VTK version.
*
* @param major: Major version.
* @param major: Minor version.
* @param major: Build version.
*
* @return True if the requested VTK version is greater or equal to the actual
* VTK version.
*/
bool VTKVersionOk(unsigned long long const& major,
unsigned long long const& minor,
unsigned long long const& build)
{
unsigned long long neededVersion =
10000000000ULL * major + 100000000ULL * minor + build;
#ifndef VTK_VERSION_NUMBER
auto ver = vtkSmartPointer<vtkVersion>();
unsigned long long vtk_version_number =
10000000000ULL * ver->GetVTKMajorVersion() +
100000000ULL * ver->GetVTKMinorVersion() + ver->GetVTKBuildVersion();
if (vtk_version_number >= neededVersion)
{
return true;
}
return false;
#else
if (VTK_VERSION_NUMBER >= neededVersion)
{
return true;
}
return false;
#endif
}
No sure if they are really that useful but at least they alert the user or allow code to run with suitable #ifdefs or try/excepts.
A good example of how this could work is scikit-image examples. Users can browse statically-generated pages and can get to an interactive environment by a single click (and 1-2 minute wait). VTK could do even better by not just showing results on static images but by providing an interactive 3D renderer.
This is all very nice if someone wants to set all this up and maintain it.
However if you are going to do this, keep in mind that there are nearly 2000 examples all nicely indexed and presented as web pages, so there would be a considerable amount of work porting to a new system.
If Kitware took it over, my feeling would be to keep the original system as much as possible to reduce the workload. This would also obviate the need to add more complexity. The system at present is stable, the only times web pages need to be updated is when new code is added or old code changed.
Really, once it is ported over there would be a bit of work setting up the web pages for the new repository, and testing the Cxx code, preferably on a Linux machine. The scripts for this are all there. User contributions are very low in number.