WASM Regression tests (file locations)

Development
1

I have some experiments with regards to regression tests for WASM. I do the following

  1. Use ExternalData_add_test and add_test
  2. Autogenerate a node.js script, which can copy data into WASM filesystem and call a function TestXXX(int argc, char* argv), where I files can be placed in the file system
  3. The output are written to WASM file system
  4. The node.js script can copy them to output

One issue. Many of our tests, just take a folder and the filename is written inside the .cxx file.

First, I will get it up and running for a test, where the input file and output files are specified in the CMake system. Do we have such a test? If yes, can you provide a good example?

Would it be an idea to always keep input and baseline in the CMake system and not written in .cxx?

I realized there is a bug in Emscripten about reading from stdio using a stream. It is though easy to accept a fixed string and execute tests like, node vtkCommonCoreCxxTests TestSomething.

2

I figure out that you have made test using preload of data in the browser. That is a different approach. Any change that we will get this to work also from Node? Would be nice with a simple example, then developers can follow and help on getting WASM tests to work

3

Hello @Jens_Munk_Hansen

I was out of office for the last week.

Emscripten provides a default memory file system backend MEMFS. With that, all file I/O from C++ assume a virtual filesystem in memory mounted at /. For VTK, we run regression tests on each MR and nightly using chrome-for-testing browser with extra plumbing in the VTK test utilities module.

With regards to test data, some unit tests want to read a data file or a baseline image and later write a difference image. So the infrastructure is capable of handling both reads and writes.

Approach 1: Using VTK testing infrastructure

If you’re writing VTK based unit tests using the vtkModuleTesting - VTK documentation cmake functions, simply add these two lines in your test cxx and the below lines in your cmake right before you call vtk_add_test_cxx and define the cmake variable VTK_TESTING_WASM_ENGINE to the path to a browser executable.

The vtkModuleTesting.cmake does all the magic of serving the wasm, and running a custom python server which implements preloading and dumping files. Just run ctest and you will see browser windows opening and closing each running your unit test.

#include "vtkRegressionTestImage.h"
...
...
int TestBar(int argc, char* argv)
{
 ...
 ...
 vtkRegressionTestImage(renderWindow); // see Testing/Rendering/vtkRegressionTestImage.h for more macros.
}

if (CMAKE_SYSTEM_NAME STREQUAL "Emscripten")
  set(_vtk_test_cxx_wasm_enabled_in_browser ON)
  set(_vtk_test_emscripten_extra_linker_args
    "-sASSERTIONS=1"
    # O2 or higher make wasm-opt terribly slow.
    # test executables take a minute to link.
    # we need emscripten to emit code that handles exit code and atexit(s)
    "-sEXIT_RUNTIME=1"
    # tests dynamically allocate memory, can easily go over the preset limit.
    "-sALLOW_MEMORY_GROWTH=1")
endif ()

vtk_add_test_cxx(MyTestExecutableName tests
  TestFoo.cxx
  TestBar.cxx
)

Approach 2: Doing it yourself

The VTK Emscripten test utilities uses a custom protocol to communicate requests to preload or dump a file from the wasm client to the server hosting the wasm module. If you want to use a node js server instead of a python server like we do in VTK, implement these routes in your node server.

Routes:

  1. /preload?file=full_path_to_file: wasm unit test makes a GET with the full path of a file in its query parameters. The server should respond with the contents of the file or 404 if a file does not exist
  2. /dump?file=full_path_to_file: wasm unit test initiates a POST with the full path of a file in its query parameters and sends the entire contents of the output file in the POST content. “Content-Length” is the bytesize of file.

Reference Client-side implementation in VTK:

  1. vtkEmscriptenTestUtilities.h
  2. vtkEmscriptenTestUtilities.cxx
  3. vtkEmscriptenTestUtilities.js

Reference Server-side implementation in VTK:

  1. The vtkTestHTTPHandler.translate_path method implements the /preload route. This responds to requests for a file.
  2. The vtkTestHTTPHandler.do_POST method implements routes like /dump, /console_output and /exit. These are triggerred when the wasm application wants to write a file to disk through fwrite, print messages to cout/cerr and exit(code) respectively.
4

While the current approach in VTK works great for most of the unit tests, there is room for improvement.

Composite dataset readers for ‘.vtm’ or ‘.vtpc’ files indirectly load other ‘.vtp’ files which are not intercepted by the vtkEmscriptenTestUtilities APIs I mentioned. Ideally, when running tests that access/write data within the browser, you’d want to intercept all C fopen, fread, fclose calls to basically request the server to send the file over network, preload it in the virtual file system, read it, popluate the buffer with the file contents and return an appropriate status code and similarly intercept fwrite.

Emscripten offers a synchronous virtual XHR backed file system. The drawback is it requires you to know the exact filename at compile time which is very limiting…

5

Thanks. That is what i figured. Would be nice if all tests were made with macros, such that we from CMake knew which files to use. When running the existing Emscripten tests, my system was hanging. I guess it was due to a missing VTK_TESTING_WASM_ENGINE, which I haven’t set.

6

Yes, it is far easier to use the system that VTK uses.

If you ran the test in verbose mode with ctest -VV .. you will see a message printed like “VTK_TESTING_WASM_ENGINE is not set. test may appear to hang but it is running a http server at url:port”

1 Like