Creating a custom docker image for VTKWeb with multi-user support (apache proxy)

Dear All,
I have a project that uses the vtk-based multi-user remote visualisation similarly to paravieweb (ParaViewWeb). The concept has been tested and works. So, currently, I am trying to create a custom docker image for this vtk-based web remote visualisation. I am trying to reproduce the process similarly to how it is described for Paraview-based custom container creation process here: ParaViewWeb .

The concept works for direct websocket connection . That is something like launching directly for fixed port: vtkpython /data/pv/pv-5.9/share/vtkjsserver/vtkw-server.py --port 1234 --host 0.0.0.0

However, when I try to switch to a multi-user configuration with apache as a proxy, the establishing of WebSocket connection fails (apparently the launcher does not receive the request to launch vtk server).

The draft files used are here: GitHub - EvgeniyaMalikova/dockerVTKWEB: docker image template for vtk-based web vis (Dockerfile, configuration and script ). The Github source that was considered: paraviewweb/Dockerfile at master · Kitware/paraviewweb · GitHub. I am not sure, however, I fully understand its structure as some files (template_config.json) seem to be not available and I guess I can skip the execution of the others (like entrypoints.sh) if I am using localhost and ws, right?

I call my image something like : docker run -v <path_to_files>:/home/pvw-user/files -p 0.0.0.0:8888:80 --name test3 --gpus all -it <image_name>:latest

I am quite new to all this and would really appreciate any explanation on how it all works within docker.
Thank you very much in advance,
Best regards,
Evgeniya

Ideally you want to check the network calls happening from your client application.

The following should be happening:

  1. POST /paraview
    a. start process and wait for readiness (if error look at logs)
  2. Response from launcher (/paraview) should provide sessionURL to connect your ws
    b. The url should be something like ws(s)://{your_host_name}/proxy?sessionId={xyz}
  3. The ws connection is established to your process going through apache

Hope that helps,

Seb

PS: You can find a lean version of the Apache/Launcher setup for docker within our trame repo. I don’t know if that will confuse you more or provide another perspective that could help.

Dear Sebastian,
thank you very much the info provided is really helpful and thanks to it the issue is solved. The problem was with the execution of the first step “POST/ paraview”

Thank you very much,
Best regards,
Evgeniya

Oh, sorry, there is another error at different place now.
The POST returns an error message:Status 503 Service Unavailable.
Session did not start before timeout expired. Check session logs.

From the launcher logs I see that launcher has received the request, generated sessionID and launched vtkw-server, which is ready for connection, but does not receives it. When I check for /data/proxy.txt file, there is no URL information in it. I guess error is between 2-3 steps, but may be you have any idea on where to look?

Launcher log
2022-05-17 14:37:21,167:DEBUG:asyncio:Using selector: EpollSelector
2022-05-17 14:38:02,325:INFO:aiohttp.access:127.0.0.1 [17/May/2022:14:37:37 +0000] “POST /paraview/ HTTP/1.1” 503 252 “http://localhost” “Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:100.0) Gecko/20100101 Firefox/100.0”

log of vtkw-server.py

e59d6d7a-d5ee-11ec-91e6-0242ac110002 //prints out session id

Interactor type
vtkInteractorStyleTrackballCamera
interactor was set up
wslink: Starting factory

Thank you very much,
Best regards,
Evgeniya

What is your launcher config looks like. You may need to increase the timeout. The sessionURL should be provided in the response of the POST. The proxy.txt file will only be updated once the launcher think the process is ready…

Sorry for late reply,

json configuration looks like:

{
"resources": [ {"port_range": [9019, 9019], "host": "0.0.0.0"} ],
"sessionData": {
"updir": "/Home"
},
"configuration": {
"log_dir": "/data/pvw/logs",
"host": "0.0.0.0",
"endpoint": "paraview",
"sessionURL": "ws://localhost/proxy?sessionId=${id}&path=ws",
"timeout": 45,
"upload_dir": "/data/pvw/upload",
"fields": ["file", "host", "port", "updir"],
"port": 9020,
"proxy_file": "/data/proxy.txt"
},
"properties": {
"dataDir": "/home/pvw-user/",
"vtkpython" : "/data/pv/pv-5.9/bin/vtkpython",
"vtk_python_path": "/data/pv/pv-5.9/share/vtkjsserver"
},
"apps": {
"cone" : {
"cmd" : [
"${vtkpython}", "${vtk_python_path}/vtkw-server.py", "--port", "${port}", "--updir","${dataDir}","--session", "${id}","--host","0.0.0.0"] ,
"ready_line" : "Starting factory"
}
}
}

Ok you have several issue in it.

First by saying "resources": [ {"port_range": [9019, 9019], "host": "0.0.0.0"} ], you can only have 1 process which means 1 user, not multiple. To extend to more user, just increase the port_range.

Second the sessionURL point to a fix localhost rather than using a template value such as SESSION_URL_ROOT or USE_HOST.

Your timeout seems reasonable for 45s for start time.

So your error seems that the launcher start the process correctly but does not detect “Starting factory” in stdout within the 45s. But looking on your config, I don’t see any issue that make me think that is truly happening. But without really seeing the issue myself it is hard to tell.

Dear Sebastien,
thank you very much. Most of the source codes that I used (except the application itself) are in Github repo. May it is the problem, that compared to other repos like (GitHub - Kitware/trame: Trame let you weave various components and technologies into a Web Application solely written in Python.) I am not using entrypoint.sh scripts. I believed the step that I have missed is taking environment vars and config-template.json to generate final
LAUNCHER_PATH=/opt/trame/config.json that is fed to wslink.launcher . I have used a fixed “localhost” and “ws” instead. Also, I am running processed as root. May be that provides an idea on what could go wrong? Is there any "correct way " to generate vtk-based remote vis docker image similarly to paraview web?

Thank you very much,
Best regards,
Evgeniya
Evgeniy

Does your application start in less than 45s? Does it print "Starting factory"?

First you need to make sure the process start and that the launcher report back the proper information. Then once that part is valid we can move to the next stage.

For me figuring out what is missing with a partial view of what is going on, it is really hard.

Dear Sebastien,
thank you very much for helping me with this. I have copied all codes for building docker image to repo (GitHub - EvgeniyaMalikova/dockerVTKWEB: docker image template for vtk-based web vis), so they can be reproduced.

The output log with the application seems to be fine, it prints out "Starting factory" and “wslink:Starting factory” . But after monitoring the update of logs, it seems that all application-related log messages are printed out only after timeout expiration set in launcher.json (that is 25 or 45 and etc.) . So looks like something is blocking the application process from execution, and increasing timeout time makes no change (that is just my guess). I will try to record a video to provide the details on what I see in logs, to give at least some idea about the problem.

Thank you so much,
Best regards,
Evgeniya

Below is a video link of what I see in logs. I hope this helps.

Best regards,
Evgeniya

Based on what you are showing, it seems that the print of “Starting factory” is not flushing.

I’m not sure which version of wslink you are using, but we use to have a -f arg. Also you can put inside your code some print("Starting factory", flush=True) to see if show up before the timeout.

When looking at your docker, you should just pip install wslink as the rest (twisted, autobahn) should come automatically as dependency.

But in truth, wslink 1.0.0+ do not depend on twisted and autobahn anymore. So I’m not sure why you are bringing them in.

I am importing wslink dependencies from vtk. So it might be the case that I make a wrong call from older vtk version 9.0.1, for example removing twisted gives me an error message like:

> # ls
> d029b79c-db47-11ec-88ed-0242ac110002.txt  launcherLog.log
> # cat d029b79c-db47-11ec-88ed-0242ac110002.txt
> Traceback (most recent call last):
>   File "/data/pv/pv-5.9/share/vtkjsserver/vtkw-server.py", line 43, in <module>
>     from vtk.web import wslink as vtk_wslink
>   File "/data/pv/pv-5.9/lib/python3.8/site-packages/vtkmodules/web/wslink.py", line 11, in <module>
>     from twisted.python         import log
> ModuleNotFoundError: No module named 'twisted'
> # pip3 show wslink
> Name: wslink
> Version: 1.6.4
> Summary: Python/JavaScript library for communicating over WebSocket
> Home-page: https://github.com/kitware/wslink
> Author: Kitware, Inc.
> Author-email: kitware@kitware.com
> License: BSD-3-Clause
> Location: /data/pv/pv-5.9/lib/python3.8/site-packages
> Requires: aiohttp
> Required-by: 
> #

Thank you very much,
Best regards,
Evgeniya

wslink.py top lines are:

r"""wslink is a module that extends any
wslink related classes for the purposes of vtkWeb.

"""

from __future__ import absolute_import, division, print_function

# import inspect, types, string, random, logging, six, json, re, base64
import json, base64, time

from twisted.python         import log
from twisted.internet       import reactor

from autobahn.twisted.websocket import WebSocketServerProtocol

from wslink import websocket
from wslink import register as exportRpc

from vtk.web import protocols
from vtk.vtkWebCore import vtkWebApplication

Best regards,
Evgeniya

Also, the project was build on top the vtkweb template that uses twisted (see web-project-templates/vtk_override_protocols.py at master · Kitware/web-project-templates · GitHub)

Maybe that is a source of the issue as flushing of print did not fix it.

Best regards,
Evgeniya

Your vtk is bringing the latest wslink which is not based on twisted.
The example was written with a wslink < 1.0.0 and therefore expect twisted. I’m not sure if it should even work.

I guess the next step would be for me to update that example for vtk 9.1. I’ll see if I can get to it sometime today.

Actually, the repo was already updated to work with vtk 9.1 and wslink >= 1.

Your code base was probably from an older version of the repo as twisted is not referenced anywhere anymore.

BUT, that piece of code is really meant for the demo not for real deployment which can mess up the connection back to the server.

Anyway, either version should be fine as long as you have consistency on your full stack. If you want to stick with twisted, just pre-install wslink<1.0.0 before installing VTK < 9.1.

If you want to use VTK >= 9.1 you must use wslink >= 1.0.0.

HTH,

Seb

Dear Sebastien,
I have tried everything (print flush, different VTKs and wslink) and it is just not working with launcher from docker image. I have also removed the “questionable piece of code” but it did not change anything. If the problem is on the client-side that was written only for a demo, is there any example of “correct” code that I take as reference and overwrite the version so it will work with launcher?

Sorry for the inconvenience, as I do not see any options here,
Thank you very much,
Best regards,
Evgeniya

Oh, Sorry for the misleading, made a mistake with versions while building. I confirm that old version of web-project-templates/vue-vtkjs-pvw-template at master · Kitware/web-project-templates · GitHub works with vtk 9.0 and wslink 0.2.0 . Still have to check for updated server with vtk 9.1 and wslink >1.0.

Dear Sebastian,
Thank you so much for digging into it,
Best regards,
Evgeniya