Custom shader to deform mesh

I have polydata that contains point_data that represents an FEA simulation’s displacement result.

I am trying update the vertex shader in PolyDataMapper to add the displacement vector to the vertex position. I believe I am close to getting it working however I am not sure I have the math right. Here is the model in paraview, note the size in x direction is ~200mm:

This is the model in my program without the shader additions:

This is the same camera view with the shader enabled:

As you can see the mesh has been streched in the positive X direction as expected. However, the magnitude of the largest displacement vector is 0.98mm but the shader seems to have stretched the mesh far more than the 0.5% expected (0.98mm of 200mm).

What I suspect is going wrong is that I have an issue with the math when adding the displacement to the vertex position. I assume vertexMC (model coordinates) is still in the same coordinates/magnitude as the raw point positions in the polydata?

The next Issue I see is when I rotate the camera the mesh becomes clipped and the red portion disappears:

What I suspect the issue is here is the math for the camera view or clipping is not taking into account the larger position of the mesh after the displacement has been added. I’m trying to figure out how I could potentially add the largest displacement vector to the camera shader calculations so that it increases the FOV. Additionally my resetCamera call focuses on the original origin of the mesh rather than the origin of the mesh with the displacement added.

Here is my code in for vtk js, I do plan to implement similar logic in VTK c++ for wasm in the future too:

        if (polyData.getPointData().hasArray('displacement')) {
          // add displacement as a custom shader attribute
          mapper.setCustomShaderAttributes(['displacement']);
          const mapperSpecificProp: any = mapper.getViewSpecificProperties();
          mapperSpecificProp.OpenGL = {
            ShaderReplacements: [],
            VertexShaderCode: '',
            FragmentShaderCode: '',
            GeometryShaderCode: ''
          };
          mapperSpecificProp.OpenGL.ShaderReplacements.push({
            shaderType: 'Vertex',
            originalValue: '//VTK::PositionVC::Dec',
            replaceFirst: true,
            replacementValue: ['//VTK::PositionVC::Dec', 'attribute vec4 displacementMC;'],
            replaceAll: false
          });
          mapperSpecificProp.OpenGL.ShaderReplacements.push({
            shaderType: 'Vertex',
            originalValue: '//VTK::PositionVC::Impl',
            replaceFirst: true,
            replacementValue: ['vertexVCVSOutput = MCVCMatrix * (vertexMC + displacementMC);', '  gl_Position = MCPCMatrix * (vertexMC + displacementMC);'],
            replaceAll: false
          });
        }

Please take a look at this code and let me know if their is anything obviously incorrect. Here is how the final GLSL of the shader:

#define attribute in
#define textureCube texture
#define texture2D texture
#define textureCubeLod textureLod
#define texture2DLod textureLod


#ifdef GL_FRAGMENT_PRECISION_HIGH
precision highp float;
precision highp int;
#else
precision mediump float;
precision mediump int;
#endif

/*=========================================================================

  Program:   Visualization Toolkit
  Module:    vtkPolyDataVS.glsl

  Copyright (c) Ken Martin, Will Schroeder, Bill Lorensen
  All rights reserved.
  See Copyright.txt or http://www.kitware.com/Copyright.htm for details.

     This software is distributed WITHOUT ANY WARRANTY; without even
     the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
     PURPOSE.  See the above copyright notice for more information.

=========================================================================*/

attribute vec4 vertexMC;

// frag position in VC
out vec4 vertexVCVSOutput;
attribute vec4 displacementMC;

// optional normal declaration
//VTK::Normal::Dec

// extra lighting parameters
//VTK::Light::Dec

// Texture coordinates
attribute vec2 tcoordMC; out vec2 tcoordVCVSOutput;

// material property values
//VTK::Color::Dec

// clipping plane vars
//VTK::Clip::Dec

// camera and actor matrix values
uniform mat4 MCPCMatrix;
uniform mat4 MCVCMatrix;

// Apple Bug
//VTK::PrimID::Dec

// picking support
//VTK::Picking::Dec

void main()
{
  //VTK::Color::Impl

  //VTK::Normal::Impl

  tcoordVCVSOutput = tcoordMC;

  //VTK::Clip::Impl

  //VTK::PrimID::Impl

  vertexVCVSOutput = MCVCMatrix * (vertexMC + displacementMC);
  gl_Position = MCPCMatrix * (vertexMC + displacementMC);

  //VTK::Light::Impl

  //VTK::Picking::Impl
}

Note that I did confirm the units for the points and displacement vector are the same (mm).

Vertices are clipped typically when the gl_Position takes values outside the [-1,1] range. The usual suspect is that MCPCMatrix. It could be that MCVCMatrix and MCPCMatrix are both built from the original dataset bounds, and not the (stretched dataset bounds). A quick way to test this theory is to shrink your model with negative displacements. If the entire shrunk dataset appears, then you’ll need to customize mapper.setCameraShaderParameters to account for the new dataset bounds.

Cc: @sankhesh

Thanks Jaswant,

Any thoughts on why the scale of the change is a lot larger? Are model coordinates normalized? They seem to actually a couple orders of magnitude to large. If I multiply displacementMC by vec4(0.01, 0.01, 0.01, 1) the displacements seem more reasonable.

I’m not sure if vtk.js normalizes the coordinates before uploading. Maybe it does some shift+scale transform prior to upload?

Yes we do something like that to handle arbitrary large coordinates or offset coordinates.

@alexDrinkwater You would likely need to account for coordinate scale and bias when working in model coordinates in the shader. Take a look at https://github.com/Kitware/vtk-js/blob/master/Sources/Rendering/OpenGL/CellArrayBufferObject/api.md#getcoordshiftandscaleenabled-- for documentation on when the model coordinates are scaled and biased.

The transform is applied here: https://github.com/Kitware/vtk-js/blob/e88aa3f37dc27528dba72af1f8d7b62704d01050/Sources/Rendering/OpenGL/PolyDataMapper/index.js#L1254-L1264 in the polydata mapper.

Hope that helps.