Enabling Cuda throws error in VTK Build via VS2017

McDave17 · July 23, 2020, 7:36pm

Hello

When I build VTK (VTKm and AcceleratorsVTKm enabled) with following options set in CMake on Windows using Visual Studio 2017 Community the build crashes with an error directive:

Error: When VTK-m is built with CUDA enabled all compilation units that include DeviceAdapterTagCuda must use the cuda compiler
Project: vtkm_cont
File: D:\Studium\vtk\src\ThirdParty\vtkm\vtkvtkm\vtk-m\vtkm\cont\cuda\DeviceAdapterCuda.h 31

I figured that using the nvcc compiler installed with Nvidia Cuda as C and CXX compiler should resolve this issue, but the error is still thrown while building.

I also enabled QtSupport and changed the VTKm_ENABLE_RENDERING variable to ON in the CMakeLists.txt under ThirdParty/vtkm/vtkvtkm, but other than that I didn’t change any configuration.

Cuda Version (output of nvcc --version):

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Thu_Jun_11_22:26:48_Pacific_Daylight_Time_2020
Cuda compilation tools, release 11.0, V11.0.194
Build cuda_11.0_bu.relgpu_drvr445TC445_37.28540450_0

VTK Version:

git commit on Master Branch (f5a8d5b7f1c7b7fab090a0e5cec5a24c19dbd1be 22. July)

VTKm submodule Version:

git commit on Master Branch (567b1fedd20452fbe474b1ae4ef0ce8d21a73939 21. July)

Thankful for any pointers toward a solution for this problem.

Best Regards,
David

McDave17 · July 27, 2020, 4:07pm

Any ideas on how to fix this? Thankful for every hint.

McDave17 · July 28, 2020, 8:49am

What exactly does it mean that AcceleratorsVTKm will be required to build static? Do I have to manually ensure that? (This message is given to me when configuring via CMake when AcceleratorsVTKm and CUDA are enabled.)

McDave17 · July 28, 2020, 11:34pm

If been fiddeling around and can’t find the error source. I tried disabeling AcceleratorsVTKm and using different compilers / linkers. I keep running into either one of or both issues:

When VTK-m is built with CUDA enabled all compilation units that include DeviceAdapterTagCuda must use the cuda compiler

or that the DeviceAdapterCuda was not defined.

Any ideas?

McDave17 · July 30, 2020, 5:13pm

Found the issue:
vtkm_filter-9.0.lib is to big for the linker. Any ideas on how to fix or why only I encounter this problem?

DavidChipman · August 3, 2020, 11:10pm

Maybe send a message to NVidia? This sounds like an internal problem in their tools. I’m sure they have developer message boards. Just a thought.

McDave17 · August 4, 2020, 11:10am

Well over there I got pointed to VTK message boards

RobertMaynard · August 4, 2020, 7:12pm

The message is informative, the build-system will make sure the library is built statically.

McDave17:

If been fiddeling around and can’t find the error source. I tried disabeling AcceleratorsVTKm and using different compilers / linkers. I keep running into either one of or both issues:
When VTK-m is built with CUDA enabled all compilation units that include DeviceAdapterTagCuda must use the cuda compiler
or that the DeviceAdapterCuda was not defined.

Any .cpp or .cxx file that includes a VTK-m header must be compiled with the cuda compiler if the CUDA backend is enabled.

You can try out https://gitlab.kitware.com/vtk/vtk-m/-/merge_requests/2186 which is the initial attempt at fixing this problem. The issue is caused by a limitation of windows where static libraries have to < 4GB in size.

McDave17 · August 5, 2020, 6:16pm

Thank you. Using that merge request resolved my issue regarding VTK-m. However the AcceleratorsVTKm Module still fails while building with following error:

197>                Ziel "CudaBuildCore":
197>                  Compiling CUDA source file ..\..\..\src\Accelerators\Vtkm\vtkmlib\ArrayConverters.cxx...
197>                  Compiling CUDA source file ..\..\..\src\Accelerators\Vtkm\vtkmlib\DataSetConverters.cxx...
197>                  Compiling CUDA source file ..\..\..\src\Accelerators\Vtkm\vtkmlib\ImageDataConverter.cxx...
197>                  Compiling CUDA source file ..\..\..\src\Accelerators\Vtkm\vtkmlib\CellSetConverters.cxx...
197>                  Compiling CUDA source file ..\..\..\src\Accelerators\Vtkm\vtkmlib\ImplicitFunctionConverter.cxx...
197>
197>                  C:\vtk\bin\Accelerators\Vtkm>"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin\nvcc.exe" -gencode=arch=compute_50,code=\"sm_50,compute_50\" --use-local-env -ccbin "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\bin\HostX64\x64" -x cu -rdc=true  -IC:\vtk\bin\Accelerators\Vtkm -IC:\vtk\src\Accelerators\Vtkm -IC:\vtk\bin\Common\Core -IC:\vtk\src\Common\Core -IC:\vtk\bin\Common\DataModel -IC:\vtk\src\Common\DataModel -IC:\vtk\bin\Common\Math -IC:\vtk\src\Common\Math -IC:\vtk\bin\Common\Transforms -IC:\vtk\src\Common\Transforms -IC:\vtk\bin\Common\ExecutionModel -IC:\vtk\src\Common\ExecutionModel -IC:\vtk\bin\Filters\General -IC:\vtk\src\Filters\General -IC:\vtk\bin\Common\Misc -IC:\vtk\src\Common\Misc -IC:\vtk\bin\Filters\Core -IC:\vtk\src\Filters\Core -IC:\vtk\bin\Filters\Geometry -IC:\vtk\src\Filters\Geometry -IC:\vtk\bin\Imaging\Core -IC:\vtk\src\Imaging\Core -IC:\vtk\bin\ThirdParty\vtkm\vtkvtkm -IC:\vtk\src\ThirdParty\vtkm\vtkvtkm -I"C:\vtk\src\ThirdParty\vtkm\vtkvtkm\vtk-m" -I"C:\vtk\bin\ThirdParty\vtkm\vtkvtkm\vtk-m\include" -I"C:\vtk\src\ThirdParty\vtkm\vtkvtkm\vtk-m\vtkm\thirdparty\taotuple" -I"C:\vtk\src\ThirdParty\vtkm\vtkvtkm\vtk-m\vtkm\thirdparty\optionparser" -I"C:\vtk\src\ThirdParty\vtkm\vtkvtkm\vtk-m\vtkm\thirdparty\diy" -I"C:\vtk\src\ThirdParty\vtkm\vtkvtkm\vtk-m\vtkm\thirdparty\lcl\vtkmlcl" -IC:\vtk\bin\Utilities\KWIML -IC:\vtk\src\Utilities\KWIML -IC:\vtk\bin\Utilities\KWSys -IC:\vtk\src\Utilities\KWSys -IC:\vtk\bin\ThirdParty\vtkm -IC:\vtk\src\ThirdParty\vtkm -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\include"     --keep-dir x64\Debug -maxrregcount=0  --machine 64 --compile -cudart static -Xnvlink -w -Xcudafe --diag_suppress=extra_semicolon /wd4702 /wd4512 /wd4510 /wd4503 --expt-relaxed-constexpr -Xcompiler="/EHsc -Zi -Ob0 /Gy /bigobj" -g   -D_WINDOWS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -DVTK_IN_VTK -D"CMAKE_INTDIR=\"Debug\"" -DWIN32 -D_WINDOWS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -DVTK_IN_VTK -D"CMAKE_INTDIR=\"Debug\"" -D_MBCS -Xcompiler "/EHsc /W3 /nologo /Od /FdAcceleratorsVTKm.dir\Debug\AcceleratorsVTKm.pdb /FS /Zi /RTC1 /MDd /GR" -o AcceleratorsVTKm.dir\Debug\vtkmTriangleMeshPointNormals.obj "C:\vtk\src\Accelerators\Vtkm\vtkmTriangleMeshPointNormals.cxx"
197>                  CUDACOMPILE : nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
197>                  nvcc fatal   : A single input file is required for a non-link phase when an outputfile is specified
197>                  C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\BuildCustomizations\CUDA 11.0.targets(772,9): error MSB3721: Der Befehl ""C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin\nvcc.exe" -gencode=arch=compute_50,code=\"sm_50,compute_50\" --use-local-env -ccbin "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\bin\HostX64\x64" -x cu -rdc=true  -IC:\vtk\bin\Accelerators\Vtkm -IC:\vtk\src\Accelerators\Vtkm -IC:\vtk\bin\Common\Core -IC:\vtk\src\Common\Core -IC:\vtk\bin\Common\DataModel -IC:\vtk\src\Common\DataModel -IC:\vtk\bin\Common\Math -IC:\vtk\src\Common\Math -IC:\vtk\bin\Common\Transforms -IC:\vtk\src\Common\Transforms -IC:\vtk\bin\Common\ExecutionModel -IC:\vtk\src\Common\ExecutionModel -IC:\vtk\bin\Filters\General -IC:\vtk\src\Filters\General -IC:\vtk\bin\Common\Misc -IC:\vtk\src\Common\Misc -IC:\vtk\bin\Filters\Core -IC:\vtk\src\Filters\Core -IC:\vtk\bin\Filters\Geometry -IC:\vtk\src\Filters\Geometry -IC:\vtk\bin\Imaging\Core -IC:\vtk\src\Imaging\Core -IC:\vtk\bin\ThirdParty\vtkm\vtkvtkm -IC:\vtk\src\ThirdParty\vtkm\vtkvtkm -I"C:\vtk\src\ThirdParty\vtkm\vtkvtkm\vtk-m" -I"C:\vtk\bin\ThirdParty\vtkm\vtkvtkm\vtk-m\include" -I"C:\vtk\src\ThirdParty\vtkm\vtkvtkm\vtk-m\vtkm\thirdparty\taotuple" -I"C:\vtk\src\ThirdParty\vtkm\vtkvtkm\vtk-m\vtkm\thirdparty\optionparser" -I"C:\vtk\src\ThirdParty\vtkm\vtkvtkm\vtk-m\vtkm\thirdparty\diy" -I"C:\vtk\src\ThirdParty\vtkm\vtkvtkm\vtk-m\vtkm\thirdparty\lcl\vtkmlcl" -IC:\vtk\bin\Utilities\KWIML -IC:\vtk\src\Utilities\KWIML -IC:\vtk\bin\Utilities\KWSys -IC:\vtk\src\Utilities\KWSys -IC:\vtk\bin\ThirdParty\vtkm -IC:\vtk\src\ThirdParty\vtkm -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\include"     --keep-dir x64\Debug -maxrregcount=0  --machine 64 --compile -cudart static -Xnvlink -w -Xcudafe --diag_suppress=extra_semicolon /wd4702 /wd4512 /wd4510 /wd4503 --expt-relaxed-constexpr -Xcompiler="/EHsc -Zi -Ob0 /Gy /bigobj" -g   -D_WINDOWS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -DVTK_IN_VTK -D"CMAKE_INTDIR=\"Debug\"" -DWIN32 -D_WINDOWS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -DVTK_IN_VTK -D"CMAKE_INTDIR=\"Debug\"" -D_MBCS -Xcompiler "/EHsc /W3 /nologo /Od /FdAcceleratorsVTKm.dir\Debug\AcceleratorsVTKm.pdb /FS /Zi /RTC1 /MDd /GR" -o AcceleratorsVTKm.dir\Debug\vtkmTriangleMeshPointNormals.obj "C:\vtk\src\Accelerators\Vtkm\vtkmTriangleMeshPointNormals.cxx"" wurde mit Code 1 beendet.
197>                Die Erstellung des Ziels "CudaBuildCore" im Projekt "AcceleratorsVTKm.vcxproj" ist abgeschlossen -- FEHLER.

and: output.txt (94.8 KB)

RobertMaynard · August 5, 2020, 6:36pm

This is generally caused when the input to the compiler is bad. That happens when quotes are missing or added in the wrong places, or if unknown parameters are passed

I would try removing flags from the link line to see what is causing the problem

McDave17 · August 6, 2020, 5:14pm

So I found the Flags causing the problem:

/wd4702 /wd4512 /wd4510 /wd4503

These are not passed into -Xcompiler as they should be.
Looking at the Accelerators\Vtkm\CMakeLists.txt this seems to be supposed to specifically do that:

if (MSVC)
  set(msvc_warning_flags
    # Generates numerous false positives with template code about unreachable
    # code
    /wd4702
    # Generates numerous warning that implicit assignment operators can't be
    # constructed. This is understood and we don't care.
    /wd4512
    # Generates numerous warning that implicit constructors can't be
    # constructed. This is understood and we don't care.
    /wd4510
    # Generates warnings when decorated names are over 4096 characters
    # This compiler warning is deprecated and no longer occurs with VS2017+.
    # So we suppress for VS2015
    /wd4503
    )
  set(vtkm_msvc_flags)
  foreach (msvc_warning_flag IN LISTS msvc_warning_flags)
    if (TARGET vtkm::cuda)
      list(APPEND vtkm_msvc_flags
        $<$<COMPILE_LANGUAGE:CUDA>:-Xcompiler=${msvc_warning_flag},${msvc_warning_flag}>)
    else ()
      list(APPEND vtkm_msvc_flags
        ${msvc_warning_flag})
    endif ()
  endforeach ()
  vtk_module_compile_options(VTK::AcceleratorsVTKm
    PRIVATE
      ${msvc_warning_flags})
endif ()

I’ve tried to fiddle around with this but didn’t seem to manage anything, any suggestions? I see this was a issue on here before:

RobertMaynard · August 11, 2020, 9:31am

The bug is that we are adding the wrong variable, it should be

  vtk_module_compile_options(VTK::AcceleratorsVTKm
    PRIVATE
      ${vtkm_msvc_flags})

Can you try that out and report back?

McDave17 · August 6, 2020, 5:56pm

That actually resolved the issue regarding the flags. Now I no longer get the to many input files error and the compilation actually runs for about 30 minutes instead of instantly failing, so this probably should be fixed in the master at some point.

However compilation is still failing, now with return code 2 instead of 1, but sadly no verbose message:

1>    1 error detected in the compilation of "C:/vtk/src/Accelerators/Vtkm/vtkmNDHistogram.cxx".
1>    CUDACOMPILE : nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
1>    vtkmNDHistogram.cxx
1>    C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\BuildCustomizations\CUDA 11.0.targets(772,9): error MSB3721: Der Befehl ""C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin\nvcc.exe" -gencode=arch=compute_50,code=\"sm_50,compute_50\" --use-local-env -ccbin "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\bin\HostX64\x64" -x cu -rdc=true  -IC:\vtk\bin\Accelerators\Vtkm -IC:\vtk\src\Accelerators\Vtkm -IC:\vtk\bin\Common\Core -IC:\vtk\src\Common\Core -IC:\vtk\bin\Common\DataModel -IC:\vtk\src\Common\DataModel -IC:\vtk\bin\Common\Math -IC:\vtk\src\Common\Math -IC:\vtk\bin\Common\Transforms -IC:\vtk\src\Common\Transforms -IC:\vtk\bin\Common\ExecutionModel -IC:\vtk\src\Common\ExecutionModel -IC:\vtk\bin\Filters\General -IC:\vtk\src\Filters\General -IC:\vtk\bin\Common\Misc -IC:\vtk\src\Common\Misc -IC:\vtk\bin\Filters\Core -IC:\vtk\src\Filters\Core -IC:\vtk\bin\Filters\Geometry -IC:\vtk\src\Filters\Geometry -IC:\vtk\bin\Imaging\Core -IC:\vtk\src\Imaging\Core -IC:\vtk\bin\ThirdParty\vtkm\vtkvtkm -IC:\vtk\src\ThirdParty\vtkm\vtkvtkm -I"C:\vtk\src\ThirdParty\vtkm\vtkvtkm\vtk-m" -I"C:\vtk\bin\ThirdParty\vtkm\vtkvtkm\vtk-m\include" -I"C:\vtk\src\ThirdParty\vtkm\vtkvtkm\vtk-m\vtkm\thirdparty\taotuple" -I"C:\vtk\src\ThirdParty\vtkm\vtkvtkm\vtk-m\vtkm\thirdparty\optionparser" -I"C:\vtk\src\ThirdParty\vtkm\vtkvtkm\vtk-m\vtkm\thirdparty\diy" -I"C:\vtk\src\ThirdParty\vtkm\vtkvtkm\vtk-m\vtkm\thirdparty\lcl\vtkmlcl" -IC:\vtk\bin\Utilities\KWIML -IC:\vtk\src\Utilities\KWIML -IC:\vtk\bin\Utilities\KWSys -IC:\vtk\src\Utilities\KWSys -IC:\vtk\bin\ThirdParty\vtkm -IC:\vtk\src\ThirdParty\vtkm -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\include"     --keep-dir x64\Debug -maxrregcount=0  --machine 64 --compile -cudart static -Xnvlink -w -Xcudafe --diag_suppress=extra_semicolon --expt-relaxed-constexpr -Xcompiler="/EHsc -Zi -Ob0 /wd4702,/wd4702 /wd4512,/wd4512 /wd4510,/wd4510 /wd4503,/wd4503 /Gy /bigobj" -g   -D_WINDOWS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -DVTK_IN_VTK -D"CMAKE_INTDIR=\"Debug\"" -DWIN32 -D_WINDOWS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -DVTK_IN_VTK -D"CMAKE_INTDIR=\"Debug\"" -D_MBCS -Xcompiler "/EHsc /W3 /nologo /Od /FdAcceleratorsVTKm.dir\Debug\AcceleratorsVTKm.pdb /FS /Zi /RTC1 /MDd /GR" -o AcceleratorsVTKm.dir\Debug\vtkmNDHistogram.obj "C:\vtk\src\Accelerators\Vtkm\vtkmNDHistogram.cxx"" wurde mit Code 2 beendet.
1>  Die Erstellung des Ziels "CudaBuildCore" im Projekt "AcceleratorsVTKm.vcxproj" ist abgeschlossen -- FEHLER.

RobertMaynard · August 9, 2020, 12:53pm

Fixed: https://gitlab.kitware.com/vtk/vtk/-/merge_requests/7155

Weird that you are getting an error code 2 on failure. Is the machine running out of memory?

McDave17 · August 10, 2020, 3:23pm

I recompiled using the newly merged master and validated that it’s still using correct msvc params. Works.

Thing is, I still get the error code 2 on failure. Interesting log lines:
204>C:\vtk\src\Common\Core\vtkSparseArray.txx(412): error : the object has type qualifiers that are not compatible with the member function "vtkArray::GetDimensions"

153> CUDACOMPILE : nvcc error : 'cl.exe' died with status 0xC0000005 (ACCESS_VIOLATION)

The access violation might also point towards memory being full? Would it make any difference to try and compile with nvlink.exe instead of cl.exe as linker?

Edit:
RAM doesn’t peak over 90% usage.

RobertMaynard · August 10, 2020, 5:03pm

That won’t work. nvlink should only be used to do device linking

McDave17 · August 10, 2020, 7:44pm

So I managed to successfully compile when combining a couple of pending merge requests. All issues I had should be fixed once this is pushed to the VTK-m master. Thank you for your help.

RobertMaynard · August 10, 2020, 7:52pm

Excellent to hear that fixes the last of your build issues.

McDave17 · August 10, 2020, 9:51pm

Sadly it seems I cheered to soon.
vtkSparseArray.txx(412): error: the object has type qualifiers that are not compatible with the member function "vtkArray::GetDimensions" still is a issue, not when compiling VTKmCore, but when compiling VTKmFilters. Any suggestions?

McDave17 · August 11, 2020, 6:33pm

This only is a issue when compiling with CUDA enabled. When CUDA is disabled the compilation is fine. In attempt of fixing this I tried using the newest commits on the master branch of both VTK and VTK-m, while compilation is still sucessfull when not using CUDA I get a error on running my project where I use VTK:

grafik

Rougly translated the simply means that the vtkmdiympi_nompi.dll could not be found. MPI is disabled. This is only when using the newest commit, and not the current release branch. Am I doing something wrong or was there maybe a bug committed?

This is my projects CMakeLists.txt, should be fine, right?
CMakeLists.txt (655 Bytes)