test timeouts vs slow debug tools like thread sanitizer

Development
1

Hi all,

I’ve recently been trying to get more VTK tests to pass under thread sanitizer. TSan can cause 10x runtime slowdown. This makes some tests timeout. Most interesting are ones like IOEnSightPython-TestEnSightGoldCombinedReader which set a custom timeout:

if (NOT VTK_TEST_TIMEOUT_TestEnSightGoldCombinedReader)
  set(VTK_TEST_TIMEOUT_TestEnSightGoldCombinedReader 300)
endif()
set_tests_properties(VTK::IOEnSightPython-TestEnSightGoldCombinedReader
  PROPERTIES TIMEOUT ${VTK_TEST_TIMEOUT_TestEnSightGoldCombinedReader})

ctest’s --timeout is not helpful, as, per its docs, it “sets a timeout on all tests that do not already have a timeout set on them via the TIMEOUT property.”

Other than editing that 300, is there an idiomatic way to lengthen the timeout?

Thanks,

Sean

2

I’m afraid not. In F3D we have a dedicated CMake var to handle the sanitizer usecase, and takes that into account for tests timetouts: f3d/cmake/f3dTest.cmake at master · f3d-app/f3d · GitHub

3

OK I’ve filed a suggestion with ctest: https://gitlab.kitware.com/cmake/cmake/-/work_items/27832

But for now, for IOEnSightPython-TestEnSightGoldCombinedReader specifically, is specifying that 300 timeout even necessary? The comment makes it seem like it’s trying to make the timeout larger, but isn’t the ctest default 1500s? Seems like this stanza could just be removed, no?

Sean

4

I dont think so, at least not in VTK.

5

Would setting a larger value in VTK_TEST_TIMEOUT_TestEnSightGoldCombinedReader when you configure build would work for you? There are about 8 tests with such manually set timeouts, so you may need to override a few more. It would be certainly nicer to have a multiplier variable in VTK to scale all these hardcoded timeout values at once.

6

Hmm, what is the default timeout then? I know in my cdash submissions I specify CTEST_TEST_TIMEOUT, but if I just run ctest on my local computer, tests that timeout always seem to timeout after 1500s. Where does that come from?

7

1500 comes from the default DART_TESTING_TIMEOUT value. This is not documented well, as pointed out in this CMake issue.

8

Nice sleuthing Andras!

So then this comment about “extending” the timeout makes little sense to me:

# This test can take much longer on Windows than other platforms, so extend the timeout
if (NOT VTK_TEST_TIMEOUT_TestEnSightGoldCombinedReader)
  set(VTK_TEST_TIMEOUT_TestEnSightGoldCombinedReader 300)
endif()

It seems this is actually decreasing the timeout vs the default 1500. Maybe ctest’s default was lower years ago?

Searching for PROPERTIES TIMEOUT, every other similar stanza also is setting < 1500.

9

VTK sets 100s timeout for its CI tests on gitlab (see for example here). Compared to that, 300s is an increase.

Instead of hardcoding 300, it should be VTK_TEST_DEFAULT_TIMEOUT * 3. Something like this (similar to what @mwestphal implemented in f3d):

if (NOT VTK_TEST_DEFAULT_TIMEOUT)
  set(VTK_TEST_DEFAULT_TIMEOUT 100)
endif()

...

# This test can take much longer on Windows than other platforms, so extend the timeout
if (NOT VTK_TEST_TIMEOUT_TestEnSightGoldCombinedReader)
  math(EXPR VTK_TEST_TIMEOUT_TestEnSightGoldCombinedReader "3 * ${VTK_TEST_DEFAULT_TIMEOUT}")
endif()

Custom timeout is used for 8 tests, so it could worth adding a helper function or macro for this.