Hi,
TL;DR: Can someone suggest a medium complex CI/CD example project that builds on Linux, OSX, and windows? I have been trying to follow the VTK implementation but it got dense fast for me. (maybe a paired down example if it exists, osx, windows, and linux targeting x64?)
One issue on windows I have found so far is that when I use download_python.cmake and pip install numpy it provides a 64/32 bit mismatched installation.
More context:
We are internally using gitlab along with the CI/CD tools. We generally use Docker for most things (with great success), however I have a need to do build on the windows directly. I started following the steps used in VTK, and it works to some extent, but the numpy/python mismatch has me stumped.
If there is another approach or a simplified example out there that someone would suggest I would be greatly appreciative.
VTK’s CI/CD extracts some information from the CMAKE_CONFIGURATION environment variable. For Windows specifically, the VCVARS* variables are also important. Our setup assumes that the runner sets VS160COMNTOOLS to where its installed Visual Studio lives. This is then used by .gitlab/ci/vcvarsall.ps1 to set up the Visual Studio toolchain environment.
Your situation seems like you have a 32bit environment loaded rather than the 64bit env.
Unfortunately, all of our projects are sharing the same infrastructure, so they’re all going to have this level of complexity.
Thank you so much @ben.boeckel. I was hopeful, but that matches up with the similar setups across the related projects i looked into.
Yes it seems super weird because I am downloading the same python file python-win64-3.8.6.tar.xz and using python -m pip install, but pip installs numpy-1.21.0-cp38-cp38-win32.whl which I would not have expected and gives an error when importing it, and trying to install the x64 version from download gives an error.
I think you are saying that VCVARS* and VS160COMNTOOLS variables would affect the how python and pip execute? I would have thought an x64 python executable would run in x64 mode all the time. Is there something in the environment (not properly set) that would be causing the mismatch? I will keep searching. Thank you so much.
Thank you for the hint, the environment appears to be the culprit.
I think the loaded environment might have something to do with it. I really don’t know; Python has so many weird behaviors that I can’t keep track of all of them, sorry.
I would too. shrug
They are made via the .gitlab/ci/wheels/prep_python_windows.ps1 script in VTK’s repository.
Thank you. It was definitely something funky with how python interacts with the visual studio variables. the scripts I am using aren’t smart enough to know if x86 or x64 is loaded, only that visual studio variables are loaded.
I definitely didn’t expect that to be the issue. making sure the vcvars loaded x64 the thing works. super funky business.
I think pip looks to see what (if any) VS toolchain is available if it needs to end up compiling C code and somewhere in there forgets to check whether it can actually use such code. I would suggest filing an issue upstream.