My vtk version is 9.1, C++.
I use vtkUnstructrudeGrid to store my data, and pick one vtkCell from the ugrid, translate to vtkPolyhedron, check if IsConvex().

put an example below, the right result of IsConvex() is true but I got false:

Indeed, it was a bug in the algorithm, and I have fixed it.
The IsConvex() function uses the algorithm from Devillers et al. I learned this algorithm from Determining Convexity of Polyhedra and tried to fix the bug. When determining “the upper face must not be vertical,” one should not use the face center (based on specific points), but should instead rely on the plane itself. Therefore, I modified the code as follows:

// 3. We get the z component of the normal of the highest face
// If this is null, the face is in the vertical plane
for (int i=0;i<3;++i){
c0[i] = c0[i]-c[i];
c1[i] = c1[i]-c[i];
}
tmp0 = c0[2]-dot(c0,n)*n[2] > c1[2]-dot(c1,n)*n[2] ? n0[2] : n1[2];
if (std::abs(tmp0) < eps)
{
continue;
}