I think Delaunay triangulation will always give you a convex hull. I’m not a mathematician, but vtkDelaunay2D (or anything that uses Delaunay triangulation) will always give you a convex output.
There are probably algorithms that will work for you, but I think that all of them will need for you to give them hints about what is “inside” and “outside” for your polygon. That might be something like a signed distance function, or distance gradients evaluated at the polygon points.
This may only be semi-related to what you were looking for, but there are lots of other posts here that may have hints, such as: How to triangulate a surface from point cloud - #4 by Paulo_Carvalho.