Changing Font Size on vtkCaptionAnnotationWidget

So I am trying to implement a caption widget and one of the properties for the caption that I would like to allow the user to change is the font size. I am currently doing this by going through the vtkCaptionRepresentation to set the text scale mode on the text actor to none and then set the font size on the caption text property;

m_CaptionRepresentation->GetCaptionActor2D()->GetTextActor()->SetTextScaleModeToNone();
m_CaptionRepresentation->GetCaptionActor2D()->GetCaptionTextProperty()->SetFontSize(m_FontSize);

This will successfully change the font size but then as soon as I do any sort of interaction with the render window (like click to rotate for example) the text reverts back to a much smaller size (what I assume must be the default size).

Also curiously (and most likely related) the border that is drawn around the caption text does not change when I change the font size. Perhaps I am just doing something wrong but as far as I can tell, this is how everyone says to change the font size.

I am using vtk 9.2.6