Thank you for the suggestions!
I quite like the std::vector idea. Python behavior would be almost exactly what we need (the only imperfection I see is that there would be no hints that the returned vector can only have 0 or 3 elements). In C++, it is OK, too (it is just a but heavier than a fixed array and always need to check the returned vector size).
Returning NaN-s cash work, too, but it is easier to forget about the value check and the check is a bit me work (need to check 3 values for NaN).