Finding Transformation Angle according to a custom axis

It is hard for me to explain my problem. I have a rectangular prism and it is placed in coordinate system with a random orientation

the rectangular prism has its own normal vector, short plane vector(short edge axis) and long plane vector (long edge axis) like main coordinate system. My job is to rotate that rectangular prism and find out its angles according to its coordinate system NOT the main coordinate system(xyz)

By doing this I can report at the end of the all transformations that the prism is rotated these angles by its short plane vector, long plane vector and normal

This is what I do:

I rotate the rectangular prism according its 3 vectors using rotateWXYZ. Till here it is great.

But at the end of the transformations, I need to report how much the rectangular prism is rotated according to its 3 vectors at start.

After all transformations I call GetOrientation() and it gives angles according to main coordinate system(xyz) But I need to know the angles according to the 3 vectors at start.

Also I call GetOrientationWXYZ() but it gives angle-axis result which is great but it doesn’t work for me. I need to find the angle-axis acording to custom axis

My be they work but I dont know how I can do it, Can anybody guide me.
I feel like I am so close to the solution but

I’ll answer this question mathematically, and from there you can probably figure out the code.

Let’s define three transformation matrices (either 3x3 or 4x4 can be used).

  • A is the original orientation of the prism
  • B is the rotated orientation of the prism
  • R is the rotation transformation

The equation that links these together is:

   B = R A

You want to know the rotation of B (rotated prism) with respect to A (original prism). This requires matrix division (or equivalently, matrix inversion):

  R = B inverse(A)

So the next question is, how do you build the matrices A and B that you can shove into this equation? The answer is, build them from the short, long, and normal vector. Let’s define:

  • [sx, sy, sz] is short vector (as a unit vector)
  • [lx, ly, lz] is long vector (as a unit vector)
  • [nx, ny, nz] is normal vector (as a unit vector)

The matrix for the original pose of the prism would be built like this (I’m assuming your three vectors are orthogonal to each other):

      / sx  lx  nx \ 
A =   | sy  ly  ny |
      \ sz  lz  nz /

B can be built in a similar fashion. I used 3x3 in this example, but 4x4 work just as well (as long as you put a 1 in the lower-right corner).

Once you have A and B, use

  R = B inverse(A)

and then extract the orientation from R. I haven’t double-checked the math above, so there is a chance that I’ve made a mistake, but it should get you on the right track.

Hi David, First of all, thank you very much for your time you spent to think and share. I deeply appreciate it. :bowing_man:
I had been lost and now I got a clue to chase :+1: I will let you know if I come to a solution.