We can do this easily using rotation transformation matrix. However, it is applicable only on origin. However, the question does not want us to rotate about C. It wants us to rotate about origin. A simple trick to solve this problem is to translate the triangle so that C becomes origin, rotate it and then translate it back to its position.

So now, we know that we have to translate x coordinate of C by -5 and Y coordinate by -2 to make it 0,0. So, we do the same with coordinates of the other vertices of the triangle. The new triangle becomes:

Now, we can apply the transformation matrix on the coordinates.

You must be wondering why am I not calculating C coordinate values. This is because the rotation is happening about C. So there will be no change in C's coordinates. Even if we calculate them, they will remain unchanged. Hence it is a waste of time to calculate values for C's coordinates.

Now the last step would be to translate the coordinates back to their original position. So, we add (5,2) to each coordinate.

Coordinate A becomes ((-3/√2 + 5) , (-7/√2 + 2))

Coordinate B becomes ((-3/√2 + 5) , (-5/√2 + 2))

Coordinate C remains 5,2

thanks for the solution