It doesn't matter if you use matrices or not. Here's a simple C ++ example without matrices, using everything that I mention in my comments:
and use:
// clear and add diagonal line for testing once: cls(); // and this do in some timer or whatever: //rot45cw(); rot90cw();
Here's the 90deg preview:

Here is the 45deg preview:

As you can see, rotating 45deg is a problem since it is not a 1: 1 display, so some cells will be displayed in multiple cells. With a fixed resolution, you can manually display 1: 1, but I doubt that it will be easily implemented algorithmically for dynamic resolution.
Yes. You only use maps (3x3) where 45-degree rotation is possible, but your problem persists, because when you map individual pixels, some will be copied again, and if you consider that you are viewing characters, it will look bad.
If I put everything together, I would prefer to use 90deg rotation only if you did not get a vector representation of the image ...
Symbol rotation can be accelerated with LUT
[Edit1] 45 degree rotation
I will teach a little and find out the solution for turns of 45 degrees. You must use another rotation core. Do not turn around circles, but circles around a circle 1/8 of a circle. To better understand a small example here:
This is a 1: 1 comparison, so no problem with that. The code in C ++ is as follows:
The tables dx,dy just similar to sin and cos . Finally, here is a preview:

but roughly, if you rotate the square around its center, it will not be as you expected !!! Here is a CCW example:
