The reason you get this is fair and simple. When the query is executed, some steps are performed. These are analysis, algebraization, optimization and compilation. Part of the algebra in this case will receive all the objects necessary for this request. Optimization will use these objects to create a better query plan, which will be compiled and executed ...
So, when you look at this part, you will see that it will scan the table for #temptable. And #temptable is defined as a way to create your table. The fact that you do some calculation on this is another matter. The column still has the nvarchar data type.
To find out how this works, you need to know how to read the request. First, all objects are extracted (from the table, the internal connection table), then the predicates (where, on), then the grouping, etc., Then the choice of columns (with cast), and then the order.
Thus, keeping in mind that you have a combination of choices, the optimizer will still handle it that way. Since your choice is subordinate to the party and joins parts of your request, this will cause this error.
Hope I made it a little clear?
source share