If all you are trying to do is determine if the process on the same computer will record the screen, you will probably find a way to do this for some things. There are certain APIs that are used by many screenwriting programs that you can intercept with EasyHook or something similar, but you will skip the rest that work differently.
UltraVNC, for example, has a mirrored video driver that you can install that gives you notifications of screen updates, as well as direct access to the video clip, which is great for recording on screen or dubbing. How will your code respond to this?
, , . , , HDCP .
, , , -, .
... . -, , - , .
.