Typically, antivirus software installs a filter driver that connects to a file system driver in the Windows kernel. Thus, all requests to the file file are first delivered to the filter, which then determines whether the request should be forwarded or rejected.
Please note that connecting the Windows API or any other API in user mode is usually not enough, as malicious software can always directly access the kernel, bypassing the connected API.
source share