I think I can be late for the party, but here is the solution. The installer team blog post explains how Restart Manager decides whether a file dialog should pop up. In particular (section "Interaction with the Windows Installer Manager", section 3.b.):
If the package is created so that the services discovered by RM are disabled due to the creation of Service * tables, then these services will not be displayed in the Files In Use dialog box.
(italics mine). Useful, but not immediately useful, because it is not so detailed. But since my service caused the same problem as the OP with
<ServiceControl Stop="uninstall" ... />
I just changed the value to both
<ServiceControl Stop="both" ... />
which was probably the only thing left that could make it βsuch that,β and boom, fireworks, magic:
MSI (s) (50:A0) [21:50:30:352]: RESTART MANAGER: Detected that application with id 6408, friendly name 'XXXX', service short name 'xxxx', of type RmService and status 1 holds file[s] in use. MSI (s) (50:A0) [21:50:30:352]: RESTART MANAGER: Detected that the service xxxx will be stopped due to a service control action authored in the package before the files are updated. So, we will not attempt to stop this service using Restart Manager
It seems that in ServiceControl , to make RM happy ending, that the service will stop before updating the files.
In retrospect, this makes sense. Since the deletion part of the upgrade is performed using the old cached MSI database, RM does not look into it to see what happens when the corresponding product is deleted. Strictly speaking, there can be several products for removal, and the installer does not require anywhere that these related products (the found FindRelatedProducts action , including the old version of the same update code) are actually associated with the service that manages in the current package. Therefore, he does not care about the action of the service during uninstallation, as it is written in the current package (in any case, this does not apply to the installation action!). Consistency requires simple and direct evidence that the service will be stopped before the files that will be used are overwritten, collecting such data only from the current package.
Thus, it is likely that RM only cares about the msidbServiceControlEventStop (0x002) flag during installation.