For brevity, I use a fake root key. In practice, replace Hive_Key with HKEY_LOCAL_MACHINE for system defaults or HKEY_CURRENT_USER for user settings. Any of these keys may exist, or both. If they both exist, the HKCU key takes precedence.
To associate an extension with a file type, you need to set a default value for the extension key ( Hive_Key\Software\Classes\.ext ) with the selected file type, setting the default key value.
The actual running program, as well as other file type data, is in the file type. File types are marked with the so-called ProgID (short for "Programmatic Identifier", which is a more readable version of the class identifier). The ProgID keys are located in Hive_Key\Software\Classes , and the approximate value for this illustration can be ext_auto_key .
ProgID can have a default value, which will be a friendly description of the file type in Explorer (for example, "Microsoft Word Document"). It is up to you to choose a description that is easy to understand for users.
ProgID may have a DefaultIcon unit where the file type icon is stored. This icon path is the default value of this key.
The ProgID key can be a subsection of the shell that will contain the context menu items in the files and the program that this context menu item will call. Like the default value for ProgID, the default value for the verb is the text that will be displayed in the context menu. The default value of this shell key is the default verb key name, which is the verb that is called when the user double-clicks the file.
These are the Verbs context menu Verbs . In our example, the verb that opens the file using Notepad will look like this: Hive_Key\Software\Classes\ext_auto_file\shell\open\command with a default value of notepad.exe %1 .
Here you put your program path. If your program is located in the PATH system, like notepad.exe, you do not need to specify the full path. In a more likely case, you need to specify the path to your exe. For testing purposes, you can simply install it as your build directory.
You asked how to check this stuff, and this can be done by first checking the default value of the extension key to get the ProgID, and then checking the shell subkey of the ProgID key to get the default, and then checking \shell\verb\command to get the path to the running program.
It may be interesting to open regedit.exe and view these registry keys for other types of files in order to better understand how it all works.
In addition, the above is all valid if the particular extension is not under the control of the program installed by default (default program) on the control panel. You can check this status by checking for the HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\ Explorer\FileExts\.EXT\UserChoice . If this is the case, you will need to deactivate the default program control before your settings take effect. This can be done by deleting this section of UserChoice .