This is a new restriction introduced in Silverlight 5. They updated the security model to make it more secure, but also explicitly allow this behavior.
Now you can run a trusted application in a browser, and not as applications other than a browser. You need to configure the application as if it were a trusted OOB application, because the "Require increased trust" and "Show installation options" settings are part of the OOB configuration.
To enable the browser in trusted applications, you need to edit the registry on the client machine:
Create / Install AllowElevatedTrustAppsInBrowser to 1
This can be found in HKEY_LOCAL_MACHINE\Software\Microsoft\Silverlight\ for 32-bit machines or HKEY_LOCAL_MACHINE\Software\Wow6432NodeaMicrosoft\Silverlight\ for 64-bit versions.
In addition, .xap files must be signed with a valid certificate, and the certificate must be installed.
A source
source share