This is not a solution that a user can make in the security model that Silverlight implements. Its "owner" of the destination site is to select the policy. There is no way to determine that the user is the owner of the destination.
The solution is to use the site hosting the Silverlight application as a proxy to access other services on the client network.
source share