The first part of your question is the expected result, version 2 of the CLR cannot load .NET 4.0 assemblies, the metadata format has changed.
The last paragraph is more difficult to explain. This may have something to do with using LoadFile (), this is a plugin way to load assemblies. One possible failure mode is that your plug-in assembly may have links to assemblies 2.0. This is pretty normal when it has a link to another assembly that was compiled to target an earlier structure. Usually this is normally resolved by the collector, it simply replaces the link 2.0 with a link to the corresponding 4.0.
You can test this theory with ildasm.exe, run it on the plug-in, and view the manifest for .assembly directives. The version number is easy to see, you will get something like
.assembly extern mscorlib as mscorlib_2 { .publickeytoken = (B7 7A 5C 56 19 34 E0 89 ) .ver 2:0:0:0 }
Always welcome Assembly.LoadFrom ().
source share