If the assembly is loaded as neutral for the domain, this means that its code can be reused in another application. If the assembly is loaded into more than one AppDomain as domain-related (default), each AppDomain gets its own copy of the code. This has several poor performance characteristics. First up is the CPU. If the build has its own image, only the first AppDomain can use its own image. All other AppDomains will need to JIT-compile the code, which can lead to significant processor costs.
Further, the JIT-compiled code is in private memory, so it cannot be transferred to other processes or AppDomains. If the assembly had an NGEN image, then the first AppDomain uses the image. All other AppDomains must JIT-compile the code, which means that the MSIL DLL for this assembly is also loaded. This is the worst case scenario from the point of view of a cold start, since disk access for this assembly will be doubled.
Downloading the assembly as a domain neutral ensures that the native image, if it exists, is used in all AppDomains created in the application. If the native image does not exist, there is still an advantage when loading assemblies as a neutral domain, because the code is compiled only once and then used by all AppDomains applications in the application.