We have Asp.net 4.5 mvc webapi, which has about 100 application domains, each of which contains an extension.
Now from time to time we have a hang api. No route even responds to api status, which returns only a string, does not respond.
When it freezes, the site has about 120 threads (which is quite normal) and about 12 GB of RAM (which is unusually high).
When we do a memory dump, we can see that the site is always in the middle of the garbage collection.
In most cases, we see that most threads hang on the stack with serialization processing code between application domains and wait for GC. We also have a lot of serialization, for example, for linking the application domain and in combination with some redis caches
The event, while waiting for about 5 minutes, does not stop hanging. Are there any known garbage collection issues associated with many application domains?
Since the site is hosted on IIS, the background GC must always be active.
When I look at the time on the GC performance counter, I see that GC almost always works
I can see that when a site freezes constantly 40% of the time in gc
When the site is in this state, I can also see that the memory is constantly increasing.
Any hints that you need to test or try to improve?
Most likely, it will be useful when updating the runtime to 4.5.2? eg:
ntdll!NtWaitForSingleObject+a
KERNELBASE!WaitForSingleObjectEx+94
clr!CLREventWaitHelper2+38
clr!CLREventWaitHelper+1f
clr!CLREventBase::WaitEx+70
clr!SVR::gc_heap::wait_for_gc_done+55
clr!SVR::WaitLonger+9e
clr!SVR::GCHeap::Alloc+224
clr!JIT_New+142
[[HelperMethodFrame]]
mscorlib_ni!System.Runtime.Serialization.ObjectManager.RegisterFixup(System.Runtime.Serialization.FixupHolder, Int64, Int64)+d1
mscorlib_ni!System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run()+128
mscorlib_ni!System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(System.Runtime.Remoting.Messaging.HeaderHandler, System.Runtime.Serialization.Formatters.Binary.__BinaryParser, Boolean, Boolean, System.Runtime.Remoting.Messaging.IMethodCallMessage)+db
mscorlib_ni!System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(System.IO.Stream, System.Runtime.Remoting.Messaging.HeaderHandler, Boolean, Boolean, System.Runtime.Remoting.Messaging.IMethodCallMessage)+1bf
mscorlib_ni!System.Runtime.Remoting.Channels.CrossAppDomainSerializer.DeserializeObject(System.IO.MemoryStream)+f8
mscorlib_ni!System.Runtime.Remoting.Messaging.SmuggledMethodCallMessage.FixupForNewAppDomain()+de8a4e
mscorlib_ni!System.Runtime.Remoting.Channels.CrossAppDomainSink.DoDispatch(Byte[], System.Runtime.Remoting.Messaging.SmuggledMethodCallMessage, System.Runtime.Remoting.Messaging.SmuggledMethodReturnMessage ByRef)+33
mscorlib_ni!System.Runtime.Remoting.Channels.CrossAppDomainSink.DoTransitionDispatchCallback(System.Object[])+92
clr!CallDescrWorkerInternal+83
clr!CallDescrWorkerWithHandler+4a
clr!DispatchCallDebuggerWrapper+1f
clr!DispatchCallSimple+88
clr!ThreadNative::InternalCrossContextCallback+2ea
[[ContextTransitionFrame]]
[[HelperMethodFrame_PROTECTOBJ] (System.Threading.Thread.InternalCrossContextCallback)] System.Threading.Thread.InternalCrossContextCallback(System.Runtime.Remoting.Contexts.Context, IntPtr, Int32, System.Threading.InternalCrossContextDelegate, System.Object[])
mscorlib_ni!System.Runtime.Remoting.Channels.CrossAppDomainSink.DoTransitionDispatch(Byte[], System.Runtime.Remoting.Messaging.SmuggledMethodCallMessage, System.Runtime.Remoting.Messaging.SmuggledMethodReturnMessage ByRef)+a0
mscorlib_ni!System.Runtime.Remoting.Channels.CrossAppDomainSink.SyncProcessMessage(System.Runtime.Remoting.Messaging.IMessage)+15d
mscorlib_ni!System.Runtime.Remoting.Proxies.RemotingProxy.CallProcessMessage(System.Runtime.Remoting.Messaging.IMessageSink, System.Runtime.Remoting.Messaging.IMessage, System.Runtime.Remoting.Contexts.ArrayWithSize, System.Threading.Thread, System.Runtime.Remoting.Contexts.Context, Boolean)+8c
mscorlib_ni!System.Runtime.Remoting.Proxies.RemotingProxy.InternalInvoke(System.Runtime.Remoting.Messaging.IMethodCallMessage, Boolean, Int32)+22c
mscorlib_ni!System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(System.Runtime.Remoting.Proxies.MessageData ByRef, Int32)+1f4
clr!CTPMethodTable__CallTargetHelper3+12
clr!CallTargetWorker2+74
clr!CTPMethodTable::OnCall+1fb
clr!TransparentProxyStub_CrossContextPatchLabel+a
[[TPMethodFrame] (SR.BusPortal.Providers.Contract.Common.IAdapterSearcher.SearchAsync)] SR.BusPortal.Providers.Contract.Common.IAdapterSearcher.SearchAsync(SR.BusPortal.Providers.Contract.Common.AdapterSearchParameters)
SR.BusPortal.Search.Steps.SearchStepOneWay`2+<SearchOneWayAsync>d__3[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext()+73