NHibernate lists the same collection on multiple threads

Question

I have a production application (IIS8, MVC5, nHibernate DAL), and I will notice that using a lot of processor resources recently. Running the application for the application pool fixes it, but after doing some diagnostic and memory dumps from the server to analyze the problem, I noticed that the sequential pattern of several threads is trying to list the same collection. The most common point is that the application checks the roles of the user. I suspect that this may be more due to the fact that this code is run for each request to check permissions, so, most likely, it will focus on it?

public IList<Role> GetRoles(string username) { var login = GetLoginForUser(username); return !login.Groups.Any() ? new List<Role>() : login.Groups.SelectMany(x => x.Roles).OrderBy(x => x.DisplayName).ToList(); } 

My CurrentUser object has a simple interface that contains user information entered from the dependency converter. I checked that UserId is present and valid, all this is pretty straight forward. When I looked at dumps, when these two requests were hung up, I received a warning that several threads were enumerating the collection. When I checked the two threads in the dump, I saw almost the same stack traces. (I renamed some details of the namespace to the stack trace, but otherwise it has not changed). The UserId (and the resulting profile) are the same in both queries, so it appears because of two separate threads that try to load the same object from the database at almost the same time.

The stack trace is lower, but I'm not sure where to go from here to fix this.

 System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.Nullable`1[[System.Int32, mscorlib]], mscorlib]].FindEntry(System.__Canon)+129 System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.Nullable`1[[System.Int32, mscorlib]], mscorlib]].TryGetValue(System.__Canon, System.Nullable`1<Int32> ByRef)+12 NHibernate.AdoNet.ColumnNameCache.GetIndexForColumnName(System.String, NHibernate.AdoNet.ResultSetWrapper)+25 NHibernate.AdoNet.ColumnNameCache.GetIndexForColumnName(System.String, NHibernate.AdoNet.ResultSetWrapper)+25 NHibernate.AdoNet.ResultSetWrapper.GetOrdinal(System.String)+e NHibernate.AdoNet.ResultSetWrapper.GetOrdinal(System.String)+e NHibernate.Type.NullableType.NullSafeGet(System.Data.IDataReader, System.String)+29 NHibernate.Type.NullableType.NullSafeGet(System.Data.IDataReader, System.String[], NHibernate.Engine.ISessionImplementor, System.Object)+16 NHibernate.Type.NullableType.NullSafeGet(System.Data.IDataReader, System.String[], NHibernate.Engine.ISessionImplementor, System.Object)+16 NHibernate.Persister.Collection.AbstractCollectionPersister.ReadKey(System.Data.IDataReader, System.String[], NHibernate.Engine.ISessionImplementor)+14 NHibernate.Persister.Collection.AbstractCollectionPersister.ReadKey(System.Data.IDataReader, System.String[], NHibernate.Engine.ISessionImplementor)+14 NHibernate.Loader.Loader.ReadCollectionElement(System.Object, System.Object, NHibernate.Persister.Collection.ICollectionPersister, NHibernate.Loader.ICollectionAliases, System.Data.IDataReader, NHibernate.Engine.ISessionImplementor)+34 NHibernate.Loader.Loader.ReadCollectionElement(System.Object, System.Object, NHibernate.Persister.Collection.ICollectionPersister, NHibernate.Loader.ICollectionAliases, System.Data.IDataReader, NHibernate.Engine.ISessionImplementor)+34 NHibernate.Loader.Loader.ReadCollectionElements(System.Object[], System.Data.IDataReader, NHibernate.Engine.ISessionImplementor)+d2 NHibernate.Loader.Loader.ReadCollectionElements(System.Object[], System.Data.IDataReader, NHibernate.Engine.ISessionImplementor)+d2 NHibernate.Loader.Loader.GetRowFromResultSet(System.Data.IDataReader, NHibernate.Engine.ISessionImplementor, NHibernate.Engine.QueryParameters, NHibernate.LockMode[], NHibernate.Engine.EntityKey, System.Collections.IList, NHibernate.Engine.EntityKey[], Bo+ab NHibernate.Loader.Loader.GetRowFromResultSet(System.Data.IDataReader, NHibernate.Engine.ISessionImplementor, NHibernate.Engine.QueryParameters, NHibernate.LockMode[], NHibernate.Engine.EntityKey, System.Collections.IList, NHibernate.Engine.EntityKey[], Bo+ab NHibernate.Loader.Loader.DoQuery(NHibernate.Engine.ISessionImplementor, NHibernate.Engine.QueryParameters, Boolean)+1e7 NHibernate.Loader.Loader.DoQuery(NHibernate.Engine.ISessionImplementor, NHibernate.Engine.QueryParameters, Boolean)+1e7 NHibernate.Loader.Loader.DoQueryAndInitializeNonLazyCollections(NHibernate.Engine.ISessionImplementor, NHibernate.Engine.QueryParameters, Boolean)+7f NHibernate.Loader.Loader.DoQueryAndInitializeNonLazyCollections(NHibernate.Engine.ISessionImplementor, NHibernate.Engine.QueryParameters, Boolean)+7f NHibernate.Loader.Loader.LoadCollection(NHibernate.Engine.ISessionImplementor, System.Object, NHibernate.Type.IType)+de NHibernate.Loader.Loader.LoadCollection(NHibernate.Engine.ISessionImplementor, System.Object, NHibernate.Type.IType)+de NHibernate.Loader.Collection.CollectionLoader.Initialize(System.Object, NHibernate.Engine.ISessionImplementor)+1c NHibernate.Loader.Collection.CollectionLoader.Initialize(System.Object, NHibernate.Engine.ISessionImplementor)+1c NHibernate.Persister.Collection.AbstractCollectionPersister.Initialize(System.Object, NHibernate.Engine.ISessionImplementor)+1e NHibernate.Persister.Collection.AbstractCollectionPersister.Initialize(System.Object, NHibernate.Engine.ISessionImplementor)+1e NHibernate.Event.Default.DefaultInitializeCollectionEventListener.OnInitializeCollection(NHibernate.Event.InitializeCollectionEvent)+16d NHibernate.Impl.SessionImpl.InitializeCollection(NHibernate.Collection.IPersistentCollection, Boolean)+1fa NHibernate.Collection.AbstractPersistentCollection.Initialize(Boolean)+2f NHibernate.Collection.AbstractPersistentCollection.Read()+d NHibernate.Collection.Generic.PersistentGenericBag`1[[System.__Canon, mscorlib]].System.Collections.Generic.IEnumerable<T>.GetEnumerator()+11 System_Core_ni!System.Linq.Enumerable+<SelectManyIterator>d__14`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext()+10c System_Core_ni!System.Linq.Buffer`1[[System.__Canon, mscorlib]]..ctor(System.Collections.Generic.IEnumerable`1<System.__Canon>)+d9 System_Core_ni!System.Linq.OrderedEnumerable`1+<GetEnumerator>d__0[[System.__Canon, mscorlib]].MoveNext()+6f System_Core_ni!System.Linq.OrderedEnumerable`1+<GetEnumerator>d__0[[System.__Canon, mscorlib]].MoveNext()+6f mscorlib_ni!System.Collections.Generic.List`1[[System.__Canon, mscorlib]]..ctor(System.Collections.Generic.IEnumerable`1<System.__Canon>)+17e System_Core_ni!System.Linq.Enumerable.ToList[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.__Canon>)+3b Company.ApplicationServices.SecurityService.GetRoles(System.String)+ef 

I am currently opening a database transaction in ActionFilter, which opens a transaction when OnActionExecuting() occurs, and then commits / rolls back the transaction when OnActionExecuted() occurs.

I use StructureMap (v2.6.4.1) for my dependency injection, and the corresponding lines for saving data are as follows.

 var cfg = Fluently.Configure() .Database(MsSqlConfiguration.MsSql2008.ConnectionString(c => c.FromConnectionStringWithKey("DatabaseConnectionString")) .CurrentSessionContext<WebSessionContext>() // ... etc etc.... .Cache(c => c.ProviderClass<NHibernate.Caches.SysCache2.SysCacheProvider>() .UseQueryCache() .UseSecondLevelCache() .UseMinimalPuts()); For<NHibernate.Cfg.Configuration>().Singleton().Use(cfg); For<NHibernate.ISessionFactory>().Singleton() .Use(ctx => { try { var config = ctx.GetInstance<NHibernate.Cfg.Configuration>(); return config.BuildSessionFactory(); } catch (SqlException ex) { ctx.GetInstance<IExceptionLogger>().Error(ex); throw; } }); For<NHibernate.ISession>().HybridHttpOrThreadLocalScoped() .Use(ctx => ctx.GetInstance<NHibernate.ISessionFactory>().OpenSession()); 

UPDATE:. I am still dealing with this and I would like some advice if this is a problem with nhibernate or how is it configured? I had the application blocked to such an extent that we had to restart the server today due to 19 separate threads trying to list the same collection.

It is mentioned below that this is probably a problem with the coverage of the SecurityService, and I agree with that. I currently have services provided through dependency injection through Structuremap (latest version 2.6 released, not yet upgraded to 3.x). The details, which I will discuss in detail below, are short but relevant for what I hope.

 public class SecurityService : ISecurityService { private readonly IRepository<Login> loginRepository; public IList<Role> GetCurrentUserRoles() { var login = GetLoginForCurrentUser(); return GetRoles(login.Name); } public Login GetLoginForCurrentUser() { //Some logic to derive the current UserId {guid} via some resources injected into this service class. return loginRepository.GetReference(loginId); } } public class NHibernateRepository<T> : IRepository<T> where T : class { protected ISession Session { get; set; } public NHibernateRepository(ISession session) { Session = session; } public T GetReference(object id) { return Session.Get<T>(id); } // Other methods typical of a repository class, nothing special } 

My dependency converter setup ....

 For<ISecurityService>().Use<SecurityService>(); For(typeof (IRepository<>)).Use(typeof (NHibernateRepository<>)); //And then the ISession is commented above. 

nHibernate is configured with an internal WebSessionContext context ISessionFactory - singleton ISession - HybridHttpOrThreadLocalScoped ISecurityService and IRepository are both left by default for Transient

Roles are cached, and if they are not found, the system makes a call to the GetRoles method in the security service, I think that I may have a problem calling GetRoles more often than necessary, but that the scope of the multiple parallel enumeration task that I have now is .

UPDATE: Therefore, I am puzzled, today I have the same problem for calling GetReference. 18 separate threads were stuck listing the same set, but this one was internal to nHibernate.

 System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.Nullable`1[[System.Int32, mscorlib]], mscorlib]].FindEntry(System.__Canon)+129 System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.Nullable`1[[System.Int32, mscorlib]], mscorlib]].TryGetValue(System.__Canon, System.Nullable`1 ByRef)+12 NHibernate.AdoNet.ColumnNameCache.GetIndexForColumnName(System.String, NHibernate.AdoNet.ResultSetWrapper)+25 NHibernate.AdoNet.ResultSetWrapper.GetOrdinal(System.String)+e NHibernate.Type.NullableType.NullSafeGet(System.Data.IDataReader, System.String)+29 NHibernate.Type.NullableType.NullSafeGet(System.Data.IDataReader, System.String[], NHibernate.Engine.ISessionImplementor, System.Object)+16 NHibernate.Type.AbstractType.Hydrate(System.Data.IDataReader, System.String[], NHibernate.Engine.ISessionImplementor, System.Object)+14 NHibernate.Persister.Entity.AbstractEntityPersister.Hydrate(System.Data.IDataReader, System.Object, System.Object, NHibernate.Persister.Entity.ILoadable, System.String[][], Boolean, NHibernate.Engine.ISessionImplementor)+3ce NHibernate.Loader.Loader.LoadFromResultSet(System.Data.IDataReader, Int32, System.Object, System.String, NHibernate.Engine.EntityKey, System.String, NHibernate.LockMode, NHibernate.Persister.Entity.ILoadable, NHibernate.Engine.ISessionImplementor)+118 NHibernate.Loader.Loader.InstanceNotYetLoaded(System.Data.IDataReader, Int32, NHibernate.Persister.Entity.ILoadable, NHibernate.Engine.EntityKey, NHibernate.LockMode, System.String, NHibernate.Engine.EntityKey, System.Object, System.Collections.IList, NHi+8c NHibernate.Loader.Loader.GetRow(System.Data.IDataReader, NHibernate.Persister.Entity.ILoadable[], NHibernate.Engine.EntityKey[], System.Object, NHibernate.Engine.EntityKey, NHibernate.LockMode[], System.Collections.IList, NHibernate.Engine.ISessionImpleme+129 NHibernate.Loader.Loader.GetRowFromResultSet(System.Data.IDataReader, NHibernate.Engine.ISessionImplementor, NHibernate.Engine.QueryParameters, NHibernate.LockMode[], NHibernate.Engine.EntityKey, System.Collections.IList, NHibernate.Engine.EntityKey[], Bo+97 NHibernate.Loader.Loader.DoQuery(NHibernate.Engine.ISessionImplementor, NHibernate.Engine.QueryParameters, Boolean)+1e7 NHibernate.Loader.Loader.DoQueryAndInitializeNonLazyCollections(NHibernate.Engine.ISessionImplementor, NHibernate.Engine.QueryParameters, Boolean)+7f NHibernate.Loader.Loader.LoadEntity(NHibernate.Engine.ISessionImplementor, System.Object, NHibernate.Type.IType, System.Object, System.String, System.Object, NHibernate.Persister.Entity.IEntityPersister)+f3 NHibernate.Loader.Entity.AbstractEntityLoader.Load(NHibernate.Engine.ISessionImplementor, System.Object, System.Object, System.Object)+22 NHibernate.Loader.Entity.AbstractEntityLoader.Load(System.Object, System.Object, NHibernate.Engine.ISessionImplementor)+12 NHibernate.Persister.Entity.AbstractEntityPersister.Load(System.Object, System.Object, NHibernate.LockMode, NHibernate.Engine.ISessionImplementor)+69 NHibernate.Event.Default.DefaultLoadEventListener.LoadFromDatasource(NHibernate.Event.LoadEvent, NHibernate.Persister.Entity.IEntityPersister, NHibernate.Engine.EntityKey, NHibernate.Event.LoadType)+84 NHibernate.Event.Default.DefaultLoadEventListener.DoLoad(NHibernate.Event.LoadEvent, NHibernate.Persister.Entity.IEntityPersister, NHibernate.Engine.EntityKey, NHibernate.Event.LoadType)+1d7 NHibernate.Event.Default.DefaultLoadEventListener.Load(NHibernate.Event.LoadEvent, NHibernate.Persister.Entity.IEntityPersister, NHibernate.Engine.EntityKey, NHibernate.Event.LoadType)+5e NHibernate.Event.Default.DefaultLoadEventListener.ReturnNarrowedProxy(NHibernate.Event.LoadEvent, NHibernate.Persister.Entity.IEntityPersister, NHibernate.Engine.EntityKey, NHibernate.Event.LoadType, NHibernate.Engine.IPersistenceContext, System.Object)+73 NHibernate.Event.Default.DefaultLoadEventListener.ProxyOrLoad(NHibernate.Event.LoadEvent, NHibernate.Persister.Entity.IEntityPersister, NHibernate.Engine.EntityKey, NHibernate.Event.LoadType)+cb NHibernate.Event.Default.DefaultLoadEventListener.OnLoad(NHibernate.Event.LoadEvent, NHibernate.Event.LoadType)+120 NHibernate.Impl.SessionImpl.FireLoad(NHibernate.Event.LoadEvent, NHibernate.Event.LoadType)+140 NHibernate.Impl.SessionImpl.Get(System.String, System.Object)+148 NHibernate.Impl.SessionImpl.Get(System.Type, System.Object)+121 NHibernate.Impl.SessionImpl.Get[[System.__Canon, mscorlib]](System.Object)+143 Intellitive.Data.Repositories.NHibernateRepository`1[[System.__Canon, mscorlib]].GetReference(System.Object)+38 

After calling GetReference, more appeared, but it is not related to the problem from what can I say?

Answer 1

It seems to me that you are using NHibernate older than 4.0.0 (released August 17, 2014). If you are using a newer version, just ignore this answer.

NHibernate encountered a concurrency problem - see here :

Sometimes our IIS process starts using a 100% processor. In the memory dump, we see that there are many threads in the Dictionary FindEntry method, which is called from the column name ColumnNameCache.GetIndexForColumnName.

This was allowed, but the patch was only merged with version 4.0.0.

The problem was that the general Dictionary gets into an infinite loop when the base collection was modified, i.e. two threads try to read the value and one writes.

From the documentation :

A dictionary can support multiple readers at once, until the collection is changed. Despite this, listing through a collection is essentially not a thread-safe procedure. In the rare case when the transfer is related to write requests, the collection should be blocked during the entire transfer. To allow access to the collection for multiple threads for reading and writing, you must perform your own synchronization.

Insecure Thread Version: https://github.com/nhibernate/nhibernate-core/blob/3.4.x/src/NHibernate/AdoNet/ColumnNameCache.cs

The same with the applied patch: https://github.com/nhibernate/nhibernate-core/blob/master/src/NHibernate/AdoNet/ColumnNameCache.cs

A longer explanation of why the dictionary is not thread safe and why IIS stops serving requests:

• http://blogs.msdn.com/b/tess/archive/2009/12/21/high-cpu-in-net-app-using-a-static-generic-dictionary.aspx
• http://improve.dk/debugging-in-production-part-2-latent-race-condition-bugs/
• ASP.NET Hang - a general dictionary of concurrency problems causes GK deadlock

Answer 2

This problem would have one culprit: Company.ApplicationServices.SecurityService , whose lifetime would exceed the WebRequest / ISession life cycle.

Some assumptions

The SecurityService method does some magical (not shown above in quesiton), calls ISession, getting the Login object.

In case Login contains groups (first iteration), it continues to iterate and load many roles.

 public IList<Role> GetRoles(string username) { // the instance of login is loaded, still referencing some ISession var login = GetLoginForUser(username); return !login.Groups.Any() // first iteration over Groups ? new List<Role>() // second iteration : login .Groups .SelectMany(x => x.Roles) // other iterations .OrderBy(x => x.DisplayName) .ToList(); } 

This challenge is obviously very expensive in terms of resource consumption. Thus, somewhere there must be some kind of caching (keep the roles on the stream at least, but maybe longer)

Most likely, the SecurityService is singleton. But this will mean that he has his own ISession. And such an ISIS is unlikely to depend on the web request (s). And that means he can have a long session.

It may happen that it returns an instance of the same Login two many different threads (many web requests are processed by different threads)

Similar story:

I would suggest taking a look at this Q and A and its discussion: Nhibernate Lazy Load exception after view exception . The problem, the cause of the problem, was different, but the solution should be the same.

Sentence:

The above assumptions, even if they are only partially correct, should help to understand this sentence below. In short, we should avoid sharing any objects from one ISession ... with other threads / requests ...

What we see is a problem with iterating an object coming from one ISession inside the other thread. The solution I really like: use the Prototype Template. (Create or download a test file to create an expensive object)

• allows you to load roles by username in one shot
• clone them (prototype) to the level we need
• cache them, later return cached clones
• introduce some update (after 2 minutes, sql cache dependency, file dependency) to keep security objects in the cache only for a short / reasonable time

There may be other approaches (e.g. all properties with NHibernateUtil.Initialize() and clone skipping) ... but I can confirm that Cloning works well for me.

Short review:

 class Group : ICloneable { ... public override object Clone() { var entity = base.Clone() as Group; entity.Roles = new List<Role>(); foreach(var r in Roles) { entity.Roles.Add(r.Clone() as Role); } ... return entity; } } class Login: ICloneable { ... public override object Clone() { var entity = base.Clone() as Login; entity.Groups = new List<Group>(); foreach(var g in Group) { entity.Groups.Add(r.Clone() as Group); } ... return entity; } } 

Nice that this draft is higher in our hands. We can adjust the cloned part as necessary. In the end, we can have a clone loaded once, regardless of any session, having only the necessary security features ... ready for caching

EXTEND: based on additional information in the extended question

I would say that, related to my guesses above, there is a criminal (at least a suspect):

Roles are cached ...

But these roles are session related. They were not suspended. These are just received via Linq on top of the Login instance returned by ISession:

 login.Groups.SelectMany(x => x.Roles).OrderBy(x => x.DisplayName).ToList() 

Each of these objects (Login, Group, Role) is still tied to the session in which he was born.

Meanwhile, at the same time, various web requests appear. Different web requests for a single login (partial views, Web API calls). Thus, in a multi-threaded environment, more web requests can relate to and work with cached roles. With them, which are still associated with an existing, open ISession, but in a different thread .

Most likely, your structure uses them intensively to decide what to show and what to hide, what to edit ...

Thus, among many web requests, there is a common object (a set of objects) associated with another ISession.

My suggestion: disconnect these objects. And the way I found it more accurate is the Protype template.

So, I believe that this should give you an idea of ​​how these problems can occur, but I want to emphasize this solution. Do not share objects associated with one session among other requests / threads. A clone is just one way. But the key is the key.