MSBuild fails to complete the TFS build (usually error C1093 / Not enough Storage)

Question

I have a very strange and complicated problem with MSBuild / TFS. I have a solution that contains about 12 different build configurations. When launched on the build server, it may take 30 minutes to build the lot, and it has been working fine for several weeks now, but now it sometimes fails.

In most cases, when it fails, it will be an error:

19: 25: 45.037 2> TestPlanDocument.cpp (1): fatal error C1093: API call "GetAssemblyRefHash" failed "0x8007000e": ErrorMessage: there is not enough storage to complete this operation. [C: \ Builds \ 1 \ ICCSim Card Test Controller \ ICCSimCTC Release \ src \ CardTestController \ CardTestController.vcxproj]

The error sometimes occurs in another file. This will not happen for each build configuration, it is very inconsistent, and sometimes even successfully creates all of them. There is not much difference between assembly configurations, basically these are just a few line changes, and, of course, they are all built locally just fine.

The API request in question is usually GetAssemblyRefHash , but not always. I do not think this is a problem, since Googling for GetAssemblyRefHash specifically brings up practically nothing. I suspect there is some kind of resource problem here, but I'm at a loss as to what: there is a lot of hard disk space (hundreds of GB), a lot of RAM (the machine initially had a minimum of 4 GB, but was dynamic, since Hyper -v - it never exceeded 2.5 GB. I raised it to a minimum of 8 GB just in case and have not changed).

I have installed verbosity of the assembly for diagnostics, and in fact it does not show anything useful, just the same error.

For reference, the build server is fully updated in all patches. It is running Windows Server 2012 R2, installed TFS 2013 and VS 2013, both of which are on Update 4.

At the moment I am at a loss and will be grateful for any help or pointers.

EDIT: just to keep people up to date, the compilation toolchain was in 32-bit mode, however, even after switching to a 64-bit problem, the problem persists.

Answer 1

Ok, I have an update! I opened a support ticket with Microsoft and was busy working with them to find out the problem. They followed the same paths as above, and came to the same conclusion: this is not a resource problem .

To shorten the long history, Microsoft now admitted that this was most likely a bug in the VC ++ compiler, which was almost certainly caused by the race condition (although this is unconfirmed). There is no word if they correct it in a future release.

There is a workaround using the / MP flag at the project level to limit the number of compiler processes open by MSBuild without completely shutting down multiple instances (for me this doubled the build time).

To do this, go to your project properties and in the "Configuration Properties" section → C / C ++ → Command Prompt you need to specify the / MP flag and then a number to limit the number of processes.

My build server has 8 virtual processors, and normal behavior is equivalent to / MP 8, but this leads to an error. For me, using / MP 4 seems enough to limit the error without increasing build time. If you see an error like this, you may need to experiment with other numbers, such as / MP 6 or / MP2.

Answer 2

I think I found the source, but I still don't know the reason.

Looking through the Microsoft Shared Source , we can find the source for GetAssemblyRefHash() :

 HRESULT CAsmLink::GetAssemblyRefHash(mdToken FileToken, const void** ppvHash, DWORD* pcbHash) { if (TypeFromToken(FileToken) != mdtAssemblyRef) { VSFAIL( "You can only get AssemblyRef hashes for assemblies!"); return E_INVALIDARG; } HRESULT hr; CAssembly *file = NULL; if (FAILED(hr = m_pImports->GetFile( FileToken, (CFile**)&file))) return hr; return file->GetHash(ppvHash, pcbHash); } 

Only two places to explore - call m_pImports->GetFile() , where m_pImports - CAssembly *m_pImports; , the other is file->GetHash() .

m_pImports->GetFile() here, and this is a dead end:

 HRESULT CAssembly::GetFile(DWORD index, CFile** file) { if (!file) return E_POINTER; if (RidFromToken(index) < m_Files.Count()) { if ((*file = m_Files.GetAt(RidFromToken(index)))) return S_OK; } return ReportError(E_INVALIDARG); } 

file->GetHash() , which is located here:

 HRESULT CAssembly::GetHash(const void ** ppvHash, DWORD *pcbHash) { ASSERT( ppvHash && pcbHash); if (IsInMemory()) { // We can't hash an InMemory file *ppvHash = NULL; *pcbHash = 0; return S_FALSE; } if (!m_bDoHash || (m_cbHash && m_pbHash != NULL)) { *ppvHash = m_pbHash; *pcbHash = m_cbHash; return S_OK; } DWORD cchSize = 0, result; // AssemblyRefs ALWAYS use CALG_SHA1 ALG_ID alg = CALG_SHA1; if (StrongNameHashSize( alg, &cchSize) == FALSE) return ReportError(StrongNameErrorInfo()); if ((m_pbHash = new BYTE[cchSize]) == NULL) return ReportError(E_OUTOFMEMORY); m_cbHash = cchSize; if ((result = GetHashFromAssemblyFileW(m_Path, &alg, (BYTE*)m_pbHash, cchSize, &m_cbHash)) != 0) { delete [] m_pbHash; m_pbHash = 0; m_cbHash = 0; } *ppvHash = m_pbHash; *pcbHash = m_cbHash; return result == 0 ? S_OK : ReportError(HRESULT_FROM_WIN32(result)); } 

We see that about halfway there he is trying to allocate a place to store the result of byte [], and if it fails, returns E_OUTOFMEMORY, which is the error code that you see:

 if ((m_pbHash = new BYTE[cchSize]) == NULL) return ReportError(E_OUTOFMEMORY); m_cbHash = cchSize; 

There are other ways to consider, but this seems like the most obvious source. Thus, it seems that the problem is that simple memory allocation is not performed.

What could be the reason for this?

• Lack of free pages of physical memory / swap
• Fragmentation of memory in the process.
• Inability to reserve a commit location for this in the swap file
• Lack of address space

At this point, I think it will be a fragmentation of memory. Have you checked three times that the Microsoft CPP compiler works in 64-bit mode? Perhaps see if you can debug the compiler (Microsoft character servers can help you here) and set a breakpoint for this line and unload the heap when that happens.

Some features of heap fragmentation diagnostics - run sysinternal VMMap when the compiler breaks and see a free list - you need three pieces of at least 64 KB for free to perform the distribution ; less than 64 kbytes, and it will not be used, and two blocks of 64 KB are reserved.