FreeConsole Behavior on Windows 8

Question

In Windows 8, we have a problem with FreeConsole. It seems to close stdio handles without closing file streams.

This may be a Windows 8 issue, or it may be that I just don’t understand the (completely absurd) way that the Windows / GUI application subsystem does things.

What's happening?

The minimum example is below. Tested with compilers: VS2005, VS2013, VS2017, using a statically coupled CRT.

#include <windows.h> #include <io.h> #include <stdio.h> static void testHandle(FILE* file) { HANDLE h = (HANDLE)_get_osfhandle(fileno(file)); DWORD flags; if (!GetHandleInformation(h, &flags)) { MessageBoxA(0, "Bogus handle!!", "TITLE", MB_OK); } } int main(int argc, char** argv) { freopen("NUL", "wb", stdout); // Demonstrate the issue with NUL // Leave stderr as it is, to demonstrate the issue with handles // to the console device. FreeConsole(); testHandle(stdout); testHandle(stderr); } 

Answer 1

After disassembling the code for FreeConsole in different versions of Windows, I solved the cause of the problem.

FreeConsole is a remarkably unmanageable feature! I really close a lot of pens for you, even if he doesn’t “own” these pens (for example, PENS that belong to the stdio functions).

And the behavior in Windows 7 and 8 is different, and 10 times changed.

This is where the dilemma arises when the fix is ​​reached:

• When stdio has a HANDLE on the console device, there is no documentary way to force it to abandon this handle without calling CloseHandle. You can call close(1) or freopen(stdout) or whatever, but if there is an open file descriptor that references the console, CloseHandle will be called on it if you want to switch stdout to the new NUL descriptor after FreeConsole.
• On the other hand, with Windows 10 there is also no way to escape FreeConsole by calling CloseHandle.
• Visual Studio debugger and Application Verifier checkbox - application for calling CloseHandle on an invalid handle. And they are right, that’s really not good.
• So, if you try to “fix” stdio before calling FreeConsole, then FreeConsole will make an invalid CloseHandle (using your cached handle, and there is no way to say that this handle is gone - FreeConsole no longer checks GetStdHandle(STD_OUTPUT_HANDLE) ). And, if you call FreeConsole first, you cannot fix stdio objects without forcing them to make an invalid CloseHandle call.

An exception, I came to the conclusion that the only solution is to use an undocumented function if the public ones simply do not work.

 // The undocumented bit! extern "C" int __cdecl _free_osfhnd(int const fh); static HANDLE closeFdButNotHandle(int fd) { HANDLE h = (HANDLE)_get_osfhandle(fd); _free_osfhnd(fd); // Prevent CloseHandle happening in close() close(fd); return h; } static bool valid(HANDLE h) { SetLastError(0); return GetFileType(h) != FILE_TYPE_UNKNOWN || GetLastError() == 0; } static void openNull(int fd, DWORD flags) { int newFd; // Yet another Microsoft bug! (I've reported four in this code...) // They have confirmed a bug in dup2 in Visual Studio 2013, fixed // in Visual Studio 2017. If dup2 is called with fd == newFd, the // CRT lock is corrupted, hence the check here before calling dup2. if (!_tsopen_s(&newFd, _T("NUL"), flags, _SH_DENYNO, 0) && fd != newFd) dup2(newFd, fd); if (fd != newFd) close(newFd); } void doFreeConsole() { // stderr, stdin are similar - left to the reader. You probably // also want to add code (as we have) to detect when the handle // is FILE_TYPE_DISK/FILE_TYPE_PIPE and leave the stdio FILE // alone if it actually pointing to disk/pipe. HANDLE stdoutHandle = closeFdButNotHandle(fileno(stdout)); FreeConsole(); // error checking left to the reader // If FreeConsole *didn't* close the handle then do so now. // Has a race condition, but all of this code does so hey. if (valid(stdoutHandle)) CloseHandle(stdoutHandle); openNull(stdoutRestore, _O_BINARY | _O_RDONLY); } 

Answer 2

caused by the fact that the previous standard (not redirected) Windows 8 console descriptors (which were returned by GetStdHandle), where they are actually pseudo-referenced, whose values ​​do not interact with other descriptors of the kernel object, so writing to this pseudo-handle after it closes FreeConsole always suffers a failure. In Win8 MS, something changed, so GetStdHandle returns a normal handle to the kernel object that refers to the driver object of the console subsystem (in fact, this driver also appeared only in Win8). Therefore, FreeConsole closes this handle. The funny thing is that CRT does GetStdHandle at startup and stores the return value somewhere inside and uses it everywhere where C functions are used that access std :: in / out / err. Since FreeConsole closed this descriptor and did not become its special pseudo-unique value - the same descriptor value can be reused by any other open descriptor of the kernel object, and you will be lucky if in this case it does not cause the file, pipe or socket, your debug outpout will go there:)