¶D3D11 refusing to stay in full screen mode
I've been working on getting full-screen mode support into a Direct3D 11 based program, and in my opinion Microsoft got a little too fancy with their new full-screen support in the DirectX Graphics Infrastructure (DXGI) layer. The idea is that DXGI handles most of the work for you now, including setting the window mode, changing the window frame and size, etc. However, DXGI's automatic handling only really works when you've got something simple like a single overlapped window with a caption bar and menu. Throw in a status bar, a side panel, and other window decorations, and it becomes more of a headache.
If you're going through this yourself, let me drop you a hint now and save you some time: don't try to call IDXGISwapChain::SetFullscreenState(TRUE) on a swap chain whose output window is a child window. You can create a windowed swap chain that way, but if you try transitioning to full-screen mode you will get DXGI_ERROR_INVALID_CALL (887A0001). AFAIK this isn't documented, and the debug runtime wasn't spitting out a message either.
The particular battle I've been fighting this evening is DXGI "helpfully" detecting that another window has appeared on top and popping out of full screen mode. This avoids the common bug in Direct3D 9 where the program doesn't notice and keeps drawing over the entire screen even though it has lost foreground status. Problem is, when you've got a case like mine where several layers of window chrome and gadgets have to be added or removed when transitioning in and out of full screen mode, it looks really broken when DXGI mysteriously flips out of full screen and the window is still stripped down for it. At least in the D3D9 case the problem is apparent after the user clicks and seeing the other program's window activate; the DXGI behavior however doesn't give a clue as to the offending window and just looks like the computer being stubborn.
That brings us to the cause of the issue in my case. I typically have WinAmp up all the time docked at the top of my desktop, so I immediately suspected it as the culprit. Now, although WinAmp is set to be a "topmost" window, it is possible for another window to also be topmost on top of it -- essentially, the topmost flag segregates windows into two tiers, but windows can be rearranged in Z order within each tier. It's easy enough to add code to force the application window topmost before going to full-screen mode, and that almost solved the problem... but it was only about 30% effective, with WinAmp somehow still ending up on top afterward sometimes and blowing the mode switch. The non-determinism was a big red flag as this should have either worked or failed.
A bit of digging around with WinDbg in the winamp.exe process revealed the problem: a call to DeferWindowPos() triggered by a display mode switch. DeferWindowPos() is the batched form of SetWindowPos(), an API for manipulating windows. SetWindowPos() is second only to CreateFont() for the worst designed API in Win32, given that it can be used to change the position, size, Z order, topmost, visibility, or activation state of a window as well as request a frame change, and takes a bitfield that specifies which of those actions you do not want it to perform. This means that it is really easy to accidentally do something with SWP/DWP() that you didn't want. In this case, the culprit was a call to DeferWindowPos(HWND_TOPMOST) on a window that was already topmost. The documentation says that HWND_TOPMOST places the window on top of all non-topmost windows, but it turns out to bring the window to the top of the Z order within topmost windows, too. The result was that a regular D3D10/D3D11 based program like those from the DXSDK samples would immediately fall out of full screen mode every time, and my program would do it occasionally depending on whether it or WinAmp responded to the mode switch first. Mystery solved.
In the end, I'm not terribly surprised since I'm using WinAmp 2.95, which is a pretty old version. I keep using it because it's simple, it's compact, and except for this case it's generally just worked. This is probably not something that could or should be solved from a D3D11-based program, since due to the race there isn't a reliable way for the D3D11 program to come out on top. It could be solved from the WinAmp side, though... I'm tempted to write a WinAmp plugin to patch the DeferWindowPos() import and avoid the call if the window is already topmost.