One of the most common folkloristic methods (and one I used for years, alas) is to use EnumWindows, do a SendMessage to each window, and look at the return result from SendMessage. What you send is a Registered Window Message (see my essay on Message Management), and if you receive this message you return TRUE. All other windows will not understand this and return FALSE. This turns out to be deeply flawed in a variety of ways.
Note that this method always worked in Win16 because it used cooperative multitasking. It is the preemptive multitasking of Win32 that makes this method fail. And it does.
The SendMessage can hang indefinitely. But if the thread that owns the handle is blocked, on a Semaphore, a Mutex, an Event, an I/O operation, or some other manner, the SendMessage will block until that thread frees up and runs. But this may never happen.. So you haven't gained any reliability.
You can solve this by using SendMessageTimeout. This more-or-less works. You will typically choose a short timeout, for example about 200ms.
But it gets worse.
Microsoft, in violation of all known Windows specifications, has created an application that does not pass messages it doesn't understand to DefWindowProc, which would return 0 for any message it does not understand. Instead, they have a component, apparently associated with Personal Web Server, which has the truly antisocial property of returning 1 for every message sent to its top-level window, whether it understands it or not. So you can't rely on a zero meaning it is not your app.
Well, this could be solved. When I needed to do this for another reason, I ended up having to return the registered window message value, which avoids the Microsoft blunder.
So we've solved the problem of timeouts and bogus messages. We know not to depend on the caption contents, and probably want to avoid worrying about the Window class name. So why doesn't this work?
Because of a much more fundamental problem: a race condition. Just like the one described in the previous section.
The code did an EnumWindows loop and for each HWND it did a SendMessage. So what happened was that application instance 1 searched for another instance of itself, didn't find one, and proceeded to come up. Meanwhile, application instance 2 searched for an instance of itself, but since instance 1 had not yet come up and created its own main window, instance 2 did not find a conflicting instance, and it proceeded to come up. Using a scheme similar to the table I used to demonstrate why the basic FindWindow method doesn't work, you can show that this method will fail for the same reason.
This is the most fundamental failure of this mechanism, and a reason it cannot be used.