Say I do a CALL with 'step over' and the routine never returns... windbg 'hangs'

Pressing ctrl+break restores windbg but on "ntdll!DbgBreakPoint", not inside my prog.
What can I do?

Sorry to ask this question here... couldn't find a better forum..
Posted on 2010-12-14 13:18:31 by aleksaZR
You may be attempting to debug a program that doesn't want to be debugged.  Seriously, check out IsDebuggerPresent and you can fathom from there.  If you are trying to debug commercial apps you can count on tricks like this being in there.  I've personally used this in the past, as well as many more advanced tricks, to prevent certain apps from being reversed...

http://msdn.microsoft.com/en-us/library/ms680345(VS.85).aspx
Posted on 2010-12-14 20:20:51 by p1ranha
aleksaZR,

Usually “step over” means “execute 'til next instruction”. That means that if that instruction doesn't execute (I know, that doesn't qualify for good literacy ;)), debugger won't stop.
Posted on 2010-12-15 03:16:50 by baldr
"Step over" means that the debugger doesn't step into the call, but executes the call and steps to the first instruction after the call.
If the call never returns for some reason, the debugger will never break again.
You'll have to "Step into" to see what goes wrong inside the call.
Posted on 2010-12-15 04:50:31 by Scali

You may be attempting to debug a program that doesn't want to be debugged.


No, I'm the author of the code.


If the call never returns for some reason, the debugger will never break again.


That sucks.. Guess I'll revert to SoftICE.
Posted on 2010-12-15 05:07:29 by aleksaZR
Quite an extreme step - using SoftIce to debug userland code is like cracking a walnut with a sledgehammer.

DEBUGGING BASICS
Since you are the author, you have a good idea of what loops might exist behind this call, you are familiar with your own program's sourcecode.
I would recommend OllyDBG, you can break out anytime and it will show you where you landed, that's enough for debugging pretty much any userland application... just take a look at the code where you landed, and if you don't recognize it (at all), you can step backwards or forwards until you do.
Save the ice for kernelside / extreme stuff :)

DEBUGGING AT RUNTIME
If you still have trouble finding this infinite loop, try adding a few "int 3" instructions deliberately into your sourcecode, in order to help you accelerate the search and also to help you get your bearings.. you might like to introduce one right before that call for example, and step into / over the code behind it.
Also consider adding some kind of log output in the "potentially offensive" functions behind this call.
You might write messages to a textfile or another application's text controls or something ;)
This can definitely help you trace the infinite loop since you'll see infinite log messages in the output stream...
Posted on 2010-12-15 06:25:48 by Homer

You might write messages to a textfile or another application's text controls or something ;)
This can definitely help you trace the infinite loop since you'll see infinite log messages in the output stream...


The OutputDebugString() API function is very nice for that.
It writes to a special debug output stream, which you can monitor with a program such as dbmon. If you use Visual Studio, it will automatically display the debug stream in your output window.
Posted on 2010-12-15 07:18:19 by Scali