I’ve been working on an application that suddenly crashes without an apparent reason. I though that it could be an buffer overrun and I decided to implement something similar to the /GS switch (C++)

I put the relevant code in the prologue and epilogue macros that are called for each procedure. To activate them simply use

      OPTION PROLOGUE:$StkGrdPrologue
      OPTION EPILOGUE: StkGrdEpilogue

I didn’t implement the checking of the return address since my goal is only to detect stack overruns, but it is relativ simple to add this feature.

I hope this helps…


Posted on 2006-01-16 10:12:55 by Biterider
Very interesting! :D

A couple feature suggestions:

1) A macro to be used on program startup, to randomly generate the cookie on runtime. Doesn't need to be truly random, I'd be happy with just RDTSC and XOR against some value. :)

2) The possibility to use a callback instead of raising a DEBUG_BREAKPOINT exception. That's because if the overrun also corrupted the SEH record, there's no way to catch it.

What do you think?
Posted on 2006-01-16 10:56:17 by QvasiModo
Hi QvasiModo

1) I initialize the cookie this way, but it can be done also using a constant value.

      xor eax, edx
      mov dStkGrdCookie, eax

2) Can you give me an example of what you mean?

Posted on 2006-01-16 11:49:03 by Biterider
1) That's OK, it's pretty much what I meant :)

2) Just replacing the INT 3 with a call to some routine. For example:

IFDEF StkGrdCallback
      call StkGrdCallback
      int 3                                      ;;Break Code here

Where StkGrdCallback could be a procedure name, a code label, or an equate to either.

The problem would be if you want to catch the stack overflow on runtime, rather than being attached with a debugger. Then you'd use SEH, but if the SEH record in the stack is corrupted this would be impossible. A callback function would do the trick.
Posted on 2006-01-16 12:31:51 by QvasiModo
On XP you could use VEH instead of SEH, but... callback doesn't seem like a bad idea :)
Posted on 2006-01-16 13:02:20 by f0dder
Attachment on the first post was updated.

Posted on 2006-01-17 01:56:13 by Biterider
Great! Thanks :)
Posted on 2006-01-17 10:18:24 by QvasiModo