This is driving me crazy, so I hope someone can help :shock:

Simply: What are the rules for starting/ending an ASM program? More specifically I'm on Win32, NASM.

You see, the book I'm following only uses macros for this, and refuses to reveal what's inside. I don't have the source code for the book, so I can't check what's behind the curtains.. Web searches only made me more confused. I think I might be onto something, but it's 2AM in the morning and I can't seem to understand what "stack frames" are, and no sources seem to mention when they are necessary.

Could anyone point me in the right direction? I'm not asking for spoonfeeding, but I seem unable to find any rules for this..
Posted on 2010-04-23 19:01:27 by !me
x86 Instruction Set Listing @ Wikipedia

The above link demonstrates how ENTER/LEAVE are actual x86 instructions and not necessarily macros, though I suppose that, internal to the CPU, they can be thought of as "macros" as they decode to execute a specific set of instructions and don't have any particular meaning unto themselves.

Stack frames, in their most basic form, are a means of preserving (the previously used portion of) and utilizing (the unused portion of) the stack during a function call. The stack itself nothing more than a pointer to a linear chunk of memory. When a particular function is done utilizing the stack, it is necessary to restore the stack (SP/ESP/RSP and BP/EBP/RBP) back to the state it was in when the function was called, so that the code being returned to isn't accessing the wrong parts of the stack... remember PUSH/POP. For Win32, you'll want to become very familiar with the STDCALL calling convention.

As for the rules of starting/ending a ASM program, that is actually a separate question and is indeed platform-specific. A common scenario is that a global/public symbol in a particular assembled/compiled object is passed to the linker as the program entry-point, and this is embedded in the binary so that the OS knows where to start executing the program. The Portable Executable (e.g. modern Win32/Windows .EXE) format uses this method, so for Win32, that's your newly-found mantra. Some linkers will implicitly assume that main/_main, or even start/_start, is the entry-point in the object(s) to be linked unless otherwise specified. Ending a Win32 application is much more simple by comparison, read up on the ExitProcess function.

Many points I've made are oversimplified for the sake educational brevity. As you delve deeper into x86 ASM, your understanding of the stack, and how it is (mis)used, will grow. Same goes for Win32/PE.
Posted on 2010-04-23 22:42:11 by SpooK
Thanks a lot for the reply, that clarified it some ;) So a stack frame is only necessary for function calls within a program, and not for the program itself.

When I said that my book used macros, I did not mean ENTER/LEAVE. The author calls them STARTUP and END or something like that. I just read about some programs ending with e.g.

leave
ret

and in some programs

xor eax, eax    ;or mov eax, 0
ret

When it comes to ExitProcess, is that used for console programs as well? Is there no way to write an OS-portable ASM program (that e.g. calculates the sum of 2 variables - something trivial)?
Posted on 2010-04-24 07:36:50 by !me
Strictly speaking, ExitProcess is not required - the os is smart enough to realize that your final RET is the exit of the program. But this is undocumented behavior and should not be relied apon it could change in the future .
Win32 Console programs should also call ExitProcess, yes, but the same remarks apply.

I am teaching bad habits.
Hah!

Posted on 2010-04-24 11:08:44 by Homer
!me,

Stack frame is only necessary when you need it.

Exiting program with ret is questionable practice, because Windows API doesn't specify exactly the context in which program entry point is reached (some reverse engineering shows that for several generations of Windows ret is OK, but this is not guaranteed in the future). ExitProcess() is the documented way.

Console programs aren't much different from GUI: they simply don't have to worry that console is allocated to them.
Posted on 2010-04-24 11:18:11 by baldr
Thank you guys, I really appreciate it :D And it's always good to know bad habits ;)

Gotta love forums :D Now I can finally start concentrating on the fun stuff
Posted on 2010-04-24 12:59:20 by !me