Hi, I am calling a MASM DLL from VB, and I seem to be running out of resources. It works correctly for a while, then dies.

I understand having to use "ret 12" for example to get MASM to properly unload the stack of passed variables from VB before returning. However, I do not quite understand what happens to global and local variables within the MASM DLL itself.

Questions:
1: Do I need to handle the stack for these manually, or is it automatically handled by MASM?
2: If MASM handles it, is there something specific like "ret 12" that needs to be used?
3: If I need to do it manually, does anyone know detailed mapping of the stack coming into the DLL so I don't unload the wrong thing?
4: If this is not my problem, does anyone know of a way that I can trace throught the DLL as it is called? I have GoBug, and VC++ as well to use.

Thanks.
Posted on 2002-02-18 18:46:52 by sceptor
sceptor,

From memory, VB uses the normal STDCALL calling convention so the normal PROC system in MASM should work fine. What you will have to know is how VB passes the parameters as it can be either BY VALUE or BY REFERENCE.

You can write MASM code to handle either with no problems, if a parameter is passed BY REFERENCE, you will be working on the address of the data which is no big deal to do.

I would suggest looking up any data you can find on the way VB passes data as the rest is simple.

Good luck,

hutch@movsd.com
Posted on 2002-02-18 19:41:15 by hutch--
I have done a lot of VB->asm dll->VB calling, and never had this problem. As hutch says, VB uses STDCALL convention, so it is no different to calling a standard windows API function.

To get a good answer, you will have to be more specific with your details. How does the VB app "just die"? What "resources" are you running out of? What parameters are you passing to the dll? Are you passing strings? What are the declares you have in the VB app for the dll?

As your dll is running in the process space of the VB app, you will not be able to directly trace through it with VC. I would suggest you pick up a debug macro library, and insert some debug statements into the dll that output to a file.
Posted on 2002-02-18 21:31:00 by sluggy
Thanks for the tips, I noticed I was passing a string by val instead of by ref.

So this leads me to even further puzzlement.
I passed the string ByVal, and oddly enough I was able to pick up the characters just like I was doing a "C" string, 1 byte per character. I thought VB used 2 bytes per character. Unicode vs. Ansi right?

Thanks for the clues.
Posted on 2002-02-19 07:10:09 by sceptor
sceptor,

You will need a VB man to properly answer that one but from memory VB converts its internal format of unicode to ansi with external calls to DLLs. I think you can pass data from VB as BYTE data which will be a lot faster than internal conversions but you will have to look up how its done in VB.

Regards,

hutch@movsd.com
Posted on 2002-02-19 20:09:08 by hutch--
Strings in VB should always be passed ByVal. What happens then is VB translates the string (which is in unicode) to ANSI, that copy is made on the heap, and a pointer is passed to the version of the string that is on the heap. If you pass the string ByRef, you will get a pointer to the real Unicode string. If you use StrPtr to pass a pointer, you will get a pointer to a BSTR.

If you use ByVal to pass a string, and your asm function needs to keep a copy of the string, then you will have to copy it from the heap to your own private buffer, as the copy on the heap is destroyed when your function returns. I would advide to NEVER use ByRef string passing, because while it may seem simple and may be exactly what you want, it tends to make the VB IDE GPF all the time when you try to access the string from your asm function.

hutch also recommended trying to pass a byte array instead. This is valid, but only if you want to go through the overhead of converting the string to and from the byte array. If you use the byte array, pass the first element of the array ByRef, this passes a pointer to the array.
Posted on 2002-02-20 18:16:41 by sluggy