I am writing an applications that requires arrays of very large size.
Several of over 16 MB, whose size I know ahead of time and
several whose size I do not know till the program is running.
I started out using the HeapCreate and HeapAlloc functions to
allocate memory.
The behavior is very odd.
The first thing I do is to use GetProcessHeap to create space
The first thing I do is to use HeapAlloc to create memory space
to read in a file of about 1.5 MB, and then use the rest of
that space to create an index file for that file. Needs aboujt
60,000 bytes
The next array I use, lets call it hG is 2exp(22) *4 in size.
That gets allocated OK in another heap created ujsing HeapCreate.
The next array I used,lets call hList is 16,000 KB in size
and I allocated that using space from the first created above.
Now it turns out that when I start running the code,
and writing to the space assigned to hList, that I get a memory
faul and where in the code the faujlt occurs seems to depend
on where I place the variable hList in the data section..
I am quite puzzled by this.
Next I made the space assigned to hG sufficiently large so that
it could accomodate hList and after the hG space was assigned
I move tothe end of its space,subtracted the amount needed
for hList and assigned hList there. Still get memory fault.

If I assign hList and all he other arrays I need in the DATA
section the code runs OK but I am puzzled as to why the
memory functions behave so oddly.

I am writing the program on an Athlon 800 MHZ machine
using Win2000 in Console mode. I have 36 GB of disk space,
512 MB of dynamic memory and as you can see-problems.

What stupid thing am I doing?
Posted on 2001-12-06 20:46:19 by nolpak
Probably incorrect indirection. It's hard to see if you are keeping straight three things: 1) the actual array and its starting point, 2) pointers to the actual arrays, and 3) heap handles.

If you have a correctly operating program, add two pointer variables, pList and pG. I think you can define them this way:

pList DWORD aList ; aList is the name of the actual array
pG DWORD aG ; aG is the name of the actual array

Then alter your program so that it uses only pList and pG. This way, you make sure you are properly using indirection.

If your program still works, use the Heap functions to initialize pList and pG, and delete the pre-runtime arrays.

CreateHeap and GetProcessHeap return heap handles.
HeapAllocate returns a memory pointer.

If you've already done this, then you need to look at the addresses you're creating. If you're doing something fancy like hashing, watch out for overflows and negative indexes.
Posted on 2001-12-07 18:37:25 by tank
Thanks for the response and the suggestion.
I tried it and no difference. In fact the memory error occured
immediately in the program rather than later on.

Everything works OK if I assign hUrlist DD 4000 dup(0)
istead of trying to do it dynamically. The way the error occurs
is when I try to write to the memory pointed to byh hUrlist
and not when I try to read it.

It is very odd.
Posted on 2001-12-07 19:32:34 by nolpak

What I would do is write up a test piece with the memory allocation scheme you are after and sample test the beginnings and ends of the seperate areas to see if you can both read and write to them.


Posted on 2001-12-07 19:35:31 by hutch--

Thanks for responding.

I can read and write correctly at the beginning and end
of the memory area
Posted on 2001-12-07 20:51:36 by nolpak
Your problems are sorta weird :). Anyway, for allocing large amounts
of data (large enough that up to 4095 bytes of alignment slack
doesn't bother you) I would go for VirtualAlloc, it's the lowest wrapper
around the paging functions I know of, and probably the fastest
as well. And oh yeah, another advantage, it allows you to specify
page protection.
Posted on 2001-12-08 19:37:55 by f0dder
Figure out the address it the mem access it faulting at, then make sure this address is withen your arrays boundrys.

Its not often that windows memory allocation fails, espically without telling you, so it might be a good idea to check the return values of the memory allocation functions.

Also, if you dont have enought disk space free for the page file to grow then you will run into big trouble, windows at this stage will start grinding to a halt.

Its easy to get bugs that crash your app later rather than sooner, espicaly if you save handles to stack (LOCAL) varibles without thinking, or with Heap functions, you have a high chance of randomly hitting elsewhere on the heap.
Posted on 2001-12-08 19:55:11 by huh
The declaration "hUrlist DD 4000 dup(0) " suggests the problem may be indirection related.

The first step is to add a pointer declaration "pUrlist DD hUrlist", keep the Urlist declaration, and change all other code to use pUrlist instead of hUrlist. Do not use the memory allocation functions until you finish this step. It will require more code because the x86 does not do data indirection via memory. The only place hUrlist should be referenced is in that pUrlist declaration.

In doing this, what you must do is: for every instruction that uses hUrlist:

; example shows optional index register, esi,
; and optional displacement, 8
cmp eax,hUrlist

you must use up a register to replace hUrlist:

; example
push ecx ; preserve register, if necessary
mov ecx,pUrlist ; (re)load array pointer, if necessary
cmp eax, ; use pointer for addressing
pop ecx ; restore register, if necessary

You must also replace "offset hUrlist" and "addr hUrlist" with "pUrlist" (with no offset or addr).

If you don't make these changes, then the code will fail again when you add in the calls to the memory allocation functions.

If you are ensuring the extra code is working, and failing when you add in the memory functions, then you're either not using the memory functions properly, or you've got problems with address calculations. A 16Kbyte array should not be a problem whether you allocate it statically (with DD) or dynamically (with HeapAllocate).
Posted on 2001-12-10 23:46:23 by tank