When using file mapping to map a very big file, how much physical memory will actually be used?
Posted on 2004-01-06 08:59:45 by optimus

When using file mapping to map a very big file, how much physical memory will actually be used?


That entirely depends on two things:

1. How much physical memory is available,
2. How much of the file you actually access.

The file mapping is handled by the VM system. So it brings in the file's data as you need it (and flushes the file data back to disk, if necessary, when it needs that physical RAM for some other purpose).

Cheers,
Randy Hyde
Posted on 2004-01-06 10:43:48 by rhyde
Generally as much as Windows can put together. MMF's have a 4K page size and Windows will map 4K blocks of memory as you read/write new sections. If there is a request for memory it will unmap that page of the file and if the page is "dirty" (has been written to) it will flush it to the disk. This allows you to map huge files (2Gig in Win98+, 1Gig in Win95) and let Windows manage the memory. You will experience granularity problems with very large files however or if you read and write to many parts of the file at once. The way Windows handles the mapping also depends on the flProtect flag specified in the CreateFIleMapping API.
Posted on 2004-01-06 10:56:25 by donkey
If I don't use FlushViewOfFile, when will the data be written back to the file?
And if a file is used for MMF, is it possible to do ordinary file operations, e.g. reading, writing, resizing, over the file at the same time?
Posted on 2004-01-07 23:04:32 by optimus
If you don't use FlushViewOfFile the data will be written
1) When the system wants
2) When you close the file

When you close the file you are sure that all data are written

You can do normal read and write but it's unsafe to resize the file
Posted on 2004-01-08 02:11:29 by greenant

If I don't use FlushViewOfFile, when will the data be written back to the file?
And if a file is used for MMF, is it possible to do ordinary file operations, e.g. reading, writing, resizing, over the file at the same time?


You have to specify the size of the file when you open it (for writing). Hence, MMFs are not appropriate for file operations when you don't know the ultimate size of the file.
Cheers,
Randy Hyde
Posted on 2004-01-08 11:10:40 by rhyde

2Gig in Win98+, 1Gig in Win95

It will probably be less than this on any 9x, as MMF are placed in the shared part of addressing space - thus competing with DLLs and other MMF. Even for NT it will be slightly less, as there's no "shared" memory region, and MMF are per-process.

Btw, the file loads aren't actually done per-page, it's done i regions. I can't remember the exact size, but it's probably 64kb... to reduce the amount of pagefaults generated.

I wouldn't do other file operations on a file while it has a memory view mapped, just like I wouldn't be using multiple handles to the same file, or let multiple threads access a file through the same handle. Some of the operations are probably safe (consult the friendly MSDN), but IMO it's just asking for trouble.

Writes to file will be done lazily, on NT there's a flushing thread that works when the system is idle, and honestly I don't care too much about 9x :).

MMF are mostfly for convenience or when you need to share memory between processes, they're somewhat slower for regular file access, and gives you less control over memory usage etc.
Posted on 2004-01-08 18:28:25 by f0dder
Hi f0dder, could be that you're right. I got the 4K page size from the "Managing Memory-Mapped Files in Win32" technical article at MSDN.

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dngenlib/html/msdn_manamemo.asp
Posted on 2004-01-08 18:54:43 by donkey
Well, page size is definitely 4k on x86 (when not dealing with the 4meg and 2meg page extensions - I don't know exactly when these are used on windows, but afaik on NT the kernel and AGP memory is/can be mapped with 4meg pages).

Not pulling in 4k at a time makes a lot of sense, even if pagesize is 4k... less exception (better speed), and better for the same reasons as file caching is good (hm, I dunno how the relationship between MMF, caching subsystem, and the kernel pagein() function is, really - time to dig around Inside win2k again, I guess). I'm pretty sure (though I only recall vaguely) that the "not pulling in only 4k" is mention in the inside win2k book, and some friends of mine who have dealt a lot with NT internals did mention it too.
Posted on 2004-01-08 19:02:01 by f0dder