I am writing a program to convert SFX archives to standard archives. I have everything working smoothly, but I have a slight predicament. You see I am currently filling the appropriate hex offsets with 0's, but I would like to delete those sections instead so that I can obtain the same results with a smaller file size.

The problem here being that my ASM knowledge is limited to the point that I cannot seem to get my head around doing this. Any help would be much appreciated.

Regards,

Lollie
Posted on 2003-12-28 14:46:11 by Lollie
Use of "File Maping" to load the program into memory using same method as the operating system. From there you could grab a pointer right underneath that sections and overlap the beginning of section with rest of file from memory buffer. That's assuming those sections are in same place. If not you could do search string to next section and still do this.
Posted on 2003-12-28 19:35:27 by mrgone
memory mapped files are somewhat slow though.

What is it exactly you want to do? Deleting a portion of a file is easy, but involves a deal of operations - you have to construct a temporary file, then copy all the wanted data from the old file. If all goes well, you can delete the original file and rename the temp file.
Posted on 2003-12-29 22:31:03 by f0dder
Well what I want to do is delete everything above a "critical" offset. I already have everything working and can obtain the critical offset fine; when it comes to deleting everything above that offset I am stumped. Would reading the data from file A then immediatly writing that data to file B be an appropriate solution?
Posted on 2003-12-30 16:52:43 by Lollie
Nah, use the SetEndOfFile function. It cuts off everything after the current file offset.
Posted on 2003-12-30 18:32:55 by Sephiroth3

memory mapped files are somewhat slow though.


?????
In my experience, memory mapped files are *much* faster than standard file I/O for most operations. This is due to the fact that file I/O uses the underlying (high-performance) VM system versus all the buffering that takes place with standard file I/O.

Please share your experiences with slow memory-mapped I/O...
Cheers,
Randy Hyde
Posted on 2003-12-31 00:08:20 by rhyde
Lollie,




TruncateFile proc file1,AtFilePos ; if AtFilePos is 10, the
; resulting file will be
; 10 bytes long
invoke SetFilePointer,file1,AtFilePos,0,FILE_BEGIN
invoke SetEndOfFile,file1
ret
TruncateFile endp

DeletePortionOfFile proc file1,StartPos,LenOfPortion
local str1[4096]:byte ; temporary buffer
local ReadPos,WritePos,bytesRead,bytesWritten
mov eax,StartPos
mov WritePos,eax
add eax,LenOfPortion
mov ReadPos
_move_another_block:
invoke SetFilePointer,file1,ReadPos,0,FILE_BEGIN
invoke ReadFile,file1,addr str1,4096,addr bytesRead,0
invoke SetFilePointer,file1,WritePos,0,FILE_BEGIN
.if bytesRead
invoke WriteFile,file1,addr str1,bytesRead,\
addr bytesWritten,0
.endif
mov eax,bytesRead
add ReadPos,eax
add WritePos,eax
or eax,eax
jnz _move_another_block
invoke SetEndOfFile,file1
ret
DeletePortionOfFile endp


Using the second function, if LenOfPortion is 10, the filesize will get 10 bytes shorter. (I'm saying this just to make sure). "StartPos" is zero-based. This means, that "0" is the first byte of the file.

Randall, maybe he means it's slower to use mappedfiles if you're going to read the whole file in no more than one time, and not much randomly.
Posted on 2003-12-31 00:43:48 by Ultrano

Randall, maybe he means it's slower to use mappedfiles if you're going to read the whole file in no more than one time, and not much randomly.


Actually, that's when mmapped files do really well.
Perhaps when copying data from one file to another (subtracting a few bytes in the middle) there would be little benefit to using mmap files (as you wind up doing the buffering manually that the file system does automatically).
Cheers,
Randy Hyde
Posted on 2003-12-31 08:22:51 by rhyde
randall, mmf are slow - it's much faster to do traditional buffered fileread/write, with appropriate sized buffers. Some of the slowness of mmf comes from the use of page fault exceptions and the underlying extra code required, I guess. Furthermore, there's the problems of address space usage, especially on 9x.
Posted on 2003-12-31 12:34:34 by f0dder
Lollie, for safety reasons perhaps you should consider copying the data you want to keep to a temp file, deleting the original file, and then renaming the temp one. Sure, t's much slower, but that way if an error occurs during the operation you don't corrupt the original file. That's the way most (if not all) archivers work.
Posted on 2003-12-31 12:51:06 by QvasiModo

randall, mmf are slow - it's much faster to do traditional buffered fileread/write, with appropriate sized buffers. Some of the slowness of mmf comes from the use of page fault exceptions and the underlying extra code required, I guess. Furthermore, there's the problems of address space usage, especially on 9x.


Source file #1 (using memory mapped I/O):


// Performance test of memory-mapped versus normal file I/O:
// Memory mapped file version:


program t;
#include( "stdlib.hhf" )
var
theFile:mmap;
destFile:mmap;
adrs:dword;

begin t;

for( mov( 0, edx ); edx< 2000; inc( edx )) do

theFile.create();
destFile.create();
theFile.open( "dictionary.txt", fileio.r );
destFile.openNew( "destfile.txt", theFile.fileSize );

mov( theFile.filePtr, esi );
mov( destFile.filePtr, edi );
mov( theFile.fileSize, ecx );
cld();
rep.movsb();
theFile.close();
destFile.close();
theFile.destroy();
destFile.destroy();

endfor;


end t;


Source file #2 (using file I/O):


program t2;
#include( "stdlib.hhf" )
var
theFile:dword;
destFile:dword;
buffer:byte[4096];
destbuf:byte[4096];

begin t2;

for( mov( 0, edx ); edx< 2000; inc( edx )) do

fileio.open( "dictionary.txt", fileio.r );
mov( eax, theFile );
fileio.openNew( "destfile.txt" );
mov( eax, destFile );
repeat

fileio.read( theFile, buffer, 4096 );
push( eax );
cld();
lea( buffer, esi );
lea( destbuf, edi );
mov( eax, ecx );
rep.movsb();
fileio.write( destFile, destbuf, eax );
pop( eax );

until( eax <> 4096 );
fileio.close( theFile );
fileio.close( destFile );

endfor;


end t2;


Turns out we're both wrong. The run times are roughly equivalent (off a few tenths of a second, that varies by run in either direction, easily attributable to system multitasking overhead).

OTOH, dealing with the file buffer using pointers is a whole lot easier for
smaller files (less than 1GB) than dealing with buffer boundaries.

As for dealing with Win9x, I'd simply suggest that if you're worried about application performance, you shouldn't be worrying about performance on Win9x - write for the latest OS and let it go at that. If someone's using Win95 and gripes about it, well, they're probably running a *really old* machine anyway and performance is going to stink no matter what :-)

Cheers,
Randy Hyde
Posted on 2003-12-31 14:29:57 by rhyde
Humm... I did some tests on CRC32 a while back, of course without file caching, as that tends to distort results a bit :). Used large buffers (256kb-ish, I think), so there wouldn't be too much r3<>r0 overhead. MMF was noticeably slower, of course because of the 3?#PF mechanism that MMF uses - good thing they don't do #PF for every 4k, but use "ranges" instead (can't remember the correct term right now).

Buffered IO can require somewhat more complex code, and MMFs are quite convenient indeed... but if you're dealing with really large files, you'll have to do buffer logic anyway (remapping the view to another portion of the file), so you might as well use normal buffered file I/O.

I'm not too worried about 9x, IMO it should never have existed ^_^ - but it's still worth keeping in mind that MMF goes into the global part of the address space on 9x, which quite limits the amount and size of files you can deal with.

Also, there's async I/O if you really need speed.

But of course MMF is fine if you don't need critical speed, don't deal with very large files, or need the capability to share MMFs between processes, they're fine - and definitely handy.
Posted on 2004-01-01 01:06:20 by f0dder
f0dder, do you know if a page fault takes longer than using ReadFile to read 4k of the file, assuming the file is read sequentially ?
Posted on 2004-01-01 06:31:37 by Dr. Manhattan
Good question - I think it will be hard to measure. The #PF for MMF brings in multiple 4k pages, just like a FileRead on 4k should take more bytes into the file cache - the overhead will be r3<>r0 and #PF processing time... with larger buffers and readfile, you should be able to minimize r3<>r0 switches.

Async I/O is interesting, but takes some coding and logic to do right... but it does seem pretty nice.
Posted on 2004-01-01 15:02:57 by f0dder


Async I/O is interesting, but takes some coding and logic to do right... but it does seem pretty nice.


Like many things, if done correctly Async I/O is very fast, but if used incorrectly it will slow a program down. There are a number of hidden costs with async, mostly kernel mode processessing time. Also creating, setting, checking, clearing the Events uses kernel mode switches/processing. Async is good for "large" reads/writes or where other processing can be done while waiting for the I/O to complete and bad for lots of "small" reads/writes.
Posted on 2004-01-01 15:35:51 by Mecurius
I can tell you this. If you use GlobalAlloc then Windows uses 4meg pages to allocate the memory though it is still loaded to memory in 4K blocks. If there are several 4k blocks than they most certainly will not be sequencial.
Posted on 2004-01-01 20:31:46 by mrgone
What do you mean by "Windows uses 4meg pages to allocate the memory though it is still loaded to memory in 4K blocks" ? Do you mean a 4Mb page can be splitted in several blocks of physical memory ?
Posted on 2004-01-02 03:01:43 by Dr. Manhattan
You understand you can have 4Meg pages or 4K pages. 4Meg page entries are in the page directory and 4k page entries are in the page table. An example is in W2K only, all 4Meg pages are allocated between I beleive it was 88000000h and 8BFFFFFF logical addresses. So specifically what I was saying before is the "GlobalAlloc" function uses 4Meg pages so that in W2K you will always find the data between these addresses. They still dole out the memory in 4K blocks though and usually not sequencial(Am I spelling that right?).
A similar example would be that the processor uses multitasking and can run each program in it's own TSS but Windows doesn't do this. It runs all programs as an extension of it's own tasks or task and MS refers to programs as processes.
Basically they got first dibs on the operating system and can do what ever they want. I will say though I would not have wanted the chor of putting together all those graphics and they do an outstanding job with it.
Posted on 2004-01-02 07:45:25 by mrgone

Async is good for "large" reads/writes or where other processing can be done while waiting for the I/O to complete and bad for lots of "small" reads/writes.

Yup... checksum/hash processing of files, or compression, etc would probably be a good example of large reads and writes.
Posted on 2004-01-02 14:23:44 by f0dder