hi everyone

i have made a small parser that can read some bytes from a file,parsing these bytes & write them to some other file,it continues performs these operations until it reaches EOF.

but instead of reading->parsing->writing, i want to read bytes,parsing them and store them in a byte array until it reaches EOF instead of wiring in another file as i doing before.

my problem is how can i reallocate that byte array & concatenate parsed bytes
while doing this whole operation.

as what i m doing now,read->parse->write to file operation is going fast, so plz advice me,how i acheive that much or higher speed with read->parse->byte array

the size of the input file to this parser me in 1-2 gb's also so plz consiseder this also.

thanks in advance and hope for good comments
Posted on 2006-10-12 11:53:43 by AssemblyBeginner
Hm, keeping 1-2GB in ram can be a pretty hard task, actually. Give some more info on the input format you're parsing, and what kind of output you parse to?
Posted on 2006-10-12 16:09:47 by f0dder
hi f0dder,

basically the input file contains some sorts of strings that are given to me and  my task is to remove that string occurences from file and store back to string(mem) instead of file and database.

any help regarding this.

Posted on 2006-10-13 11:16:48 by AssemblyBeginner
- how do you receive input? A file, a memory block, database queries?

- must you sort contents, or do you "only" need to store "good strings" while skipping "bad strings"?

If it's from file, you have basically two options: ReadFile or memory-mapped files. Memory mapped files tend to be a bit more convenient, but also require a bit more CPU (depending on the exact situation). Since you need to deal with and store huge amounts of data, you can't just map the entire file, but will have to process is in chunks - and then you might as well do the chunk reading with ReadFile instead.

Do you need to process line-by-line, or as a stream?

Are the "bad words" simply static words, or more like regular expressions?

Posted on 2006-10-13 11:24:52 by f0dder
i am reading from a file using readfile function without mapping and parse the bytes in small chunks and retreive back parsed bytes

now want to store that parsed bytes in memory until eof.

the words list is static.
Posted on 2006-10-13 11:40:24 by AssemblyBeginner
what i m doing is this in simple way

do until eof
    invoke readfile chunk of bytes.......
    invoke parse  chunk of bytes......
    invoke writefile  chunk of bytes...

and want to acheive

do until eof
    invoke readfile chunk of bytes.......
    invoke parse  chunk of bytes......
    ;instead of writing to file store parse byes into memory
Posted on 2006-10-13 11:56:21 by AssemblyBeginner
Okay :) - before processing a file, can you estimate how large the output will be? You can start by preallocating that amount of memory, then expand later on.

If the output is small compared to your input, HeapAlloc + HeapReallocate might be an option.

If the output is also very large, I suggest you implement something custom based on VirtualAlloc, probably chains of blocks. But how to do that depends a lot on what format the output has. I think Donkey implemented something custom with VirtualAlloc once, might be useful to get him to join in on this thread :)
Posted on 2006-10-13 11:59:43 by f0dder