Hi, how do you store data that you use in your apps? I am using the following format in one of my apps, it works, but how could it be improved? I tend to memory map it and then use the data from there. Is that better than allocating heap memory?

The format is similar to the following:


-------------------------------------------------------------
| DWORD - File version. | File header.
| DWORD - Total chunks. |
| DWORD - Offset of main data chunk. |
| DWORD - Offset of strings chunk. |
-------------------------------------------------------------

-------------------------------------------------------------
| DWORD - Length of main data chunk. | Main data chunk.
-------------------------------------------------------------
| WORD - Unique ID of entry. |
| DWORD - Some value. |
| WORD - Some value. | This is a structure
| WORD - Some value. | repeated n times.
| WORD - Some value. |
| BYTE - Related string ID. |
-------------------------------------------------------------

-------------------------------------------------------------
| DWORD - Length of strings chunk. | Strings chunk.
-------------------------------------------------------------
| BYTE - String ID. |
| BYTE - Length of string (bytes) | n times.
| BYTE(s) - String (ASCII) |
-------------------------------------------------------------


Everytime I write I must update the offsets, but they allow me quick access to later chunks...
I used BYTE and WORD size values to reduce the overall file size, but I have just been reading about aligning data in your programs, should data in a file like this be aligned too? If so, would it be better to just use DWORDS to have a bigger file size but quick operations?
Hope I make some sense.
Thanks for any advice,
~Adam
Posted on 2004-07-11 15:59:26 by adamjjackson
Efficient file layout depends a great deal on the use and needs of the file.
Since you are using bytes for the string IDs I assume this is a very small file, there can only be 256 strings at any given point in time and 65535 data chunks.

Trying to squeeze size might be a waste of effort because all file allocations happen in increments of the disk's cluster size. If the file was 768 bytes, it would actually occupy 4096 bytes of physical space on any NTFS volume (I don't know what the default cluster size on fat32 is).

Most alignment performance issues occur when the data in memory itself is not aligned. Moving the data from disk into properly aligned structure should be all that you need to do.
Posted on 2004-07-11 18:13:04 by Mecurius
Hmmm, okay. Once it is memory mapped I get the address of the first structure and use that as a pointer to some struct I have defined. I'm not really sure if that counts as "Moving the data from disk into properly aligned structure". It's not a major issue, just want to do things the right way!
Thanks
Posted on 2004-07-12 03:38:19 by adamjjackson
adamjjackson,

This looks very much like a database. If it is, treat it as such. I would recommend TSUNAMI RECORD MANAGER. You find it here:
TSUNAMI RECORD MANAGER

Regards, P1 :cool:
Posted on 2004-07-12 09:13:26 by Pone
If the data layout is fairly simple, just use a STRUCT to help, otherwise use several structs, as long as the loader can undo whatever the saver does...
Posted on 2004-07-12 09:40:19 by Homer
adamjjackson,

If you will be using this as an ongoing project and the size of your data file will grow, I couldn't agree more with Pone! TRM is fast, "free", and very powerful. It won't hold your hand, but it will do just about anything you might want.

farrier
Posted on 2004-07-12 19:11:45 by farrier
I might take a look at it. Initially I did try using a database - I used ODBC thingies with an access database - I found it to be really clunky and slow. Since it isn't really a big project, I thought it would be simple enough to make something myself :) And since it's just a hobby I thought I might learn something too!
Cheers
Posted on 2004-07-13 04:16:10 by adamjjackson