I have mentioned it before on a few occassions, but now it's time to dedicate a thread to it.
The BHM file format is a format that was originally developed for the Bohemiq demosystem. I have recently decided to make the file format open, and release the code under the BSD license. I have created a project on SourceForge for this:
http://bhmfileformat.sourceforge.net
The summary of the project reads:
"A generic hierarchical file format. BHM can be seen as the binary equivalent of XML. It allows flexible hierarchies of any type of data, in a platform-independent way. BHM however trades XML's readability for compactness and speed of processing."
That pretty much covers the idea of BHM. It is tree-based, like XML, and it is flexible in the use of node types/tags, like XML. However, it stores everything in binary, rather than in human-readable ASCII, which makes the file a lot more compact, and faster to parse. This is especially interesting for large arrays of data, such as vertex buffers, textures, audio etc.
For more information, see the attached README.txt from the project.
In short, the file format is meant to be platform-independent. An exporter for 3dsmax6 is included in the project source code.
I am currently working on a platform-independent example program that will read such exported files from 3dsmax6, and display the scene, complete with animation (the skinned claw model in my D3D Engine example is a BHM file modeled in 3dsmax6. I will be using this file for the BHM example code aswell).
Exporters for other modelers should also be possible, and everyone is free to join the project to add support for other modelers, and expand the BHM format to support more 3D/animation primitives.
The BHM file format is a format that was originally developed for the Bohemiq demosystem. I have recently decided to make the file format open, and release the code under the BSD license. I have created a project on SourceForge for this:
http://bhmfileformat.sourceforge.net
The summary of the project reads:
"A generic hierarchical file format. BHM can be seen as the binary equivalent of XML. It allows flexible hierarchies of any type of data, in a platform-independent way. BHM however trades XML's readability for compactness and speed of processing."
That pretty much covers the idea of BHM. It is tree-based, like XML, and it is flexible in the use of node types/tags, like XML. However, it stores everything in binary, rather than in human-readable ASCII, which makes the file a lot more compact, and faster to parse. This is especially interesting for large arrays of data, such as vertex buffers, textures, audio etc.
For more information, see the attached README.txt from the project.
In short, the file format is meant to be platform-independent. An exporter for 3dsmax6 is included in the project source code.
I am currently working on a platform-independent example program that will read such exported files from 3dsmax6, and display the scene, complete with animation (the skinned claw model in my D3D Engine example is a BHM file modeled in 3dsmax6. I will be using this file for the BHM example code aswell).
Exporters for other modelers should also be possible, and everyone is free to join the project to add support for other modelers, and expand the BHM format to support more 3D/animation primitives.
ChunkInfo has a uint Type and TreeNodeInfo has a uint Type
I've only read the readme but it seems like TreeNodeInfo.ChunkID correlates with ChunkInfo.ID, in which case the repeated ChunkInfo.Type is unnecessary.
Also, why not combine the two structures and just have
If I'm understanding correctly TreeNodeInfo and ChunkInfo have a 1 to 1 relationship, as a result normalizing with the ID relation is inefficient.
Actually making datasize and total size part of the data pointed to by dataoffs would be the most efficient.
File Header:
Tree Node Info Array:
[#of children]
[#of childrend]
...
[#of childrend]
Binary Data: (each entry 16byte aligned)
dataoffs1: [8byte pad]
dataoffs2: [8byte pad]
...
dataoffsN: [8byte pad]
?
I've only read the readme but it seems like TreeNodeInfo.ChunkID correlates with ChunkInfo.ID, in which case the repeated ChunkInfo.Type is unnecessary.
Also, why not combine the two structures and just have
typedef struct {
unsigned int numchildren;
unsigned int type;
unsigned int datasize;
unsigned int totalsize;
unsigned int dataoffs;
} TreeNodeInfo;
unsigned int numchildren;
unsigned int type;
unsigned int datasize;
unsigned int totalsize;
unsigned int dataoffs;
} TreeNodeInfo;
If I'm understanding correctly TreeNodeInfo and ChunkInfo have a 1 to 1 relationship, as a result normalizing with the ID relation is inefficient.
Actually making datasize and total size part of the data pointed to by dataoffs would be the most efficient.
File Header:
Tree Node Info Array:
[#of children]
[#of childrend]
...
[#of childrend]
Binary Data: (each entry 16byte aligned)
dataoffs1: [8byte pad]
dataoffs2: [8byte pad]
...
dataoffsN: [8byte pad]
?
The tree structure is decoupled from the chunks.
The tree just describes how the data is structured (the size of the data, or the offset from the start of the chunk don't seem to be relevant to the structure of the data, they'd just make the tree more bloated). The chunks merely store the data (which is why that's where the size is stored... You don't need to look at the tree to load the data and find the next chunk). They are not required to be in any specific order (which can give you more freedom in modifying files).
If you were to combine them, you'd need to scan through the whole file just to retrieve the tree.
With the current file layout, you can simply load the whole tree in one go, and rebuild it in memory.
You can then use this tree to determine which chunks you may be interested in, and then retrieve them from the rest of the file.
The type field in the chunk info is not strictly required, but it doesn't hurt to store it there. It might be useful when dealing with broken files.
One concern I do have though... we didn't decide on using 64-bit integers back when we designed the format. This means that a single chunk is limited to 4 GB. Other than that, there are no limitations though.
Perhaps an extended chunk format should be designed to handle such large chunks. By making use of the dataoffset value, you can put extended info in the chunk header, and remain backwards compatible.
The tree just describes how the data is structured (the size of the data, or the offset from the start of the chunk don't seem to be relevant to the structure of the data, they'd just make the tree more bloated). The chunks merely store the data (which is why that's where the size is stored... You don't need to look at the tree to load the data and find the next chunk). They are not required to be in any specific order (which can give you more freedom in modifying files).
If you were to combine them, you'd need to scan through the whole file just to retrieve the tree.
With the current file layout, you can simply load the whole tree in one go, and rebuild it in memory.
You can then use this tree to determine which chunks you may be interested in, and then retrieve them from the rest of the file.
The type field in the chunk info is not strictly required, but it doesn't hurt to store it there. It might be useful when dealing with broken files.
One concern I do have though... we didn't decide on using 64-bit integers back when we designed the format. This means that a single chunk is limited to 4 GB. Other than that, there are no limitations though.
Perhaps an extended chunk format should be designed to handle such large chunks. By making use of the dataoffset value, you can put extended info in the chunk header, and remain backwards compatible.
Hum, I've played around a bit with OpenGL under FreeBSD, but I found that although I have an Intel G31 chipset in my FreeBSD box, which technically is an SM2.0 part, and has OpenGL 2.0 support under Windows... in FreeBSD I am not that lucky.
I wanted to use GLSL to implement the skinning in a vertexshader, but ARB_VERTEX_SHADER is not supported. It does support ARB_VERTEX_PROGRAM, but at this point I'm not sure if I want to implement it with that.
I think I will just concentrate on the Windows-version of the sample program for BHM loading and animation for now, and worry about other OSes later. In theory the GLSL version should also work on FreeBSD/linux, I just cannot test it on my own machine at this point.
Perhaps once the GLSL version is done, it's not that much extra work to make an ARB_VERTEX_PROGRAM version, in theory I would only have to rewrite a single shader.
I wanted to use GLSL to implement the skinning in a vertexshader, but ARB_VERTEX_SHADER is not supported. It does support ARB_VERTEX_PROGRAM, but at this point I'm not sure if I want to implement it with that.
I think I will just concentrate on the Windows-version of the sample program for BHM loading and animation for now, and worry about other OSes later. In theory the GLSL version should also work on FreeBSD/linux, I just cannot test it on my own machine at this point.
Perhaps once the GLSL version is done, it's not that much extra work to make an ARB_VERTEX_PROGRAM version, in theory I would only have to rewrite a single shader.
Played around a bit more, and I'm definitely giving up on FreeBSD now... According to glxinfo, my Intel chip+driver supports ARB_vertex_buffer_object, but when I try to use it, I get a vague crash... not even during the creation of the VBOs, but after that.
So I thought perhaps my code was broken... so I decided to download some VBO tutorial and build that on my box... Did the exact same thing.
Another thing that shows how D3D has 'deformed' me... There doesn't seem to be a 'default' math library for OpenGL, something like D3DX. The basic matrix/vector math in the OpenGL API is very limited. There are a few 'usual suspects' as addons for OpenGL, such as GLU, GLUT and GLEW.
But it seems like there's no 'usual suspect' for a sort of D3DX replacement.
In fact, even the datatypes of OGL themselves are barely datatypes... you just get flat arrays of GLfloat and related types, which are supposed to represent colour, vertex, normal, texcoord, matrix and that sort of data.
On my travels I found various attempts at such a library, most of them in various stages of incompleteness... I did however find two that were reasonably promising... GLM and CML.
I think I'll give CML a try, it gave the best first impression of the two.
So I thought perhaps my code was broken... so I decided to download some VBO tutorial and build that on my box... Did the exact same thing.
Another thing that shows how D3D has 'deformed' me... There doesn't seem to be a 'default' math library for OpenGL, something like D3DX. The basic matrix/vector math in the OpenGL API is very limited. There are a few 'usual suspects' as addons for OpenGL, such as GLU, GLUT and GLEW.
But it seems like there's no 'usual suspect' for a sort of D3DX replacement.
In fact, even the datatypes of OGL themselves are barely datatypes... you just get flat arrays of GLfloat and related types, which are supposed to represent colour, vertex, normal, texcoord, matrix and that sort of data.
On my travels I found various attempts at such a library, most of them in various stages of incompleteness... I did however find two that were reasonably promising... GLM and CML.
I think I'll give CML a try, it gave the best first impression of the two.
nvMath suits my needs and style. It's buried deep in nvSDK9.5/10.5 (is not announced anywhere or listed in google) . It doesn't have SSE, but might be worth a look. It was easy to convert the stuff to left-hand.
Played around a bit more, and I'm definitely giving up on FreeBSD now... According to glxinfo, my Intel chip+driver supports ARB_vertex_buffer_object, but when I try to use it, I get a vague crash... not even during the creation of the VBOs, but after that.
So I thought perhaps my code was broken... so I decided to download some VBO tutorial and build that on my box... Did the exact same thing.
I suddenly had this suspicion of what might go wrong in FreeBSD... As I said, it didn't crash DURING the creation of the VBOs, but it crashed at a later point.
It crashed when I entered the glutMainLoop().
I left my VBOs in a bound state though... as I use VBOs exclusively throughout my program.
In Windows this works correctly, and judging from the sample code I've found around the web, it is common practice to not explicitly call glBindBuffers() with 0 to 'unbind' the buffer after use.
As long as you don't do anything that requires geometry from something other than a VBO, it should be perfectly valid to leave the buffers bound.
However, that may be the problem with the runtime in FreeBSD. It may be using some kind of draw calls in the background when glutMainLoop() starts and opens the window, because that's where it went wrong.
So, after the initial initialization, I 'unbound' my VBOs, to see what happens. Instead of nothing happening at all, I could now see the glut window appearing, and a single frame being rendered, before it crashed.
Apparently I was on to something there...
So I figured... there might be some more VBO-incompatible stuff going on in glutSwapBuffers() or such... So after my draw calls I again 'unbound' my VBOs, and yay, it works.
That still doesn't get me very far, as I still lack GLSL support in FreeBSD... but at least the extensions I DO have actually work now :)
It also begs the question: who is right? If the FreeBSD runtime is the 'correct' way and leaving VBOs bound is not the way to do it, it just 'accidentally' works on some implementations... then I guess there's a LOT of broken code out there.
There's no way FreeBSD's implementation is correct. The specs state clearly that only specific geometry-defining calls are affected by bound VBOs, just as you expected.
There's no way FreeBSD's implementation is correct. The specs state clearly that only specific geometry-defining calls are affected by bound VBOs, just as you expected.
That part is clear yes, but the question is more: Does FreeBSD (or GLUT) use geometry-defining calls in parts of its runtime (eg, it may use a textured quad to render to the desktop window)? And if so, is that allowed?
I use freeglut on Windows, and the standard glut that is bundled with Mesa on FreeBSD, so I wonder if they are different. I could try using freeglut on BSD...
However, since I use Mesa (with the Tungsten Intel drivers), X.Org and KDE, one would think that the environment is pretty much identical to a linux installation, I wonder what happens there (then again, the VBO sample code I tried was probably developed for linux aswell).
I'd check the srccode of said libs and x-server, and also post this in a bug-tracker or forum/maillist.
I'd check the srccode of said libs and x-server, and also post this in a bug-tracker or forum/maillist.
I've reported it to the FreeBSD team... I'll leave it up to them to figure out whether it's FreeBSD-specific, Intel-specific, Mesa-glut specific or otherwise.
Hummm... it looks like AMD's OpenGL drivers still aren't all that great.
I added a simple FPS counter to my code, to get an idea of how well things are running so far.
I rendered a single triangle, using static VBOs, which should theoretically be the fastest way to render them.
First I tried on my laptop with Intel X3100. It got around 1000 fps, in Vista 32-bit.
Then on my desktop with Radeon HD5770. About 1500 fps in Vista 64-bit. Then I tried in Windows 7 64-bit, closer to 2000 fps.
Now I was getting curious.... Apparently there's quite a bit of CPU overhead somewhere, because the GPU should be WAY faster than the X3100, not just 1.5-2x as fast (especially since I also have a 1.5 GHz Core2 Duo in my laptop and a 3 GHz Core2 Duo in my desktop). What's more, if I run the ENTIRE BHM scene in D3D9/10/11 (as opposed to just a single triangle), animated, per-pixel lit and everything, I get about 6000 fps still, on the HD5770.
So I tried it in XP, which is generally slightly faster in windowed mode (gets about 7000 fps on my HD5770). Now I got about 3500 fps... that's more like it, but still a far cry from D3D9 performance.
Now AMD's drivers have a bit more overhead than nVidia's anyway... Because on an old Pentium 4 system I have, with a GeForce 9800GTX (which is quite a bit slower than a HD5770), I would get 7000 fps in Vista/Win7, and 8000 fps in XP.
So f0dder ran the OpenGL thingie for me on his machine with Windows 7 and a GeForce. He got over 5000 fps... So apparently the overhead is mostly in AMD's OpenGL implementation. Even on nVidia's implementation, it's not *quite* as low overhead as D3D, but still it's considerably better than AMD.
So... what do I conclude from this so far?
- Contrary to popular belief, basic driver overhead in D3D is lower than in OpenGL, in practice.
- Windowed mode 3d graphics have slightly more overhead on Vista/Windows 7 than on XP.
- The difference in overhead between Vista/Windows 7 and XP is larger with AMD than with nVidia.
- The overhead in AMD's drivers is higher than in nVidia's drivers.
- In OpenGL, the difference in overhead between AMD's and nVidia's drivers is significant.
In practice it probably won't make that much of a difference though.
Namely:
- 1000 fps means 1 ms per frame
- 8000 fps means 0.125 ms per frame
- 60 fps means 16.667 ms per frame
So, if we assume all of the difference fo rendering at 1000 fps vs 8000 fps to be overhead, then that means the driver will have 0.875 ms overhead per frame.
Now, if you were to translate that to the 60 fps figure, an extra 0.875 ms overhead per frame would only only make a difference of about 3 fps. So it's very marginal.
Once the OpenGL code can also render and animate the entire BHM file, I can do a more direct comparison. Frankly I don't expect the framerate to drop much, if at all, when it has to render the entire scene. I'll just have to wait and see if I'm right about that.
I added a simple FPS counter to my code, to get an idea of how well things are running so far.
I rendered a single triangle, using static VBOs, which should theoretically be the fastest way to render them.
First I tried on my laptop with Intel X3100. It got around 1000 fps, in Vista 32-bit.
Then on my desktop with Radeon HD5770. About 1500 fps in Vista 64-bit. Then I tried in Windows 7 64-bit, closer to 2000 fps.
Now I was getting curious.... Apparently there's quite a bit of CPU overhead somewhere, because the GPU should be WAY faster than the X3100, not just 1.5-2x as fast (especially since I also have a 1.5 GHz Core2 Duo in my laptop and a 3 GHz Core2 Duo in my desktop). What's more, if I run the ENTIRE BHM scene in D3D9/10/11 (as opposed to just a single triangle), animated, per-pixel lit and everything, I get about 6000 fps still, on the HD5770.
So I tried it in XP, which is generally slightly faster in windowed mode (gets about 7000 fps on my HD5770). Now I got about 3500 fps... that's more like it, but still a far cry from D3D9 performance.
Now AMD's drivers have a bit more overhead than nVidia's anyway... Because on an old Pentium 4 system I have, with a GeForce 9800GTX (which is quite a bit slower than a HD5770), I would get 7000 fps in Vista/Win7, and 8000 fps in XP.
So f0dder ran the OpenGL thingie for me on his machine with Windows 7 and a GeForce. He got over 5000 fps... So apparently the overhead is mostly in AMD's OpenGL implementation. Even on nVidia's implementation, it's not *quite* as low overhead as D3D, but still it's considerably better than AMD.
So... what do I conclude from this so far?
- Contrary to popular belief, basic driver overhead in D3D is lower than in OpenGL, in practice.
- Windowed mode 3d graphics have slightly more overhead on Vista/Windows 7 than on XP.
- The difference in overhead between Vista/Windows 7 and XP is larger with AMD than with nVidia.
- The overhead in AMD's drivers is higher than in nVidia's drivers.
- In OpenGL, the difference in overhead between AMD's and nVidia's drivers is significant.
In practice it probably won't make that much of a difference though.
Namely:
- 1000 fps means 1 ms per frame
- 8000 fps means 0.125 ms per frame
- 60 fps means 16.667 ms per frame
So, if we assume all of the difference fo rendering at 1000 fps vs 8000 fps to be overhead, then that means the driver will have 0.875 ms overhead per frame.
Now, if you were to translate that to the 60 fps figure, an extra 0.875 ms overhead per frame would only only make a difference of about 3 fps. So it's very marginal.
Once the OpenGL code can also render and animate the entire BHM file, I can do a more direct comparison. Frankly I don't expect the framerate to drop much, if at all, when it has to render the entire scene. I'll just have to wait and see if I'm right about that.
Single-triangle scenes are far from a viable benchmark :P
I've had such scenes run at 22-25k fps, 720p; means nothing.
Or again, 12500 fps to render a whole CSS map in a 64x64, with texlod-biasing.
I've had such scenes run at 22-25k fps, 720p; means nothing.
Or again, 12500 fps to render a whole CSS map in a 64x64, with texlod-biasing.
Single-triangle scenes are far from a viable benchmark :P
I've had such scenes run at 22-25k fps, 720p; means nothing.
Or again, 12500 fps to render a whole CSS map in a 64x64, with texlod-biasing.
They're a viable benchmark for the overhead because they measure everything in the OpenGL runtime and the rendering framework, with minimal actual geometry.
All the states are processed, lights and geometry are set, screen is cleared every frame, etc.
So I measure all the overhead, and the actual time spent rendering the scene is negligible, as it's just a single triangle (which, unlike NOT rendering anything, doesn't give the driver any opportunity to 'cheat' and skip setting states altogether, if it has some kind of lazy/optimized state machine).
And yes, I know you can do better, you always can. You tell that in every post. Problem is, I don't care. It's not a contest.
These "overhead benchmarks" vary greatly even between minor driver revisions. And I was telling that there are drivers that can transform and lit a whole scene two times faster than the one-triangle bench. This overhead is not "0.875ms lost time", it gets masked once you start drawing actual geometry.
These "overhead benchmarks" vary greatly even between minor driver revisions. And I was telling that there are drivers that can transform and lit a whole scene two times faster than the one-triangle bench. This overhead is not "0.875ms lost time", it gets masked once you start drawing actual geometry.
I don't think you understood my post. I didn't say anything like that. But I'm not going to bother to explain and clarify.
I let you do your thing... Why don't you let me do mine...
Hey, since I got back into opengl, I have to ask you about your file format.
I just implementing code for MS3D extension (skinmesh with skinweights) and I was wondering if you had expressed a format for animation bones and keyframes, and what it looks like.
If it looks nice, I'll write import/export code and support it.
Well, I don't know all the details from the top of my head...
But roughly, it supports standard keyframing animation, based on the rotation and position keys as 3dsmax uses them.
The skeleton is exported the same way (3dsmax even exports the actual geometry for it), so you get a bunch of objects with animation keys attached to them.
Skinning in an object is implemented by having a reference to the bone names in the skeleton. So each object knows which bones will affect it.
You then animate the skeleton the same way as regular objects (but don't draw it, obviously), which gives you the resulting bone matrices in world space. These will then be used as a matrix palette, which is indexed in the shader.
But roughly, it supports standard keyframing animation, based on the rotation and position keys as 3dsmax uses them.
The skeleton is exported the same way (3dsmax even exports the actual geometry for it), so you get a bunch of objects with animation keys attached to them.
Skinning in an object is implemented by having a reference to the bone names in the skeleton. So each object knows which bones will affect it.
You then animate the skeleton the same way as regular objects (but don't draw it, obviously), which gives you the resulting bone matrices in world space. These will then be used as a matrix palette, which is indexed in the shader.
Last night, I did a quick port of the basic BHM file code to FreeBSD.
The OpenGL engine can now load the geometry and the camera from the BHM file, on both Windows and FreeBSD (and theoretically it should also work on similar OSes such as linux, Solaris, OS X etc).
There are now two steps left to complete the BHM sample:
- Interpolation of the animation keyframe data.
- Shaders for the skinning animation.
So far, I'm not too happy with the cml library that I chose as the underlying math library. It's rather clunky to use, since it isn't that easy to just take an array of OpenGL float data, and interpret it as a vector or array (like how D3DX seamlessly blends in with the standard D3D types).
I haven't exactly decided what to do about that yet.
Before cml, I had made my own basic data wrappers, which work more like D3DX. So far they were just for data storage, and they don't actually support any math operations yet. I could extend these wrappers and use cml internally, so the code remains easy and intuitive to read.
I could also give GLM a look, it may integrate with OpenGL better than cml does.
Or I could just give up on my idea to keep it clean and simple, and just build the entire math lib myself.
It almost looks like the OpenGL world needs such a math lib. Thing is just that I don't feel I'm the right person to build and maintain such a library, and especially not as part of this project.
The OpenGL engine can now load the geometry and the camera from the BHM file, on both Windows and FreeBSD (and theoretically it should also work on similar OSes such as linux, Solaris, OS X etc).
There are now two steps left to complete the BHM sample:
- Interpolation of the animation keyframe data.
- Shaders for the skinning animation.
So far, I'm not too happy with the cml library that I chose as the underlying math library. It's rather clunky to use, since it isn't that easy to just take an array of OpenGL float data, and interpret it as a vector or array (like how D3DX seamlessly blends in with the standard D3D types).
I haven't exactly decided what to do about that yet.
Before cml, I had made my own basic data wrappers, which work more like D3DX. So far they were just for data storage, and they don't actually support any math operations yet. I could extend these wrappers and use cml internally, so the code remains easy and intuitive to read.
I could also give GLM a look, it may integrate with OpenGL better than cml does.
Or I could just give up on my idea to keep it clean and simple, and just build the entire math lib myself.
It almost looks like the OpenGL world needs such a math lib. Thing is just that I don't feel I'm the right person to build and maintain such a library, and especially not as part of this project.