Now that the CPUInfo library is on its way, I'll get back to developing the D3D11 engine and related tasks again. The previous thread about abstracting 64-bit had become a bit of a blog/brainstorming session far beyond its original scope.
Since the 64-bit part is covered, I figured I'd make a new thread which is more appropriate for such topics.

This thread exists both for me write down ideas so I can find them back later, get my thoughts straightened out and all, and for you to follow what it is I'm currently working on, and ask questions or offer suggestions if you like.

My plans so far are (not necessarily in any order):
- Port over some of the remaining code from the original D3D8/9 codebase (mainly the scene loading code).

- Continue work on the BigMesh code, which allows easy manipulation of geometry, before converting it to D3D (or theoretically any other format). This is a candidate for an open source project, as it can mostly be written as reusable and multiplatform code.

- Update the shadowmapping code to more modern algorithms with more uniform mappings.

- Toy around with DirectCompute and CS4.0 to try and get some tessellation going (I'm thinking of something REYES-like, not necessarily realtime, but maximum image quality, I've already did some experiments with Cuda while waiting for either OpenCL or CS to become available).

...

Well, that's it for now. I'm sure I'll think of more as I go along.

Completed so far:
- Work on the BHM file format a bit more. This is also a candidate for an open source project. The format works much like any archive format, except that it isn't directly file-oriented. It has binary chunks and headers that describe them. Think of it as a binary version of an XML file. It's very convenient for importing and exporting structured sets of data, such as 3d geometry. You can also include textures, mp3s and other files.
I have written an exporter for 3dsmax6 for the 3d format I've defined using BHM. I think it may be useful to other people to have a working importer/exporter set, which they can extend with new functionality, or modify for use with their own engines or other modelers. With any luck, it will actually become some kind of standard 3d format, which for some reason never worked out so far.
- Finish up the Dependency Walker routine I started, to try and get some more useful/robust error messaging when the engine fails to load (this code is open source and available in the CPUInfo project repository on SourceForge.net).
Posted on 2009-10-12 07:48:18 by Scali
3d formats. The only thing that can hold them properly is the original modeler you use :P, as modelers keep+generate their data in absolutely different ways. My conclusion for my stuff was to transcode binaries of the data I import into a temporary bloated RAM-based multi-mesh, and then have shaders/materials pick and pack select data of parts themselves to be saved to file. A simple example:

class XMI_BoneAnim1 : public XMESH{
public:
ILVBO vboPos;
ILVBO vboFull;

ILTEX tex0;

virtual void AttachToPasses(PASSLIST* p){
p->depth->add(this);
p->albedoAndLight->add(this);
}

virtual void RenderDepth(){
ilUseProgram(progDefault);
ilSetUniformsVS(1,&g_Matrix_MVP);
ilDrawVBO(vboPos);
}
virtual void RenderAlbedoAndLight(){
ilSetTexture(0,tex0);
ilUseProgram(progBoneAnim1);

ilSetUniformsVS(1,&g_Matrix_MVP);


ilDrawVBO(vboFull);
}


virtual void TranscodeLWO(XSurface* xs){
xs->ConvertQuadsToTris();

int numv = xs->NumVerts;
int numt = xs->NumTris;

ufwrite4(numv);
ufwrite4(numt);
ufwritestr(xs->GetUVChanTexture('COLR',0));

ufwrite(xs->AllTris,numt*3*4); // write indices

vec3* pos = xs->AllVerts;
vec3* norm= xs->AllNorms;
vec4* uv0 = xs->GetUVChan('COLR',0);
vec4* boneID= xs->GetVTXMap('WGHT',"ILX_BoneIDs");
vec4* boneWT= xs->GetVTXMap('WGHT',"ILX_BoneWTs");
for(int i=0;i<numv;i++){
VTXDCL_Type4 v;
v.pos = pos;
v.norm= norm;
v.uv0 = uv0.xy();
v.boneID[0] = (U8)(boneID.x);
v.boneID[1] = (U8)(boneID.y);
v.boneID[2] = (U8)(boneID.z);
v.boneID[3] = (U8)(boneID.w);
v.boneWT = boneWT;
ufwrite(&v,sizeof(v));
}
}

virtual void LoadFromFile(){
int numv = XMDT_fread4();
int numt = XMDT_fread4();
tex0 = ILXLoadTexture(XMDT_freadstr());

void* indices = XMDT_fread(numt*3*4);
void* verts  = XMDT_fread(numv*sizeof(VTXDCL_Type4));
vec3* pos = XMDT_ExtractVtxPos(verts,numv,sizeof(VTXDCL_Type4));

vboPos = ilCreateStaticVBO(pos  ,numv,vtxDecl_Pos,indices,numt*3*4,true);
vboFull= ilCreateStaticVBO(verts,numv,vtxDecl_Type4,indices,numt*3*4,true);
delete[] pos;
}

virtual ~XMI_BoneAnim1(){
ilDeleteVBO(vboPos);
ilDeleteVBO(vboFull);
ilDeleteTexture(tex0);
}

};


Can easily define collision-hulls, nav-meshes, lights, null-proxies, LOD layers. The code has access to the same data that the modeler is working with, and the material is made to mimic the modeler's node-based shading (or vice-versa) to keep coherency of the art.
Posted on 2009-10-12 16:03:44 by Ultrano
At the end of the day, it's all just geometry and materials.
All my engines have used pretty much the same format for about 8 years I'd say. I don't think it's very likely that there will be fundamental changes to this anytime soon, as the hardware still works the same. I just add extra elements from time to time. But my file format is very modular, so it's easy to extend it (and remain backward compatible).
I've only made an exporter for 3dsmax so far, because that's what I was using at the time, but I don't see any problems making exporters for other modelers.
Importing it back into the modeler is a different story, but that is not really something I'm interested in. It's always best to keep your original artwork in a format native to the modeler, because it's virtually impossible to export ALL the data of the modeler and import it again. And that data is not interesting for the 3d engine either.

The 'fallback' I have is that every node in the file can be named. In my Java engine, I just import the name of the material, and then use a custom xml configuration which defines a material native to the Java engine, rather than trying to shoehorn 3dsmax' materials into Java somehow.
Posted on 2009-10-13 02:42:47 by Scali
Materials... can have multiple texture-layers for each channel, uv-maps, vertex-maps for coloring, blending between tex-layers, morph-targets, weightmaps, smoothing-groups; partial maps that cover multiple materials and parts; layers; parenting and pivots.
It's a far cry from .obj, and almost all studios do try to keep their in-modeler shading node-net close to the ingame one, afaik. There's got to be a reason :) .
Of course, there's no need to re-import the data into the modeler; scripts should be written to modify stuff there.
Posted on 2009-10-13 03:37:48 by Ultrano
I know what materials can do in a modeler. But until recently there was NO way you would use anything even remotely similar in a realtime renderer. Usually you'd just 'bake' most of you lighting/materials into one or two textures. After all, the modeler is also responsible for generating a lot of the content, not just storing it.
Most of this data is of no use in a realtime situation.

In fact, these days many studios have turned things around. Instead of trying to match the engine's shading to the modeler, they try to use the actual engine's shaders in the modeler. In that case, yes it makes a lot of sense to keep both trees as close as possible.

However, I don't see how any of that has anything to do with the file format itself. It's mostly a semantic matter of how you choose to model, and what you will then export to the file and import into the engine. My file format can support various approaches. You could store the whole shading tree with all its nodes and bits. But you could also choose to only export the bare necessities.
It all depends... My Java engine obviously cannot do the same things that my D3D11 engine can, at least, not in realtime. So I use simplified shading methods. That however does not mean that I have to completely remodel the artwork, let alone use a completely different fileformat. I just ignore some of the data, or pre-bake it into additional textures rather than solving everything in realtime. There's little point in using the actual shading nodes, since it cannot be evaluated quickly enough in Java. You'd want to use a pre-optimized fixed single-pass shading routine. As long as I can link those by material name (Material X in modeler == Material X in Java), that's fine. Writing a complete shader compiler that can compile 3dsmax shading trees into fixed, optimized single-pass shaders in Java would go WAY beyond the scope of the engine. Besides, I don't believe in that approach. Just like D3D Warp or SwiftShader... Sure, they can compile shaders to x86 code, and the generated code is pretty optimized for what it is and does... Problem is, it's never going to be anywhere near as fast as a simplified hand-crafted routine, which you cannot express in regular D3D shaders.

So I concentrate on the ACTUAL shading in the engine, and the modeler is mainly there to shape and animate the geometry. The actual shading doesn't have to be a realistic representation of how it will look in the engine. Once the geometry and animation are imported into the engine, the shading can be fine-tuned there, with a WYSIWYG approach.
In theory I could also create a wrapper around my engine so it can be loaded as a viewport plugin in 3dsmax, but that is not a priority for now.
Posted on 2009-10-13 03:59:50 by Scali
I've made a start with the BHM File Format project on SourceForge:
http://sourceforge.net/projects/bhmfileformat/
Posted on 2009-10-26 11:54:08 by Scali
On a slightly related note... I already discussed the Vista Platform Update here: http://www.asmcommunity.net/board/index.php?topic=29455.msg208695#msg208695
It is now out of beta, and available to everyone on Windows Update, so that means that Vista now has official DirectX 11 support. And it also works fine on non-English versions of Vista now.
Posted on 2009-10-28 06:20:09 by Scali
Since DX11 is officially released now, I've decided to compile a full release package of the DX9, DX10 and DX11 engines in 32-bit and 64-bit format:
http://bohemiq.scali.eu.org/Engine20092810.rar

I make use of D3D_FEATURE_LEVEL_9_1 or higher, so if you have Vista or Windows 7 and a DirectX 9 videocard, it will use the DirectX 11 API.
In the startup dialog you can see info on whether it runs 32-bit or 64-bit, which API version, and what hardware features.
Eg, on my system I see this:


So, 64-bit, using the DirectX 11 API, but my 8800GTS only supports DirectX 10 features.
Posted on 2009-10-28 16:05:21 by Scali
Since there apparently was a false positive of a worm in my archive, I've removed the launcher executable, and left the rest untouched:
http://bohemiq.scali.eu.org/Engine20090311.rar
This no longer triggered the antivirus software.

Rename the Engine32.bin and Engine64.bin to exe if you want to run them.

If you still want to use the launcher, it's open source, and you can get it from the CPUInfo project (http://cpuinfo.sf.net), see the Windows/Launcher3264 folder.

I did download the rar file from the server and inspected the Engine.exe in a disassembler. As I suspected, I couldn't find any code that shouldn't be there. The Launcher3264 code just looks suspect to certain antivirus software apparently. Well, it's open source, I have nothing to hide :)
Posted on 2009-11-05 03:59:56 by Scali
Hi,
Three suggestions, if I may:
1) The 'Adapter' label should say 'display' since it shows all display options.
2) The adapter combobox should say "display name @ adapter name" not the other way around, since we are selecting displays, not adapters.
3) IMHO, the 'Aspect ratio' option should be a combobox with selectable ARs derived from resolutions supported by selected display, and then it should filter the 'Resolution' combobox's items. I know it's a bit more work to do but many games today do it this way so the users are somewhat 'trained' to use it in this manner.

And 1 bugreport:
It doesn't start on my WinXP pro 64bit. Both the 64- and 32-bit versions say "Unable to load any of the supplied engine libraries!"
Posted on 2009-11-05 11:11:55 by ti_mo_n
It's a matter of perspective... This dialog was 'designed' in the DX8/9 era, when multimonitor wasn't really used yet... besides, those APIs actually DID enumerate multiple adapters.
Originally the idea was mostly for people with multiple adapters to be able to select the adapter they wanted to use.
Perhaps these days it's better to select a display.

Aspect ratio... I personally HATE games that only show resolutions for a single aspect ratio at a time. Usually I know what resolution I want, but I am not at all bothered about the aspect ratio (it isn't relevant really, as long as the engine autodetects it properly, so you don't get stretched images).
The idea here is that it tells you the aspect ratio of the resolution you selected, and if you want to adjust it, you can manually override it by specifying something else in the two edit boxes. Eg, I have a 16:9 widescreen connected, but I want to render in 1024x768... then normally I'd get stretching, because it assumes it's a 4:3 resolution. So I override it with 16:9 and I get good images again.
In other words, those fields are for the aspect ratio of your SCREEN, not of the resolution. By default it assumes that pixels are isomorphic, so a resolution like 800:600 is 4:3, so that's what it will put in the box. However, if you have a 16:9 screen connected, in 800:600 you will have anisomorphic pixels. So you can set 16:9 in the aspect ratio field to correct for that.

Speaking of filtering by aspect ratio... yesterday I tried to play Shattered Horizon... it didn't enumerate my native resolution of 1280x1024. Must be because it's the only resolution that's not a standard 4:3 or 16:9/16:10, but 5:4. All the more reason to not want filtering like that. I want to see ALL resolutions.

As for your bugreport... I think your computer isn't up to date. Try installing the August 2009 DirectX redistributable, and the VC++ 2008 SP1 redistributable. I know for a fact that it runs on XP Pro x64, because I have two systems here with XP x64 installed, and I tried it on both (my own Core2 Duo with 8800GTS, and a Pentium 4 with a Radeon X1900XTX).
Posted on 2009-11-05 11:23:45 by Scali
(...)In other words, those fields are for the aspect ratio of your SCREEN, not of the resolution

Oh, OK. Now I can see how it works. Previously I was referring to the dialog box itself.

As for your bugreport... I think your computer isn't up to date. Try installing the August 2009 DirectX redistributable, and the VC++ 2008 SP1 redistributable. I know for a fact that it runs on XP Pro x64, because I have two systems here with XP x64 installed, and I tried it on both (my own Core2 Duo with 8800GTS, and a Pentium 4 with a Radeon X1900XTX).

OK, it runs now. Installing DXaug2009 solved the problem.
But why do you require DX August 2009 while the code uses functions available in d3dx9_33 (April 2007)? (it even runs with d3dx9_32 , although the texturing is incorrect). Is there any specific reason for requiring the "_42" version or you just want to use the latest one?
Posted on 2009-11-05 23:55:12 by ti_mo_n

OK, it runs now. Installing DXaug2009 solved the problem.
But why do you require DX August 2009 while the code uses functions available in d3dx9_33 (April 2007)? (it even runs with d3dx9_32 , although the texturing is incorrect). Is there any specific reason for requiring the "_42" version or you just want to use the latest one?


Short answer: DirectX 11 requires it.
Long answer: I like to always use the latest SDK and runtime, because this contains the latest bugfixes and optimizations. The biggest reason is the shader compiler. Microsoft recommends using the latest D3DX one, since it gets updated every few months, adding new features and improving the optimization algorithms.
Currently I use a single set of shaders, which compile in D3D9, D3D10 and D3D11. I don't think they'd work with older D3D9 compilers at all, since the syntax has changed a lot over the years. And even if I changed the code to make it compile, it would probably be considerably less optimal than when using the latest compiler, and it would break compatibility with D3D10 and D3D11.

I could stick with the August 2009 SDK for now, since it does everything I need for D3D9, D3D10 and D3D11, but I will probably update anyway, whenever a new SDK is released. Not sure when that will be, the previous SDK releases were mainly aimed at beta and final D3D11 support. Microsoft probably doesn't have any major updates to the SDK scheduled for a while.
Posted on 2009-11-06 04:32:28 by Scali
I've done some more work on the Dependency Walker routines... I got it to open a Dll, walk down all its imports, and check in the respective Dll's export tables. What surprised me though was that some functions could not be found. Eg, in user32.dll, it couldn't find SetClassLongPtrW. A relatively new function.
When I did a manual search for the string in the file, it turns out that the code was right: There is no mention of the string "SetClassLongPtr" anywhere in the file.
However, when using the real Dependency Walker (depends.exe), it finds those exports without a problem. So they do exist.

That puzzled me for a moment... I suppose I'll have to read the COFF specs again, perhaps there are other ways to store exported function names... perhaps unicode?
If anyone has some pointers on this, let me know.
Posted on 2009-11-09 02:40:51 by Scali
Let me guess, you're ending up with the 32bit version of USER32.DLL?

C:\Users\f0dder>strings c:\Windows\System32\user32.dll |grep SetClassLongPtr
SetClassLongPtrA
SetClassLongPtrW

C:\Users\f0dder>strings c:\Windows\Syswow64\user32.dll |grep SetClassLongPtr

C:\Users\f0dder>
Posted on 2009-11-09 02:45:58 by f0dder
No, I tried both, to be sure... In fact, I've even tried on a 32-bit XP system... "SetClassLongPtr" just isn't in user32.dll...?
32-bit or 64-bit shouldn't matter, the function should be in both versions, right?

Edit: Now I'm confused... Did I really test both?
Looking at winuser.h, SetClassLongPtr is #defined to SetClassLong in the case of Win32. So yes, the 32-bit dll won't have that function...
But I was looking at the 64-bit one... the one in Windows\system32... weird. Or does Windows silently open the 32-bit one in Windows\syswow64 anyway, just because I'm running a 32-bit process (I'm just doing CreateFile(), not an actual LoadLibrary() or anything)? I've looked through the PE Coff docs again, and don't find any mention of alternative ways of storing import/export name tables... Perhaps Windows just substitutes \syswow64 with \system32 automatically for 32-bit processes, so even stuff other than LoadLibrary() will always look at the 32-bit stuff.
That would also explain why I couldn't see it with a text editor? Because they were also 32-bit processes, and would also silently be redirected to the 32-bit file... Is your strings/grep stuff 64-bit by any chance?

Well, at least I've gotten one step further: in 32-bit the "SetClassLongPtr" just doesn't exist. I bet the other few functions that came up as 'missing' are also 64-bit only, and are #defined to something else in 32-bit.
I'll recompile the stuff as 64-bit and see what happens. If it magically works then, that would prove the theory that 32-bit processes will automatically look at Windows\syswow64 even when you think you're in \system32.
That would also mean that it's slightly more complicated to handle 32-bit and 64-bit than I thought. I don't just need to look in different paths, I actually need to be either a 32-bit or a 64-bit process (or always use a 64-bit process, assuming that can look in syswow64 aswell, and solve 32-bit dependencies).

Would also explain why there's a 64-bit version of Dependency Walker. Not because it would be faster or because they need fancy 64-bit arithmetic and 4+ GB memory usage... but simply because you're not looking at the proper DLLs otherwise. Ofcourse I tested with the 64-bit version, didn't occur to me to also try the 32-bit one. I bet it can't find those exports either... unless they did something smart to work around it :)

Edit 2:
I found the 'smart' way, it seems... There is a function Wow64DisableWow64FsRedirection();
Posted on 2009-11-09 03:34:06 by Scali
Well, seems like you answered your own questions - welcome to the wonderful world of silent redirection... pretty leim that MS didn't just throw the 64bit stuff in a "system64" folder and let the system32 be as-is, imho. Heck, using blabla64.DLL would also have been nice :)
Posted on 2009-11-09 06:20:20 by f0dder
Oh yea... For some reason I assumed that the redirection would only be done inside API calls, like LoadLibrary().
I agree that system64 would have been a much better solution. The explanation that Microsoft gives is that system32 is hardcoded in so many applications that they thought changing it would be more trouble than it's worth. Makes me wonder... You need to recompile to 64-bit anyway... might aswell do a search&replace for system32->system64 while you're at it. What applications are we talking about anyway? It's not like most applications have any business poking around in system32...

At any rate I think it's a VERY good idea to check the architecture that a Dll is compiled for. Dependency Walker warns about mixing x86 and x64 modules aswell... If I had a check like that in the first place, I would have figured out what was going on earlier.
64-bit applications have a "PE32+" header anyway, so I do need to handle them separately.

I'd like your opinion on something though:
With Wow64DisableWow64FsRedirection() and proper detection of the architecture of each model, I suppose I could create a 32-bit executable that will work for 64-bit modules aswell.
Should I write code that will work in both 32-bit and 64-bit? Or should I write it in such a way that when compiled for 32-bit it will work only for 32-bit modules, and in 64-bit only 64-bit modules?
My preference is with the former.
Posted on 2009-11-09 07:04:09 by Scali
Okay, getting somewhere...
It does a recursive scan, and currently it just outputs whatever problems on stdout.
It works quite well, quite fast too.
I've found problems with one dll so far, which for some reason doesn't seem to have the ordinals I'm looking for. Not sure what's going on there. Oh well, not too important right now.
I've also added a simple check to see if each imported DLL matches the machine that the importing module is using.

Done:
- Do a complete scan of all the DLL paths according to Windows rules (currently I only check \Windows\system32, good enough for proof-of-concept, but not quite correct).
- Simplify some of the PE32/PE32+ handling code.... There's only a few fields that actually matter, for the most part, the headers are equal.
- Create some kind of datastructure in which the library can log its messages, so the calling application can easily read them and display them to the user.
- Figure out why that one DLL has problems with its ordinals.
- Create some kind of header/lib to handle the dynamic loading of Wow64-related functions.
- Optimize the search routines to use binary search.

Some things to fix/improve:
- Support special trickery like SxS and special SafeDll-whateveritwascalled registry setting.
Posted on 2009-11-09 16:39:13 by Scali
While I tie up the last loose ends of the dependency walker thing, I think it's time to also start looking forward to the next steps in development.

I think it will go something like this:
- Extend setup dialog (or add about box or such) and include some basic data from CPUInfo library (gratuitous promotion of CPUInfo and BHMFile).
- BHMFile can load into engine now, but skinning needs to be re-implemented to display it properly.
- Create a lightweight OpenGL engine that loads BHMFile and uses CPUInfo, as a kind of open source multiplatform benchmark? (more gratuitous promotion of CPUInfo and BHMFile).

Done:
- Implement depedency walker in the engine launch code (the part that determines D3D9/10/11 compatibility).
Posted on 2009-11-10 07:07:45 by Scali