id Tech Forums

id Tech 4 (Doom3/Prey/Q4) => id Tech 4 Engine Coding => Topic started by: bitterman on February 09, 2016, 07:19:20 AM

Title: Migration from D3 to BFG (simple way to track game resources)
Post by: bitterman on February 09, 2016, 07:19:20 AM
Is there a simple way to track which game resources are called by the idTech4 engine (especially at startup)?

May be some kind of logs/console commands?

I'm trying now to solve the problem "a minimal set of own resources" for BFG (without any vanilla code - for great GPL'tice).

P.S. A minimal assets for D3 give me a crash into D3BFG/RBDOOM.
Title: Re: A simple way to track game resources?
Post by: bitterman on February 10, 2016, 07:32:55 AM
Seems like "+set logFile 1" is working at the moment.

Now I see a strange things with r_showTris.
Hmm, seems like they changed a relative part of render code...

related thread about D3:
http://idtechforums.fuzzylogicinc.com/index.php?topic=236.msg2070#msg2070
Title: Re: A simple way to track game resources?
Post by: The Happy Friar on February 10, 2016, 09:54:44 AM
motorsep's gotten D3BFG code running with non-BFG assets, maybe he'll post the min required stuff.  :)

I do know the BFG will crash much easier then stock D3 in relation to assets & it won't put out a useful error message when related to assets part of the time.
Title: Re: A simple way to track game resources?
Post by: motorsep on February 10, 2016, 03:40:10 PM
All I did was ran engine in debug with no assets and watched what crashed it, one by one adding missing assets.

Somewhere in the framework/ folder (common.cpp or something else), there is basically a list of assets that get loaded on start up. That's something to start with.
Title: Migration from D3 to BFG (simple way to track game resources)
Post by: bitterman on February 10, 2016, 09:26:25 PM
Yes, thanks.

Well, I've got a totally black window with working console and huge wall of warnings.

It's works even without any swf stuff (main menu, hud, pause etc.).

But now a problem is to see anything ingame.

There is a similar problem:
http://idtechforums.fuzzylogicinc.com/index.php?topic=335.msg3504#msg3504

Any map which loaded by map/testmap is still black.

This is really scare:)


P.S. There is a problem with r_showTris in original BFG:



Title: Re: Migration from D3 to BFG (simple way to track game resources)
Post by: The Happy Friar on February 10, 2016, 10:33:05 PM
Are your assets binarized?
Title: Re: Migration from D3 to BFG (simple way to track game resources)
Post by: bitterman on February 10, 2016, 11:41:42 PM
No.

Do you think the problem is in that?
Title: Re: Migration from D3 to BFG (simple way to track game resources)
Post by: motorsep on February 11, 2016, 09:20:18 AM
Quote from: bitterman on February 10, 2016, 11:41:42 PM
No.

Do you think the problem is in that?

Your assets should be binarized automatically on map load. If they don't get binarized, something is wrong.

The list of assets in C++ code gets binarized first. Those are a must have assets. Then you load a box map and see the warnings. Better run it with +set com_smp 0 to turn off threading to see _all_ prints in the console. Add each asset that is being asked for in those warnings/errors.
Title: Re: Migration from D3 to BFG (simple way to track game resources)
Post by: bitterman on February 12, 2016, 12:52:06 AM
Let's refine:

1. Get the source code (D3BFG/RBDOOM) and compile it (as debug).

2. Create 'base' folder into appropriate folder (eg. Win32/Debug/base).

3. Put into 'base' our own folders (af, def, script, maps, newfonts etc.) and default.cfg.
At this moment custom game resources are not binarized, map created in D3 Editor/Radiant, and original game resources are not present.

4. Run .exe.

After that I have a black app window with working console.

5. Type 'map game/testmap.map' (also try 'dmap game/testmap.map').

After that I have a black app window. Console still working.

QuoteYour assets should be binarized automatically on map load. If they don't get binarized, something is wrong.

This moment is not clear.
What about fs_bulidresources, buildGame, fs_loadPriority etc.?
No need to use it in this case?

Will testmap.resources be created automatically (via 'map' command) in maps/game/?
Title: Re: Migration from D3 to BFG (simple way to track game resources)
Post by: The Happy Friar on February 12, 2016, 07:28:50 AM
.resources is the id tech 5 "pak" file, that's done manually (I think).  The binarizied stuff goes in to to the base/generated folder & follows the folder structure of the assets.  IE textures/wall/wall.tga should be in generated/textures/wall/wall.bsomething.  Something is different based on what's binarized.
Title: Re: Migration from D3 to BFG (simple way to track game resources)
Post by: motorsep on February 12, 2016, 10:24:18 AM
Quote from: bitterman on February 12, 2016, 12:52:06 AM

This moment is not clear.
What about fs_bulidresources, buildGame, fs_loadPriority etc.?
No need to use it in this case?

No, those are to make .resources (pk4-like files for idTech 5)

Quote from: bitterman on February 12, 2016, 12:52:06 AM
Will testmap.resources be created automatically (via 'map' command) in maps/game/?

No, it won't. You don't need .resources at all.

What engine should do is load all assets pertaining to the map when you loading that map, binarize them and save them into base/generated/

Btw, you don't need to put base/ next to you binary in Win32/Debug/

Use +set fs_basepath X:\games\my_doom3bfg_mod\ for your debug cmd line in MSVC to point debug .exe to your base\  (base\ would be located in X:\games\my_doom3bfg_mod\ )

X:\games\my_doom3bfg_mod\ can be anything/anywhere
Title: Re: Migration from D3 to BFG (simple way to track game resources)
Post by: bitterman on February 13, 2016, 04:43:26 AM
Sorry, guys, my fault.
.resources like .pk4.
binary like .bimage.

Yes, and thanks for basepath.

update:

Title: Re: Migration from D3 to BFG (simple way to track game resources)
Post by: The Happy Friar on February 13, 2016, 08:00:04 AM
You could enable the tris view by default & call it an "Art style".  :)
Title: Re: Migration from D3 to BFG (simple way to track game resources)
Post by: bitterman on February 14, 2016, 10:18:43 AM
Showtris always remind me Braben's Elite-like style.

Moreover it would be solve texture problems :-)

update:

Some troubles with new fonts. As I see tool for D3 fonts is not work at now.

What is idFont for Rage? Can't find any link. It's from ID ftp?

Update:

Ok, it's from rage toolkit.
Is idfont.exe from this package available?
Title: Re: Migration from D3 to BFG (simple way to track game resources)
Post by: bitterman on February 17, 2016, 12:38:49 AM
const char * DEFAULT_FONT = "Arial_Narrow";

static const float old_scale2 = 0.6f;
static const float old_scale1 = 0.3f;


struct glyphInfo_t {
byte width; // width of glyph in pixels
byte height; // height of glyph in pixels
char top; // distance in pixels from the base line to the top of the glyph
char left; // distance in pixels from the pen to the left edge of the glyph
byte xSkip; // x adjustment after rendering this glyph
uint16 s; // x offset in image where glyph starts (in pixels)
uint16 t; // y offset in image where glyph starts (in pixels)
};


Which of these values are used by default in BFG-version?

48.dat - is it 48x48px per symbol?

What size of fontimage.tga is needed? It is power of two texture?
Title: Re: Migration from D3 to BFG (simple way to track game resources)
Post by: motorsep on February 17, 2016, 07:38:53 AM
RAGE SDK has font tool that generates .dat and .tga from ttf fonts. BFG has exactly the same font system as RAGE.
Title: Re: Migration from D3 to BFG (simple way to track game resources)
Post by: bitterman on February 17, 2016, 09:21:54 PM
Quote from: motorsep on February 17, 2016, 07:38:53 AM
RAGE SDK has font tool that generates .dat and .tga from ttf fonts. BFG has exactly the same font system as RAGE.

Yes, just a small thing the size of about 35 Gb (and need to buy a game?).

Thanks, motorsep, but let's try something else :)

Where should I place 48.tga to get the 48_...bimage?
And what about alpha-channel in .tga?

Thanks.
Title: Re: Migration from D3 to BFG (simple way to track game resources)
Post by: motorsep on February 17, 2016, 09:41:48 PM
Quote from: bitterman on February 17, 2016, 09:21:54 PM
Yes, just a small thing the size of about 35 Gb (and need to buy a game?).

Sure, it's probably on sale for $5 or going to be on sale for $5.

Do you have Doom 3 BFG (it was on sale many times already too)?
Title: Re: Migration from D3 to BFG (simple way to track game resources)
Post by: bitterman on February 17, 2016, 09:52:58 PM
Wow, it was too fast! :))

Yes, I have D3BFG from Steam and D3 in jewel.

35 Gb it's too much anyway.

Now I have some .dat but when I try to replace it via fs_game it's works incorrect.

I think it's because .dat is custom and .tga is still original:

basedev\generated\images\newfonts\arial_narrow\48#__0400.bimage

Now I looking for my fontimage.tga -> 48#__0400.bimage.
Title: Re: Migration from D3 to BFG (simple way to track game resources)
Post by: motorsep on February 17, 2016, 10:58:38 PM
.dat has glyphs' info and their location on the .tga canvas. So yeah, they need to match. Same deal as in old Doom 3, except in idTech 5 utf-8 is supported and everything is on one .tga canvas.
Title: Re: Migration from D3 to BFG (simple way to track game resources)
Post by: bitterman on February 17, 2016, 11:20:35 PM
/*
==============================
idFont::LoadFont
==============================
*/
bool idFont::LoadFont() {
idStr fontName = va( "newfonts/%s/48.dat", GetName() );
idFile * fd = fileSystem->OpenFileRead( fontName );

        .....

idStr fontTextureName = fontName;
>>> fontTextureName.SetFileExtension( "tga" ); <<<

fontInfo->material = declManager->FindMaterial( fontTextureName );
fontInfo->material->SetSort( SS_GUI );


Hmm, ".tga"? Not ".bimage"?

And why "FindMaterial"? I see no decls for fonts in .mtr.

idStr fontTextureName = fontName ="newfonts/%s/48.tga"?

FindMaterial( "newfonts/%s/48.tga" )?

update:

related link (about old fonts and q3font):
http://wiki.thedarkmod.com/index.php?title=Font_Conversion_%26_Repair

As I think q3font is useless for new BFG fonts because .dat format was revised.

Is there a simple way to browse (e.g. after fix) .bimage files in an external viewer?
Title: Re: Migration from D3 to BFG (simple way to track game resources)
Post by: bitterman on February 18, 2016, 07:39:24 AM
motorsep, can you test this one on D3BFG/RB, please?
Title: Re: Migration from D3 to BFG (simple way to track game resources)
Post by: motorsep on February 18, 2016, 09:38:42 AM
Engine makes .bimage from TGAs. You don't even need to be bothered with base/generated/ folder because engine has it all done automatically. All you need to worry about is getting correct source assets (assets in the formats old Doom 3 uses).

Is there any good anonymous file sharing service where someone maybe uploaded RAGE font tool ? ;)
Title: Re: Migration from D3 to BFG (simple way to track game resources)
Post by: bitterman on February 18, 2016, 10:34:49 AM
It's funny to use source, gimp, blender, other GPL-stuff and this uploaded tool.

But it's interesting how it looks. Maybe someone have an screenshots with idfont.exe.

update:

It's close.
Title: Re: Migration from D3 to BFG (simple way to track game resources)
Post by: motorsep on February 18, 2016, 10:45:59 AM
Quote from: bitterman on February 18, 2016, 10:34:49 AM
It's funny to use source, gimp, blender, other GPL-stuff and this uploaded tool.

Not an issue when using it for free modding.

Quote from: bitterman on February 18, 2016, 10:34:49 AM
But it's interesting how it looks. Maybe someone have an screenshots with idfont.exe.

Cmd line tool, like many small utils that ever came from idSoftware.
Title: Re: Migration from D3 to BFG (simple way to track game resources)
Post by: bitterman on February 18, 2016, 10:55:10 AM
Look at screen above :-)
Title: Re: Migration from D3 to BFG (simple way to track game resources)
Post by: bitterman on February 18, 2016, 10:20:57 PM
Few notes about changes between D3 and D3BFG:

- work with game resources (loading order, binarize, packs) - critical;

- guis: a set of swf menus (shell, intro, pause, dialog etc.) and swf hud - critical;

- fonts: new .dat format - critical;

- added few new materials whiсh hardcoded by engine (eg. for ps3, xbox360) - non critical;

- scripts: added weapon_flashligth & weapon_flashligth_new - non critical;

The list is not complete.
Title: Re: Migration from D3 to BFG (simple way to track game resources)
Post by: bitterman on February 24, 2016, 12:19:18 AM
I think the most difficult SWF is dialog.bswf. An impressive Action Script code.

Not sure about D3 but in BFG a GDM (game dialog messages) is a complex layer of interaction between C++ code and SWF (GUI).

And it uses SetGlobal/GetGlobal (swf variables as I think).

That theoretically allows to  jump on any (not hard-coded) menu handlers. E.g. not only CAMPAIGN -> NEW GAME -> GAME TYPE -> ... but in any sequence.

****************update:

After extract resources I see in /basedev/renderprogs next folders:

cgb (Cg NVIDIA?)
gl (with old or revised D3 .vfp files)
glsl
hlsl

Questions:

1. In what format I must create own shaders for BFG?
Anybody got an working GLSL/HLSL shaders?

2. How to use non-ARB shaders?
I can't see any new vertexProgram/fragmentProgram (glsl,hlsl,cg) into .mtr (except .vfp).

*******************

Hmm, as I see in RenderProgs_GLSL.cpp .cg (and perhaps .hlsl) was converted by engine.

GLuint idRenderProgManager::LoadGLSLShader( GLenum target, const char * name, idList<int> & uniforms ) {

idStr inFile;
idStr outFileHLSL;
idStr outFileGLSL;
idStr outFileUniforms;
inFile.Format( "renderprogs\\%s", name );
inFile.StripFileExtension();
outFileHLSL.Format( "renderprogs\\glsl\\%s", name );
outFileHLSL.StripFileExtension();
outFileGLSL.Format( "renderprogs\\glsl\\%s", name );
outFileGLSL.StripFileExtension();
outFileUniforms.Format( "renderprogs\\glsl\\%s", name );
outFileUniforms.StripFileExtension();
if ( target == GL_FRAGMENT_SHADER ) {
inFile += ".pixel";
outFileHLSL += "_fragment.hlsl";
outFileGLSL += "_fragment.glsl";
outFileUniforms += "_fragment.uniforms";
} else {
inFile += ".vertex";
outFileHLSL += "_vertex.hlsl";
outFileGLSL += "_vertex.glsl";
outFileUniforms += "_vertex.uniforms";
}


Still not clear.

Title: Re: Migration from D3 to BFG (simple way to track game resources)
Post by: bitterman on February 26, 2016, 01:31:30 AM
What does it mean?

idLib::Printf( "While linking GLSL program %d with vertexShader %s and fragmentShader %s\n"

Why GLSL are linked with ARB?

Is only GLSL-progs not enough to work with BFG?

And if game resources are still covered by EULA then why shader asset (base/renderprogs/) come with GPL code?

***************** update *****************

There is related info about shaders "RBDoom 3 engine (shader programs)":

http://idtechforums.fuzzylogicinc.com/index.php?topic=204.msg1745#msg1745 (http://idtechforums.fuzzylogicinc.com/index.php?topic=204.msg1745#msg1745)