id Tech Forums

id Tech 4 (Doom3/Prey/Q4) => id Tech 4 Engine Coding => Topic started by: trebor on December 30, 2015, 07:29:50 AM

Title: RBDOOM-3-BFG 1.1.0 preview
Post by: trebor on December 30, 2015, 07:29:50 AM
I used the christmas holidays to add a bunch of renderer features to DOOM 3 BFG.

Here is the changelog:

- True 64 bit HDR lighting with adaptive filmic tone mapping

- Enhanced Subpixel Morphological Antialiasing
   For more information see "Anti-Aliasing Methods in CryENGINE 3" and the docs at http://www.iryoku.com/smaa/

- Filmic post process effects like Technicolor color grading and film grain

- Fixed issues with Mesa drivers and allowed them to use shadow mapping

- Defaulted fs_resourceLoadPriority to 0 so it is not necessary anymore to specify when running a modification

- Additional ambient render pass to make the game less dark similar to the Quake 4 r_forceAmbient technique << -- this was really necessary IMHO and will be improved with automatic placed cubemaps aka IBL light probes soon

I only imlemented SMAA 1x so far. It was necessary because real HDR with 16 bit floats and MSAA didn't work well with all those post process effects like heat haze.
I went into many driver problems using MSAA.
However the SMAA code is a very good starting point for the other modes like SMAA T2x which is essentially what UE4 does.


You can download it from https://dl.dropboxusercontent.com/u/22890478/RBDOOM-3-BFG/RBDOOM-3-BFG-1.1.0-preview1-win32-20151224-git-090fbf0.7z


Screenshots how it should look like:

RBDOOM-3-BFG 1.1.0 lighting preview Deltalabs 1
http://imgur.com/a/E7MZ9

RBDOOM-3-BFG 1.1.0 lighting preview Phobos 1
http://imgur.com/a/VG3KO

Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: motorsep on December 30, 2015, 09:22:30 AM
Nice one! That makes me want to play Lost Missions after all :)
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: motorsep on December 30, 2015, 11:37:07 AM
- Additional ambient render pass to make the game less dark similar to the Quake 4 r_forceAmbient technique << -- this was really necessary IMHO and will be improved with automatic placed cubemaps aka IBL light probes soon

How do you plan on doing this efficiently ?

I have been poking at Unity and probes there have to be manually placed on the shadow boundaries. Someone made a tool to place light probes on a grid, a la Quake lightgrid, but Unity guys said it's too inefficient and eats up too much of resources.
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: trebor on December 31, 2015, 07:23:09 AM
That should be easy with id Tech 4. Just start at the center of each BSP area and then move to the next area portal in a certain distance.
The BSP is still kinda useful. However I already implemented some basic code. You will also be able to manually place and move env probes ingame.
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: motorsep on December 31, 2015, 09:16:29 AM
That should be easy with id Tech 4. Just start at the center of each BSP area and then move to the next area portal in a certain distance.
The BSP is still kinda useful. However I already implemented some basic code. You will also be able to manually place and move env probes ingame.

I see. I was thinking of maybe utilizing .proc file data (static shadow volumes). So probes don't have to be located on the shadow's boundary like in Unity ?
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: raynorpat on December 31, 2015, 12:59:22 PM
Sounds more like Source's env_cubemap then Unity.
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: trebor on December 31, 2015, 03:17:44 PM
http://gdcvault.com/play/1015312/Light-Probe-Interpolation-Using-Tetrahedral

Another interesting source is Multi-Scale Global Illumination in Quantum Break 
http://advances.realtimerendering.com/s2015/   
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: romulus_ut3 on January 01, 2016, 12:36:16 AM
And as always, the performance is choppy on AMD cards. :(
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: motorsep on January 01, 2016, 03:59:32 AM
And as always, the performance is choppy on AMD cards. :(

Did you update driver ? I recall with 15.11 (or whatever it was the latest) issues with idTech 5 based games and AMD were gone!
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: trebor on January 01, 2016, 06:32:25 AM
I admit that my HDR implemention is not optimal no matter what card you are using.

See https://github.com/RobertBeckebans/RBDOOM-3-BFG/issues/278

This is just a preview and not the final 1.1.0 release.
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: argoon on January 01, 2016, 10:24:30 PM
Fantastic work trebor,  i just want to ask one thing, how can i tweak the ambient light?  I just think the game became to bright now, yes finally seeing detail on the shadows is very nice and have the normal maps also work on them is fantastic, but totally removing the darkness that Doom3 add is just not good to me, the game imo loses the sense of dread and becomes just a game of shooting monsters in lite corridors, because of this the lower amount of enemies encounters really become aparente now.

I also seem to have some banding problems in some lite surfaces.

My GPU - AMD R9 270X using the new 15.11 drivers

BTW didn't vanilla idtech 4 removed polys in shadow? afaik because they were pitch black anyway.  If now this is not true anymore that means the game lost one of its performance optimization tricks? And so we should expect worse performance?


edit - I should learn to read i didn't realized you literally used the r_forceAmbient  cmd, saw it on the console, it was at 0.2 i brought it down to 0.05, bang the better of both worlds, darker shadows but still able to see detail on them. :D

Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: Phrozo on January 02, 2016, 01:58:52 AM
Nice, keep up the good work.
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: trebor on January 02, 2016, 07:28:27 AM
Fantastic work trebor,  i just want to ask one thing, how can i tweak the ambient light?  I just think the game became to bright now, yes finally seeing detail on the shadows is very nice and have the normal maps also work on them is fantastic, but totally removing the darkness that Doom3 add is just not good to me, the game imo loses the sense of dread and becomes just a game of shooting monsters in lite corridors, because of this the lower amount of enemies encounters really become aparente now.

I also seem to have some banding problems in some lite surfaces.

My GPU - AMD R9 270X using the new 15.11 drivers

BTW didn't vanilla idtech 4 removed polys in shadow? afaik because they were pitch black anyway.  If now this is not true anymore that means the game lost one of its performance optimization tricks? And so we should expect worse performance?


edit - I should learn to read i didn't realized you literally used the r_forceAmbient  cmd, saw it on the console, it was at 0.2 i brought it down to 0.05, bang the better of both worlds, darker shadows but still able to see detail on them. :D

r_useHalfLambertLighting 0 reenables the culling of triangles that are not facing into the light's direction. However it looks crappy with Half-Lambert lighting and I think in general that traditional Lambert N*L lighting in Doom 3 just sucks.
Half-lambert fakes a bit of the amount of indirect lighting which Doom 3 is completely missing. I think the lighting in Doom 3 is completely broken after learning so much about PBR.

However the old lighting system is the result of the hardware limitations in 2004 and it was designed to even work on a Geforce 3.
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: argoon on January 02, 2016, 09:26:30 AM
r_useHalfLambertLighting 0 reenables the culling of triangles that are not facing into the light's direction. However it looks crappy with Half-Lambert lighting and I think in general that traditional Lambert N*L lighting in Doom 3 just sucks.
Half-lambert fakes a bit of the amount of indirect lighting which Doom 3 is completely missing. I think the lighting in Doom 3 is completely broken after learning so much about PBR.

However the old lighting system is the result of the hardware limitations in 2004 and it was designed to even work on a Geforce 3.

Thanks for the suggestion, but i concur with you, i was just curious if this new lighting affected that particular optimization, also i don't think anyone playing with this is using a old dx8 or dx9 GPU to need the extra performance from the triangle culling.

Knowing that you are about to implemente PBR into idtech 4.5 makes me smile, i can't wait to see how it will look on Doom3 (after some material modifications i'm sure). :)

I would not call the Doom 3 lighting broken, like you said it was made even before GPU's were fully programmable so is impressive how much juice was John Carmack able to squeeze from that hardware and make a real time lighting system that even today looks good. 
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: Manson on January 03, 2016, 07:53:53 AM
Everything i got now is black screen on linux.

Ok. Turning HDR off gives me image back.:) Turning r_antiAliasing and r_useFilmicPostProcessEffects on/off i didn't see any difference at all.

My be my graphics card is too old.
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: The Happy Friar on January 03, 2016, 01:29:18 PM
Try a "reloadengine" or exiting the game & loading it back up.  That's the only way I've found to reliably make sure video settings (besides the basics included in the stock game) read the changes.

Might not also hurt to rename the "doom 3.exe" to something else (I use "mickeymouse.exe").
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: Chromatose on January 05, 2016, 07:28:40 AM
Oh man RBD3BFG has come on hugely since I last tried.  Awesome stuff!
I've just been browsing the issue tracker on GitHub and I've noticed you have plans to support PBR-conformant materials, which is super exciting.  One question I did have is regarding the decision to eventually utilise displacement-mapping/tessellation.  I totally respect that the parallax-based method that Doom III modders have used up until now is a bit of a nightmare in terms of shader complexity, so please don't think I'm just having a moan about not implementing it; but the examples of tessellation-based displacement I've seen have absolutely butchered performance and looked really of weird and ugly.  Firing up Unigine Heaven and watching rock faces warp around as they adjusted complexity was genuinely a bit nauseating haha.

It could just be that I'm looking at bad examples, mind you.  I suppose I was wondering if it was something you'd toyed with yet, and if so, do you reckon it could work to the same extent that, for instance, modded DIII currently does parallax?
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: wintch on January 05, 2016, 01:44:47 PM
Might not also hurt to rename the "doom 3.exe" to something else (I use "mickeymouse.exe").

Take into account that the binary naming is used by video card drivers in order to set specific optimizations and/or flags to get the game working.
Carmack mentioned once, that nvidia drivers were using quake1 filename to limit openGL extensions and make the game work, otherwise he would need to make a patch for it. I can't find the link now  :'(, but if something strange happens when renaming doom3.exe, you know what to do
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: argoon on January 05, 2016, 05:37:39 PM
Oh man RBD3BFG has come on hugely since I last tried.  Awesome stuff!
I've just been browsing the issue tracker on GitHub and I've noticed you have plans to support PBR-conformant materials, which is super exciting.  One question I did have is regarding the decision to eventually utilise displacement-mapping/tessellation.  I totally respect that the parallax-based method that Doom III modders have used up until now is a bit of a nightmare in terms of shader complexity, so please don't think I'm just having a moan about not implementing it; but the examples of tessellation-based displacement I've seen have absolutely butchered performance and looked really of weird and ugly.  Firing up Unigine Heaven and watching rock faces warp around as they adjusted complexity was genuinely a bit nauseating haha.

It could just be that I'm looking at bad examples, mind you.  I suppose I was wondering if it was something you'd toyed with yet, and if so, do you reckon it could work to the same extent that, for instance, modded DIII currently does parallax?

Tesselation is faster than POM (parallax occlusion mapping) or Horizon Mapping, and looks better especially at geometry corners, i know you add a bad experience with tesselation , like i did, and this i attribute to Nvidia sponsored games (almost all AAA games), they just use too much tessellation or extreme tessellation based effects like the Nvidia hair tech, why they do this, because their GPU's (high end GPU's) are better at it than AMD, but very well optimized tessellation (16x or 32x is sufficient, Nvidia uses 64x all the time) and applied only where it makes sense (on Crysis 2 they used 64x even on flat surfaces), makes no major visual diference to extreme tessellation and is faster then the same effect using POM, this is so for the simple reason that GPU's are good at rendering many millions of triangles, even smaller ones, but not so good at ray tracing that is something POM is based on, this is why tesselation is a major selling point in dx11+ and not POM or Horizon mapping.

p.s - Forgot to say Unigine Heaven is not a good example of optimal tessellation usage, is a benchmark so has tessellation tossed in at random just to stress PC's, there's surfaces there that really didn't needed tessellation and a normal map would suffice, for example the ropes and the rooftops and there's others.   
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: nbohr1more on January 05, 2016, 11:31:09 PM
Both Parallax and Tessellation can be done badly.

The problem with both effects is that people want to see "the effect" so it's often implemented in a very garish way
where the geometries are exaggerated from a subtle normal map.

Good Tessellation (or Parallax) is a subtle effect and will give a scene an almost intangible improvement and (in the case of Tessellation)
you'll also get a performance boost due to more efficient LOD management.

The reason that Carmack gave for not including Truform (DX8 Tessellation)?

"The Meshes would have silhouettes that didn't match the shadows."

And lo, most Doom 3 modders make "Shadow Meshes" which cause that mismatch issue anyway for the sake of performance (and I'll bet that vanilla Doom 3 has a few too).

Honestly, I believe he was planning on having Tessellation in Doom 3 because it was demoed along with early Radeon demos.
The blocky heads were perfect candidates to exaggerate the benefits.
He probably stopped working on it or yanked it after ATI employees leaked his E3 Demo.

The only thing I wonder about is whether it would be too expensive to adjust normal map projections in the shader to accommodate the new geometry without distortion.
Of course that's only a concern if you bother to render normal maps on top of tessellated geometry rather than just using tessellation in place of normal maps.
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: The Happy Friar on January 06, 2016, 06:57:32 AM
Take into account that the binary naming is used by video card drivers in order to set specific optimizations and/or flags to get the game working.
Carmack mentioned once, that nvidia drivers were using quake1 filename to limit openGL extensions and make the game work, otherwise he would need to make a patch for it. I can't find the link now  :'( , but if something strange happens when renaming doom3.exe, you know what to do

That's why I've been doing it, ATI/AMD drivers have some hacks to "fix" issues with D3 that causes other issues with HDR, bloom, & what not.
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: Chromatose on January 06, 2016, 10:02:48 AM
The only thing I wonder about is whether it would be too expensive to adjust normal map projections in the shader to accommodate the new geometry without distortion.
Of course that's only a concern if you bother to render normal maps on top of tessellated geometry rather than just using tessellation in place of normal maps.

I was kind of under the impression by what Treb was talking about on the issue tracker was that you'd just have a displacement map defined in your material, which would feed either your normals or tessellation depending on setup, so it would be an either-or type of affair.

Good Tessellation (or Parallax) is a subtle effect and will give a scene an almost intangible improvement and (in the case of Tessellation)
you'll also get a performance boost due to more efficient LOD management.

Yeah, I think that's what I love so much about Doom III's S-POM implementation.  The effect was nigh intangible, and I couldn't tell what was making the new textures look so grounded in the rest of the world until I realised that they were all self-shadowing and occluding correctly.  It was a pretty incredible effect insofar as it didn't really 'stick out', but just made everything look way more convincing.
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: MrC on January 06, 2016, 04:52:50 PM
Awesome stuff! Really looking forward to the PBR stuff, I might not understand how it works but sure hope it'll help performance.

Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: argoon on January 07, 2016, 05:45:26 AM
Awesome stuff! Really looking forward to the PBR stuff, I might not understand how it works but sure hope it'll help performance.

I don't think PBR will help with performance, all new graphical tech is always heavier than the past one but looks better, PBR (physical based rendering) is just a fancy name to say shaders/materials will now be more physically correct, like you make materials on 3D tools like Blender, Modo, Maya, etc, by changing physical properties, you will now be able to do the same on a PBR rendering engine.

In this example we have the Unity 3D PBR workflow more commonly known by metallic workflow, there's also the specular workflow that is very similar.

(http://www.alanzucconi.com/wp-content/uploads/2015/06/metallic.png)

More examples

(http://i1.wp.com/www.pacocasares.com/wp-content/uploads/2014/04/graphChart.jpg)     
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: MrC on January 07, 2016, 06:23:16 AM
Cool, thanks for the info argoon - very informative!
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: argoon on January 07, 2016, 10:04:18 AM
No problem :) i'm just as happy as you for knowing that eventually Doom 3 BFG will have the ability to render PBR materials, no pressure Treb take the time you want.
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: trebor on January 08, 2016, 01:49:50 PM
r_forceAmbient adds quite a lot ambient light so I thought that it is a good starting point to bring SSAO into the game.

SSAO preview/comparison shots
http://imgur.com/a/eIdOA
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: MrC on January 08, 2016, 02:46:28 PM
Very cool, I for one really like the r_forceAmbient feature - been playing through Lost Mission again, trying different settings etc... (will report any bugs) and seeing so much more for the first time, looks like it works really well with SSAO. Reminds me of Quake4 multiplayer, where they used ambient light to brighten things up. Really awesome work Trebor. By the way, I know some people here are divided when it comes to editors and I know this is off topic but any plans down the road to re-add DoomEdit and/or the other tools?
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: motorsep on January 08, 2016, 02:55:46 PM
SSAO preview/comparison shots
http://imgur.com/a/eIdOA

I wonder how Doom 3 would feel with all those effects turned on :)  Having spatialized audio would be an icing on the cake!

Is it just conventional SSAO technique or HBAO ?
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: motorsep on January 09, 2016, 03:21:52 PM
@trebor: Would it it be hard to add self-shadowing from normal maps? Would it have any significant impact on performance ?
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: romulus_ut3 on January 09, 2016, 09:58:08 PM
Stupid question. Can there be an AMD friendly Soft Shadowing be implemented? For example GTA V has AMD's CHS, which is probably better optimized for AMD. Though it's for DirectX 11 and this is OpenGL for which AMD is simply notorious as far as drivers are concerned.
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: trebor on January 12, 2016, 04:29:24 PM
SSAO has a bad quality and HBAO is too expensive. I implemented a relatively new algorithm by a Nvidia research group: http://graphics.cs.williams.edu/papers/SAOHPG12/
I use a newer version from 2014. The image quality would usually cost up to 16 milliseconds each frame but with a killer optimization it can be achieved in 2-3 ms.

Results: http://imgur.com/a/zjecr
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: motorsep on January 12, 2016, 05:33:49 PM
SSAO has a bad quality and HBAO is too expensive. I implemented a relatively new algorithm by a Nvidia research group: http://graphics.cs.williams.edu/papers/SAOHPG12/
I use a newer version from 2014. The image quality would usually cost up to 16 milliseconds each frame but with a killer optimization it can be achieved in 2-3 ms.

Results: http://imgur.com/a/zjecr

The AO-only images look really cool! The game is dark in general, so I am guessing the effect is subtle in dark scenes (at least on those screenies).

Yeah, I remember that one - the sample build from their website overheated my GPU and PC shut down :P

I am guessing SSAO and derivatives (of good quality) are expensive in general :(
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: argoon on January 13, 2016, 02:20:30 PM
SSAO has a bad quality and HBAO is too expensive. I implemented a relatively new algorithm by a Nvidia research group: http://graphics.cs.williams.edu/papers/SAOHPG12/
I use a newer version from 2014. The image quality would usually cost up to 16 milliseconds each frame but with a killer optimization it can be achieved in 2-3 ms.

Results: http://imgur.com/a/zjecr

Taking out the normal OpenGL performance hit how does this AO algoriyhm run on a AMD card?

Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: trebor on January 13, 2016, 07:07:22 PM
SSAO has a bad quality and HBAO is too expensive. I implemented a relatively new algorithm by a Nvidia research group: http://graphics.cs.williams.edu/papers/SAOHPG12/
I use a newer version from 2014. The image quality would usually cost up to 16 milliseconds each frame but with a killer optimization it can be achieved in 2-3 ms.

Results: http://imgur.com/a/zjecr

Taking out the normal OpenGL performance hit how does this AO algoriyhm run on a AMD card?

We will see :)

I tweaked the HDR tone mapping in the meantime: http://imgur.com/a/dKfLH
There you can see the difference between real 16-bit precision HDR true colors and the old crap and how important it is to accumulate the lights in linear RGB color space.
This is gamma-correct rendering now.
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: Chromatose on January 14, 2016, 07:56:09 PM
Wow, I'm kind of amazed at the difference between the two.  I was vaguely aware that Id originally used Blinn-phong shading because of performance consideration, but it never really hit home how much of a difference just having correct tonemapping could make.  Especially in the first image where with default settings, the two marines look like they're made of plastic, but with proper tonemapping they suddenly look halfway convincing.
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: romulus_ut3 on January 19, 2016, 01:14:22 PM
Today I compiled the whatever there's in the master branch and found totally gibberish graphics being displayed on an AMD card..
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: motorsep on January 20, 2016, 10:13:59 AM
@trebor: is this what you used for SSGI http://graphics.cs.williams.edu/papers/DeepGBuffer14/ ?

Btw, is Vulkan already supported in drivers for Intel/Nvidia/AMD for Win 7 and newer (and Linux and OSX) ?

Also, on the subject of Vulkan API, http://www.phoronix.com/scan.php?page=news_item&px=AMDGPU-Vulkan-Driver-Only
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: 2ton on January 21, 2016, 08:10:58 AM
I read that bit of news on phoronix the other day too. A little worrying, especially when you have 3 amd cards, none (to my knoledge) are 1.2. Still though, it might not meant that non amdgpu cards are completely out in the dark though.
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: trebor on April 11, 2016, 02:46:31 AM
I uploaded new binaries so you can test the latest HDR implementation.

https://github.com/RobertBeckebans/RBDOOM-3-BFG/releases/tag/1.1.0-preview3
Title: Re: RBDOOM-3-BFG 1.1.0 preview
Post by: bitterman on July 13, 2016, 12:12:22 PM
In last RBDoom on Github I see:

Quote
blender/scripts/addons

What is it? What is mean "export map json"?

Don't see any info in readme.

Thanks.