News:

One Minute Game Review by The Happy Friar: https://ugetube.com/@OneMinteGameReviews
Also on Rumble: https://rumble.com/c/c-1115371

idTech 4 (aka Doom 3 tech) Discord Server! https://discord.gg/9wtCGHa

Main Menu
Menu

Show posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Show posts Menu

Messages - revelator

#1
So no not related to idtech as such unless you count compilers :P
been working on fixing a rather bizarre bug in the TDM gcc compiler, first i found out that the dev used CreateMutex to lock the same named mutex in two places which ofc wont work as you would get the ERROR_ALLREADY_EXIST flag and the mutex would deadlock, so i rewrote that part but ran into a new bug which only seems to affect glib2 based sources like gdk-pixbuf2. turns out glib2 now also uses a mutex locking mechanism which collides with the one in TDM gcc basically deadlocking it again urgh... This should actually not be possible as far as i know as the TDM gcc code uses a named mutex with a unique identifier, so how it does this is beyond me unless they screwed up royally writing the locking code. so far all attempts at making it behave have met with failure and im at my wits end.
so if anyone wants to take a look feel free to contact me.
#2
id Tech 4 Discussion / Re: Why so few uses of Id Tech 4?
September 22, 2020, 03:59:13 AM
Well i got the UV problem fixed and readded the AVX intrinsics.
Still lacking the SMP changes from TDM, code was quite a mess and some functions had to be changed rather heavy handedly i remember.

The problem with the bad UV's was a silly one in the GLSL code, i forgot to detach the shaders when the program object had been created (most of the earlier GLSL examples actually forgot to do that) which also caused numerous bugs with them  :)). Well i guess we can be excused, neither of us had much experience at the time with GLSL so that it actually worked to some degree was in itself pretty amazing heh.
#3
More thread work.

Removed thread affinity again windows handles this just fine itself.

Event triggers have been fixed so that it now actually works on the backgroundthread and does not require Sys_Sleep anymore ( fix from TDM ).

My thread exit code turned out to be safe enough after some small modification to actually run before recreating the thread handles in case of map change or reload so it is used there as well now.
If the handle allready exists it will simply reuse it :) else it will wait for the thread to exit and then recreate it.
Found one small use for Sys_ThreadName in the event trigger it was normally set to NULL to create an unamed event handle, instead im now feeding it the current threads name so we can keep track of it.
#4
SO heres TDM's thread routine used in my project.

/*
===================
CreateThreadStartRoutine
===================
*/
typedef std::pair<xthread_t, void *> CreateThreadStartParams;
DWORD WINAPI CreateThreadStartRoutine( LPVOID lpThreadParameter ) {
    std::pair<xthread_t, void *> arg = *( ( CreateThreadStartParams * )lpThreadParameter );
    delete ( ( CreateThreadStartParams * )lpThreadParameter );
    return arg.first( arg.second );
}

/*
==================
Sys_Createthread
==================
*/
void Sys_CreateThread( xthread_t function, void *parms, xthreadPriority priority, xthreadInfo &info, const char *name, xthreadInfo *threads[MAX_THREADS], int *thread_count ) {
    Sys_EnterCriticalSection();
LPVOID threadParam = new CreateThreadStartParams( function, parms );
    HANDLE temp = CreateThread( NULL, // LPSECURITY_ATTRIBUTES lpsa,
                                0, // DWORD cbStack,
                                CreateThreadStartRoutine, // LPTHREAD_START_ROUTINE lpStartAddr,
                                threadParam, // LPVOID lpvThreadParm,
                                0, // DWORD fdwCreate,
                                &info.threadId );

    info.threadHandle = ( intptr_t )temp;

    if ( priority == THREAD_HIGHEST ) {
        SetThreadPriority( ( HANDLE )info.threadHandle, THREAD_PRIORITY_HIGHEST ); //  we better sleep enough to do this
    } else if ( priority == THREAD_ABOVE_NORMAL ) {
        SetThreadPriority( ( HANDLE )info.threadHandle, THREAD_PRIORITY_ABOVE_NORMAL );
    } else {
        // if we hit this then the programmer forgot to set a default thread priority.
        SetThreadPriority( ( HANDLE )info.threadHandle, GetThreadPriority( ( HANDLE )info.threadHandle ) != THREAD_PRIORITY_ERROR_RETURN );
    }
    info.name = name;

    if ( *thread_count < MAX_THREADS ) {
        threads[( *thread_count )++] = &info;
    } else {
        common->DPrintf( "WARNING: MAX_THREADS reached\n" );
    }
    Sys_LeaveCriticalSection();
}


And here is the function to kill the threads at exit.

void Sys_DestroyThread( xthreadInfo &info ) {
    DWORD dwExitCode, dwWaitResult, dwThreadCount;
    HANDLE dwThreadHandle[MAX_THREADS];

    // no threads running so nothing to kill.
    if ( !info.threadHandle ) {
        return;
    }
    Sys_EnterCriticalSection();

    // give it a little time
    Sys_Sleep( 1000 );

    // get number of threads to wait for.
    for ( dwThreadCount = 0; dwThreadCount < MAX_THREADS; dwThreadCount++ ) {
        // create an array of handles for WaitForMultipleObjects.
        dwThreadHandle[dwThreadCount] = ( HANDLE ) info.threadHandle;

        // wait for the handle to be signaled.
        dwWaitResult = WaitForMultipleObjects( dwThreadCount, dwThreadHandle, TRUE, INFINITE );

        // signal handlers for WaitForMultipleObjects.
        switch ( dwWaitResult ) {
        case WAIT_ABANDONED_0:
// Major problem somewhere mutex object might have been killed prematurely.
            idLib::common->Printf( "Mutex object was not released by the thread that owned the mutex object before the owning thread terminates...\n" );
            break;
        case WAIT_OBJECT_0:
// The condition we want.
            idLib::common->Printf( "The child thread state was signaled!\n" );
            break;
        case WAIT_TIMEOUT:
// Thread might be busy.
            idLib::common->Printf( "Time-out interval elapsed, and the child thread's state is nonsignaled.\n" );
            break;
        case WAIT_FAILED:
            // Fatal this condition would crash us anyway so might as well let it, yeah right...
idLib::common->Printf( "WaitForMultipleObjects() failed, error %u\n", ::GetLastError() );
return; // get the hell outta here!
        }

// Get thread exit status and close the handle.
        if ( ::GetExitCodeThread( dwThreadHandle, &dwExitCode ) != FALSE ) {
            ExitThread( dwExitCode );
            if ( CloseHandle( dwThreadHandle ) != FALSE ) {
                dwThreadHandle[dwThreadCount] = NULL;
            }
        }
    }
    Sys_LeaveCriticalSection();
}


it is called in one place only at the end of Sys_Quit just before ExitProcess(0);
This is done to make sure nothing is still hooked when exiting the thread and only runs at game shutdown.

The entercriticalsection and leavecriticalsection additions are there to make sure the process owns the running thread.

The last function delegates the running threads on multicore machines eg. > 2 cores.

void Sys_SetThreadAffinity( bool mainthread ) {
    SYSTEM_INFO info;

    // check number of processors
    GetSystemInfo( &info );

    // single core machine so but out.
    if ( info.dwNumberOfProcessors < 2 ) {
        return;
    }

    // set thread affinity for main thread on core 1 or 3
    if ( mainthread ) {
        switch ( info.dwNumberOfProcessors ) {
        case 1:
            SetThreadAffinityMask( GetCurrentThread(), ( 1 << info.dwNumberOfProcessors ) );
            break;
        case 3:
            SetThreadAffinityMask( GetCurrentThread(), ( 3 << info.dwNumberOfProcessors ) );
            break;
        default:
            break;
        }
    } else {
        // set affinity for other threads on core 2 or 4
        switch ( info.dwNumberOfProcessors ) {
        case 2:
            SetThreadAffinityMask( ( HANDLE )threadInfo.threadHandle, ( 2 << info.dwNumberOfProcessors ) );
            break;
        case 4:
            SetThreadAffinityMask( ( HANDLE )threadInfo.threadHandle, ( 4 << info.dwNumberOfProcessors ) );
            break;
        default:
            break;
        }
    }
}


If say we have a quad core it would delegate the main thread on core 1 and 3 and the second thread on core 2 and 4.
Doom3 uses only 2 threads, one runs the game and another for background file reads.

void idFileSystemLocal::StartBackgroundDownloadThread() {
    if ( !backgroundThread.threadHandle ) {
        Sys_CreateThread( ( xthread_t )BackgroundDownloadThread, NULL, THREAD_NORMAL, backgroundThread, "backgroundDownload", g_threads, &g_thread_count );
        if ( !backgroundThread.threadHandle ) {
            common->Warning( "idFileSystemLocal::StartBackgroundDownloadThread: failed" );
        }
    } else {
        common->Printf( "background thread already running\n" );
    }

// give the async thread an affinity for the 2 or 4'th core.
Sys_SetThreadAffinity();
}


and here

int WINAPI WinMain( HINSTANCE hInstance, HINSTANCE hPrevInstance, LPSTR lpCmdLine, int nCmdShow ) {
#ifdef _DEBUG
    SetCurrentDirectory( "C:\\Doom\\Doom 3" );
#endif
    const HCURSOR hcurSave = ::SetCursor( LoadCursor( 0, IDC_WAIT ) );

    // tell windows we're high dpi aware, otherwise display scaling screws up the game
    Sys_SetHighDPIMode();
    Sys_SetPhysicalWorkMemory( 192 << 20, 1024 << 20 );
    Sys_GetCurrentMemoryStatus( exeLaunchMemoryStats );

    win32.hInstance = hInstance;
    idStr::Copynz( sys_cmdline, lpCmdLine, sizeof( sys_cmdline ) );

    // done before Com/Sys_Init since we need this for error output
    Sys_CreateConsole();

    // no abort/retry/fail errors
    SetErrorMode( SEM_FAILCRITICALERRORS );

    for ( int i = 0; i < MAX_CRITICAL_SECTIONS; i++ ) {
        InitializeCriticalSection( &win32.criticalSections[i] );
    }

    // get the initial time base
    Sys_Milliseconds();

#ifdef DEBUG
    // disable the painfully slow MS heap check every 1024 allocs
    _CrtSetDbgFlag( 0 );
#endif

    Sys_FPU_SetPrecision( FPU_PRECISION_DOUBLE_EXTENDED );
    common->Init( 0, NULL, lpCmdLine );

#ifndef ID_DEDICATED
    if ( win32.win_notaskkeys.GetInteger() ) {
        DisableTaskKeys( TRUE, FALSE, FALSE );
    }
#endif
    Sys_StartAsyncThread();

    // hide or show the early console as necessary
    if ( win32.win_viewlog.GetInteger() || com_skipRenderer.GetBool() || idAsyncNetwork::serverDedicated.GetInteger() ) {
        Sys_ShowConsole( 1, true );
    } else {
        Sys_ShowConsole( 0, false );
    }

    // give the main thread an affinity for the first or 3'rd core.
    Sys_SetThreadAffinity( true );

    ::SetCursor( hcurSave );
    ::SetFocus( win32.hWnd );

    // main game loop
    while ( 1 ) {
        Win_Frame();

        // run the game
        common->Frame();
    }

    // never gets here
    return 0;
}


Also need an extern for it in Sys_Public.h

void Sys_SetThreadAffinity( bool mainthread = false );

#5
Strutt your stuff! / Re: TDM version of gcc-9.2.0
September 16, 2020, 08:09:27 AM
updated gcc to 9.3.0 binutils to 2.35.
One major snag turned up in regards to the TDM patchset, you cannot use ASLR with the TDM based compilers as the mechanism used to allow throwing exceptions across dll boundaries from static libraries rely on a set base adress (ASLR randomizes these).

This means various AV solutions will likely flag anything built with it as a major threat and try to delete it.
There is no threat however and pretty much any mingw gcc before the aslr update will get flagged as well, so you need to whitelist your programs with the AV or risk loosing them.

SO much for progress indeed sigh.
#6
If someone is good with gui code i could use a hand in fixing some old bugs with venoms doom3 menu.
I once had most of them fixed but it was long ago when idtech4 had just been released and i have since lost the fixed code.
One bug in particular was pretty annoying, you could not overwrite savegames with venoms menu and there was another one that screwed up game skill selection.

Also need a special version for sikkmod's options and i plan on removing the broken features from sikkmod like SSAO (breaks skyportals and alpha entities) and soft shadows (huge FPS sink and looks ghastly on some AMD cards with black and or blue outlines).
#7
Slowly recreating my lost work but since im officially retired development is slow.
New changes: Thread affinity is now used but in a different way than original.
Doom3 only uses 2 threads one for the main engine and one for the server, i created a function to auto delegate the main thread on core 1 and 3 and the server on core 2 and 4.

Thread exit was newer used in the windows version since win does not like threads being killed, so i had to create a function that makes sure all threads have run to completion then it would check for stuck threads and end each one at sys exit (and only at sys exit (doing it anywhere else might actually crash your PC)). It basically newer gets that far though as windows handles thread exit internally pretty well, but in case it gets stuck it will kill the stuck thread handles safely.

The hybrid ARB/GLSL backend has also had a major overhaul and now works perfect, you can seamlessly switch between ARB and GLSL interactions at runtime without any hickups now.
The GLSL backend uses half lambertian so looks a fair bit better than ARB. It also works just fine with sikkmod though you cannot use sikkmods parallax occlusion shader as that one is an interaction shader (world drawing) so if you want to toy with that you need to turn of GLSL. All the other effects work though. Or you could rewrite sikkmods POM shader in GLSL :) as the backend supports loading external GLSL as well.

Also added AVX and AVX2 support again, still missing the SMP changes from darkmod but eventually ill get there.

Planned: Adding dentons bloom code to the game code for base doom3 and the expansion (so that the broken bloom cvar actually gets used).
Dentons bloom is pretty sleek and does not look over the top like many other implementations so it would probably fit well.

Sadly my main engine does not have the internal editors anymore as it was based on MH's Doom3 port. For the linux guys this is pretty moot as the editors are MFC based and so not portable anyway.
#8
Tbh we could do away with it completely these days, pretty much any card today can run idtech4 with ok fps.
The original code was used as a way to detect older cards capabillities and set performance settings based on that,
but back then the biggest card we had was a geforce 3 ultra and even that struggled running it on ultra.

New capabilities added by developers have pushed idtech further ofc,
but still the biggest problems with performance these days stem from the engine being from a time where a lot of the grunt work was done on the cpu.
fhDoom might actually change that if and when someone completes codepaths for GPU skinning.
#9
I allready use a variant of this codepiece from darkmod using enumerators instead of boolean, still usefull though :)
#10
hmm the .msh is most likely a mesh file like used in orbiter, the .skn one seems to a 3D body mesh skin data file like used in the sims.
Could try to see if you can open them with the tools from either.

Might also be from photoshop https://reverseengineering.stackexchange.com/questions/19952/any-ideas-on-how-to-decode-photoshop-liquify-msh-files
#11
id Tech 4 Discussion / Re: Why so few uses of Id Tech 4?
February 01, 2020, 08:02:34 PM
scrounging up what i could from this disaster, i started adding in the SMP and timing changes to dhewm as well as the multicore renderer.
Its a big job though so might take a while.
#12
thanks ill have a look at it  ;)
The simd changes needed a lot more headers in the math sources than what was originally in darkmod since those where pulled from precompiled.h originally.
the SMP changes need some std functions that i cannot guarantee will exist in all versions of msvc or even gcc so i will need to be carefull there.
#13
added the base code for the darkmod SMP changes to dhewm, still a lot of work needed but its closer now.
Lots of stuff from darkmod runs through here so its easy to make mistakes and add stuff that does not belong there (lightgems framebuffers ad nauseum etc and so forth).
Since dhewm relies on SDL threads im not even certain if i can do it, main async thread runs at a different ticrate than normal 3 vs 60.
#14
i added AVX and AVX2 to the dhewm code, but you need to revisit the SDL2 libraries provided since it does not support AVX2 and it should (it has been availiable since version 2.0.4 but this version of 2.0.4 does not support it).
The timing changes will be added later because of differences in the code related to SDL which im not used to working with.
#15
darkmods smp changes are easy to get at, they are all commented in the idlib math source code.
There is however one snag since they removed old MMX and ALTIVEC paths there are a few places with commented code for that which needs to be reactivated.

The timing code makes use of some std:: functions which may not be in all versions of MSVC so be carefull there.

The multithreading code however was lost which indeed sucks  :-\  and also unfortunatly the hybrid GLSL backend.

The smp and timing changes are probably the ones you would be most interrested in, since they provide the biggest noticable performance boost.

The AVX and AVX2 smp functions are used for stencil shadow volumes, which have allways been a major ressource hog, so the boost is quite welcome there.