Creating an OpenGL Overlay

Apprentice
Posts: 12
Joined: 2006.12
Post: #1
Hi,

I want to create an app that displays an overlay on top of a game using OpenGL. There some programs in the Windows-world which can do this (Xfire, Teamspeak Overlay). Afaik they are doing this by hooking some OpenGL (or Direct3D) calls and replacing them with their own code.
I´ve searched a bit around, and the easiest way to do this seems to be the Application Enhancer: http://unsanity.com/haxies/ape
It´s quite easy to use, I´ve already managed to hook gl_Begin and draw my own stuff in the window of a running game, but hooking gl_begin isn´t very efficent, since it gets called once for each object per frame.
So my question is: Do you know an OpenGL function which gets called (in most games) once per frame? Or is there even a better method than doing it like this?

Thanks Smile
Quote this message in a reply
Sage
Posts: 1,199
Joined: 2004.10
Post: #2
You might look into the stuff that is called at the end of a frame, like glFinish and glFlush. Or if the app's Cocoa, [NSOpenGLContext flushBuffer].
Quote this message in a reply
Sage
Posts: 1,234
Joined: 2002.10
Post: #3
CGLFlushDrawable().
Quote this message in a reply
Apprentice
Posts: 12
Joined: 2006.12
Post: #4
Hm, okay, thank you, I´ll try that out...
So I think I´d have to hook all these functions to support as many games as possible?
Quote this message in a reply
Sage
Posts: 1,199
Joined: 2004.10
Post: #5
Probably just the function arekkusu mentioned. But obviously, test test test.
Quote this message in a reply
Apprentice
Posts: 12
Joined: 2006.12
Post: #6
Okay, many thanks, I´ll check it...
Quote this message in a reply
Apprentice
Posts: 12
Joined: 2006.12
Post: #7
Hmm...

Hooking "CGLFlushDrawable" is working for two of my old OpenGL projects, one of them calls [NSOpenGLContext flushBuffer], the other one calls glFinish() at the end of each frame.

I´m using this code to draw a rect in the top left frame over the game:

Code:
static CGLError hook_CGLFlushDrawable(CGLContextObj ctx)
{    
    DrawOverlay();
    
    return orig_CGLFlushDrawable(ctx);
}

void DrawOverlay(void)
{
    glDisable(GL_DEPTH_TEST);
    glDepthMask(GL_FALSE);
    glColor3f(1.0f,1.0f,1.0f);
    ViewOrtho(500, 500);
    glBegin(GL_QUADS);
    glVertex2f(0, 0);
    glVertex2f(0, 100);
    glVertex2f(100, 100);
    glVertex2f(100, 0);
    glEnd();
    ViewPerspective();
    glEnable(GL_DEPTH_TEST);
    glDepthMask(GL_TRUE);
}

void ViewOrtho(int x, int y)
{
    glMatrixMode(GL_PROJECTION);
    glPushMatrix();
    glLoadIdentity();
    glOrtho( 0, x , y , 0, -1, 1 );
    glMatrixMode(GL_MODELVIEW);
    glPushMatrix();
    glLoadIdentity();
}

void ViewPerspective(void)
{
    glMatrixMode( GL_PROJECTION );
    glPopMatrix();
    glMatrixMode( GL_MODELVIEW );
    glPopMatrix();
}

But the only games (beside my OpenGL projects) are the newer Feral games like Tomb Raider or Race Driver 3. In all other games I´ve tried I´m only getting the white rect during startup, or when the games are showing the intro video.
I´ve tried UT2004, CoD 1&4, Quake 3/ioQuake3, Urban Terror, Wolfenstein:ET, but none of these games work correctly with my overlay, I´m only getting it during startup with some of them...
I´ve also hooked glFlush, glFinish, glSwapAPPLE and aglSwapBuffers, but it doesn´t change anything...
What the hell are all these games calling when they reach the end of their drawing function?! Do you know any functions beside these I´ve already hooked? Or is there maybe a mistake in my overlay drawing code?

Thanks in advance for your help Smile
Florian
Quote this message in a reply
Sage
Posts: 1,234
Joined: 2002.10
Post: #8
You need to define "work correctly".

Is your code still being called in newer games, and the rendering doesn't appear? Or is your code not being called at all.

There are a few hundred other things in the context state that could cause your rendering to be broken. For example, if the game left a vertex shader enabled when they swap. You're not checking for that and disabling it. Your rendering is going to use whatever state vector they left set up.
Quote this message in a reply
Apprentice
Posts: 12
Joined: 2006.12
Post: #9
Oh, thanks for the hint.
CGLFlushDrawable seems to get called in all games, but the rendering doesn´t work.
I thought I was hooking the wrong functions, but CGLFlushDrawable seems to be okay.
So the problem is within my rendering code...
Thanks for your help Smile
Quote this message in a reply
Apprentice
Posts: 12
Joined: 2006.12
Post: #10
okay, I think I got it now...
Calling
Quote:glDisable(GL_CULL_FACE);
before I draw the overlay makes it appear in most games... Thanks four your help Smile
Quote this message in a reply
Apprentice
Posts: 12
Joined: 2006.12
Post: #11
Hi,

I got a new problem... Annoyed
Since the APE seems to be a bit buggy (it made my iMac hang up when I tried to boot, only removing APE in the single user mode helped) I decided to use another method: Compiling my code as Dynamic Library and loading it with the dyld environment variable "DYLD_INSERT_LIBRARIES" when an application starts.
This works for functions which get called directly in the games code, e.g. glBegin or glEnable, for some games [NSOpenGLContext flushBuffer] works, too.
But I´m not able to override "CGLFlushDrawable()" using this method. I think it doesn´t work, because no game seems to call "CGLFlushDrawable" directly. "CGLFlushDrawable" seems to get called by NSOpenGL or AGL or GLUT. Just adding the function "CGLFlushDrawable" to my code works for Application Enhancer, but not when I load the code using dyld.
Do you know whether there is a way to override CGLFlushDrawable using the dyld-method? Or is there maybe a better method than using dyld? I found this code, but it doesn´t work for x86-Macs and it´s quite difficult. I could also try APE again, but I don´t want that the users of my app have to download APE to be able to use my app, so dyld seems to be the better way...
Quote this message in a reply
Sage
Posts: 1,234
Joined: 2002.10
Post: #12
Can you clarify your goal here? You want to do something like Xfire, where you display a chat dialog or some other UI composited on top of an arbitrary OpenGL application?

I think this is difficult to do in a robust way.

It sounds like you're interposing specific functions (CGLFlushDrawable etc) and then inserting draw commands into the app's context. This approach is doomed to failure, because if you use the app's context, you need to save/restore their state around your drawing. And there's no future-proof way for you to save all of the state correctly-- even if you carefully push/pop/get/set every state variable that exists today, a future extension may add a non push-poppable bit that will break your rendering.

A more robust way would be use your own context for the drawing. You can set all the state there without worrying about what the app is doing. The problem with that approach is that you need to attach your context to the app's current drawable. You can do this if you interpose at the NSGL or AGL level and the app runs in a window, but you can't do it at the CGL level or if the app is fullscreen, because there's no public API to get the fullscreen drawable a context is attached to.
Quote this message in a reply
Apprentice
Posts: 12
Joined: 2006.12
Post: #13
arekkusu Wrote:Can you clarify your goal here? You want to do something like Xfire, where you display a chat dialog or some other UI composited on top of an arbitrary OpenGL application?

Yep, thats what I want to do Wink

arekkusu Wrote:It sounds like you're interposing specific functions (CGLFlushDrawable etc) and then inserting draw commands into the app's context. This approach is doomed to failure, because if you use the app's context, you need to save/restore their state around your drawing. And there's no future-proof way for you to save all of the state correctly-- even if you carefully push/pop/get/set every state variable that exists today, a future extension may add a non push-poppable bit that will break your rendering.

A more robust way would be use your own context for the drawing. You can set all the state there without worrying about what the app is doing. The problem with that approach is that you need to attach your context to the app's current drawable. You can do this if you interpose at the NSGL or AGL level and the app runs in a window, but you can't do it at the CGL level or if the app is fullscreen, because there's no public API to get the fullscreen drawable a context is attached to.

Using this method (overriding CGLFlushDrawable) worked quite good for me when I did it in App Enhancer. It worked for all Quake3/ioQuake3-based games, many old OpenGL-Games and for example Doom3, Quake 4, Call of Duty 4 and many more. I only had problems with UT2004-based games (they simply don´t show my overlay), but support for a few games is still better than no overlay at all.
That isn´t my main problem, currently my problem is that I can´t override "CGLFlushDrawable" with dyld as described above...
If this is simply not possible, then I´ll try to override the functions in AGL/NSOpenGL which flush/swap the buffer(s). The problem is, that I don´t know every function which can do that.
Are there some more beside glFinish, glFlush, glSwapAPPLE, aglSwapBuffers, glutSwapBuffers and [NSOpenGLContext flushBuffer]?

Thanks for your help Smile Grin
Quote this message in a reply
Post Reply 

Possibly Related Threads...
Thread: Author Replies: Views: Last Post
  OpenGL ES creating a 2D mesh soulstorm 0 3,103 May 20, 2009 02:37 AM
Last Post: soulstorm
  GLUT Overlay vinodpatel2006 8 6,201 Feb 26, 2008 07:30 PM
Last Post: kuon_
  creating chessboard using opengl darshnee 6 6,631 Mar 1, 2006 10:51 AM
Last Post: unknown