System freezing issue under Mac OS X using realtime priority

[rjvbertin]

I wonder if anyone has encountered the following annoying phenomenon under Mac OS X, and/or has a solution to propose.

In short, to try to get fluid animation, I use the code below to enter realtime priority just before I start drawing a frame. I suppose to be running at FrameRate herz, determined empirically, using SDL in OpenGL, patched to by synched with the VBL. FrameRate typically is somewhat less than 60.
The way I understand the documentation, this code requests realtime priority for the specified duration (enough for a single frame), with a few constraints.

This works (see also below). HOWEVER, if a bug causes a crash and (I suppose) we're still in realtime priority, a deadlock occurs in which the system tries to do something (show the quit dialog?? I don't use the SDL "parachute") but cannot. This FREEZES UP THE WHOLE system. Even if I still have focus on the calling terminal, or if I have a remote terminal running, the only solution is to powercycle. Note that even some events like clicking the window close button can cause a crash inside SDL_GL_SwapBuffers().

Am I doing something wrong? Should I force an exit from realtime after drawing the frame -- but how??

Curiously, when I do not enter realtime mode, my process is not killed when the same programme crash occurs: it remains in a sort of semi-limbo/semi-functional state, and no longer responds to SIGINT (^C). Is this a "known issue" with SDL on/or Mac OS X??

There *are* some benefits to using realtime: my application is a visual psychophysics stimulus programme (for doing fundamental science ;^)), and gives feedback to the user with system beeps (until now, I only know to ring the terminal bell, which causes some overhead that perturbs the animation). There is also a known issue where once every 30-some seconds one looses one or more frames, possibly due to the update(8) process -- this seems somewhat less when using realtime.

Thanks!
RenÈ

Here's the code: I start drawing immediately afterwards.


#if defined(__MACH__) /* && !defined(DEBUG) */
static struct thread_time_constraint_policy ttcpolicy;
static unsigned char called= 0;
if( CloudCity_Attempt_RealTime ){
if( !called && CloudCity_Attempt_RealTime ){
int ret, bus_speed, mib[2] = { CTL_HW, HW_BUS_FREQ };
size_t len;
extern double FrameRate;

len = sizeof( bus_speed);
ret = sysctl (mib, 2, &bus_speed, &len, NULL, 0);
if (ret < 0) {
fprintf( stderr, "sysctl query bus speed failed (%s)\n", serror() );
}
else{
fprintf( stderr, "System cpu/bus_speed==%d\n", bus_speed );
ttcpolicy.period= bus_speed / FrameRate;
ttcpolicy.computation= bus_speed / 330;
ttcpolicy.constraint= bus_speed / 220;
ttcpolicy.preemptible= 1;
}
called= 1;
}
if( called ){
errno= 0;

/* Do a sync() to flush all open disk buffers before entering realtime priority. We do
\ this since even if there are no bugs in this routine itself, SDL_GL_SwapBuffers()
\ can crash in some circumstances, and we do not know exactly at what time we get
\ out of realtime mode. Crashing in realtime mode usually means the system ends up
\ in a deadlock situation, requiring a powercycle. Syncing means that at least we minimise
\ the risk of filesystem corruption.
\ There doesn't seem to be a penalty for doing a sync here.
*/
sync();

if( thread_policy_set( mach_thread_self(),
THREAD_TIME_CONSTRAINT_POLICY, (int*) &ttcpolicy, THREAD_TIME_CONSTRAINT_POLICY_COUNT
)!= KERN_SUCCESS
){
if( called< 255 ){
fprintf( stderr, "Couldn't set realtime priority: %s\n", serror() );
called+= 1;
}
}
}
}
#endif

[arekkusu]

One question, this is the main thread?

[rjvbertin]

Yes. I only use a single thread. I'm not sure if SDL does or does not create another one when running in OpenGL mode, though. I think it does (the top command shows 2 threads). I suppose that makes a difference?

[arekkusu]

I'm not a multithreading guru, but in my experience it is very easy to deadlock your entire system with threaded OpenGL. If you create a context, and then submit rendering commands to it from a second thread, things work fine UNTIL the main thread also submits commands (for example, due to regular OS events like window resizing, moving, minimizing, etc) at which point, you are screwed. Not quite sure if this is what is happening with your fullscreen context at crash time (the OS is supposed to tear down fullscreen contexts on force-quit or crash. It won't at, for example, a gdb breakpoint tho... which is where the remote ssh comes in.)

To avoid this you have to use locks around every point where GL commands can be submitted (pthread_mutex_lock is fine.)

Try a bit of printf-ing to see what the current thread is when your GL commands are submitted, to verify it is a multithreading issue first.

[rjvbertin]

I'm not a multithreading guru, but in my experience it is very easy to deadlock your entire system with threaded OpenGL. If you create a context, and then submit rendering commands to it from a second thread, things work fine UNTIL ...

I can imagine this. I'd have to delve into the SDL "underworld" to see in what thread it creates the OpenGL context. I am sure that my own code only uses a single thread, and as you know, SDL does not (currently) provide wrappers to the GL calls that could change thread behind my back. To my knowledge, the only reason SDL creates a separate thread on Mac OS X is to simulate VBL syncing when *not* using OpenGL.

This whole argument is a bit redundant as the system freezes only occur when I use realtime. The exact same code runs fine when I don't (with the exception of the 'limbo phenomenon'). And again: the only reason for it being a multithreading issue is if SDL_GL_SwapBuffers does something in another thread.

Not quite sure if this is what is happening with your fullscreen context at crash time (the OS is supposed to tear down fullscreen contexts on force-quit or crash. It won't at, for example, a gdb breakpoint tho... which is where the remote ssh comes in.)

I'm not running fullscreen. That, I already postponed to the moment I'm sure of my code :) And no, a remote ssh is of no help when one of my system freezes occurs.


To avoid this you have to use locks around every point where GL commands can be submitted (pthread_mutex_lock is fine.)


I'll try this (though I've become a bit more careful with trying new features ;^))

R.

[rjvbertin]

I've looked at this multi-threading/OpenGL thing a bit more closely.

SDL_GL_SwapBuffers() calls:

void QZ_GL_SwapBuffers (_THIS) {
[ gl_context flushBuffer ];
}


So OpenGL things do not run in another thread (*probably*). There is however a separate thread that does the event handling. Is it necessary to propagate a received (abortive/fatal) signal to all other threads?? Are there multithreading issues with signal handling, that would require using sigaction() rather than signal()? Cf these remarks in SDL_systhread.c :

#if !defined(MACOSX) /* pthread_sigmask seems to be missing on MacOS X? */
/* List of signals to mask in the subthreads */
static int sig_list[] = {
SIGHUP, SIGINT, SIGQUIT, SIGPIPE, SIGALRM, SIGTERM, SIGCHLD, SIGWINCH,
SIGVTALRM, SIGPROF, 0
};
#endif /* !MACOSX */


And concerning pthread_mutex_lock(): how do I create the mutex I must pass -- is there any way/need to specify that I want to lock the current thread (pthread_self(), or is that automatic as it seems to be)? Wouldn't pthread_mutex_trylock() be better/safer?