Mouse cursors in games

⌘-R in Chief
Posts: 1,254
Joined: 2002.05
Post: #1
I was looking at mouse cursors in games earlier today, and was trying to come up with a reason why many games draw the cursor in GL instead of using the system cursor. I'm still not entirely sure, but it has been posited that, on OS X at least, it's really not necessary.

The main reason I could think of is locking the cursor to within the window bounds. I, at first, figured that you couldn't do this on OS X, because the best you can do is warp the cursor back within the window's bounds which would cause noticeable "bouncing" and would still make it all-too-easy to accidentally click outside of the window.

So I created a test project and tried it, and sure enough, the "bounce" is visible and annoying. So instead of ever letting the system move the mouse, I thought to completely control its position, but still let the system draw it, instead of having to draw it in GL. So I created another path which does just that. (Using CGAssociateMouseAndMouseCursorPosition and then warp the mouse with CGWarpMouseCursorPosition every time I receive mouse deltas in NSApplication's -sendEventSmile.

The second solution works better, but the problem is that the mouse movement no longer uses the OS's nice accelerated curve. In other words calculating a new mouse position is not as simple as adding the delta to the previous position because the delta is in hardware terms (I believe) and thus the mouse movement is hyperactive. In my case multiplying by a scalar (0.5) is acceptable, but it'll vary user-to-user, and is still not a nice curve like we've gotten used to since whatever-OS-version-Apple-changed-it-in.

So my questions:
1) Has anybody done lock-to-window-bounds? If so what approach did you take and why?

2) I'm preeetty sure it's not possible to use the OS X acceleration curve (there's no API for it), so if a curve is desirable (is it, in a game?) then I'd need to actually create one, but heck if I can figure that out. I guess I could try to plot the hardware deltas and location deltas and reverse it. Anyone tried?

3) If simply using a linear scalar "sensitivity" value… that's straightforward, but am I forgetting anything?


Here's the example project I cooked up:
http://www.sethwillits.com/temp/CursorTest.zip



And the code for the archives:

Code:
//
//  GBApplication.m
//

#import "GBApplication.h"


// There are two modes:
// 1) Warping only when the mouse goes outside of the window
// 2) Disassociating the mouse and cursor and warping every event.
//
// The problem with #1 is that the mouse can "bounce" at the edge of
// the window. It'll show be outside of the window and then warped
// within its bounds, and that warp is visible.
//
// Disassociating with #2 solves that problem, but introduces a
// lesser one: the mouse movement, being based off of hardware
// deltas, is no longer system-control and does not have the nice
// curve translating hardware deltas into smooth movements. This
// creates the need for a "sensitivity" parameter. Here it is
// linear 0.5, but that will certainly be wrong from user to user
// (and perhaps mouse to mouse)?
//

#define DISASSOCIATE_MOUSE 1



@interface GBApplication (Private)
- (CGRect)screenRectForWindowContent:(NSWindow *)window;
- (BOOL)isMouseLocation:(CGPoint)mouseLocationTL outsideOfWindow:(NSWindow *)window;
- (CGPoint)mouseLocation:(CGPoint)mouseLocationTL insideOfWindow:(NSWindow *)window;
- (BOOL)warpMouseToLocation:(CGPoint)mouseLocationTL event:(NSEvent **)event;
@end




#pragma mark  
@implementation GBApplication

- (id)init
{
    if (!(self = [super init])) {
        return nil;
    }
    
    
    #if DISASSOCIATE_MOUSE
        CGAssociateMouseAndMouseCursorPosition(NO);
    #endif
    
    
    // "Synthesized" events (ones that don't come from hardware),
    // more specifically, ones from CGWarpMouseCursorPosition(),
    // trigger a "suppression interval" during which events from
    // local hardware (the user's physical mouse) are ignored for
    // a default of 0.25 seconds. We really don't want that, so
    // we turn it off.
    // This function is deprecated but there's no way around it.
    CGSetLocalEventsSuppressionInterval(0.0);
    
    
    return self;
}




- (void)sendEvent:(NSEvent *)event
{
    // If mouse moved event:
    // 1) check if it's outside of the window bounds
    // 2) if so, warp the cursor position to inside the window
    // 3) create a new event to represent that mouse move with the new point
    // 4) send the event along
    
    
    if ((event.type == NSMouseMoved) || (event.type == NSLeftMouseDragged) || (event.type == NSRightMouseDragged) || (event.type == NSOtherMouseDragged)) {
        NSWindow * window = [[self windows] lastObject]; // Only one in this test app
        CGPoint mouseLocationTL = CGEventGetLocation([event CGEvent]);
        
        #if DISASSOCIATE_MOUSE
            
            // Need to move the mouse ourselves every time.
            
            // I believe the OS uses a curve, not a scalar value, but
            // I don't think we can get the system's curve function
            // to use. So we use a linear "sensitivity" scalar here.
            mouseLocationTL.x += event.deltaX * 0.5;
            mouseLocationTL.y += event.deltaY * 0.5;
            
            mouseLocationTL = [self mouseLocation:mouseLocationTL insideOfWindow:window];
            [self warpMouseToLocation:mouseLocationTL event:&event];
            
        #else
            
            // Only warp if the mouse is outside of the window
            if ([self isMouseLocation:mouseLocationTL outsideOfWindow:window]) {
                mouseLocationTL = [self mouseLocation:mouseLocationTL insideOfWindow:window];
                [self warpMouseToLocation:mouseLocationTL event:&event];
            }
        
        #endif
    }
    
    
    [super sendEvent:event];
}

@end





#pragma mark  
@implementation GBApplication (Private)

- (CGRect)screenRectForWindowContent:(NSWindow *)window
{
    NSSize inset = NSMakeSize(10, 10); // for demonstration
    NSRect windowContentScreenRect = [window convertRectToScreen:[(NSView *)[window contentView] frame]];
    windowContentScreenRect = NSInsetRect(windowContentScreenRect, inset.width, inset.height);
    windowContentScreenRect.origin.y = ([NSScreen mainScreen].frame.size.height - windowContentScreenRect.origin.y - windowContentScreenRect.size.height);
    return windowContentScreenRect;
}



- (BOOL)isMouseLocation:(CGPoint)mouseLocationTL outsideOfWindow:(NSWindow *)window;
{
    CGRect windowContentScreenRect = [self screenRectForWindowContent:window];
    return (!CGRectContainsPoint(windowContentScreenRect, mouseLocationTL));
}



- (CGPoint)mouseLocation:(CGPoint)mouseLocationTL insideOfWindow:(NSWindow *)window;
{
    CGRect windowContentScreenRect = [self screenRectForWindowContent:window];
    mouseLocationTL.x = MIN(MAX(mouseLocationTL.x, CGRectGetMinX(windowContentScreenRect)), CGRectGetMaxX(windowContentScreenRect) - 1);
    mouseLocationTL.y = MIN(MAX(mouseLocationTL.y, CGRectGetMinY(windowContentScreenRect)), CGRectGetMaxY(windowContentScreenRect) - 1);
    return mouseLocationTL;
}




- (BOOL)warpMouseToLocation:(CGPoint)mouseLocationTL event:(NSEvent **)event;
{
    // Warp the cursor
    CGError error = CGWarpMouseCursorPosition(mouseLocationTL);
    
    // Replace the event using the new location
    if (error == kCGErrorSuccess) {
        NSPoint mouseLocationBL = NSMakePoint(mouseLocationTL.x, [NSScreen mainScreen].frame.size.height - mouseLocationTL.y);
        *event = [NSEvent mouseEventWithType:(*event).type
                                    location:mouseLocationBL
                               modifierFlags:(*event).modifierFlags
                                   timestamp:(*event).timestamp
                                windowNumber:(*event).windowNumber
                                     context:(*event).context
                                 eventNumber:(*event).eventNumber
                                  clickCount:(*event).clickCount
                                    pressure:(*event).pressure];
        
        return  YES;
    }
    
    return NO;
}


@end




Code:
//
//  GBView.m
//

#import "GBView.h"


@implementation GBView

- (void)resetCursorRects
{
    NSImage * image = [NSImage imageNamed:@"Cursor"];
    [image setSize:NSMakeSize(32, 32)];
    
    NSCursor * cursor = [[[NSCursor alloc] initWithImage:image hotSpot:NSMakePoint(10, 5)] autorelease];
    [self addCursorRect:self.bounds cursor:cursor];
}


- (void)drawRect:(NSRect)dirtyRect
{
    [[NSColor blackColor] set];
    NSRectFill(dirtyRect);
}

@end




Code:
//
//  AppDelegate.m
//

#import "AppDelegate.h"

@implementation AppDelegate
@synthesize window = _window;


- (void)applicationDidFinishLaunching:(NSNotification *)aNotification
{
    [self.window setAcceptsMouseMovedEvents:YES];
}

@end
Quote this message in a reply
Luminary
Posts: 5,143
Joined: 2002.04
Post: #2
You can get the acceleration curves from IOKit, but it's not immediately obvious to me how to use them.
Quote this message in a reply
Moderator
Posts: 3,572
Joined: 2003.06
Post: #3
(Jun 11, 2012 02:33 AM)SethWillits Wrote:  1) Has anybody done lock-to-window-bounds? If so what approach did you take and why?

Yes, I do the lock-to-window-bounds thing, but do not use the system cursor. Hide it, then CGWarpMouseCursorPosition to keep it in the center of the window, unseen while drawing the cursor using the GL. Why?! Because... Wacom tablet is why. What I discovered is that either the Wacom driver or the OS is/was buggy as all heck. I don't know if this has changed in recent OS versions because I haven't fully tested the problems. One major problem was that I discovered some sort of "ghost cursor" operating in parallel behind the scenes. I could have the cursor visibly bound by the game window and apparently working just like your test project but when clicking, the ghost cursor may be in the background clicking on the dock, not bound to the window at all! Another problem would be when hiding the mouse, if it was in a resizable window, the resize control was still active, as was the window's title bar. Another major problem was with multiple monitors, apparently the cursor was bound to some rectangle other than the screen rectangle on other monitors, which was bizarre. Then in full screen there were numerous other problems.

So, the "best" solution, after much testing, researching and experimenting, turned out to be to hide the OS cursor, warp it to the center of the active view, and draw the cursor using the GL, using deltas for position. I really should test the new Wacom drivers in Snow Leopard one of these days to see if things have improved. It'd be easy to just say the game isn't compatible with Wacom tablets, but there are a lot of people who use them as primary input devices.
Quote this message in a reply
⌘-R in Chief
Posts: 1,254
Joined: 2002.05
Post: #4
I did a search late last night and saw your mention of Wacom tablet stuff back in 2007. Why would you use a Wacom tablet in a game? Seems awkward.

Quote:Another problem would be when hiding the mouse, if it was in a resizable window, the resize control was still active, as was the window's title bar.

Meaning what? You can easily exclude the title from the allowed bounds (which I did), and turn off resizing?

Quote:Another major problem was with multiple monitors, apparently the cursor was bound to some rectangle other than the screen rectangle on other monitors, which was bizarre.

That to me really sounds like a bug in your code…

Quote:Then in full screen there were numerous other problems.

And here, I can't think of anything that would be any different at all.
Quote this message in a reply
Moderator
Posts: 3,572
Joined: 2003.06
Post: #5
> Why would you use a Wacom tablet in a game?

They come with a mouse in addition to the stylus.

> Meaning what? You can easily exclude the title from the allowed bounds (which I did), and turn off resizing?

The Wacom mouse goes right through to the title. I was referring to when a window has resize turned on.

> That to me really sounds like a bug in your code…

Sorry it sounds that way to you, but it was either buggy Wacom drivers or buggy OS working with the Wacom. The rectangle outside of the main monitor was even inverted in the y (vertical axis) where the *system* cursor would vertically move opposite, which had zip to do with my code. It was bloody awful, so again, better solution was to constrain the Wacom and warp to center of the view to avoid nastiness.

> And here, I can't think of anything that would be any different at all.

???
Quote this message in a reply
⌘-R in Chief
Posts: 1,254
Joined: 2002.05
Post: #6
Ah, right. Wacom mouse. (What's the advantage of that?)

Unless all of those bugs are only with the Wacom mouse, I honestly don't understand how they could exist. The code is trivial and if that code is somehow buggy in the ways you suggest, then the OS would be seriously broken. (iow, if the OS lying about which rect is which, then that's ridiculous. I can believe that a 2007 Wacom driver didn't calculate something properly though.)

We'd need test cases to show it one way or the other, now.
Quote this message in a reply
⌘-R in Chief
Posts: 1,254
Joined: 2002.05
Post: #7
(Jun 11, 2012 07:45 AM)OneSadCookie Wrote:  You can get the acceleration curves from IOKit, but it's not immediately obvious to me how to use them.

So I found IOHIDGetMouseAcceleration() and IOHIDSetMouseAcceleration(). That's a value between 0 and 4 which I'm guessing is part of the power in the curve.

I've found functions and property names such as NXGetMouseScaling and kIOHIDPointerAccelerationTableKey, but I'm not yet sure how to use them. Still looking.

It looks like I really want to somehow end up calling

Code:
void IOHIPointing::scalePointer(int * dxp, int * dyp).
// Description:    Perform pointer acceleration computations here.
//        Given the resolution, dx, dy, and time, compute the velocity
//        of the pointer over a Manhatten distance in inches/second.
//        Using this velocity, do a lookup in the pointerScaling table
//        to select a scaling factor. Scale dx and dy up as appropriate.

If I can do that, I'm golden?
Quote this message in a reply
Moderator
Posts: 3,572
Joined: 2003.06
Post: #8
(Jun 11, 2012 10:43 AM)SethWillits Wrote:  Ah, right. Wacom mouse. (What's the advantage of that?)

The tablet sits next to your keyboard where you would normally have a mouse pad. When you're not using the stylus, you put it down and you use the Wacom mouse instead, so you don't have to move the tablet out of the way every time you want to switch to mouse control. To the user, it operates just like a normal mouse, except that it sits on the tablet and uses the tablet wireless.

(Jun 11, 2012 10:43 AM)SethWillits Wrote:  Unless all of those bugs are only with the Wacom mouse, I honestly don't understand how they could exist. The code is trivial and if that code is somehow buggy in the ways you suggest, then the OS would be seriously broken. (iow, if the OS lying about which rect is which, then that's ridiculous. I can believe that a 2007 Wacom driver didn't calculate something properly though.)

I have no idea if it is/was the Wacom and/or the OS. It seems like it should be trivial, but when working with the Wacom, my experience was that it was definitely not trivial because of all the bugginess. I can say that I was not particularly impressed with the code in the Wacom API/SDK, so clearly suspicion goes toward them, but I didn't write their drivers and I am not familiar with the Apple API routines needed for such a task, so I am not in a position to lay blame.

The bottom line for me is that Wacom plus mouse cursor is/was buggy when it comes to my needs for games. As I already said, my solution was to constrain the Wacom cursor to a small portion of the active view and warp the cursor to the center and draw the cursor in GL and work off deltas. I don't know if the need for this technique is the same now. I spent a week working and verifying that this was necessary to my own satisfaction back in 2007 and have not bothered reinvestigating the situation since -- heck I felt frustrated to have had to waste that time back then! I don't feel like repeating those test cases right now since my mouse code currently works sufficiently for my needs.

The only solid thing I can suggest is that if you wish to have mouse code that reliably works with the widest variety of user configurations, the Wacom tablets may be a fly in the ointment to be aware of.

(Jun 11, 2012 10:43 AM)SethWillits Wrote:  We'd need test cases to show it one way or the other, now.

If you need test cases and this is something which you feel is important, the best thing to do would be to go buy a Wacom tablet and test for yourself. I found it to be a miserable experience with unusual bugs hiding in unexpected places, which is why I gave up and did the GL mouse draw thing, but YMMV.

Also, I seem to recall there is some Wacom specific bug workarounds in the SDL codebase which you might try digging around for.
Quote this message in a reply
Moderator
Posts: 3,572
Joined: 2003.06
Post: #9
Curiosity got the better of me after reflecting on this a bit, so I dug out the Wacom and plugged it in. I tested it with your test project and it does indeed appear to work correctly, at least with only one display plugged in. I didn't beat it up with any extensive testing, such as multiple display configurations, resolutions, other CG mouse calls, full screen, etc. etc.

Obviously Wacom and/or Apple addressed at least some issues with the Wacom since I last tested, which means I am going to have to revisit all that miserable testing when I get a chance sometime. Annoyed
Quote this message in a reply
⌘-R in Chief
Posts: 1,254
Joined: 2002.05
Post: #10
Heh. Yeah, I don't have a second display handy at the moment to test with, but it should work as is with a normal mouse at least.

I spent a few hours this morning trying to figure out the acceleration curve stuff and I gave up. There doesn't appear to be any public way to call IOHIPointing::scalePointer() and although I figured out how to get the curve table I'd have to basically copy all of the code called by scalePointer() and that seems like a bad idea.

So I think, if the cursor does get locked to the window using the second method in the demo project, it'll simply have to have a linear sensitivity scalar and call it macaroni. There is the possibility that I'm mistaken and the deltas themselves aren't linear 1-to-1 to hardware movements, and are in fact actually accelerated as well, but do need to be scaled down. I'll have to explore that later.
Quote this message in a reply
Post Reply 

Possibly Related Threads...
Thread: Author Replies: Views: Last Post
  Cursors in Cocoa ededed 4 3,972 Aug 11, 2002 07:11 AM
Last Post: ededed