GL_LINE_SMOOTH looks ugly

[Vertizor]

I'm trying to apply anti-aliasing to my GL_LINES but it comes out looking ugly. By that I mean the lines just look thicker not smoother. The extra pixels added to buffer the jagged edges have no alpha to them so they're completely solid and thus effectively makes my lines look thicker yet still aliased.

My test program is written in Cocoa, I'm using a subclassed NSOpenGLView which I use Interface Builder to set up:
Renderer: Hardware
Color/Alpha buffer: 8888 ARGB
Depth buffer: 24 bit
Stencil buffer: 8 bit
Accum. buffer: 8888 ARGB

I've also tried it with Stencil and Accum buffers set to none.

I'm hoping someone just knows of a sure fire way to get nice anti-aliasing setup, better than what I'm doing now:

glHint (GL_LINE_SMOOTH_HINT, GL_NICEST);
glEnable (GL_LINE_SMOOTH);

I can post a screenshot of what it looks like now if it helps.
EDIT: Heh, ok so I don't know how to take a screenshot of a GL window. Grab.app won't work.

[CarbonX]

Take a look at this, particularly the part about GL_LINE_SMOOTH_HINT.

OpenGL blue book (glHint) (http://rush3d.com/reference/opengl-bluebook-1.0/ch05.html#id5495369)

[skyhawk]

use glAArg

[arekkusu]

First, which renderer is "Hardware"? GL_*_SMOOTH doesn't work on some hardware.

Second, you need to enable blending to get smoothing to work.

Third, GL window screenshots are broken (http://www.idevgames.com/forum/showthread.php?t=6581) but you can get a screencapture of the entire screen and crop it in Photoshop. Or you can add code (http://www.idevgames.com/forum/showthread.php?t=5750) to copy the window to Quartz when you minimize, after which you can get a screenshot properly (big PITA though.)

[Vertizor]

Take a look at this, particularly the part about GL_LINE_SMOOTH_HINT.

OpenGL blue book (glHint) (http://rush3d.com/reference/opengl-bluebook-1.0/ch05.html#id5495369)
I should have also mentioned that I did try GL_FASTEST instead of GL_NICEST. Both give me the same results unfortunately.

[Vertizor]

use glAArg
A quick Google search and some reading puts me into this conclusion: glAArg is a band aid fix, doesn't look pretty. I appreciate your input none the less. It's just not one I'd prefer.

[arekkusu]

See also this source code (http://homepage.mac.com/arekkusu/bugs/glLineWidth.dmg) for a working example of GL_LINE_SMOOTH (on hardware that supports it, anyhow.)

[Vertizor]

First, which renderer is "Hardware"? GL_*_SMOOTH doesn't work on some hardware.
I'd like to know myself, IB simply says "Hardware." Twist my arm for a little more effort and I can tell you that OpenGL Profiler tells me:

Launch Arguments:
GL Context ID: <hex value>
GL_RENDERER: ATI Radeon 9200 OpenGL Engine

Second, you need to enable blending to get smoothing to work.
Looking into that after this post.

Third, GL window screenshots are broken (http://www.idevgames.com/forum/showthread.php?t=6581) but you can get a screencapture of the entire screen and crop it in Photoshop. Or you can add code (http://www.idevgames.com/forum/showthread.php?t=5750) to copy the window to Quartz when you minimize, after which you can get a screenshot properly (big PITA though.)
Thanks for the tip, not too worried about it right now but it'll come in handy someday.


I also read about how to use the GL_MULTISAMPLE_ARB extension for fullscreen AA, looking at sample code though, it appears I can narrow it to just my 2D primatives. But using this extension is a bit more work, which I'll get to eventually. Any input on the GL_MULTISAMPLE_ARB extension will be appreciated if anyone has past experience with it.

[Vertizor]

Woo Hoo! You were right arekkusu. All that was needed was enabling GL_BLEND and picking the right blend function. For anyone who cares, I found this to be the best setting for points and lines:

glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
glEnable(GL_BLEND);

Oddly enough I found this while Googling and it lead me to a Microsoft webpage (http://msdn.microsoft.com/library/default.asp?url=/library/en-us/opengl/glfunc01_4vs3.asp) (sorry Mac fans - that's all of us isn't it? *sheepish grin*)

[arekkusu]

Lines should look OK on 9200 (with the "wrong spec" behavior (http://homepage.mac.com/arekkusu/bugs/invariance/HWAA.html)). Points were still broken last I looked (10.3.4, I think.)

Multisampling can't be used selectively because it's busted on most hardware (once you create the context with FSAA, you can't disable it; see note (http://developer.apple.com/qa/qa2001/qa1268.html).) It also introduces a lot of point/line sizing problems (http://homepage.mac.com/arekkusu/bugs/invariance/FSAA.html).

And, we Mac people prefer the platform-agnostic official documentation (http://www.opengl.org/documentation/spec.html).