Polish Voting by Judges

[Carlos Camacho]

I've done my best to gather a range of judges, whom should have a wide range of machines. However, 2 judges, including myself and running G4 Machines at 400MHz. In the event we are unable to launch a game, should the Polish vote be 1, or should the vote simply be removed? Your input is welcomed.

My thoughts is, I'd rather see a message pop up that says, "Sorry, but this game has detected you are running a G3. The minimum requirements for the game are
a G4." And then exit gracefully. Much better than a black screen, and then nothing.

Another example is a game that does not scale well to lower-end machines. If there are settings for, say resolution, textures on/off, etc to help squeeze some extra speed for the game, thereby making it playable, then that would seem like something to reward a dev (in the Polish category.)

Anyhow, the main question -- what to do with games that simply won't run on the Judges machine. (I have suggested a first step to them which is to test said game on a friend's machine, or co-workers.)

[JustinFic]

I think a good solution would be to first not count the vote, and then have some sort of flat deduction from the final score.

Obviously if a game doesn't run on a judge's computer it should be penalized, but depending on the size of the panel, just one score of 1 could knock a game completely out of the Polish category.

If you have 4 judges on a scale from 1 to 5, even one score of 1 means a whole point off your final score. If the score instead doesn't count, and you are deducted a more reasonable amount (.1, .25, .5, depending on the size of the panel) an otherwise well polished game still has a chance if it doesn't run on one computer.

It would suck to see flashbacks of Vectorized (my two favorite entries were penalized enough to where they had absolutely no mathematical chance of winning.)

[igame3d]

How about the judge reads the system requirements and instructions posted with the game download?

[Carlos Camacho]

>How about the judge reads the system requirements and instructions posted with the game >download?
Well, if they did, and the game still doesn't work, then what?

[igame3d]

>How about the judge reads the system requirements and instructions posted with the game >download?
Well, if they did, and the game still doesn't work, then what?

Games should be tested on the judges equipment in advance, and the devs should be informed of issues and machine specs in advance. This issue shouldn't be left to the last minute.

We've had over 5,000 downloads, and about 5 "doesn't run" complaints, do you know how many system specs we've receive with those complaints so that we could trouble shoot the problem? ZERO! Even from devs.
How are we supposed to polish an invisible problem on a random computer?

I don't see in the rules or judging section "games must be playable on 5 year old computers running stock video cards to win"...but I do see that in this thread, not literally but its being hinted at.

Thats like having a Macintosh game dev contest in 1985 and saying "Games must work on Apple II".
For perspective OS 8.5 was released in 1999, same year as the 400mhz G4, but we don't make games that even support OS 9 anymore. Why is hardware any different?

How can a judge honestly review today's games, on the last centuries computers?

Since the standard for the games has been raised, therefore the standard for the minimal equipment of the judges computers should be raised.
Disqualify the judges' computers, not the games.
Respect the work of the developers, not the "appointed" temp job of favored person x.

Think about it, these judges would in effect give a low polish score to HALO, because it won't run on their machines. Would that be a fair vote? Is HALO not polished?
For perspective add to this, TRON 2.0, World of Warcraft, Unreal Tournament 2004, Homeworld 2, Doom III....and just keep rattling off names million dollar games that "ye olde mac" can't play. Guess the collective Billions of dollars they've made means zilch in the eyes of the "Can't play it, give it a low polish score" mentality.

Do the right thing, since you didn't set a minimum system spec at the start of the contest, raise the minimum system spec for judging to the minimum system spec of the games people have made for this contest.

[arekkusu]

How are we supposed to polish an invisible problem on a random computer?
As a developer releasing software into the wild, it is your responsibility to test it on all of your target hardware. That means getting access to a lab, or a lot more beta testing (with a list of "must run" machines, checking off each config as you get testers.)

If that's not possible (and yes, it is hard), there are some things you can do. For example, if you fail to get a GL context, dump a bunch of system info out and alert the user to email it to you.

[geezusfreeek]

Five judges, toss the lowest score and the highest score, average the other three.

(Scale to match total number of judges.)

[igame3d]

As a developer releasing software into the wild, it is your responsibility to test it on all of your target hardware. That means getting access to a lab, or a lot more beta testing (with a list of "must run" machines, checking off each config as you get testers.)

If that's not possible (and yes, it is hard), there are some things you can do. For example, if you fail to get a GL context, dump a bunch of system info out and alert the user to email it to you.

Where is the list of "must run"machines for this contest?
Where is the "target hardware"?
Doesn't exist, does it?

Lets make a list based on Carlos' 400mhz G4.
G4/400 (Sawtooth) introduced 1999.10.13
G4/450 introduced 1999.08.31
G4/500 introduced 1999
G4/450 MP introduced 2000.07.19
G4/500 MP introduced 2000.07.19
G4/466 introduced 2001.01.09
G4/533 introduced 2001.01.09
G4/533 dual introduced 2001.01.09
G4/667 available 2001.02
G4/733 available 2001.02
G4/733 introduced 2001.07.18
G4/867 introduced 2001.07.18
G4/800 dual announced 2001.07.18
G4/800 introduced 2002.01.28
G4/933 introduced 2002.01.28
G4/1 GHz dual introduced 2002.01.28
G4/867 MGHz GeForce4 MX introduced 2002.08.13
G4/1.0 GHz dual Radeon 9000 Pro introduced 2002.08.13
G4/1.25 GHz dual Radeon 9000 Pro introduced 2002.08.13
G4/1.0 GHz GeForce4x introduced 2003.01.28
G4/1.25 GHz dual Radeon 9000 introduced 2003.01.28
G4/1.42 GHz dual Radeon 9000 introduced 2003.01.28.13
G4/1.25 GHz single 256/80/Combo introduced 2003.06.23
G4/1.25 GHz dual 256/80/Combo introduced 2003.06.23
1.6 GHz announced 2003.06.23
1.8 GHz announced 2003.06.23
1.8 GHz single introduced 2003.10.19
1.8 GHz dual 512/160 introduced 2003.11.18
2.0 GHz dual 512/160 announced 2003.06.23
1.8 GHz dual 256/80 introduced 2004.06.09
2.0 GHz dual 512/160 introduced 2004.06.09
2.5 GHz dual 512/160 introduced 2004.06.09

Thats 32 different TOWER configurations between 1999 and 2004.
That does not include, imacs, ibooks, emacs, and powerbooks. Or any of the G3 line.

We've personally and succesfully tested this software on 6 of our personal Macs and 2 pc's. One of those macs, is 50mhz below the minimum system spec, and the game ran fine when i followed my own instructions of changing resolution. Why it doesn't work on random machine X which is above spec is still unknown.

If over 5,000 downloads isn't enough "beta testing" I don't know what is.
If fellow developers can't provide adequate system info when reporting a bug, the 1 out of 1,000 regular users who reports a bug obviously can't be expected to do much better.

If "polish" score boils down to 5 macs, 2 of which are 31 revisions from "modern", then the host of the contest should provide the system specs for those 5 Macs as "targets", before people spend time coding or are in a crunch to rewire their work to fit the last minute situation ambush.

[arekkusu]

Where is the list of "must run"machines for this contest?
Where is the "target hardware"?
Those are two different questions, and I was only responding to the second one. The target hardware is determined individually by each developer according to their game requirements and should be specified in their readme.

The first question is open, I make no attempt to answer it. I agree with you though that the judge's machine specs should be make public in advance. It certainly would have changed my test plan (and likely, score results) in the Vectorized contest if I had known in advance that Brad Oliver would be testing with a dual-card-dual-monitor setup.

[PowerMacX]

As a developer releasing software into the wild, it is your responsibility to test it on all of your target hardware. That means getting access to a lab, or a lot more beta testing (with a list of "must run" machines, checking off each config as you get testers.)

Personally, I've had a hard time getting any feedback at all from over one thousand downloads. Not getting bug reports doesn't mean that the game worked fine for everyone, just that I won't know about it = won't know it needs fixing!

I tried my best to make sure my entry ran even on a Rage 128 Pro (16MB), but that's the bare minimum and really on the edge, not only for my entry, but for practically any OpenGL based game.

[skyhawk]

best beta testing, releasing it to the IMG forums and say "hey guys, bitch at me".

seriously, they are great for feedback

[PowerMacX]

My entry's final version (http://www.idevgames.com/forum/showthread.php?t=7339&page=2) is up, feedback from you guys would be really appreciated!
BTW, Carlos, did the Judges vote already? If not, did they download all the entries at the beginning of the contest, a few days ago... yesterday? Also, are they using the old download page or the new system (I haven't updated my info there yet)

[Carlos Camacho]

>BTW, Carlos, did the Judges vote already?
Only 3 have voted thus far. I've held off in asking them during the first week. Since the Poliosh category works the same way as the other categories, understand it isn't the amount of votes that you get. (Just wanted that to be made clear.)

Anyhow,　please come back to a bit of reality. In the start of the contest, I have a list of say 10 people to be final judges. These people say yes. No contract is signed of course. I could post their spec machines, but.....

Now we get to the end of the contest. That list of 10 is down to 3. What can I do? Sue them for being busy? They are after all, people who work fulltime in this business. (Read the EA thread to see what that is like.) Yes, in a perfect world, all 10 judges would have brand new machines, and they would indeed see the contest out until the end. So, that is basically why I don't mention anything about judges during the contest. Lucky for me, I know ahead of time that most likely we will have judges that go AWOL, or whose commercial projects are behind schedule, etc, so I have a backup list of judges who can assist this community. Those people have been contacted and are chipping in. I'm extremely confident that they will give this community their best effort in helping us recognize the best Polished games.

So to rephrase, would it have made a big difference if you got a list of say 30 different machines (all 1st judges and backup judges) at the start of the contest? Although devs usually run fast and latest machines, my guess is that the list of machines would be similiar to a random sampling of people in this community or gamers. Would knowing that Carlos, with his old machine, 'cuz he is too poor to buy a new machine, due to iDevGames ;) have people at the start of the contest saying, "I'd like to have Judge CC removed." Kind of selecting the jury. :D

Cheers

Not sure if I am explaining myself right. I think that arekkusu has a good grasp of it though.

Sorry if that's not really the reply everyone is looking for... a bit tough trying to juggle so much. I do agree for Mini-contests, where the time frame is so short, posting judges Mac specs is do-able. Since all 4 judges that were asked, came through at the end.

Cheers,

[ERaZer]

So, igame3d, are you saying that a game that runs on more computers(but only Macs, since this is a Macintosh game contest) shouldn't get rewarded?

And yes, I think the Mac version of Halo should get knocked down a notch becouse of it's high system requirements and low possibilites to change that(ie, make the graphics look uglier).

[phydeaux]

As a developer releasing software into the wild, it is your responsibility to test it on all of your target hardware. That means getting access to a lab, or a lot more beta testing (with a list of "must run" machines, checking off each config as you get testers.)

If that's not possible (and yes, it is hard), there are some things you can do. For example, if you fail to get a GL context, dump a bunch of system info out and alert the user to email it to you.

I think the problem with the latter part of what you're saying is that at this point, it's too late. Possibly, the judge could send you this information, you fix it, judge runs again, but people really don't have time for this if it's gotten to this point.

For a professional piece of software, yes, it's very sloppy to not test on a huge lab with many different configurations. You can probably rent time out of one of these, though obviously for a contest of this scale, it's not as if any of us are at liberty to do that. Even if the judges post their specs, it doesn't mean any of us might have access to said machine, so that probably won't help (if you didn't even know about the dual-monitor bug, it wouldn't have helped to have been told a judge had a dual-monitor setup.)

I agree with Justin- if a judge that can't run the program, don't consider his vote, but give some deduction per each judge that can't run the program. The amount of penalization is up to you, though one easy soution might be -25% off your total for each failed computer.


From my personal experience for this contest, by the way, I've only received a few comments about crashing, and for the majority of them, I've either been ignored when asked for the debug files that would tell me what was wrong, or the person who crashed was anonymous, and I couldn't contact them. So even trying to ask the community for help is no guarantee you will get anything. Again, I believe an entry for a contest of this scale should be graded differently performance-wise than professional software.

[BeyondCloister]

Maybe the best solution is that if a judge cannot run the game due to performance issues then they just do not vote on that game. That way the score for the game is not affected due to the developer being rich and the judge poor.

A side note: no idea where you get the idea that developers have the latest kit - I've never worked in any development job where the computers have been state of the art. As for my home computers, I've only ever twice had a brand new just released model (Amiga 1200 when the magazines still had it classed as only a rumour, and iMac G4 about a month after it was released.) However neither of these computers were top of the range models.

[aarku]

Am I the only one who read the thread title to be Polish (people from Poland) Voting by Judges? :blink:

That aside, I think at the minimum a handful of developers spent a great deal of time testing their games on as many different Macs as they could. I personally spent a great deal of time getting things running on an old Rage 128, and trying to figure out why sometimes fullscreen would switch resolutions on a handful of people's machines. I also spent time testing other entries on really weird systems to help them out with compatibility. (Beige 266mhz G3 Desktop with Mac OS X 10.2.0 and a Radeon 7500 PCI, anyone?)

It may be impossible for some people's games to run on old systems, however. I do not know how those should be judged, but I just wanted to point out that it seems strange that developers who took time away from adding bells and whistles to their game to work on compatibility get treated the same in the polish category as an entry that focused on said bells and whistles. Does that make sense?

-Jon

[igame3d]

The best solution was presented on the judging page
The judges will use the following criteria to guide them:
The game was straight-forward to install.
The game's icons were graphicly pleasing and visible.
The game's introduction or splash screen was presented professionally.
The game's menu and setup screens/windows were intuative.
The game felt like it belonged on a Mac (it uses established conventions for Macintosh GUI applications).
The game ran smoothly.
The game quit gracefully and caused no problems afterwards

7 items worth 0.7142857142857143 points for each YES answer, left up to the hundreds of people who voted.

This would have avoided this entire last minute fiasco.

Also sending feedback to the dev through the voting page would have helped, sending them feedback 120 days after the contest starts is fairly useless, many devs will not
continue to work on their games after contest. So what good is that delayed feeback?

Also my suggestion at the beginning of this contest for feedback on the udevgames site, intstead of splitting things up, would have benefitted the developers. Its ludicrous to think people will sign up at a game developer site just to post feedback when they are not interested in game development themselves.

There is no link to iDevGames from the download page.
There is no link to iDevGames from the voting page.
There is no link to iDevGames from the udevgames home page.
etc etc etc etc
How are people downloading these games going to know where to send feedback?
The iDevGames read me? About .0001% of downloaders will bother to read the game read me, .0001% of those will bother to read the iDevGames read me.


best beta testing, releasing it to the IMG forums and say "hey guys, bitch at me". seriously, they are great for feedback
I think the Gokuban thread would prove you wrong. Request for feedback posted 10-28-04, total replies= 5, total devs who participated in feedback=1.
Also feedback from idevgames does not guarantee that the game won't run on some judges machine. Do you see a crash report in the "Dead Days" thread? No you don't.
It took over 3,000 public downloads to get the first crash report.

Also udevgames competitors can't be trusted:
observe this comment left at a game site.
Sucks big time
If all (32 competitors x number team members), left comments like that around the web about each UDG entry...you know what this contest would look like to the eys of the public?

They are after all, people who work fulltime
And what about the people who participate in this contest?
Do you see them all as jobless slobs?

Rerturning to arekkusu's comments about renting a lab, 98% of us don't live within pissing distance of Apple computer. The labs don't exist for the majority of the devs.
Nor is the expense of renting one even feasible in this contest. Nor is the ability for many people to tell their bosses : "Hey I'm taking a day off work to test my freeware"
Or their school "My education is not as important as this contest".
I can see Tobi telling the old people he takes care of "Fend for yourself feeble ones, I've got beta testing to do!"

If one is releasing shareware or commerical product, you get ca$h money from each person who pays for the software when it works on their machine, and people who want to pay for the software send in information regarding problems because they have a need for the software.
Your suggestions are really your own "high ideals" and not reality, because, even Apple can't release software that works on all its computers, and they have hundreds of millions of dollars in resources, and the man power. This can easily be illustrated if you read macfixit, and Apple's support pages. The most recent 10.3.6 update is causing trouble with hundreds of thousands of dollars worth of hard drive equipment and openGL slowdowns from 30fps to 1fps on Blizzard games, more issues are likely to develop since the upgrade is only 7 days old.

Sorry if that's not really the reply everyone is looking for

Likewise

[MattDiamond]

I mostly agree with igame3d's points; our resources are limited and we don't always get the feedback we need. It's kind of annoying to be told otherwise.

But here's how I look at it: being judged is always a crapshoot. Good judges try to be fair but ultimately it's a subjective process. If my game runs on 99.9% of the machines out there, but it happens to crash on one of the judge's machines because I never was able to test on that particular setup, then that's just bad luck. I'd better get over it. I can improve my odds by putting up betas early, sticking to lowest-common-denominator OpenGL functionality, etc. But nothing is guaranteed.

In my opinion if a game advertises itself as not meeting a judge's specs then the judge probably shouldn't vote on that one. If it doesn't have minimum specs written out, or it says that it should run on their machine, but it runs badly or crashes, then polish will be very low from that judge. It might not even be the developer's fault, it might be something very strange about that judge's computer, or due to a random electrical surge. But that's just life! When a skater falls down in the Olympics their scores are low, even if it was because their laces broke.

This isn't just a theoretical topic for me; I had a bug that rendered an important part of my game invisible on some machines. I managed to fix it a few days later but I'm still not sure what the problem was, never reproduced it on my machine. For all I know the judges all downloaded the version with the bug. Oh well!

Just my opinion.

[Zwilnik]

As far as the testing vs resources situation is concerned, uDev is like a scrunched version of a typical project. There's no way you'll be able to test on all the unusual frankemacs out there and there's bound to be an issue on one or two of them after you release a game, no matter how much you test.

As far as my judging went, the only project that I couldn't play was one that wasn't on the downloads list, the rest all at least started up and showed screens. I'd like to have waited until the last possible second to test them, but there's this great big "things to do and bugs from earlier releases to fix" list I'm working on ;)

[Carlos Camacho]

>Am I the only one who read the thread title to be Polish (people from Poland) Voting by >Judges?
Actually, a distinguished judge replied to me, "I wonder why Carlos would think I was from Poland." So, you are not alone.

>Do you see them all as jobless slobs?
No, not what I meant.

Anyways, here is what I shall go with. I've asked judges to report any games that they were not able to test due to (a) download being offline (b) their machine not meeting required specs (c) game not running at all. I will take their list of untested games, and remove their votes. Thus, they only vote on games that they were able to properly fully test.

[PowerMacX]

I know this is probably asking too much but, assuming the Judges didn't erase the games after downloading & testing them, would it be possible to know which version/release they judged? (At least for games in which the version is either part of the compressed download filename or in plain sight in the ReadMe)

...or, just when did they download the games, so every developer can figure out the version by themselves...

[Carlos Camacho]

>I know this is probably asking too much but
A bit too much for this year's rushed contest. But next year, I will leave a space next to each entry so they can enter in the version # reviewed. Good suggestion. Remind me in a few months!

[PowerMacX]

Remind me in a few months!
Will do :)