same deal as last year

[skyhawk]

all stats hosted on my site

http://hkasoftware.com/skyhawk/udg2003/

if anyone else would like to donate more stats or list, I would be glad to host them

[NCarter]

Thanks, skyhawk. I was going to do the same myself, but you've saved me a lot of trouble! :)

[Jake]

Thanks!!!

[DaFalcon]

Yes, thanks Skyhawk :-)

In case anyone can do some statistical analysis in Excel (correlation between the various categories and overall score, for example) I've dropped it all into a spreadsheet at work:

http://www.snakequest.com/other/uDevGame03.xls

What is cool to see here is the graph showing all of the games' scores broken down by category. You can see just how tight the competition to get into the top 5 is, and how every little bit of polish, every great sound, every wonderful graphic could be the thing that pushes you up in the ranks!

[KittyMac]

Thanks Skyhawk!

I was wondering what place PI came in the Sound & Music category. My musician will be happy to hear that we came in 6th.

Cheers,
Rocco

[Carlos Camacho]

Thanks for helping everyone. You still have 2002? I thought the link might be broken.

In related news. I plan to set up an independent website for the contest, so that it will have a permanent home, and thus you will all be able to look back on the past year's contests.

And yes, I am interested in getting some thought-provoking analysis of what this all means like we got in 2002.

Cheers,

[Carlos Camacho]

Ah, I see you have /2002/ up as well.

Cool.

Argo. scored higher than Kiki from last year in overall. I'll like to know if this year's entries had a higher polish than last year. :)

Gamep play score fro 2002 and 2003 were the same :)

Cheers,

[aarku]

Has anyone averaged all the games categorically and compare those numbers to last year and the year before? It'd be interesting to see how those numbers are changing.

-Jon

[Holmes]

Originally posted by Camacho
Ah, I see you have /2002/ up as well.

Cool.

Argo. scored higher than Kiki from last year in overall. I'll like to know if this year's entries had a higher polish than last year. :)

Gamep play score fro 2002 and 2003 were the same :)

Cheers,

In fact several games from this year scored higher than Kiki overall including Chopper, primate plunge, and Tower of Tears. Lightning's Shadow and Yoink we also close. Congratulations to these devs. I was shocked to see this! So many high quality games! (or voters stuffed full of sugar and morphine?)

[skyhawk]

maybe that means this year people weren't as stern and hardcore about their votes. maybe they were more willing to throw all 5s around

[DaFalcon]

[Mandatory Disclaimer from someone who took too many stats classes in college for business: Please note that this makes the assumption that scores across years can be compared. This is not necessarily true because different votes may have voted (with different preferences towards higher or lower scores) and the same voters may have changed their voting habits. Even so, it is interesting and does tell a good story when taken with a grain of salt]

Well, comparing 2003 to 2002, there has been improvement in every category, most notably in sound. A look at the numbers:


Gameplay Graphics Orig. Polish Sound
Avg. 2002 Score: 2.89 2.94 3.00 2.85 2.53
Avg. 2003 Score: 2.91 3.03 3.01 2.88 2.73

% Improvement: 1% 3% 0% 1% 8%

This makes sense because sound was the category that most obviously needed improvement, so many developers found people to make music for them, or they at least put some thought into the sound effects used.

Overall, scores improved 3% from an average of 14.2 to 14.56.

Now, this is all looking at averages. We can also see that the bar was raised in 2003 for the best overall games:

-=2002=-
Kiki the Nanobot 19.59
The Belt 19.5
Galder 19.46
MAFFia 19.31
Black Shades 19

-=2003=-
Argonaut - 2149 20.1
Chopper 19.91
Primate's Plunge 19.8
Tower of Tears 19.77
Lightning's Shadow 19.4

These top 5 games show a 2%-3% increase year-to-year.

So is this 3% "inflation" a product of voters' changing voting patterns? I say no, it is a reflection of the overall increased quality of games this year over last, slight as it is. So good job everyone, not only this year, but last year too.

[geezusfreeek]

[Edit: DaFalcon has made a much better analysis farther down the page. Don't even bother with mine. :p ]

I decided I would see how closely each category was was associated with the overall score. That is, how much does each category deviate from the overall score? Here are the steps I took:

A) Instead of using the sum of the categories for the overall score, I used the average.

B) I calculated the sums of the scores in each category:
Overall: 125.18
Gameplay: 125.13
Graphics: 130.09
Sound: 117.26
Polish: 123.90
Originality: 129.54

C) I got the absolute value of the difference of each category and the overall score.
Gameplay: 0.05
Graphics: 4.91
Sound: 7.92
Polish: 1.28
Originality: 4.36

These numbers are rather small and close together. What does this mean? It means that high ratings in any one category don't necessarily mean high ratings overall. There were theories that the polish category was just the sum of the other categories, thus it probably reflected the overall score the most. This appears to be untrue.

Please correct me if my logic is off track.

[DaFalcon]

There are actual formulas for finding how closely two variables are related, but alas that knowledge slipped right out of my ears the minute my final final in stats ended.... It is probably something simple, too. Even then, you must remember that correlation doesn't necessary imply causation.

Maybe I should go to sleep so I can wake up and speak normal human english again :-)

[Carlos Camacho]

You guys amaze me.

I'm curious to know if devs from 2002 did better or not.

[DaFalcon]

Okay I calculated the correlation coefficient of each of these variables (okay, so Excel did it for me). [If you want to read about cc, read this: http://coastsociology.org/204/204_corr.htm ]

That is the relationship between variables... If a higher polish always happened with a lower sound score, that would be a negative relationship. Here, however, are only positive relationships. This is as expected. The scale here is -1 to +1, with +1 meaning that every time one goes up a certain amount, the other goes up by a certain (different) amount [the values could be graphed on a straight line and every point would fall on the line].

NOTE!!! "A correlation between two variables does not necessarily mean that one variable causes another to change. It just means that as one variable changes the other changes also."

So even though the relationship between graphics and polish is 0.91, it could just mean that teams that are good at creating graphics are also good at polishing the game. BUT it COULD mean that voters are saying, "oh, this game has nice graphics, it is really polished". We may never know for sure :-)

Here are the relationships:
Gameplay Graphics Original Polish
Gameplay x
Graphics 0.773 x
Original 0.517 0.389 x
Polish 0.892 0.917 0.451 x
Sound 0.829 0.809 0.501 0.895
As you can see, the strongest relationship is between graphics and polish. Sound also seems to be related fairly strongly to everything except originality, which is not strongly related to anything (relatively speaking). Polish was also strongly related to gameplay. Graphics and gameplay also were related.

Here are the relationships between each category and the overall score for the game:
Gameplay 0.923
Graphics 0.902
Original 0.637
Polish 0.960
Sound 0.928
The strongest relationship here is easily between polish and the overall score. In other words, increase your polish and you may end up with a higher overall score. (Or, it could be that polish means "good in every category" to voters, and that you can't do specific things to increase your polish score) This makes sense because polish was closely related to almost every category.

Interestingly, originality does not really influence the overall score nearly as much as any other category. So you can create a clone but do it REALLY well and still get a high overall score. I checked and for the top 5 games, originality made up only 15-19% of the overall score (it was 15% of argonaut). In "lower ranked" games, originality made up a much higher % of the score, as much as 30% of it (Pantheon).

So there you have some statistical analysis of 2003 from Alan. Have fun :-)

[geezusfreeek]

Ah, thank you! I was dying for a better analysis than mine. No sarcasm.:D

And the results are much more interesting than my own. Especially since the correct way showed that polish is in fact very closely related to the overall score.

[geezusfreeek]

Hmm... after reading that link you posted, I saw that squaring the correlation coefficient would give the percentage of variation. I don't really understand why you have to square it. The values look like percentages already, but, trusting what that page said, I did this so I could more easily understand the results.

Relationships Between Each Category:
Gameplay Graphics Original Polish
Gameplay x
Graphics 59.8% x
Original 26.7% 15.1% x
Polish 79.6% 84.1% 20.3% x
Sound 68.7% 65.4% 25.1% 80.1%

Relationships Between Categories and Overall
Gameplay 85.2%
Graphics 81.4%
Original 40.6%
Polish 92.2%
Sound 86.1%

Same results as yours (of course... I used yours). I just find these easier to interpret.

[DaFalcon]

Yes, you are right... percentages are *much* easier to interpret. So if your polish was ALL 5's across the board, there is a 92% chance that your overall score would also be a 5. Of course, good luck getting everyone to give you a perfect polish score :-) Anyway, I believe it -- if a game is good enough that *everyone* thought it deserved a 5 in polish, then every other category (with the possible exception of originality) is probably also that good. Thanks, geezusfreeek :-)

[codemattic]

shocking that originality counts for so little - and that sound counts more than graphics or even gameplay. Go fig.

[DaFalcon]

Originally posted by codemattic
shocking that originality counts for so little - and that sound counts more than graphics or even gameplay. Go fig.

I think people are misinterpreting these numbers :-)

This does not say that to get your overall score we multiply originality by 0.4, you sound by 0.86, etc....

All it means is that if you look at two games with an overall rating of 18, one may have a sound score 24% higher than another game with an overall rating of 18.
So with originality, all this tells us is that knowing a person's originality score does little to help us predict the overall score in the contest. Whereas knowing a person's polish score tells us that his overall score is likely within +/- 8% of a certain overall score.

These numbers are interesting, yes. But maybe not as useful as some of us want them to be :)

All values count exactly the same -- 20% -- when computing the overall score!