Raw Voting Data Now Available (ie da Feedback)

[Carlos Camacho]

I have uploaded, in CSV format, everyone's Public & Peer Votes. The two Peer files will show you how your fellow devs thought of your game. The Public votes contains the feedback of public, for those that left feedback -- minus their name, IP, and email. Please note I left their scores for you to glance at when reviewing their feedback. The FINAL scores which we post are the OFFICIAL results, so don't have a cow if you notice differences in your RAW data vs the Auditor's data.

The Public files also contain the demographics info. This is the info that I hope people will be able to analyze to give us a good perspective of our voting public.

http://www.idevgames.com/files/CSVData.zip

Just for fun, if you'ld like to post some of the memorable feedback messages from your voters, feel free in this thread.

One last thing... As mentioned above, the OFFICIAL results are still with the Auditors, along with the Judges Results. I will ask all of you NOT to discuss who won what so as not to spoil our Award's Show.

Anyhow, I am posting the data now, because everyone did a fine job.... and it is one thing now off my list of to-dos (just 88 more to go!)

Cheers,

[MattDiamond]

This is great! Carlos, the availability of these scores and comments, survey details etc is a big step forward for the uDG contest. Thank you. (I assume we'll see the polish scores later on... I also hope we'll see peer's comments at some point, not just their scores.)

I recommend all developers, particularly the ones who weren't finalists, to look at their scores. Without these it's just a win/lose contest. These let you see that some people did appreciate your game, and in my case most of the feedback was either useful or at least an opinion I could respect even if I disagreed. Some of them are things I'd fixed by contest end, and in fact my scores went up in the second week showing that maybe I was on the right track.

Highlights from my comments:
A couple people thought it wasn't fun. Fair enough. A few liked it quite a bit. "My mom is adicted(sic) to games like this. I'll pass it along." One person points out that it's only good for English-speaking players; I had resigned myself to that when I started, but the comment was deserved.

Best comment:

I respect all the hard work that goes into these games. This is meant to be constructive criticism.
First impression: Word Search. Boring.
Good: It's not just word search! Nice zen style puzzle game. Game play style is great. Music fits relaxed style of play.
Bad: Straight line should be default not freeform. Who wants to worry about mouse accuracy? Cover (title screen) does NOT fit style of play. Yes, it looks like a cheesy word search book cover, but... that's not really what this is. The simple understated play screen is what it's all about. The timer for the timed play is either too fast or needs an audible warning as it gets near the end. No options for size of board and no added bonuses etc make short term goals a little simplistic for continued play. What about having a slow timer that that triggers the letters falling into the blanks. In other words you can rack up a multiplier if you get many words before the other letters fall.
Overall: Simple but great. One of my favorites.

This person totally understood my game, including my cover page "theme", took it all apart and gave useful feedback. Wow.

Also got a good comment from someone about adding more polish to the game using Apple technologies like Rendezvous, OpenGL, transparency. I'm wondering if it was Rocco actually, since he breathes that stuff and knows better than most people what it can add to a game. Regardless of who it was, much appreciated.

Worst comment:

It is really annoying because the columns move down at different rates there is no skill to the game it is just luck

There's plenty of skill in the game, *particularly* in managing the columns. Gave me my lowest scores too. Oh well.

[Zwilnik]

Ah the fun of reviews. I rather liked Word Search but unfortunately, no matter how good the game, there'll always be someone who just "doesn't get it".

[skyhawk]

Ah the fun of reviews. I rather liked Word Search but unfortunately, no matter how good the game, there'll always be someone who just "doesn't get it".
are you kidding, some people just didn't get that you didn't need to dodge every bullet on screen in Icarus... I think it is not a fault of the game as much as the gamer

[BeyondCloister]

Impossible to figure out. Even the quick start guide didn't work for me.
Well as I suspected a lot of people did not get Industrial Revolution.

However I'm not at all bothered as I was making it for those who did such as:

Awesome game! Best strategy game I've seen, and a good historical recreation of the factories of the early industrial rev. Very original concept. One of the most complex games in udev 2004. I love the NWN-like UI. Congratulations!
It makes it all worth while.

Very nice, addictive game, somewhat educational,
Now if only Carlos was stilling going for a most educational game special award.

All I need to decide now is whether to concentrate on finishing the game, working on the board game version or playing with Torque Engine.

[willThimbleby]

It is amusing the range of views you get, you can definitely tell who is having a bad day. One comment from someone who gave me all 1s
Those sounds hurt, the graphics are "retro" but not polished at all. And what is with the physics? It is cool but it hurts.
Ah well. The next comment in the file was for all 5s
I just love the retro feel of this. Reminds me of some of the games around when I was younger :)
You just can't please everyone, can you. Sadly most of my comments are people grumbling about little things. But it is great having the feedback.

Most fun comment:I'm going to delete it from the computer.... No, wait, just another race....

[aarku]

I compiled the number of votes for each game. The average was 59 votes.

Here is the html:
http://www.tc.umn.edu/~czec0035/udevgames/uDG2004-number-of-votes.htm

Here is the csv
http://www.tc.umn.edu/~czec0035/udevgames/uDG2004-number-of-votes.csv

Cheers
-Jon

[Carlos Camacho]

Thank youYou are welcome. I hope everyone can use their data to progress as developers.

also hope we'll see peer's comments at some point, not just their scores.
Sadly, you won't because Peer voting didn't inlcude a feedback field. I had figured that with the forum everyone was already getting their feedback from peers. And yes, you will get the Polish data, as long as everyone promises not to stalk the Judges if they are not happy with the scores they got.

These let you see that some people did appreciate your gameIndeed. Some games that didn't win were very much liked.. even from the Judges from what they said.

Matt, I think your game was really well-thought out. With those "Rocco" technologies, and Polish, I REALLY think you had a Freeverse-type of game on your hands. I STRONGLY advise you not to put the game down, but go forward. I'm available for any game asset help.

As for your best Comment. The person, who is also an entrant, gave such in-depth reviews to every game if I recall! Amazing! I was so impressed, that I've asked him to be my special guest on MacRadio.

there'll always be someone who just "doesn't get it".Too true, and feedback like that helps you build a thick skin. Might such feedback from the contest help some of you to reconsider adding/editing to your postmortems?

All I need to decide now is whether to oncentrate on finishing the game
Andrew, like Matt, I think you have a game with much potential. You can stop working on it and play around with learning that RTS tool, and not release anything for some time. Or you can work on the feedback, and looking at all aspects of your game, and turning it into a good shareware title. It too has Freeverse written all over it. Please don't stop working on it! Again, my assistance is also available for game asset help.

Well, I really look forward to seeing analysis of things like "What type of games did people over 30 like the best?", "What is the most sought after 'classic' game?"

It would also be fun to hear what your average was for the field, "What would you pay for this game?"

Cheers,

[NeonSurge]

Hey, Jordan from the NeonSurge team (makers of SolarQuest) wrote a PHP script to parse the CSV files. We threw it up on our server for everyone to see. Please take a look at it, Jordan followed the judging scores instructions the closest he could. Check it out!

http://neonsurge.com/udevgames/

Please note, this doesn't take into account duplicates because of the same IP address, or other things that will be in the official vote. So do not take this as the FINAL vote, just as a estimate. Thanks!

- Andrew and Jordan

[MattDiamond]

Sadly, you won't because Peer voting didn't inlcude a feedback field.
Whoops, my bad. Of course I was thinking of the regular voting comment field.

And yes, you will get the Polish data, as long as everyone promises not to stalk the Judges if they are not happy with the scores they got.

Feel free to anonymize it, especially if the judges weren't universally aware that their comments might be attributed.

Matt, I think your game was really well-thought out. With those "Rocco" technologies, and Polish, I REALLY think you had a Freeverse-type of game on your hands. I STRONGLY advise you not to put the game down, but go forward. I'm available for any game asset help.

It means a great deal to me to hear you say that. I selected that particular game idea because I thought it had some long-term potential, and I am heartened to hear that some of that came out in the not-so-final product. I'm definitely considering further work on it in 2005.

Back to your regularly scheduled topic...

[ThemsAllTook]

Hey, Jordan from the NeonSurge team (makers of SolarQuest) wrote a PHP script to parse the CSV files. We threw it up on our server for everyone to see. Please take a look at it, Jordan followed the judging scores instructions the closest he could. Check it out!

http://neonsurge.com/udevgames/

Please note, this doesn't take into account duplicates because of the same IP address, or other things that will be in the official vote. So do not take this as the FINAL vote, just as a estimate. Thanks!

- Andrew and Jordan

This list is a bit misleading. I stared at it for a long time before realizing that the scores weren't normalized.

It may be just as well, though, as it probably shouldn't even be up:

I will ask all of you NOT to discuss who won what so as not to spoil our Award's Show.

- Alex Diener

[Carlos Camacho]

I will ask all of you NOT to discuss who won what so as not to spoil our Award's Show.
I'm glad somone said that. It's a nice list, but shouldn't be up right now... Especially as it may lead to confusion once the real winners are posted.

Matt, have you played that Mac Scrabble game? It is freeware or shareware. I can't recall. I'm pretty sure it is marketed straight by the dev. Anyways, if you could add that kind of polish to your game, then it would be super. Also, think of a way that foreign dictionaries can be added. Because not only is that nice for native speakers of that language, but you can then tap into people wanting games to learn foreign languages. Cheers.

[JustinFic]

This is the most addicting game I have ever played in my life. Hide it from humanity.

:lol:

Hiring an artist could really have brought this game to a commercial level.

Very much noted. After this contest I now have an unbreakable respect for artists and the assets they create. Drawing sprites was like pulling teeth.

[Carlos Camacho]

This is the most addicting game I have ever played in my life. Hide it from humanity.I saw that myself. Good one. I have a few easter eggs in those comments for a few of you too. ;)

[BeyondCloister]

I forgot to mention, it looks like 5% of the people who downloaded Industrial Revolution voted on it and judging by the feedback and suggested acceptable prices I would be able to hit the 2% registration rate I've got with other platforms in the past .

[DaFalcon]

I am working on a little spreadsheet to get some interesting idea such as "what was the average female score for a given game in each category" and also what did each level of casual to hardcore gamer rank the game. I haven't decided how I am going to break down age yet, so that isn't included. An interesting category might be those English speakers versus non-English speakers (or non native English speakers) ... I will get to see which games had the greatest differential between those two categories, and those developers would probably realize that localization is needed for a game like that in order for it to reach its widest potential audience. Also, my results will only be as good as the data entered: hopefully nobody lied on the survey results, and hopefully there was a random sampling of those who responded and those who did not respond to the survey, to make this a bit more meaningful. Should I post the results once I finish, or should I wait until winners are announced??

[DaFalcon]

Well, here is some interesting information that shouldn't influence anyone's thoughts as to who actually won anything in the contest.

This is the gender information (but keep in mind that for some of these games only one or two "registered females" voted, so this is by no means representative!):

Top 5 games that males liked more than females:

CubicsRube Male 58.54%
BugThug Male 27.28%
WordBeGone Male 17.10%
Ark-Annoyed Male 9.31%
Snowball Male 1.09%

Read the above data as follows: "Male voters gave an average of 58.54% higher score on Cubic's Rube than female voters" or "The average male score was 27.28% higher than the average female score for Bug Thug".

The main reason CubicsRube is at the top of the list is because only one female voted on it, and she gave it all ones.

Interestingly, these were the ONLY games that males awarded higher scores to than females. It seems females either had more forgiving scores in general, or the games produced for this contest simply had more appeal to females who voted in the contest.

The top eleven games females liked more than males, using the same formula for the percentage are as follows:

Starport Female 55.00%
Splock Female 47.50%
RoboRubOut Female 41.71%
Feathered Soccer Female 35.56%
PictureFrameX Female 34.03%
Icarus Female 31.29%
FrogSpawned Female 30.16%
DeadDays Female 26.07%
Nanocrisis Female 25.56%
WildLandsHockey Female 25.12%
ToadMuncher Female 24.74%

As before, the top four games on this list received perfect votes from all females who voted on them, usually just one or two. Nanocrisis also received a "perfect" 5 score from all females, but the male vote was so high that the percentage still dropped it down to ninth on the list.

Games where NO ("registered") FEMALES voted on them:

BitRacer
Bumper Cars
Gokuban
Goldgeier
Industrial Revolution
MadTak
Okugai
Rescue
Shards
ThunderBlast

Obviously, the samples were too small for any of these numbers to mean much. But I still find them interesting anyway.

[DaFalcon]

Well, here is some more of 'da feedback from 'Da Falcon. As before, this is interesting information that shouldn't influence anyone's thoughts as to who actually won anything in the contest.

This is the native English speaker information. The following games had much higher scores (>10%) from non-native English speakers than native english speakers:

Splock (Non) 24.36%
FrogSpawned (Non) 23.40%
ToadMuncher (Non) 22.88%
Goldgeier (Non) 21.64%
Feathered Soccer (Non) 18.49%
OneNeum (Non) 16.25%
DeadDays (Non) 14.86%
WordBeGone (Non) 14.13%
BitRacer (Non) 12.94%
ShineBug (Non) 11.72%

And this next list is a list of games that might have benefitted from localization, or else they just appealed to an English speaking audience better. Or maybe it was random chance, there's always that.

Industrial Revolution (Engl) 28.21%
Okugai (Engl) 26.48%
Nanocrisis (Engl) 20.67%
WildLandsHockey (Engl) 5.90%
Snowball (Engl) 5.52%

In general, non-native English speakers gave higher scores than native English speakers. (So if you appeal to more female and non-native English speakers, will you get more of their generally higher votes? Maybe. Hmnnnnn.)

[DaFalcon]

Well, here is a final analysis of 'da feedback from 'Da Falcon. As always, this is interesting information that shouldn't influence anyone's thoughts as to who actually won anything in the contest.

Now I want to look at how games appealed to casual vs. hardcore gamers. With some exceptions, games fell into a bell curve with scores peaking at one level of "hardcoreness" and falling off around that level. I'm going to look at the top games in each category.

First, here are the overall rankings (including graphics and sound/music)

Here are the top ten rated games by Casual (1) gamers in overall score:

Nanocrisis
Starport*
Icarus
ToadMuncher
ShineBug
Okugai
SolarQuest
OneNeum
Snowball
Splock

*[Starport had some odd numbers due to low voter turnout on this game, and so few of the votes had survey information attached, and those that did gave the game abnormally high scores, compared to average scores for this game -- so you will see it on my lists often. RoboRubOut, PictureFrameX, Goldgeier, and Splock also had some abnormally non-representative numbers for these categories]

Here are the top ten rated games by Semi-casual (2) gamers in overall score:

Starport*
ToadMuncher
Goldgeier
Okugai
ShineBug
BitRacer
Nanocrisis
Feathered Soccer
Rescue
Icarus

Here are the top ten rated games by Semi-hardcore (3) gamers in overall score:

Okugai
Rescue
SolarQuest
ShineBug
MadTak
Splock
RoboRubOut
WildLandsHockey
Snowball
KillDrCote

Here are the top ten rated games by self-proclaimed Hardcore (4) gamers in overall score:

RoboRubOut
Icarus
PictureFrameX
Starport*
ShineBug
OneNeum
SolarQuest
DeadDays
Snowball
Okugai**
Rescue**
WildLandsHockey**

** These three games all tied for 10th place

For this category, I also want to look at the top ten games for each level of "casualidity" ( :p ) in GAMEPLAY ONLY, so here goes!

Here are the top ten games in GAMEPLAY as rated by Casual (1) gamers:

Starport*
PictureFrameX
WaterTower3D
Icarus
Nanocrisis
DeadDays
OneNeum
ToadMuncher
KillDrCote
Snowball

Here are the top ten games in GAMEPLAY as rated by Semi-casual (2) gamers:

Starport*
ToadMuncher
Industrial Revolution
BitRacer
Goldgeier
KillDrCote
Nanocrisis
OneNeum
Feathered Soccer
Snowball

Here are the top ten games in GAMEPLAY as rated by Semi-hardcore (3) gamers:

KillDrCote
WildLandsHockey
Rescue
Snowball
Okugai
SolarQuest
Goldgeier
MadTak
Industrial Revolution
Nanocrisis***
OneNeum***

***Tied for tenth place

Here are the top ten games in GAMEPLAY as rated by self-proclaimed Hardcore (4) gamers:

WildLandsHockey
Icarus
RoboRubOut
Starport*
PictureFrameX
OneNeum
KillDrCote
DeadDays
Okugai
Snowball****
SolarQuest****
ShineBug****

****Tied for tenth place

So, I hope these lists were of some interest to all of you curious cats out there. If you want me to mine for more data, well, I'd have to create some all-new spreadsheets, but if you ask real nice I'll probably do it anyway. I like playing with numbers, even if they aren't the most meaningful numbers.

[Jaisor]

Maybe I am just dense, but most of that info does not make any sense nor point to me! But like I said, maybe its me.

[skyhawk]

makes a ton of sense to me, apparently Icarus appeals to real casual gamers, and real hardcore gamers, but nothing in the middle

[DaFalcon]

It probably isn't just you, Jaisor :-) I added a bit of explanation for what the percentages are supposed to mean for male/female english/non-english scores. As for the top ten games for each level of casual or hardcore gamer, I'm not sure how useful it is, but it should at least be self explanatory. In case it isn't, the survey asked voters to rate themselves as more casual or more hardcore gamers, and my list averages the scores for voters in each category then presents the rankings. I am not presenting the actual numbers to keep these lists from giving any real information about who may or may not have won the contest in any given category, especially because they would probably mislead anyone who took them that way: I am only counting voters who answered the survey, but all votes count for the giving of awards. Does that make sense?

[DaFalcon]

It makes perfect sense, Skyhawk! Your game is all about being black or white, one way or another, all or nothing. So those "semi-casual" and "semi-hardcore" gamers felt you were discriminating against them and voted accordingly. :D

Actually, Skyhawk, your game had appeal to the "middle" gamers too, but relative to the other entries, yes, very casual and very hardcore gamers preferred Icarus.

[Najdorf]

I liked icarus, but I was too much disappointed by its length. I mean, you have fun playing it, but its a 30 second fun!!! "wow, this game is cool! oh... its finished... ****!" That really annoyed me, but i refrained myself from giving you all zeros which I should have :-P

[MattDiamond]

WordBeGone relied rather heavily on the English language in the actual gameplay, yet was rated higher by non-native English speakers? That's weird. But perhaps not since the uDG voting pages are themselves in English so there is an implicit filter on the sample. It's really a "non-native English speakers who understand English well enough to participate in the voting" statistic. Anyway, I'd be careful about concluding anything from the those particular statistics.

[Carlos Camacho]

hopefully nobody lied on the survey results,
I went through all the data, and removed obvious data that was out of place, such as people who posted they were "120" years old, or would pay "$1m" for a game.

Concerning the data, I learned alot about how to gather it this year, and how to improve it for next year. What I will do is NOT place it in the voting process, But rather, AFTER the voting is done, have it set up as it it's own survey, and have all questions mandatory. To be considered for a door prize, you will need to do this survey. But once done, you won't have to do it again and again as with this year. Yeah, I wrote on the survey pages that voters ONLY needed to provide the data once, but some people added it more than once.

One other problem. I noticed that some females would state their name, but leave Gender as Male. This is because the default was set to Male, so it took a tiny effort for them to change the Gender option. I went ahead and fixed that for obvious voters who were females, but when voters didn't supply a name, there was no way to know their gender.

Anyways, this year is a "start." We can't read TOO much into the data, but it is fun. Next year, we will need to do this part of the contest better.

Should I post the results once I finish, or should I wait until winners are announced??
If you could, please try to do it in a report form. Listing stats, as the last poster said, makes some people's heads spin. If you can make summaries, or conclusions, then it is much easier I think.

Cheers!

[skyhawk]

I liked icarus, but I was too much disappointed by its length. I mean, you have fun playing it, but its a 30 second fun!!! "wow, this game is cool! oh... its finished... ****!" That really annoyed me, but i refrained myself from giving you all zeros which I should have :-P
never fear, I have decided (as in there is a 90% probability) that I will for sure make 1 more level of at least 3-5 minutes (minimum) in length. It will take place in tartarus.

[PowerMacX]

Aaron's comments are really well thought! But I think I found a misplaced vote: in Splock, the comment seems to be about Shards?

Well, I took the time to calculate each category averages... but obviously I can't post that, so I'll only say: so close :cry: !

Anyway, while I can't post those scores (they aren't even official results), I can post my own "Category":

If the game you just voted for was distributed as shareware, what would be an appropriate price?
Game.Author,Price
Nanocrisis.Alex_Ni,15.05
Snowball.Aaron_Sullivan,12.34
Okugai.Juan_Garcia,12.23
KillDrCote.Justin_Ficarrotta,11.95
Rescue.Neil_Carter,11.88
DeadDays.Tobias_Opfermann,11.79
BugThug.Jonathan_Czeck,11.58
OneNeum.Rene_Bauer,11.50
Icarus.Brian_Ramagli,11.40
IndustrialRevolution.Andrew_Sage,11.37
BitRacer.Will_Thimbley,11.30
ShineBug.Charles_Goran,11.19
ToadMuncher.Peter_Leahy,11.18
SolarQuest.Andrew_Farley,11.15
Splock.Sean_Monahan,10.73
WaterTower3D,10.58
Ark-Annoyed3D.Mike_Plouffe,10.27
WildLandsHockey.Kevin_Dromereski,10.22
Starport.Justin_Northcott,9.75
MadTak.Joseph_Duchesne,9.57
FeatheredSoccer.Ludovico_Rossi,9.23
WordBeGone.Matt_Diamond,9.06
Goldgeier.Kemal_Yun,9.04
ThunderBlast.Kyle_Leary,8.97
Frogspawned.Iain_Bethune,7.80
PictureFrameX.Alan_Dickson,7.65
RoboRubOut.Kevin_Linderman,7.25
CubicsRube.David_Rzepa,6.72
Shards.Nathan_Goings,5.67
Gokuban.Eric_Hoaglin,5.50
AstroKnights.Stuart_Barth,4.59
BumperCars,1.43

OK, not very useful, but still...

Note: before calculating these averages, I deleted all amounts above and including $40 (otherwise, some games' results would be very unrealistic, since many had "$50", and even "$100" - for a shareware game?). Perhaps I should have used a smaller value?

[Carlos Camacho]

So, is that saying that the public feels most shareware should fall between $10 and $15 ;)

I think the "high" values must have been cast by "parents" or "grandparents" :D

To improve your data, place game genre. And tell us if game genre has any impact on what the public is willing to pay.

Cheers,

[PowerMacX]

So, is that saying that the public feels most shareware should fall between $10 and $15 ;)

Yes, I made a very surprising discovery :lol: !

To improve your data, place game genre. And tell us if game genre has any impact on what the public is willing to pay.

If there aren't going to be any changes to the raw data, I have a lot of *complex* correlation analysis I can make, but since those take time I was going to wait until the data was set in stone... If it is, I'll start right away... well, not right away since it's 4:24 AM here in Argentina and I should get some sleep, but today.

I forgot to ask: I know that for privacy reasons IP, names and email addresses were stripped, but do you have any "arbitrary code" to distinguish votes from the same voter? With that info we could find out interesting stuff like:
* How many games did the average voter rate?
* Did most voters rate only specific genres (only puzzles or only arcades or 3D...)
* Are there any 5-5-5 votes for game X, 1-1-1 for everyone else voters?

[Carlos Camacho]

>to distinguish votes from the same voter?
Indeed I do, but I didn't pull out such info from the voting system. I will at some point if it would be helpful.

Cheers,

[Zekaric]

Something I noticed is that the voter details page mentioned that it only needed to be filled out once, when I took a gander at the results I noticed a lot of my voted didn't include the voter details because I only did them for a few times and then never again.

[Zwilnik]

Remember that the suggested price will be biased slightly by there being a larger balance of shareware developers in the voting mix, so it would probably tend towards the non too surprising result of $10-$15.

[PowerMacX]

Average Scores & Number of Votes (vs Time)

Date Gameplay Graphics Audio Votes
Nov-09 3.654 3.808 3.551 234
Nov-10 3.424 3.664 3.327 217
Nov-11 3.457 3.674 3.295 129
Nov-12 3.455 3.597 3.221 77
Nov-13 3.404 3.640 3.167 114
Nov-14 3.380 3.589 3.287 129
Nov-15 3.794 3.912 3.706 34
Nov-16 3.245 3.628 3.234 94
Nov-17 3.808 3.923 3.385 26
Nov-18 4.044 4.133 3.689 45
Nov-19 3.371 3.671 3.286 70
Nov-20 3.000 3.096 2.890 73
Nov-21 3.029 3.360 3.131 175
Nov-22 3.370 3.467 3.250 184
Nov-23 3.597 3.702 3.435 124


Graph: here (http://www28.brinkster.com/ignacio3d/udg2004stats.html)


The sections with big variations seem to be the result of the low number of votes at that time.
More votes at the beginning and end of the voting period than in the middle.
Average daily score for all categories seemed to go _down_, not much and not necessarily statistically significant, but surprising since many entries fixed bugs/added stuff during the voting period.
The average daily score for Graphics was always higher than Gameplay's, and Gameplay's daily average was higher than Audio's (except on Nov. 21)


Edit: Direct linking to the graph didn't work, so I made a page with all the graphs - link updated.

[PowerMacX]

Average Scores by Voter Type
Are you a gamer or developer? Gameplay Graphics Audio
Gamer 4.31 4.36 4.12
Gamer; Developer 3.92 3.95 3.73
Developer 3.69 3.70 3.46
Not specified 3.15 3.41 3.07


Graph here (http://www28.brinkster.com/ignacio3d/udg2004stats.html) (same as before)

People who considered themselves as "Gamers" (only) gave the highest scores, and those who considered themselves "Developers" gave the lowest (from those who answered the question). Those who didn't answer the question gave the lowest scores of all.

Interesting... :sneaky:

[PowerMacX]

Average Scores & Number of Votes by "Gamer Type"

1=Casual 4=Hardcore Gameplay Graphics Audio Votes
Not specified 3.137286759 3.39642567 3.054427295 1231
Casual (1) 4.201680672 4.168067227 4.016806723 119
Semi-Casual (2) 4.222222222 4.281481481 4.074074074 135
Semi-Hardcore (3) 3.945454545 4.066666667 3.733333333 165
Hardcore (4) 4.226666667 4.253333333 4.053333333 75

Graph here (http://www28.brinkster.com/ignacio3d/udg2004stats.html)

Those who didn't answered the question about what kind of player they are gave the lowest scores, followed by those who classified themselves as "Semi-Hardcore"

Correlation Analysis
Gameplay Graphics Audio Votes
Gameplay 1
Graphics 0.993976237 1
Audio 0.997226678 0.990538771 1
Votes -0.978115058 -0.981032998 -0.960675511 1


When grouped by "Gamer Type", all categories' scores have a strong inverse relationship to the number of voters. Also, Gameplay, Graphics & Audio have a strong direct relationship between them.

[PowerMacX]

Correlation Analysis (based on each entry average scores)
Gameplay Graphics Audio Originality Story Overall Price
Gameplay 1
Graphics 0.77503823 1
Audio 0.80941343 0.94937233 1
Originality 0.57114400 0.58839934 0.63655862 1
Story 0.35554912 0.49295929 0.55910539 0.49428052 1
Overall 0.85332665 0.92678016 0.95789815 0.76790004 0.66771110 1
Price 0.78441473 0.83907031 0.79797292 0.58630223 0.58644433 0.86428460 1



Overall scores have a strong relationship with Graphics & Audio, a relationship to "Estimated Price", and practically no relationship to Story & Originality (note: for a relationship to be considered relevant, it should have a correlation value of +0.8 or higher (or -0.8/lower).
Gameplay scores have a relationship with Audio (and, to some extent, with "Estimated Price" and with Graphics).
Graphics scores have a strong relationship with Audio scores, and a relationship with "Estimated Price".
Audio scores seem to have a very weak relationship to "Estimated Price".
Story and Originality have practically no meaningful relationship between them.


That's all the stats I have for now. Anything interesting? :)

[skyhawk]

shweet.

if you have anymore, keep posting them, I LOVE reading this stuff

[kemalyun]

I may seem stupid, but what is CSV-format? Also I can't unstuff that file. Can anyone send me a "readable" file by email? I'd also like to read those funny comments ...

[skyhawk]

have you tried opening it in text edit

[DaFalcon]

Story and originality were peer categories, so that may help explain some of why the didn't correlate: they were very different samples. And you showed that "developers" tend to vote differently than "gamers", at least when averaged together :-)

[MattDiamond]

Still looking forward to the judge's scores/comments in the polish category... (No hurry though.)

[Carlos Camacho]

CSV=Comma Separated Variable
It is a regular text file, with the fields/variables divided by a comma. You can use Excel, or a Spreadsheet to open it in a nice to read display. Probably other ways to view CSV in a nice formatted way as well.

>Still looking forward to the judge's scores/comments in the polish category
OK, keep reminding me. Also, keep in mind that their comments were not for EVERY game. For the most part, some mentioned a game or two, or commented on the contest as a whole. Next year, I will add a comment field after each game, for the judges who have the time to comment on Polish.

Well, there is a lot of data in this thread. It would be VERY good for someone to summarize it in a report, as was done in uDG 2002. Anyone?

[PowerMacX]

Some new tables/charts: http://www.reversecode.com/powermacx/udg04stats.html

Please note that some of the charts & decimal places in the table's values are slightly different from what I previously posted (not much, ie: my previous comments about them are still valid). Why? Well... there is this game called l1neum, which is called different in it's .csv file so that messed up some data when I sorted it, and in the end I forgot to add the votes of one game to my basic "all votes" spreadsheet. BTW, I think that the "number of votes" stat presented earlier (not by me) had incorrect values for the same reason (l1neum/OneNeum).

Note: For the 2D vs 3D charts, I assumed that BumperCars was 2D, but I never played it, so if it actually was 3D I'll have to update that chart. I think Carlos said that the game will be disqualified, but I made all my stats before I read that post, so the votes for BumperCars are included. I really don't want to calculate everything again but in any case, there are about 10 votes for this game so it wouldn't change the end results much.

New charts: Gamer Type vs. Age, Number of Votes by Age and 2D Games vs. 3D Games.

[PowerMacX]

Story and originality were peer categories, so that may help explain some of why the didn't correlate: they were very different samples. And you showed that "developers" tend to vote differently than "gamers", at least when averaged together :-)

Yes, that's why the charts only compare the public categories. Perhaps I should add an additional correlation table with only developer's votes taken into account?
The problem with this is that, for instance, many of my votes in the public categories (those I can recognize by my comments) didn't have any info! I entered it once the first time I voted, but in many of my votes it didn't show up. :wacko:

Also, I would like to complete the correlation table with Polish scores, as soon as that info is available.
Edit: I just found out the official results Excel files, so I'll add Polish to the stats tomorrow.

[Carlos Camacho]

Comments from Judges (this is for Matt who asked)

Ingemar Ragnemalm (Co-Author of da "Tricks" book)
An impressive lineup, with many good games. Some would be good shareware titles, and some promise great future titles from the same author. I have an urge to submit something myself some day... a nice way to promote Mac games. Great contest!
Comments: Most of it seems really good. What could be better? Not too much. I would like the ""polish"" part to put some more emphasis on the general feeling of "finished game", and re-warding games that help the player figuring the games out. Some were very good at this, others required the user to quit and read the manual first. At least one game could not be downloaded at all (BumperCars) and a few did not work. Couldn't at least the broken download be found and reported? (Editor's Note: We did our best with Bumper Car)

Ian Lynch Smith (Freeverse)
Best thing to happen to Mac game development since support for the 2 button mouse!

Glenda Adams (Aspyr Media)
Very high quality entries this year! I especially liked the variety of different game types and originality. Several with exceptional polish as well.

Michael S. Austin (CTSP Games)
Seems like the entries have overall improved this year. Great Job on all parts! I was asked to vote on polish, and to me, that is a start to finish impression of the game. The game should be easy to use and intuitive, as well as pleasing to the senses. Always take time to fix the small details! They are the ones that make an average game great.
Great job, Carlos and all others involved. Truly, this contest is motivating and drawing new people into our industry. Judging by the turnout and quality of some of the entries in this years contest, you are having a great effect on the future of gaming! Congratulations!

Sam Lantinga (Blizzard)
I am very impressed with the quality of games in general. There were many interesting and fun entries. It would have been nice to have some way to note which games I was not able to run. For those games I resorted to checking screenshots and evaluating the packaging. (Editor's Note: Sam Judges early and before instructions were given on how to deal with games that would not run)

Labriet Daniel (A developer with 8 arms)
Rescue looked really nice -- needs to be finished, & needs better control. huhu!

Nathan Lamont (Bigger Planet)
I couldn't find Bumper Cars to download. Splock had no sound. Thunderblast had technical difficulties and so was difficult to judge. I attempted to judge on "polish" alone, which to me means presentation and seamlessness, disregarding all other qualities. Thus I tended to be more forgiving of lower quality art if it was presented and used well. Thanks for inviting me to judge.

George Warner (Apple)
Solor Quest was the only game I gave a 5. Kill Dr. Cote & BugThug were excellent also. I judge games on: art, controls, difficulity (balance), documentation, fun, GUI, originality, physics, replayability, sound (& music), story and "extras" on a scale of 0-9.

Aaron Fothergill (One of the Unusual Suspects)
The quality of entry is getting better each year. Some very nicely put together games this year, and there are some games that have a great future potential.

Well, overall high marks to the games and the contest. :D

Cheers,

[Carlos Camacho]

I should mention two other points... Some judges didn't leave a remark -- in case you were wondering.

Also, I approached a "good number" of people who I felt were qualified to be judges due to their activities in the Mac game market. A good deal of them turned me down for lack of time. In many cases, I'm sure that was the reason, and yet in some cases, I could feel "the brush off. " I am too professional to say who we should give our Grinch Awards to, but I raise this point because the people in the list above, are the ones that REALLY care about this community, Mac games, and what we are doing IMHO. Think about it.

[JustinFic]

Note: For the 2D vs 3D charts, I assumed that BumperCars was 2D, but I never played it, so if it actually was 3D I'll have to update that chart.

...

New charts: Gamer Type vs. Age, Number of Votes by Age and 2D Games vs. 3D Games.

BumperCars was 2D.

The stats I'm finding particularly interesting are the average Audio scores for 2D vs 3D games. Is this because of a tendency for the teams behind 3D games to also put more effort into the music, or is there a tendency for voters impressed with a game to vote it higher in all categories?

Or does having 3D graphics truly make the music sound better? :p

[PowerMacX]

BumperCars was 2D.

The stats I'm finding particularly interesting are the average Audio scores for 2D vs 3D games. Is this because of a tendency for the teams behind 3D games to also put more effort into the music, or is there a tendency for voters impressed with a game to vote it higher in all categories?

Or does having 3D graphics truly make the music sound better? :p

If you look at the last table, Correlation (when grouped by game), you may notice that the biggest correlation is between Audio & Graphics (0.94937). This may be an indication of people trying to rate the game high "overall", and not having it as a separate choice leaving them no other way than just giving it good scores in the available public categories. Of course, there where many games with good music & graphics, but other than your "3D graphics making the music sound better" :P or "3D teams putting more effort into the music" theories I can't find a better explanation for the 2D vs. 3D scores.
Once I get the Polish scores with a bit more precision (more than 2 decimals) I'll add them to the correlation table too. Also, please note that the Peer categories' scores I used didn't include self-votes, but the official results seem to.