XML too slow, best way to go binary

Member
Posts: 117
Joined: 2010.09
Post: #1
Hi all,

My app loads an XML which it then parses to sort out texture atlases etc.
Whilst this appears almost instant on the simulator , on the device it is slow (and as the XML grows ) will only get slower :-(((
My supposed (!) solution was to "somehow" have in DEBUG mode reading of the XML but also writing out the file in binary (can this be done?, if I do fwrite() where does the file actually get stored)....
Then in !DEBUG it simply pulls in the binary file only.

My other question is during the compile how can I make the xcode project include the XML in DEBUG but not in release; and in release include the binary but not the XML in the build.

If there's a better way; or "nicer" solution please tell! :-))

Cheers
Quote this message in a reply
Sage
Posts: 1,482
Joined: 2002.09
Post: #2
I've found that a pretty easy way to do it is to generate headers full of static data structures from whatever data you need. That aren't even parsed then, just dumped directly into memory when the executable loads. Sure you can't load them at runtime, but most iPhone games don't really need that.

I would also recommend setting up build rules to generate the headers as part of the build process.
For example, for a terrain based game that I was working on I had a Ruby script that was generating all of the level geometry and graphics data offline. It spit out a header file using a simple erb template. I have a hard time believing that generating an XML file would have been much simpler, and as a bonus you don't need to parse the XML structure to put it into C data structures because it's already done!

I shortened it up a bit using "..." so you could see the structure better.
https://gist.github.com/778131

Scott Lembcke - Howling Moon Software
Author of Chipmunk Physics - A fast and simple rigid body physics library in C.
Quote this message in a reply
Member
Posts: 117
Joined: 2010.09
Post: #3
Hi Skorche,

Not sure exactly what you mean?
Also I have lots of level data i.e. I need to load a level, bin it and then load another. So nothing can stay resident if you see what I mean (I presume you are suggesting basically embedding the data at compile time).
If I have mis-understood; let me know.
Might have to 'test' the fwrite and see what happens!!

Cheers
Quote this message in a reply
Sage
Posts: 1,482
Joined: 2002.09
Post: #4
Yeah, it's basically all embedded at compile time. As long as you don't have many megabytes of level data it works just fine. You can still load the big memory hogs like sounds and textures at run time. I've generally found that my level data/geometry is a fraction of the size of the textures and other media that get's loaded.

Scott Lembcke - Howling Moon Software
Author of Chipmunk Physics - A fast and simple rigid body physics library in C.
Quote this message in a reply
Member
Posts: 117
Joined: 2010.09
Post: #5
Hmmmm,
I have many many levels.
And as it's a Zelda'esque type game you can go back and forth between maps & textures.
Hence the need to load only what's needed.
fwrite() here I come (doomed to failure I know!)

Cheers
Quote this message in a reply
Member
Posts: 117
Joined: 2010.09
Post: #6
junk
Quote this message in a reply
Sage
Posts: 1,482
Joined: 2002.09
Post: #7
You could always try Cocoa's built in serialization. It's not super fast, but it is pretty easy and flexible. Probably much faster than XML, much slower than fread(), but much easier than both.

If you want to go the fread() route, I might recommend using Ruby to generate your binary data files. Array#pack() gives you tons of options for packing binary data with different endianness and type. It also makes a much better language for hand editing data structures in as it's much more expressive than XML, has a simpler, easier to read syntax, and it's actually a programming language which means you can define methods to help out with repetitive data or embed calculations.

The following ruby snippet writes arrays of CGRects. (assuming that the iPhone is big-endian, I can never remember)
Code:
BINARY_DATA = ""

def write_array(format_string, arr)
    # write the count of the data
    BINARY_DATA << [arr.length].pack("N")
    
    # write the data elements
    arr.each do|data|
        BINARY_DATA << data.pack(format_string)
    end
end

write_array("gggg", [
    [1,2,3,4],
    [2*8, 3*Math.sin(6), 4, 5],
    some_variable_defined_somewhere,
])

write(BINARY_DATA)

If you throw the helper functions into another file, you have what looks more like a structured data file than an executable program. Nice and easy to edit. I'm sure Python can work similarly too.

Just make sure you write your helper functions in a way that makes it easy to add/remove data elements later or provide default values, etc. With XML, you can provide default values for missing elements at load time. With binary data, it's much easier to make sure that your writer does that work instead.

To read the data into C you'd just need to read the first 4 bytes into an integer, use that to allocate an array of GCRects and load them directly into that.

Scott Lembcke - Howling Moon Software
Author of Chipmunk Physics - A fast and simple rigid body physics library in C.
Quote this message in a reply
Member
Posts: 204
Joined: 2002.09
Post: #8
Firstly, you might try finding the best XML parser for your data:

http://www.raywenderlich.com/553/how-to-...ne-project

Secondly, I haven't tried them myself yet but you might look into protobuf:

http://code.google.com/p/protobuf/
Quote this message in a reply
Member
Posts: 117
Joined: 2010.09
Post: #9
Hi all,

Thanks for the suggestions.
I have a fast XML parser (tinyXML) it's just the size of the XML that's becoming the problem.
I've given it some thought and embedding the data structure with #include or incbin would be fine.
But it's actually generating the data from the XML that's the issue now.
I am using Zwoptex to produce the XML; so it's 'converting' it's output somehow that's the problem.
No experience of Ruby etc. and hence the iphone load XML, iphone write binary seemed a quicker and simpler approach...

Cheers
Quote this message in a reply
Member
Posts: 117
Joined: 2010.09
Post: #10
Hi all again,

Ok!. I tested fwrite() and AMAZINGLY it worked i.e. it writes a file!!!
I assumed that fwrite() would simply fail on simulator as I *assume* you aren't allowed to just 'write files' as though in an open system.
So now (in theory) I can read my XML, create a CSV (for #include) or maybe a binary for incbin!!
In theory anyway! :-)))
So how do I get xcode to only include the XML in debug and not in device mode????

Is no-one else surprised fwrite() worked? :-)))

Cheers
Quote this message in a reply
Post Reply 

Possibly Related Threads...
Thread: Author Replies: Views: Last Post
  Why is my iPad game so slow? Applewood 53 28,998 Jan 3, 2011 04:54 PM
Last Post: Applewood
  emulator slow when rendering from a second thread captainfreedom 1 2,925 Jan 30, 2010 05:05 PM
Last Post: ChrisD
  Simple application slow down, FPS drops Newbrof 15 8,057 Sep 2, 2009 12:51 PM
Last Post: AnotherJake
  Chipmunk is too slow? jaguard 14 8,240 Jan 22, 2009 04:49 AM
Last Post: jaguard
  mac mini simulator - slow frame rate sumiguchi 3 3,879 Dec 18, 2008 08:56 AM
Last Post: SafariAl