Profiling is fun when it shows you something big that’s easy to fix.  Profiling is a lot less fun when everything is equally slow.  But luckily in my case, there was something big to fix.  :)

I noticed the other day that editing roads in MMORPG Tycoon 2 had become extremely slow.  (Well.. actually I noticed that roads didn’t draw at all.  It was only after I fixed them not drawing that I noticed that they were slow to edit).   This was odd, because I have videos of early road editing, where it hardly touched the frame rate at all on a system much slower than the one I’m using now.  So after some profiling, I found that creating their procedural geometry was taking up about 75% of the whole processing time for the frame when they were created.  This was extremely bad when editing them, especially (because they’re regenerated every frame, when they’re being edited, or when the ground under them is being edited).

Those of you who were at Freeplay will remember the part of my talk where I discussed optimising games, and when to do it (or rather, when not to do it).

For those who weren’t there, the tl;dr version is this:  don’t optimise anything unless you have timing numbers that proves it’s slow relative to the rest of your program, and against which you’ll be able to judge your progress.  Ideally, you get these numbers by using a profiler of some sort.  (Visual Studio has one built in, modern Xcode uses Instruments while old Xcode used Shark, under Linux you have gperf, etc)

What I didn’t say in that talk (and which I’ve regretted ever since), is that there’s a second criteria which can make it acceptable to optimise some code, even if you don’t have performance numbers.  Here is that other criteria:

Monte Carlo Profiling

1. Make a build of your game with optimisations turned on (a “release” build, or “retail” build, or “dist” build, or whatever you call it), but with debugging symbols still present.
2. Run this build through your debugger of choice.
3. At any time of your own choosing, pause the execution of the game, and examine the call stack shown in your debugger.
4. Repeat step 3 nine more times (for a total of ten pauses and ten callstacks).
5. If in seven of the ten total times you paused the game you were in the same area of code, then you may spend time optimising that area of code, even without first profiling it.

The Monte Carlo test is basically very quick and ad hoc profiling.  Profiling does exactly this same process, but instead of doing it ten times, it does it millions of times, far faster than you’re able to do it manually.  It avoids having to go to the bother of setting up proper profiling, and will still point out anything that’s extremely slow in your code.

But back to my story.  In my case, I did do real profiling.  Here’s what it looked like in Instruments:

That graph is of CPU usage.  (Also, it totally breaks the site frame layout, but I don’t care.  I want you to see the details.  :) )  The spike at the start is the game starting up (and generating all the world geometry all at once).  The low part in the middle is just me looking around a bit, and then the heavy bit that I have highlighted is me editing a road — just dragging one of its endpoints around the map at random.

As you can see from the screenshot, it’s spending about 30% of the frame time in vsMeshMaker::BakeTriangleVertex, 20% of its time in vsFloor (called from vsMeshMaker::BakeTriangleVertex), 16% of the frame time in vsVector2D::SqLength() (also called from vsMeshMaker::BakeTriangleVertex)…  you get the idea.  Add it all up, and we were spending about 75% of our CPU time inside BakeTriangleVertex.  That BakeTriangleVertex function was really, really heavy for some reason, and I needed to figure out why.

Basically what vsMeshMaker::BakeTriangleVertex does is to look at all the triangle vertices in a procedurally generated model, and to decide (one at a time) whether they should be merged with other nearby vertices to make a smoothly curving surface.  The hardest part of this is actually finding other nearby vertices.  Previously, I had sped this up by dividing up space into grid squares, and storing each vertex into the appropriate grid space.  Then when I wanted to know about nearby vertices, I only had to check the other vertices in the same grid space, not every other vertices in the model.

But as it turns out, those grid spaces were too large, especially for large objects like roads, and so I was still comparing far too many vertices for merging.  So I increased the number of grid spaces (and made each grid space smaller, so there would be fewer vertices to test in each), so that instead of having an 8x8x8 grid and needing to test every vertex in those large squares, we have a 32x32x32 grid and only need to test every vertex in much smaller squares.  I had also noticed that very simple functions like vsFloor and vsVector3D::SqLength() were showing up in the profiling, and so made those functions inline, to hopefully reduce the cost of calling them.  This led to this graph:

This one is fascinating.  Notice that the amount of CPU being used has dropped substantially, compared to the previous one.  Also note that functions like “vsFloor” and “vsVector3D::SqLength” have vanished, since I made them inline (they’re now included in the caller’s time).  Increasing the resolution of my vertex lookup grid had sped things up a lot — we’re down from 75% of our frame time, down to an average of 40% of our frame time (and made a huge improvement to the frame rate).  But here’s the interesting thing that I hadn’t noticed previously:

Look at that graph, how it increases over time.  I wasn’t doing anything unusual in-game, just rapidly dragging one end of the road back and forth over a single area of terrain.  But the more time I spent editing the road, the slower editing became.  This eventually pointed me to a bug in my code, where some of the merging work from previous frames was being left-over to do again on all subsequent frames, even though it didn’t accomplish anything on the new frames.  So I fixed that bug, and profiled again:

Great!  Getting rid of that leftover repeated work brought us down from 40% frame time (on average, during a long editing session) to about 2.6% frame time, low enough that you don’t even notice a frame rate hit in-game.

The problem now was that my grid was so fine, that it actually uses a lot of memory.  A 32x32x32 grid has about 32,000 spaces in it, and each of those needed storage to hold an arbitrary number of vertices.  And I had one of these for every road (and indeed, every building, tree, etc).  That builds up fast!  And while 2.6% is low enough to be acceptable, I would have liked it to be even lower.  Let’s face it, my placeholder “road” model is ridiculously simple.  I’d really like this code to be much faster than it is now, to handle the much more detailed meshes that I’ll be throwing at it later.

So I set out to replace the grid with a new structure:  an octree.  For those who are interested, this implementation has been ported back into the main VectorStorm trunk, as VS_PointOctree.h, inside the VS/Utils/ directory.  It’s basically a tree structure which expands as more data is added to it, so it should behave far more predictably than the grid structure.  What’s more, it expands as data is added, instead of being a fixed (huge) size, which means that smaller models use much less space, exactly as you’d hope.  Profiling isn’t much different (down to about 2%), but memory usage is drastically lower.  So I call that a win.  And in future, I may return to this code to see if I can’t whittle it down even a little further, once it starts moving up into a prime spot in the profiling again.

I’ve also been doing other stuff recently, but I reckon that this blog post is enough of a novel already, so I’ll save the other stuff for tomorrow.

Today, MMORPG Tycoon 2 has finally become a multithreaded game.  It’s not ubiquitous by any means, but it’s taken the first step;  the ground clutter is now being generated in a background thread.  The only bit of clutter-related code which is now occurring in the main thread is copying the final geometry into the buffer that OpenGL renders from.  Now all the setup work as you move around the map is being handled in separate threads.

This means a couple of things.  For one, it means that MMORPG Tycoon 2 can finally use a little more of the CPU time from everyone with a multi-core computer (which is just about everyone with a computer from the last six years or so), and it also means that as more tasks can be moved out into other threads, I’ll be able to do more and more processing without hurting the game’s frame rate.

I’ll also note that Mac OS X’s support for semaphores is half-broken, and Win32′s support for semaphores is bizarre and generally unrelated to traditional semaphores.  Makes me a sad coder.  (for the non-coders in the audience:  semaphores are a tool for synchronising multiple threads, to help keep them from stepping on each other’s toes)

More details tomorrow.  Or later today, I guess, technically.  And semaphores will eventually be ported back into the VectorStorm trunk, if anyone wants to look at them.  Also fixed some bugs in the vsTask class, and added features to the vsMutex.  Will need to move all those changes back to the live trunk in the near future.  Maybe over the weekend.

In brief, what was happening before was that MT2 (and all VectorStorm games) would render every object in the scene one at a time, setting up OpenGL to render that object individually (“Turn fog on.  Turn lighting on.  Turn zbuffer on, turn writing into color channels on, turn glow effect off, etc.”), even though 95% of those settings were the same on every object in the scene.

Now what happens is that each object registers those state changes onto a new object (the “vsRendererState”), and immediately before rendering, all those queued state changes are made.  Previously if (for example) the first object in a scene is using a texture and the second object is also using a texture, I’d be telling OpenGL “okay, turn on texture support”, render the first object, then “turn off texture support now”, and then “okay, turn on texture support again” before rendering the second object.  Now, it leaves texture support on until it actually starts rendering an object which isn’t using a texture;  it only changes the settings which are actually changing, instead of just naively setting everything.

Note that to make this work, I had to remove compiled display list support.  There’s simply no good way to keep track of OpenGL’s internal state when compiled display lists are being used.  But compiled display lists were really replaced by VBOs long ago, so this shouldn’t cause concern.  It’s similar to when I removed CPU-side shaders, and overlays.  Old technologies which have been superceded by faster, better ones.  :)

The screenshot today is what happens when there’s a bug in the code (in this case, not applying rendering state changes before performing post-render effects).

I’ve also been playing a bit with gDEBugger, an OpenGL profiler/analyser/etc, which is now free (was previously quite expensive to license).  It’s nowhere near PIX in terms of power and insight, but it’s as close as we have under OpenGL at the moment.  And you can’t argue with the price.  And it works on all the major platforms, which is pretty awesome to see.  I’ve already made some sizeable rendering speed boosts for VectorStorm just on the basis of its insights into MMORPG Tycoon 2, and will have more in the near future.  Once I have these mostly sorted out, I’ll port them back into trunk as well.

This tool has also pointed out that lots of the techniques VectorStorm’s core rendering architecture use have been declared obsolete as of OpenGL 3.1.  Which surprised me, since I kind of designed VectorStorm for OpenGL 1.4 (which was the most that my ancient laptop could handle at the time);  I really didn’t think that the interfaces I was using would last all the way up to 3.1!  But it does suggest that someday I should be thinking about looking forward to the more modern revisions of OpenGL;  purging some of my old rendering architecture, and replacing it with more modern approaches.  That won’t be until after MT2, though, unless MT2 really calls for some modern rendering techniques.  (I’d really like to have some basic shadows.  But otherwise, I’m not too fussed over fancy rendering techniques for MT2).

First, I’ve updated the VectorStorm engine so that it’ll compile successfully against iOS SDK v4.3.  I’ve also got both the standard VectorStorm Testbeds and MMORPG Tycoon 2 compiling under the newly released XCode 4.  My opinion of XCode 4 is complicated;  I can understand the general outcry from many developers, but there’s lots of cool stuff in there too.  Whether the cool stuff outweighs the concerns, I’m not really sure.  But for the next couple of days, I’ll be using XCode 4 as my day-in day-out compiler to see how well I adapt.

Second, I’ve begun implementing some threading primitives into VectorStorm.  These are not yet compiling under Windows (since Microsoft has an entirely different threading model than UNIX-based systems), but I do now have both mutexes and threads working under OS X.  I’ll get the Windows version working tomorrow.  These little tools are going to be critical for MMORPG Tycoon 2, due to the amount of processing that it has to do;  it’s just going to be a lot easier to put some of that work into a separate thread.

Do be aware, though, that none of the rest of VectorStorm is written to be threadsafe.  These multithreading classes should really only be used for simple tasks that don’t involve calling into the various VectorStorm libraries at all.  If you have lots of calculations to be done in the background, then that’s fine.  But probably best not to try to allocate memory or start playing sounds from inside a thread or anything.  :)

I’ve fixed the VectorStorm engine bug which was causing problems with the middle mouse button and right mouse buttons (often swapping the two, or causing them not to work at all).  It was an embarrassing mistake;  I had just mis-assigned labels for the controls, causing some minor swapping when loading saved control preferences.  I’ve made this bug fix both within MMORPG Tycoon 2 (where it was a major problem), and within the trunk VectorStorm library (where it was less of a problem, since no standard VectorStorm games were using mouse control).  If any of you have been looking at or using the VectorStorm library, you should go get those fixes.  If not, don’t worry about it too much, just be aware that I’m back to the compiler again.  :)

I’ve also begun poking with Tycoon code again.  Not a whole lot;  am just slowly easing myself back into it.

I’ve made a bunch of little tweaks to the VectorStorm engine today.  Most notably:

When doing a scene update, the camera now updates last, instead of first.  Previously, the camera would pick where to be before the game had a chance to take any actions, which usually led to the game camera lagging one frame behind whatever it was supposed to be following.  (Of course, now you’ll have the opposite problem;  the game will no longer be able to tell where the camera is, since it won’t have moved into position until after the game finishes its update)  In practice, the real solution is for the game itself to take responsibility for updating the camera whenever is appropriate for that game.  But as a default, having the camera update last is usually less-bad than having it update first.  So I’ve changed it.

I noticed that vsFont wasn’t applying text colors to strings which were justified as “centered” or “right”;  they were always coming through as pure white.  This is now fixed, in current trunk.

I also added another “primitive” function for creating renderable textured rectangles (as vsFragments that can be attached to a vsSprite or vsModel), this time supporting tiled textures.  You can specify how big the tiles should be, and can optionally specify a rotation for the texture.  Check for vsMakeTiledTexturedBox2D() in VS_Primitive.h, for that.

I’ve also been poking with the bloom effect again;  I noticed that the trunk version looks pretty ugly at large screen sizes.  I think I have it balanced nicely again, but haven’t committed these changes to the public repository yet;  probably later tonight or tomorrow, once I’m certain that performance is still okay.

Finally, I’ve just finished scaling down the Asteroids testbed game.  It’s been misbehaving ever since I converted it over to Box2D, because its objects were moving too fast for Box2D to cope with.  So I’ve reduced everything by a factor of ten, and now things are moving at a reasonable speed again.  I’ll be checking that in a bit later on as well.  Oddly, I’m still occasionally having shots bounce off of asteroids, without receiving a collision callback.  Mind you, I’m using an ancient version of Box2D;  maybe I should update that someday.

Again, this photo is using exactly the same graphics as the past few days, just now you can use multitouch gestures to zoom and rotate the interface.  (Most of my time went into the zooming.  Surprisingly difficult to get the maths right for that, in this case.)

It feels.. surprisingly natural.  I didn’t think it’d feel this good to move an object around on the screen this way.  There’s probably a bestselling game just in that interaction, if you could figure out a way to wrap a game around it (and if somebody else hasn’t already done it.  I’m sure someone has).  I’m not sure exactly what game you could wrap around this interaction, but it does feel really, really nice.

Right now, VectorStorm supports only two simultaneous touch events.  But now that I have two working, it’d be really easy to add up to about ten.  But in this case, I only need two for the pinch/zoom interface.  So I’ll probably make do with what’s in there now.

Tomorrow, I may take a stab at sound support.  Or more likely, I’ll spend time on the game code itself, again.

I’ve been doing further development on the “automatic save/load” system that I’ve talked about before;  the system which automatically handles loading and saving of game data, without me having to write custom code for everything the way that I did back in previous games.  For comparison, in MMORPG Tycoon 1, nearly half of my total development time went into the save/load functionality — it’s extremely complicated, and extremely bug-prone, so anything that I can do to make the saving/loading process easier and safer is going to be a huge win, overall!

This “how do we load and save large amounts of data” issue, incidentally, is one that professional game development teams everywhere have struggled with forever, and I’ve seen a lot of different solutions to it.  But I’ve gotta say, this solution I’ve been working on is by far the best that I’ve ever seen within the C++ language;  it imposes few restrictions on the code I write, and can just automatically figure out how to save files out based upon inspecting the saveable variables I expose within each class.

What I’ve done today is to make this system recursive.  That is, you can now have an object which contains other objects, which contain other objects.  Or to give a more concrete example:

In MMORPG Tycoon 2, there’s an object for each of a character class’s abilities.  So if he has three different types of attack, that’s three code objects which store information about how much damage those types of attack do, etc.  Those three ability objects are stored inside another object, an “Ability Set”, which lists all the abilities available to one character class.

With this new stuff I’ve written today, it’s now possible for me to simply call “AbilitySet->Save(“Filename”)”, and have the AbilitySet save itself out, plus all of the Abilities which are inside of it, into a single neat data file.  Similarly, “AbilitySet->Load(“Filename”)” loads that data file from disk, recreating the AbilitySet object itself and all of the abilities that it should contain.  This is all typesafe — the AbilitySet can only contain objects of “Ability” type (or derived from that type), and no typecasting is required at any point during the process.

Of course, the next step is going to be to have the CharacterClassPrototype have an AbilitySet inside itself, so that when I load a class prototype from disk, it automatically loads the abilityset, and the abilityset’s contents.  And then a ClassManager will be set to contain ClassPrototypes, so that when I save the ClassManager, it automatically saves out all of the class prototypes inside it, and so forth and so on.  Eventually, everything in the game should be included within this data hierarchy, so a single call to “Save()” or “Load()” at the top level will be able to automatically handle saving or loading the complete game state, all from a single line of code.

So I’m pleased.  :)

What you’re looking at in this screenshot is a not-yet-activated starting point, being rendered as a glowing outline, behind a tree.  Anyone who’s played with the MS1 build of Tycoon2 will know that one of the big rendering problems that it had was that glowing objects would often glow through opaque objects that were in front of them.  As you can see, this is no longer the case.  VectorStorm now automatically ensures that objects are drawn in the correct order, so you won’t incorrectly see obscured glowing effects.

To do this, I had to (warning:  the rest of this post contains dangerously boring technical details) change how VectorStorm renders.

Originally, VectorStorm was designed for rendering simple 2D vector graphic games.  As such, it was designed with a system of “layers”.  Your game could configure how many layers it wanted to use, and register sprites or other objects onto any layer you liked.  Within each layer, objects were drawn in order, with later objects (or later layers) drawn on top of previous layers.  With vector graphics, since you only had additive lines and no solid objects, the whole issue of which line was on top really didn’t matter;  the layers were really used just for putting objects under different cameras.  For example, in The Muncher’s Labyrinth, the game world was in one layer, and the HUD was in a different layer, just so the two could be rendered using different cameras and I didn’t need to worry about moving the HUD around to match the camera.  When I added solid rendering for ThunderStorm, the layers worked really well to ensure that some objects would always draw in front of other objects.

Each renderable object would have a “Draw()” function called on it, being given a pointer to a display list to which it could append its own rendering commands.  This was (in my own opinion) simple and elegant, and it was great for a while.

But now that VectorStorm is rendering in 3D, layers don’t really make sense any more.  We still have them, but in 3D we have a z buffer which automatically lets the closest object be drawn, regardless of whether it was drawn first or last, so it’s usually no longer important to manage which objects are drawn first or last, except for in special circumstances (glow effects, transparency, etc).

But the real problem is that we now often have objects which have different drawing styles within a single object.  For example, the cursor in MT2 (and MT1, for that matter) is made up of two parts;  a dark background, and a glowing outline.  Under the “just add your commands to this display list” approach, each object has to render all of its geometry at once.  This meant that I couldn’t (for example) draw the background portion of the cursor early, and the glowing part later on;  they’d both always render at the same time.

Another possible issue is that of rendering partially transparent objects.  These see-through objects are always a problem with modern 3D games, since z-buffers can’t be used to render them (z-buffers assume that everything is opaque, and so won’t allow you to draw anything behind a pane of glass, for example, if you draw the glass first).  The traditional way to handle these translucent objects is to sort them from back to front, then draw them in that order, after you’ve drawn all the non-translucent objects.  But with the simple-and-elegant way that I was handling drawing of objects, there was no way to sort the objects into any different order than they were already in.

Anyhow, enough of the problem statement.  :)

What I’ve changed is that renderable objects in VectorStorm are no longer given a pointer to the game’s vsDisplayList when it’s time for them to render.  Instead, they’re now given a vsRenderQueue object.  On that vsRenderQueue, they can provide their rendering instructions in one of several ways, and the vsRenderQueue can then reorder the drawing as required, to make sure that (for example) the glowing and transparent parts of objects are drawn last, while their opaque portions can still be drawn earlier.

The other change is that vsLayer is now completely gone.  Instead, we now have vsScenes, which work very much like vsLayers did;  they’re just collections of renderable objects.  Like before, your game can have any number of vsScenes, and they’re rendered in order.  Right now, they share a single z-buffer and render target between them all, but setting it up so that the different scenes can have separate renders is certainly possible in the future, if I ever need that feature.

You can probably tell by how I’m gushing, but I’m absolutely thrilled and relieved to have this all working at last.  Lots of stuff in MT2 (particularly the very old code) is currently working via a “compatibility” mode that I’ve put in the vsRenderQueue, and I do still need to convert those bits of code over to using the new systems instead.  But this change has illuminated a whole bunch of unreliable rendering code that was lurking in the GUI systems until now, and the stricter system should keep me from writing terrible code again in the future.

Right now, these changes are only in my MT2 development branch — they haven’t been propagated back into trunk, largely because I’d probably have to make a lot of big adjustments to the testbed apps to make them render under the new systems, and I’m feeling like taking a bit of a rest, after this ten-hour coding session.  But I’ll update trunk sometime in the foreseeable future, and will post about it when I do.