Profiling is fun when it shows you something big that’s easy to fix.  Profiling is a lot less fun when everything is equally slow.  But luckily in my case, there was something big to fix.  :)

I noticed the other day that editing roads in MMORPG Tycoon 2 had become extremely slow.  (Well.. actually I noticed that roads didn’t draw at all.  It was only after I fixed them not drawing that I noticed that they were slow to edit).   This was odd, because I have videos of early road editing, where it hardly touched the frame rate at all on a system much slower than the one I’m using now.  So after some profiling, I found that creating their procedural geometry was taking up about 75% of the whole processing time for the frame when they were created.  This was extremely bad when editing them, especially (because they’re regenerated every frame, when they’re being edited, or when the ground under them is being edited).

Those of you who were at Freeplay will remember the part of my talk where I discussed optimising games, and when to do it (or rather, when not to do it).

For those who weren’t there, the tl;dr version is this:  don’t optimise anything unless you have timing numbers that proves it’s slow relative to the rest of your program, and against which you’ll be able to judge your progress.  Ideally, you get these numbers by using a profiler of some sort.  (Visual Studio has one built in, modern Xcode uses Instruments while old Xcode used Shark, under Linux you have gperf, etc)

What I didn’t say in that talk (and which I’ve regretted ever since), is that there’s a second criteria which can make it acceptable to optimise some code, even if you don’t have performance numbers.  Here is that other criteria:

Monte Carlo Profiling

1. Make a build of your game with optimisations turned on (a “release” build, or “retail” build, or “dist” build, or whatever you call it), but with debugging symbols still present.
2. Run this build through your debugger of choice.
3. At any time of your own choosing, pause the execution of the game, and examine the call stack shown in your debugger.
4. Repeat step 3 nine more times (for a total of ten pauses and ten callstacks).
5. If in seven of the ten total times you paused the game you were in the same area of code, then you may spend time optimising that area of code, even without first profiling it.

The Monte Carlo test is basically very quick and ad hoc profiling.  Profiling does exactly this same process, but instead of doing it ten times, it does it millions of times, far faster than you’re able to do it manually.  It avoids having to go to the bother of setting up proper profiling, and will still point out anything that’s extremely slow in your code.

But back to my story.  In my case, I did do real profiling.  Here’s what it looked like in Instruments:

That graph is of CPU usage.  (Also, it totally breaks the site frame layout, but I don’t care.  I want you to see the details.  :) )  The spike at the start is the game starting up (and generating all the world geometry all at once).  The low part in the middle is just me looking around a bit, and then the heavy bit that I have highlighted is me editing a road — just dragging one of its endpoints around the map at random.

As you can see from the screenshot, it’s spending about 30% of the frame time in vsMeshMaker::BakeTriangleVertex, 20% of its time in vsFloor (called from vsMeshMaker::BakeTriangleVertex), 16% of the frame time in vsVector2D::SqLength() (also called from vsMeshMaker::BakeTriangleVertex)…  you get the idea.  Add it all up, and we were spending about 75% of our CPU time inside BakeTriangleVertex.  That BakeTriangleVertex function was really, really heavy for some reason, and I needed to figure out why.

Basically what vsMeshMaker::BakeTriangleVertex does is to look at all the triangle vertices in a procedurally generated model, and to decide (one at a time) whether they should be merged with other nearby vertices to make a smoothly curving surface.  The hardest part of this is actually finding other nearby vertices.  Previously, I had sped this up by dividing up space into grid squares, and storing each vertex into the appropriate grid space.  Then when I wanted to know about nearby vertices, I only had to check the other vertices in the same grid space, not every other vertices in the model.

But as it turns out, those grid spaces were too large, especially for large objects like roads, and so I was still comparing far too many vertices for merging.  So I increased the number of grid spaces (and made each grid space smaller, so there would be fewer vertices to test in each), so that instead of having an 8x8x8 grid and needing to test every vertex in those large squares, we have a 32x32x32 grid and only need to test every vertex in much smaller squares.  I had also noticed that very simple functions like vsFloor and vsVector3D::SqLength() were showing up in the profiling, and so made those functions inline, to hopefully reduce the cost of calling them.  This led to this graph:

This one is fascinating.  Notice that the amount of CPU being used has dropped substantially, compared to the previous one.  Also note that functions like “vsFloor” and “vsVector3D::SqLength” have vanished, since I made them inline (they’re now included in the caller’s time).  Increasing the resolution of my vertex lookup grid had sped things up a lot — we’re down from 75% of our frame time, down to an average of 40% of our frame time (and made a huge improvement to the frame rate).  But here’s the interesting thing that I hadn’t noticed previously:

Look at that graph, how it increases over time.  I wasn’t doing anything unusual in-game, just rapidly dragging one end of the road back and forth over a single area of terrain.  But the more time I spent editing the road, the slower editing became.  This eventually pointed me to a bug in my code, where some of the merging work from previous frames was being left-over to do again on all subsequent frames, even though it didn’t accomplish anything on the new frames.  So I fixed that bug, and profiled again:

Great!  Getting rid of that leftover repeated work brought us down from 40% frame time (on average, during a long editing session) to about 2.6% frame time, low enough that you don’t even notice a frame rate hit in-game.

The problem now was that my grid was so fine, that it actually uses a lot of memory.  A 32x32x32 grid has about 32,000 spaces in it, and each of those needed storage to hold an arbitrary number of vertices.  And I had one of these for every road (and indeed, every building, tree, etc).  That builds up fast!  And while 2.6% is low enough to be acceptable, I would have liked it to be even lower.  Let’s face it, my placeholder “road” model is ridiculously simple.  I’d really like this code to be much faster than it is now, to handle the much more detailed meshes that I’ll be throwing at it later.

So I set out to replace the grid with a new structure:  an octree.  For those who are interested, this implementation has been ported back into the main VectorStorm trunk, as VS_PointOctree.h, inside the VS/Utils/ directory.  It’s basically a tree structure which expands as more data is added to it, so it should behave far more predictably than the grid structure.  What’s more, it expands as data is added, instead of being a fixed (huge) size, which means that smaller models use much less space, exactly as you’d hope.  Profiling isn’t much different (down to about 2%), but memory usage is drastically lower.  So I call that a win.  And in future, I may return to this code to see if I can’t whittle it down even a little further, once it starts moving up into a prime spot in the profiling again.

I’ve also been doing other stuff recently, but I reckon that this blog post is enough of a novel already, so I’ll save the other stuff for tomorrow.

In all this time, this blog has remained pretty much unchanged. I guess I was finally feeling like it was time to air the room and let in a little more light, so welcome to the new layout. Clean, simple, and relatively unobtrusive.

Of course, for those of you who read via the RSS feed, there’ll be no visible difference. But for everyone else, hopefully the site will be a bit easier on the eyes, particularly for my longer bouts of rambling.

Today, MMORPG Tycoon 2 has finally become a multithreaded game.  It’s not ubiquitous by any means, but it’s taken the first step;  the ground clutter is now being generated in a background thread.  The only bit of clutter-related code which is now occurring in the main thread is copying the final geometry into the buffer that OpenGL renders from.  Now all the setup work as you move around the map is being handled in separate threads.

This means a couple of things.  For one, it means that MMORPG Tycoon 2 can finally use a little more of the CPU time from everyone with a multi-core computer (which is just about everyone with a computer from the last six years or so), and it also means that as more tasks can be moved out into other threads, I’ll be able to do more and more processing without hurting the game’s frame rate.

I’ll also note that Mac OS X’s support for semaphores is half-broken, and Win32′s support for semaphores is bizarre and generally unrelated to traditional semaphores.  Makes me a sad coder.  (for the non-coders in the audience:  semaphores are a tool for synchronising multiple threads, to help keep them from stepping on each other’s toes)

More details tomorrow.  Or later today, I guess, technically.  And semaphores will eventually be ported back into the VectorStorm trunk, if anyone wants to look at them.  Also fixed some bugs in the vsTask class, and added features to the vsMutex.  Will need to move all those changes back to the live trunk in the near future.  Maybe over the weekend.

Well, it turns out that I hadn’t fixed it at all;  I’d merely hidden it in a few situations.  As you can see from the screenshot on the left, the problem was still visible in other places (left corner of the graveship, top of the column, and more subtly, roof of the inn).

I believe that I’ve now well and truly solved the problem;  certainly all those problem areas are now rendering more appropriately, as seen in the screenshot on the right.  So yay.  As I mentioned before, these fixes are in the “vsMeshMaker” utility class in the VectorStorm library.  These fixes will be brought into the live trunk sometime in the next few days, once I’m satisfied that the bug is actually fixed for real this time, and hasn’t just scampered off to hide under some different rock.

In other news, I spent a good half hour trying to figure out why I haven’t been able to post thumbnails of screenshots since moving to the new server.  Turns out that WordPress really wants to have php5′s gd module installed, and I had neglected to install that on the new server. Boo to WordPress’s support sites, which don’t mention that it needs this module.

In non-news, I’m still poking away at MMORPG Tycoon 2 and making (extremely) slow but steady progress.  There hasn’t been much to show for a while, as I’ve been re-plumbing its innards to use regular VectorStorm library functionality wherever possible, instead of using custom rendering code everywhere.   VectorStorm now supports out-of-the-box features (batched rendering, etc) which I had previously implemented explicitly within custom game-side rendering logic in MMORPG Tycoon 2.  Simpler codebases lead to faster development!  But simplifying codebases takes a lot of time.  Especially when you’re doing it in your spare time.

1. One sprite to add.
2. One sound-effect to find and trigger
3. Add instructions.  (Probably will just be text floaters within the level, because that’ll be fastest to do)
4. Maybe think about making the title screen less awful.
5. Arrange the Windows build.

There were a few things I wanted to have in the game which simply aren’t going to get done in time, and the graphics are extraordinarily rough.  But by and large, I’m kind of pleased with how this is turning out.

In brief, what was happening before was that MT2 (and all VectorStorm games) would render every object in the scene one at a time, setting up OpenGL to render that object individually (“Turn fog on.  Turn lighting on.  Turn zbuffer on, turn writing into color channels on, turn glow effect off, etc.”), even though 95% of those settings were the same on every object in the scene.

Now what happens is that each object registers those state changes onto a new object (the “vsRendererState”), and immediately before rendering, all those queued state changes are made.  Previously if (for example) the first object in a scene is using a texture and the second object is also using a texture, I’d be telling OpenGL “okay, turn on texture support”, render the first object, then “turn off texture support now”, and then “okay, turn on texture support again” before rendering the second object.  Now, it leaves texture support on until it actually starts rendering an object which isn’t using a texture;  it only changes the settings which are actually changing, instead of just naively setting everything.

Note that to make this work, I had to remove compiled display list support.  There’s simply no good way to keep track of OpenGL’s internal state when compiled display lists are being used.  But compiled display lists were really replaced by VBOs long ago, so this shouldn’t cause concern.  It’s similar to when I removed CPU-side shaders, and overlays.  Old technologies which have been superceded by faster, better ones.  :)

The screenshot today is what happens when there’s a bug in the code (in this case, not applying rendering state changes before performing post-render effects).

I’ve also been playing a bit with gDEBugger, an OpenGL profiler/analyser/etc, which is now free (was previously quite expensive to license).  It’s nowhere near PIX in terms of power and insight, but it’s as close as we have under OpenGL at the moment.  And you can’t argue with the price.  And it works on all the major platforms, which is pretty awesome to see.  I’ve already made some sizeable rendering speed boosts for VectorStorm just on the basis of its insights into MMORPG Tycoon 2, and will have more in the near future.  Once I have these mostly sorted out, I’ll port them back into trunk as well.

This tool has also pointed out that lots of the techniques VectorStorm’s core rendering architecture use have been declared obsolete as of OpenGL 3.1.  Which surprised me, since I kind of designed VectorStorm for OpenGL 1.4 (which was the most that my ancient laptop could handle at the time);  I really didn’t think that the interfaces I was using would last all the way up to 3.1!  But it does suggest that someday I should be thinking about looking forward to the more modern revisions of OpenGL;  purging some of my old rendering architecture, and replacing it with more modern approaches.  That won’t be until after MT2, though, unless MT2 really calls for some modern rendering techniques.  (I’d really like to have some basic shadows.  But otherwise, I’m not too fussed over fancy rendering techniques for MT2).

First, I’ve updated the VectorStorm engine so that it’ll compile successfully against iOS SDK v4.3.  I’ve also got both the standard VectorStorm Testbeds and MMORPG Tycoon 2 compiling under the newly released XCode 4.  My opinion of XCode 4 is complicated;  I can understand the general outcry from many developers, but there’s lots of cool stuff in there too.  Whether the cool stuff outweighs the concerns, I’m not really sure.  But for the next couple of days, I’ll be using XCode 4 as my day-in day-out compiler to see how well I adapt.

Second, I’ve begun implementing some threading primitives into VectorStorm.  These are not yet compiling under Windows (since Microsoft has an entirely different threading model than UNIX-based systems), but I do now have both mutexes and threads working under OS X.  I’ll get the Windows version working tomorrow.  These little tools are going to be critical for MMORPG Tycoon 2, due to the amount of processing that it has to do;  it’s just going to be a lot easier to put some of that work into a separate thread.

Do be aware, though, that none of the rest of VectorStorm is written to be threadsafe.  These multithreading classes should really only be used for simple tasks that don’t involve calling into the various VectorStorm libraries at all.  If you have lots of calculations to be done in the background, then that’s fine.  But probably best not to try to allocate memory or start playing sounds from inside a thread or anything.  :)

I’ve fixed the VectorStorm engine bug which was causing problems with the middle mouse button and right mouse buttons (often swapping the two, or causing them not to work at all).  It was an embarrassing mistake;  I had just mis-assigned labels for the controls, causing some minor swapping when loading saved control preferences.  I’ve made this bug fix both within MMORPG Tycoon 2 (where it was a major problem), and within the trunk VectorStorm library (where it was less of a problem, since no standard VectorStorm games were using mouse control).  If any of you have been looking at or using the VectorStorm library, you should go get those fixes.  If not, don’t worry about it too much, just be aware that I’m back to the compiler again.  :)

I’ve also begun poking with Tycoon code again.  Not a whole lot;  am just slowly easing myself back into it.

I’m not sure precisely what I need to do to avoid this, since the sort of asset re-use that was key in making this awful vista is also going to be required in MMORPG Tycoon 2.  Maybe if there were any sort of shadows being cast it’d help.  Or maybe if the trees weren’t quite so awful.  Or if the rocks weren’t simply amorphous blobs wedged into a monochromatic height map.  I don’t know.  I saw this vista in a modern MMO the other day, and was immediately seized by the terror of “how do I avoid creating this, myself?”

Anybody have thoughts?