I wanted to make a point about the flexibility of the system, so I present you with the letter “A” stamped into the terrain.  Doing this required no code changes;  just creating the brush texture (which I’ve pasted over the screenshot in the image), and editing two data files (one to tell the cursor how to use the image, and one to add it as an action bar button).  All up, it took about five minutes.  (About half as long as it took me to write this post)

I don’t imagine that anyone will actually want to create a world with the letter “A” stamped into the terrain.  But this approach is extremely flexible, and should make it much easier for folks to customise the interface to make it more convenient for themselves and/or others.

Now I just need to make the GUI not be quite so terrible, visually.

Today I started hooking up properly moddable terrain editing tools.  This will both allow me to develop terrain editing capabilities more quickly (and with less code!) and also allow end users who are interested to do so, to create their own terrain editing abilities, including adding additional options to the toolbars.

What I’ve done is set up terrain tools to be able to reach image files, and use those images to determine how they affect the terrain when applied.  50% grey means no change, lighter colours go up, darker ones go down.  By changing the ramping or the shape, you can get terrain editing to automatically make different types of terrain shapes.  In the screenshot here, I have made a much steeper hill than is normally easy to create, just by using a tighter gradient than normal (gradient image shown on the right).

I’ve also exposed a lot more internal variables to be modified via simple text data files.  Tomorrow, I’ll be filling this out a bit further;  adding steep chasms, craters, and similar shapes.

So here’s something I’ve been waiting an awful long time to see; I’ve finally reached the end of the massive AI refactoring job that has been consuming so much of my spare time over the past few months (that’s overstating it; I’ve really only been feeling up to outside-of-work coding for a few hours per week, so this hasn’t actually taken up all that much of my time), and have merged it back into the main development trunk of MMORPG Tycoon 2.

What does this mean? Well.. not a lot, immediately; mostly it’s doing the same things that it was doing before, just with much more well-organised code. There are probably a few new bugs in the AI which weren’t there before (as this new implementation hasn’t been as extensively tested as the old one). But the architecture of the system is much, much better now.

For example, the handling of human-controlled PCs and AI-controlled PCs move through almost identical code paths. This means that human-controlled PCs now receive experience points and level up and get quest completion credit, the same way that AI-controlled ones do. The Bartle type of a PC is now stored on the subscriber, not on the individual character.. which means that no matter how many alts a subscriber makes, they’ll all have the same Bartle type, as you’d expect (which wasn’t the case, before). Similarly, the various desires (“socialise” vs “gain level”, etc) are now on the subscriber instead of on the PC, so a subscriber could now theoretically meet a desire to socialise by logging in with a different character, which was not previously possible.

But more important than that, all of the AI logic has been pulled apart into clean(er) modules, so it should now be much, much easier for me to insert new AI behaviours, such as adventuring as a party or as a guild.

I have also begun work on supporting different xp allocation mechanisms. Previously, MMORPG Tycoon 2 allocated experience points to every PC who thought they were attacking a monster when it died, whether or not they actually ever hit it (and similarly, any PC who chose to run away after damaging the monster would receive no XP if it was killed immediately afterward). Now, monsters keep track of who has damaged them, and award XP to those PCs. This also will be important for supporting party-based adventuring.

So what’s up next? Well, I’m really wanting to push ahead with some world editing interface work, and I’m thinking really hard about whether the MMORPG Tycoon 1-style spline roads are really a good choice, here, or whether a more traditional “tile grid”-based road system would make more sense for a game that’s played from this close up. The problem with spline roads is that a slight movement to a long road can require huge amounts of terrain modification — often I have to rebuild almost a quarter of the visible terrain in order to cope with a path that’s now taking a slightly different direction. Maybe the simple tile grid layout would help localise changes more. Although I really do like the freeform paths, I must confess.

Anyhow, moving on! There’ll be some general life news later in the week, as well. Further details then!

Profiling is fun when it shows you something big that’s easy to fix.  Profiling is a lot less fun when everything is equally slow.  But luckily in my case, there was something big to fix.  :)

I noticed the other day that editing roads in MMORPG Tycoon 2 had become extremely slow.  (Well.. actually I noticed that roads didn’t draw at all.  It was only after I fixed them not drawing that I noticed that they were slow to edit).   This was odd, because I have videos of early road editing, where it hardly touched the frame rate at all on a system much slower than the one I’m using now.  So after some profiling, I found that creating their procedural geometry was taking up about 75% of the whole processing time for the frame when they were created.  This was extremely bad when editing them, especially (because they’re regenerated every frame, when they’re being edited, or when the ground under them is being edited).

Those of you who were at Freeplay will remember the part of my talk where I discussed optimising games, and when to do it (or rather, when not to do it).

For those who weren’t there, the tl;dr version is this:  don’t optimise anything unless you have timing numbers that proves it’s slow relative to the rest of your program, and against which you’ll be able to judge your progress.  Ideally, you get these numbers by using a profiler of some sort.  (Visual Studio has one built in, modern Xcode uses Instruments while old Xcode used Shark, under Linux you have gperf, etc)

What I didn’t say in that talk (and which I’ve regretted ever since), is that there’s a second criteria which can make it acceptable to optimise some code, even if you don’t have performance numbers.  Here is that other criteria:

Monte Carlo Profiling

1. Make a build of your game with optimisations turned on (a “release” build, or “retail” build, or “dist” build, or whatever you call it), but with debugging symbols still present.
2. Run this build through your debugger of choice.
3. At any time of your own choosing, pause the execution of the game, and examine the call stack shown in your debugger.
4. Repeat step 3 nine more times (for a total of ten pauses and ten callstacks).
5. If in seven of the ten total times you paused the game you were in the same area of code, then you may spend time optimising that area of code, even without first profiling it.

The Monte Carlo test is basically very quick and ad hoc profiling.  Profiling does exactly this same process, but instead of doing it ten times, it does it millions of times, far faster than you’re able to do it manually.  It avoids having to go to the bother of setting up proper profiling, and will still point out anything that’s extremely slow in your code.

But back to my story.  In my case, I did do real profiling.  Here’s what it looked like in Instruments:

That graph is of CPU usage.  (Also, it totally breaks the site frame layout, but I don’t care.  I want you to see the details.  :) )  The spike at the start is the game starting up (and generating all the world geometry all at once).  The low part in the middle is just me looking around a bit, and then the heavy bit that I have highlighted is me editing a road — just dragging one of its endpoints around the map at random.

As you can see from the screenshot, it’s spending about 30% of the frame time in vsMeshMaker::BakeTriangleVertex, 20% of its time in vsFloor (called from vsMeshMaker::BakeTriangleVertex), 16% of the frame time in vsVector2D::SqLength() (also called from vsMeshMaker::BakeTriangleVertex)…  you get the idea.  Add it all up, and we were spending about 75% of our CPU time inside BakeTriangleVertex.  That BakeTriangleVertex function was really, really heavy for some reason, and I needed to figure out why.

Basically what vsMeshMaker::BakeTriangleVertex does is to look at all the triangle vertices in a procedurally generated model, and to decide (one at a time) whether they should be merged with other nearby vertices to make a smoothly curving surface.  The hardest part of this is actually finding other nearby vertices.  Previously, I had sped this up by dividing up space into grid squares, and storing each vertex into the appropriate grid space.  Then when I wanted to know about nearby vertices, I only had to check the other vertices in the same grid space, not every other vertices in the model.

But as it turns out, those grid spaces were too large, especially for large objects like roads, and so I was still comparing far too many vertices for merging.  So I increased the number of grid spaces (and made each grid space smaller, so there would be fewer vertices to test in each), so that instead of having an 8x8x8 grid and needing to test every vertex in those large squares, we have a 32x32x32 grid and only need to test every vertex in much smaller squares.  I had also noticed that very simple functions like vsFloor and vsVector3D::SqLength() were showing up in the profiling, and so made those functions inline, to hopefully reduce the cost of calling them.  This led to this graph:

This one is fascinating.  Notice that the amount of CPU being used has dropped substantially, compared to the previous one.  Also note that functions like “vsFloor” and “vsVector3D::SqLength” have vanished, since I made them inline (they’re now included in the caller’s time).  Increasing the resolution of my vertex lookup grid had sped things up a lot — we’re down from 75% of our frame time, down to an average of 40% of our frame time (and made a huge improvement to the frame rate).  But here’s the interesting thing that I hadn’t noticed previously:

Look at that graph, how it increases over time.  I wasn’t doing anything unusual in-game, just rapidly dragging one end of the road back and forth over a single area of terrain.  But the more time I spent editing the road, the slower editing became.  This eventually pointed me to a bug in my code, where some of the merging work from previous frames was being left-over to do again on all subsequent frames, even though it didn’t accomplish anything on the new frames.  So I fixed that bug, and profiled again:

Great!  Getting rid of that leftover repeated work brought us down from 40% frame time (on average, during a long editing session) to about 2.6% frame time, low enough that you don’t even notice a frame rate hit in-game.

The problem now was that my grid was so fine, that it actually uses a lot of memory.  A 32x32x32 grid has about 32,000 spaces in it, and each of those needed storage to hold an arbitrary number of vertices.  And I had one of these for every road (and indeed, every building, tree, etc).  That builds up fast!  And while 2.6% is low enough to be acceptable, I would have liked it to be even lower.  Let’s face it, my placeholder “road” model is ridiculously simple.  I’d really like this code to be much faster than it is now, to handle the much more detailed meshes that I’ll be throwing at it later.

So I set out to replace the grid with a new structure:  an octree.  For those who are interested, this implementation has been ported back into the main VectorStorm trunk, as VS_PointOctree.h, inside the VS/Utils/ directory.  It’s basically a tree structure which expands as more data is added to it, so it should behave far more predictably than the grid structure.  What’s more, it expands as data is added, instead of being a fixed (huge) size, which means that smaller models use much less space, exactly as you’d hope.  Profiling isn’t much different (down to about 2%), but memory usage is drastically lower.  So I call that a win.  And in future, I may return to this code to see if I can’t whittle it down even a little further, once it starts moving up into a prime spot in the profiling again.

I’ve also been doing other stuff recently, but I reckon that this blog post is enough of a novel already, so I’ll save the other stuff for tomorrow.

So work has been busy, and I haven’t had nearly as much time to code on things here as I would have liked, but I’ve made some interesting progress over the past week or two.

The first thing to mention is that I believe I’ve finally cracked the nut of making the game’s height map look good;  no more weird diagonal diamond patterns visible across changes of surface facing!  (this involves re-triangulating the surface to try to minimise artefacts)  This was especially bad before at sharp creases, such as you can see at the bottom of this extreme hill, or around the edges of darker areas, at the crest of the hill.  It was also often very visible on the mountains at the region boundaries.  But now, it’s pretty much bang on every time.  I’ve been doing this “re-triangulate the mesh to try to hide those diamond patterns” thing for an awfully long time, but it was only today that I finally found the right combination of heuristics to actually make it work reliably.

I’ve also moved from a simple directional-lighting light model to a wrap-around lighting model, which gives the game a much softer look, which also helps with the smoothness.  And is just a nicer look overall, in my opinion.

I’ve also started implementing more ground shaping tools.  Most notably is the ground leveller.  Basically, it tries to flatten out the terrain under your cursor.  There are versions which will only raise terrain, only lower terrain, and which will go in both directions.  This is great for setting up flat surfaces to place buildings upon.  (Since buildings only place their centre point on the ground, it’s common for their corners to be floating, or for the ground to be embedded through the building geometry.  Still TODO is to have buildings automatically flatten the ground under them, when placed.

Oh, and incidentally, grass now flows downhill, instead of always east-west.  I don’t think you can make it out in this screenshot (I’m too far away), but trust me.  :)

I still need to do thinking about how to set up the overall ground editing interface.  The “paint on” interface is nice, but even with the work being farmed out to background threads, it’s hitting the frame rate too hard, when the user paints with a large brush.  I’ve been thinking about having the terrain editing tools work with a grid lattice, which I could easily update in real-time.  When the user is happy with his changes, he could click an “Accept” button, to assign the terrain rebuilding to a game developer.  This would make it work similarly to how placing buildings works;  giving commands to actually be executed by minions.  And it would also solve the problem of needing to rebuild the terrain every frame;  instead, it could just happen in the background, when the developer is actually performing the task.  Need to think about this more;  I’m not sure how the user would specify terrain types on a grid lattice, for example.

I took a stab at continuing with player AI, but rapidly banged my head against the current code setup.  I think I’m going to have to heavily refactor the code, before I can add anything more to it.  It works right now, but it was written in the fastest possible way, way back for my self-induced “Milestone 1″, and at this stage, I’m stumped as to how to add parties to the system, without breaking the quest-following behaviours that are already in there.

Very minor update today, since I didn’t have much time to code, but got a few things implemented.

First, VectorStorm materials now have an optional “layer” value.  This value defaults to zero.  Within a scene, material batches are drawn in order, according to increasing “layer” values.  This allows us to force transparent geometry, such as the extending arrow effect, to draw later than opaque geometry, such as the ground surface.  Note that this only works when rendering is performed via the material batching interface on the vsRenderQueue;  materials applied by directly setting them within a vsDisplayList will render exactly as requested within the vsDisplayList.

Second, I’ve been removing lots of old, dead code.  Most notably, I’ve now removed the carcass of the old combat system.  Now that players and monsters are attacking each other using via configurable attacks, the fake old system really isn’t necessary any longer.  The Player AI really needs some serious refactoring;  I’m probably going to spend the weekend trying to massage the code into a simpler and more extensible state.

Here’s what’s new:

• Set up a generic “task management” system for MMORPG Tycoon 2.  Systems generate “tasks”, and tasks are assigned out to worker threads, which work on those tasks, and then let the main game know when the tasks finish.  Heightmap building now uses these generic worker threads, instead of having their own threads.  Right now, there are eight worker threads.  But in theory, it’s probably best to have the same number of threads as the user’s CPU has cores, potentially plus one.
• Clutter generation stuff occurs using dedicated threads.  I need to convert these over to use the new generic worker threads.
• The game now waits for all queued up tasks to complete, before rendering the next frame.  This fixes some problems I was having before, where different blocks of terrain were updating at different times and not matching each other, just due to how long they took to calculate in the background threads.
• Fixed some issues with my OS X semaphore implementation (since OS X doesn’t support unnamed semaphores, for unknowable reasons).
• Worked around a renderer bug, where anything which had no texture was being drawn opaque, regardless of its material settings.  This affects the live trunk, too;  I’ll bring the fix across at the same time as the various threading fixes I’ve mentioned in the past week or so.  (If anyone needs a quick workaround in the meantime, just remove the line “m_state_SetBool( vsRendererState::Bool_Blend, false );” from vsRendererSimple::SetMaterial() on VS_RendererSimple.cpp, line 958.)

And now, I’m completely sick of dealing with the complexity of threads, and so the next thing on my list will be more player AI.  I think I’m going to work on making players assemble into parties that do quests together.  That ought to be entertaining.

And it raises some interesting questions, which actually come up in the development of real MMORPGs; how to handle characters who are of differing levels, but who want to team up — ignore it?  Boost the level of the lower character?  Lower the level of the higher player?  Not allow teaming between those characters?  And worse, what do we do when a character completes a quest multiple times, accompanying different party members.  Do they get the reward multiple times?

My understanding is that most modern MMORPGs now give credit for completing a quest again in this sort of situation, but older MMORPGs often will not give repeated completion XP.  I wonder whether I need to make this configurable for players, or whether I should just take the modern approach and assume that that’s what everyone will want.

Lots of stuff to think about!

Apologies for all the similar screenshots lately;  it’s what happens when I’m working on the behind-the-scenes tech, instead of new features.

So I’ve finally finished converting the world across to being generated in a background thread.  This means that as the player moves around the world, there are no longer any frame rate stutters as new blocks of terrain are created in front of the player.  I’ve also dramatically simplified the world rendering code, which comes as a tremendous relief.

So now that I’m not worrying about performance, I’m going to be diving back into more detailed world-building.  Which should mean more interesting screenshots in the near future!

So today I’ve done phase two of the multithreading of clutter.  Yesterday, each “clutter tile” kept its own thread which would sleep while waiting for a “build some new clutter” job to arrive.  Today, there is now just one “worker thread” which handles all clutter rebuilding tasks.  This behaves a lot more nicely (particularly for people with fewer cores on the CPU), and should scale a lot better.  And it’s noticeable faster again to rebuild tiles of ground clutter, even compared against how fast it was yesterday.  From my point of view, the system side of ground clutter is now basically finished.  All that remains to do is to generate different types of clutter depending upon the terrain type (right now, it generates grass on everything, even bare mountain faces)

The plan is that tomorrow and over the weekend, I’ll be bringing this same system over to edge clutter, and to the base height maps as well.  This should substantially improve the frame rate jitters that I currently get when a new block of world terrain comes into view.

Today, MMORPG Tycoon 2 has finally become a multithreaded game.  It’s not ubiquitous by any means, but it’s taken the first step;  the ground clutter is now being generated in a background thread.  The only bit of clutter-related code which is now occurring in the main thread is copying the final geometry into the buffer that OpenGL renders from.  Now all the setup work as you move around the map is being handled in separate threads.

This means a couple of things.  For one, it means that MMORPG Tycoon 2 can finally use a little more of the CPU time from everyone with a multi-core computer (which is just about everyone with a computer from the last six years or so), and it also means that as more tasks can be moved out into other threads, I’ll be able to do more and more processing without hurting the game’s frame rate.

I’ll also note that Mac OS X’s support for semaphores is half-broken, and Win32′s support for semaphores is bizarre and generally unrelated to traditional semaphores.  Makes me a sad coder.  (for the non-coders in the audience:  semaphores are a tool for synchronising multiple threads, to help keep them from stepping on each other’s toes)

More details tomorrow.  Or later today, I guess, technically.  And semaphores will eventually be ported back into the VectorStorm trunk, if anyone wants to look at them.  Also fixed some bugs in the vsTask class, and added features to the vsMutex.  Will need to move all those changes back to the live trunk in the near future.  Maybe over the weekend.