Them’s Fightin’ Herds is an advanced 2D fighter based on Lab Zero’s Skullgirls game engine. The project started with humble beginnings with a group of friends who were passionate about fighting game mechanics (and also cute animals beating each other up).

The game featured a very large scope, from its very complex characters, to a dynamic music system, to a highly complex visual lobby system complete with a co-op/competitive mini-game called The Salt Mines to a narrative story mode featuring complex AI fights. Even with all of these overlapping modes, the game did end up quite stable.

Serious work began on the project in 2016 after a successful IndieGoGo campaign raised nearly 600k. I quit my day job and started planning the broad strokes of the game. It was a challenging project, to say the least. None of us has professional experience in the games industry at the time. Much trial and error ensued.

I served as the lead, and often sole, engineer on the project from start to finish. I had to design almost everything that was not provided as a feature from Skullgirls. Even those features had to be heavily modified and sometimes even rewritten to fit into our new game flow.

We had some BIG ideas; far outside of the scope of what Skullgirls did. We weren’t experienced enough to know exactly how big they were at the time. The Pixel Lobby was the first Big Feature that everybody on the team wanted. A place where you could go to hang out with your friends in a casual environment, organize tournaments and even play mini-games.

The Pixel Lobby, and especially salt mines, turned out to be a massive undertaking with long hours to get every little detail correct and stable. Anybody who has dealt with making 24/7 multiplayer servers knows it’s no joke. Not only did they have to support complex state for all players involved, but they had to run for months uninterrupted. This meant an engineering, verification and testing process far beyond anything I had previously done. I’m glad to report that it was a success, and the TFH lobby servers are one of the most resilient pieces of software I have built to date.

Salt Mines, in particular, too a huge amount of effort to ensure that it would not break over the course of running games for months. It had to coordinate game rules, organize matches, handle NPC life times, dish out rewards and engineer various anti-cheat measures to ensure nobody could cheese the system.

The full stream of this play-through of Salt Mines is available on Youtube here

I very much had to learn “on the fly” throughout the project, and “learn the hard way” for many approaches. The biggest takeaway I learned is what every good engineer eventually learns: simple always wins. Of course, if you can’t make it simple, at least find some way to make it understandable. Distributed matchmaking was an example of the latter.

That was another valuable lesson: knowing when to apply complex tooling to solutions. I used a finite state machine creator with a code generator to craft and maintain several features. For some problems, this turned out to be totally unnecessary and just cluttered the resulting code. For a select few, mostly involving networking protocols where LOTS of things can wrong at any time, they were a godsend. The rule of thumb, for this project, turned out to be: if more of your branches involve error handling than not, you should consider a code generation solution like this.

One of the features I’m most proud of is the Replay Widget. This features a media player style control panel that includes the ability to rewind the match using GGPO’s rollback capabilities.

Near the end of the project, when players actually began stress testing, that’s when the real work started. I would often set up multi day soak tests, stressing servers to their limit and running dozens of clients at once on multiple computers with sandboxing or simulated clients.

After shipping the game, we developed a devoted fan base, which is something I am particularly proud of. We worked closely with them to improve the game over the years and continue to add new features and polish old ones.