Introduction

In June this year, Good Old Games had a promotion where they gave away the classic RTS game Total Annihilation for free. I never played the original game before, but I had played games on the open source engine inspired by it, Spring, back in the late 2000s (proof). So me and some friends fired it up to try it out on LAN.

Turns out it is pretty hard to get the networking going, which is expected for an old game. So we turned instead to what we knew, and downloaded the main way people have been running Spring games for over a decade, SpringLobby.

While we were able to play some games, there were crashes, frequent random errors, and performance issues. So I decided it might be a fun side project to work on a replacement.

Goal and Approach

Implement a desktop app replacement for SpringLobby that can connect to uberserver (the main Spring lobby server implementation), using the Spring lobby protocol, and which can run the Spring application using the interface of passing in the script.txt file. The program should also at least point users in the direction of where to get resources such as maps and games to play.

I'm going to implement this using Clojure, a JVM Lisp that emphasizes data. There is a great talk by the language designer Rich Hickey on the types of programs the language is targeting, and I think this one fits directly into that category of "situated" programs.

It will need to have a responsive UI for user input, a TCP connection to the lobby server, as well as some method of tracking resources on disk that will change over time, plus update lists of new resources that can be acquired, not to mention actually starting and monitoring the engine process itself.

There are many language features, as well as libraries in Clojure and Java that provide the building blocks for all of this. I hope to show some of what's available, as well as some pitfalls I've found along the way.

First Steps

I started with a basic UI using a library I've been keeping an eye on, cljfx, which turns JavaFX into something resembling a React application, where you describe how state is rendered into the UI. Any changes you make to the state are watched by the renderer, which adjusts the UI accordingly. This is all done based on a Clojure Atom. You can see it started off small.

Screenshot of Initial UI

Around the same time, I started talking to a local uberserver instance. I needed to make some changes to get it working on Ubuntu 20.04 and add enable debug logging, so I forked the repo and made changes here. If you want to see the project in action, you can run this server yourself and point the lobby to localhost.

Once I had the server running, I added the Aleph networking library, which provides an easy wrapper over Netty for sending and receiving messages on a TCP connection. I also used gloss both for splitting the TCP data into messages, as well as parsing some of the encoded pieces of the messages. I think it may be possible to do more message parsing using gloss, but for now I just dispatch on the first word of each message, which works for this protocol.

Running Spring

After that, I took a break from things for a month, and when I returned I added the 7-Zip-JBinding library for parsing most maps, and extracting engine archives. I also added basic exec of the Spring executable, and some more parts of the protocol, like adding bots.

Screenshot of UI With Battle and Bots

One thing I ran into, the Spring process would hang at the same point in the logs every time. I wasn't reading the stdout or stderr streams from the process, so I added that to try to figure out what was going wrong. Turns out that fixed it, I guess a buffer was being filled

        (let [^"[Ljava.lang.String;" cmdarray (into-array String command)
              ^"[Ljava.lang.String;" envp (fs/envp)
              process (.exec runtime cmdarray envp isolation-dir)]
          (async/thread
            (with-open [^java.io.BufferedReader reader (io/reader (.getInputStream process))]
              (loop []
                (if-let [line (.readLine reader)]
                  (do
                    (log/info "(spring out)" line)
                    (recur))
                  (log/info "Spring stdout stream closed")))))
          (async/thread
            (with-open [^java.io.BufferedReader reader (io/reader (.getErrorStream process))]
              (loop []
                (if-let [line (.readLine reader)]
                  (do
                    (log/info "(spring err)" line)
                    (recur))
                  (log/info "Spring stderr stream closed")))))
          (future
            (.waitFor process)

Parsing Maps

Parsing map files involves multiple archive formats (Zip and 7-Zip), a custom binary format, a custom config text format for older maps, as well as executing Lua for the new format.

I started with the old SMD format, which also seems to be the same format that the script.txt is written in. There's a library for creating grammars for parsing called instaparse that often makes describing custom formats like these fairly straightforward. In this case the grammar came out to roughly ten lines plus a number of lines to postprocess it into a format that's easier to work with.

However, there is a possible case with instaparse where parsing never terminates. So I wrap its execution with a library called clojail, which basically runs a Thread, and calls .stop if it doesn't complete in time.

For the new mapinfo.lua format though, it's more difficult, since the map data is now stored as Lua code. I'm sure this is fine in the engine itself when the Lua is being executed, but it's a pain to deal with outside of that context. Thankfully, there's a Java library for executing Lua code, luaj. This, along with some mocking of the Spring internals that sometimes leak into map "data", allows parsing across all maps I've tested.

(defn read-mapinfo [lua-source]
  (let [globals (JsePlatform/standardGlobals)
        lua-chunk (.load globals (str mocks lua-source))
        res (.call lua-chunk)]
    (table-to-map res)))

Now for the most complex piece of all, the actual map data file .smf. This is a binary file with a header describing the layout. Fortunately, there's a great library smee/binary that fully supports formats like this with headers. There is one more source of complexity here, since the map files can be laid out in any order, so we need to use the offsets and lengths for each section. Once the file is parsed into its component pieces though, we are only partly done: we still need to parse the minimap image.

The minimap image is stored as the pixels of a DXT1 compressed image, one of the algorithms for DDS. As far as I can tell, Java support for this format is... almost nonexistant. Not to mention, the map format doesn't store the whole file, but just the pixes, which happen to be a magical 699048 in length.

After some failed attempts, I managed to grab the bytes needed to extract the .dds image. The last step is to convert from DDS to a Java Image, either through a library or by manually creating from raw pixels.

I looked at a number of other implementations of this, like BALobby and smf_tools. smf_tools uses the Squish library to uncompress the minimap data into pixels. So I stumbled across JSquish, a small Java implementation of the Squish compression. While it doesn't seem to be in Nexus or other package management, a jar of it exists here. Now I can fully extract the minimap bytes into an Image, for diplay or saving to disk in a more standard format like .png.

Not sure there's a better way, perhaps maybe just using JNI to call some C++ or Lua implementation. But that might just lead to more reliance on these obscure formats in the first place.

Resources

Now we have some basic processing of local resources, but it would be useful to be able to get new resources like games to play. One method of doing so is also custom to Spring, called Rapid. Rapid seems like a way to deal with small changes between game versions without rolling a new archive, which can be quite overall hundreds of megabytes or larger. In other words, similar to git in many ways. The .sdp format is binary and can be parsed with the same binary library mentioned earlier. To actually download packages, I just shell out to the pr-downloader executable.

Other resources can be downloaded with http, if you know where to look. I use the main clj-http library which wraps Apache HttpComponents, and this allows easy download to a file, and progress monitoring. At first I tried to build urls based on specific content to look for, but there are issues such as not knowing what case the file will have, or if it is .sdz or .sd7. Now it periodically fetches and parses a few known websites that are usually html or xml.

One thing I'm trying out in the project, is the "extensible" part of EDN, the Clojure data format. I added custom tags for java.io.File which I use in basically every resource. The custom tag allows these objects to be written to and read from config files without any extra parsing steps

(defn read-file-tag [cs]
  (io/file cs))

(def custom-readers
  {'spring-lobby/java.io.File #'spring-lobby/read-file-tag})

(defmethod print-method java.io.File [f ^java.io.Writer w]
  (.write w (str "#spring-lobby/java.io.File " (pr-str (fs/canonical-path f)))))

Development Workflow

A brief intermission to talk about development workflow. The goal is to have as tight a feedback loop as possible, while recovering from errors in order to not restart the repl for often days at a time. I mainly rolled my own here to learn a bit and get more control. There is tons of potential from just using Clojure's vars and refs. I've previously used others like component and system, and have heard of others like mount but haven't used it.

The main reloading library is tools.namespace which is used for most reloading as far as I can tell. Basically, when refresh is called, all old namespaces (usually one file maps to one namespace) are unloaded, then the code is recompiled from what's now in the files on disk which should recreate the namespaces, and an optional :after function is called. I use a file watching library called hawk to do the file watching and some teardown (like stoping periodic tasks started with chime as well as restoring the program state.

Basically, the program uses one atom for its state, and various threads makes changes to it. Another feature that I didn't know about until recently is add-watch which calls a function whenever the state of a ref like an atom changes. I (ab)use this for things like updating config files when the user makes changes in the UI, or loading the minimap from disk when the current map changes.

  (add-watch state-atom :state-to-edn
    (fn [_k _ref old-state new-state]
      (doseq [{:keys [select-fn filename]} state-to-edn]
        (let [old-data (select-fn old-state)
              new-data (select-fn new-state)]
          (when (not= old-data new-data)
            (spit-app-edn new-data filename))))))

There are a few situations where reloading would not update the running program, that needed special treatment. The UI view, the UI event handler, and the TCP client handler. For these, in dev mode, I use (var-get (find-var ...) to dynamically get the new version when namespaces are refreshed. That way, the program will keep its entire state, keep rendering the UI, talking to the server, etc., even after recompile, which is usually less than a second. This works fairly well, although compile errors sometimes cause namespaces to not be reloaded, and various things will break until full compilation happens again.

Another caveat with cljfx, if you are using an atom directly and not using the pure event handling, be careful not to deref the state atom again within the component rendering fns. I had an issue with the for a while which caused double rendering, and only found and fixed it recently.

I would be remiss if I didn't mention my editor setup, which is Neovim with the Conjure plugin as well as parinfer. It's not perfect, but it allows me to connect the editor to the running repl and evaluate code, which is very nice for prototyping. Since starting the project, I learned about another interactive dev tool, reveal, but I haven't tried it out yet.

Debugging

While I don't use a full step-through debugger, there's a nice lightweight library hashp that lets you insert #p before any form, which will print the data when that form is evaluated, plus the file, line, and function. I've seen debuggers in Calva as well as Cursive. More recently I've seen the flow-storm debugger which looks cool.

File Scanning

My first attempt to updating resources over time was to use file watchers with something like hawk which I use in the development loop. However, this proved to be very slow and used way too much CPU to track all the Spring folders. I think it's due to the number of files, especially with git folders, extracted archives, and rapid packages. So I abandoned using file watchers and switched to periodically checking files in a few select directories. While it doesn't necessarily react to file changes as quickly, it does check after downloading or extracting resources, or when joining a battle.

Current State

The project is fairly usable for my purposes right now, albeit it assumes some prior knowledge for usability. It also might not play entirely nice with the other main lobbies like SpringLobby or Chobby due to not implementing unitsync yet. So I don't know if it's fully compatible with any of the main servers yet.

Despite that disclaimer, the lobby allows download of various game resources, registering and loggin in to various servers, starting or joining battles, playing with bots or other players, choosing starting positions (no start position rectangles yet, todo), and customizing game options. Here's a fairly recent screenshot:

I'll describe some enhancements I'd like to make, which should also be kept in the changelog.

Advancements

Now that we have feature parity with SpringLobby in a number of areas, I've started finally adding the improvements I set out to do in the first place. One annoyance, the "Random" starting positions didn't seem to be actually random, but rather just a hash or something. So I added shuffling of team ids in this case, so my friends and I can start in different unknown locations.

Another issue, depending on where the uberserver is located, the wrong IP address for the host may be used. So I added an override, since it is just a part of script.txt ultimately, although the lobby protocol uses it in battle open.

GitHub Actions

Based on the great example app cljfx/hn I added some GitHub actions to build the platform-specific jars and installers for Windows, Linux, and Mac.

Future

Since it's part of the lobby protocol, I do want to implement chatting sometime, despite the prevalence of Discord.

I'd like to add TLS support for the client to server communication, but I'm not sure how difficult it will be to do, since it looks somewhat nonstandard.

Another friend tried out the game, but was very frustrated by the controls, due to familiarity with Starcraft. Customizing keys is supported but I think I'll need to add a UI around it.

One thing I haven't touched is interacting with native code directly using JNI. I tried to use Java based solutions as much as possible, but I think it may be necessary for some thing like Spring's unitsync to generate consistent hashes of resources. I'm keeping an eye on the upcoming Foreign Linker API related to this as well.

I also try to capture these in CHANGELOG.md.

This Website

To make these web pages, I used, you guessed it, another Clojure library called Cryogen. After a few false starts I figured out the URI paths and how to export to the gh-pages branch so it will be hosted here. I was also able to find examples of how to use a GitHub action to build the page and commit to gh-pages automatically when master changes. This, along with running a local server as described here, makes the process fairly smooth.

Conclusion

This project started out as scratching an itch with playing an open source game, but it turned into an exploration of the Clojure (and Java) ecosystem for making a desktop app at the end of 2020. There were readily available libraries for the wide variety of technologies needed for this project: git, 7-Zip, custom binary formats, TCP and the message format, parsing HTML, parsing custom text formats, executing Lua code, and even the obscure compression library Squish.

A huge shout out to everyone working hard to make the Clojure ecosystem as impressive as it is!