Welcome to the new Simian Zombie! It looks almost the same as it used to, and it works in pretty much the same way, but the back-end is now completely different.

The old Simian Zombie used WordPress as its blogging engine and was hosted on BlueHost. This combination resulted in terrible response times. Fetching and rendering the site took an average of around 5 seconds (testing suggested a range from 2 to 9 seconds), which is abysmal. Worse than using WordPress’ combination of PHP and MySQL was sharing a database server with a site that was either appallingly written or hugely popular; in either case, it kept taking down the database.

The new Simian Zombie is hosted on a VPS provided by Digital Ocean. It uses Gobble as its blogging engine, which is written in Go, uses the filesystem for permanent storage, and serves posts from an in-memory cache. A full render of the same blog page takes around 2 seconds on average (with a range from 1 to 3 seconds). Most of that time is used up by the JavaScript syntax highlighter; fetching the rest of the page usually takes around half a second. It probably helps that there’s 50% less HTML in the Gobble layout.

It took longer to re-format the content from the old site into correct Markdown than it did to write Gobble. Around half of the posts lost their formatting during the automated HTML-to-Markdown conversion process, so I had to add it all back in. I’ve been reluctantly working on fixing the posts since I announced Gobble, which was two months ago. After that was done, I had to ensure that all of the various downloads and images still worked (I’m sure I’ve probably missed a few links).

I’m hoping that Gobble holds up to being dogfooded. I’m also hoping that I haven’t lost anything during the transition.

My last post on this subject wasn’t overly informative, so here’s some more information about my attempt to install an email server on Linux. Note that this post will be of no use to you if you are trying to do this yourself as I entirely failed to get it working.

First of all, you’ll want to install the email server. There are a few of these out there, but if all you want to do is be able to receive email from a few different domains you’ll probably end up opting for Postfix. Postfix isn’t really a full email server. It can receive email but can’t send it, doesn’t offer any kind of client, doesn’t support POP3 or IMAP access or anything else. To get any of this you’ll need to install mailutils and sendmail and Dovecot and half a dozen other programs. I don’t want any of this.

Some command-line wizardy later and you’ll have Postfix installed. Tweak some config files, and…

It doesn’t work.

You need to change your MX records to point at your server. Tweak the DNS, check that the changes have propagated, and…

It doesn’t work.

Perhaps the firewall is blocking the ports? You need to look at the documentation for iptables. iptables seems to be a hideous mess of a program, so you look for something else and find UFW. UFW is very easy to use. Some simple configuration, and…

It doesn’t work.

I had a server that would send email to itself quite happily, but failed the MX test from mxtoolbox.com. Email from anywhere else failed to reach the server. There’s no obvious way to diagnose this because Postfix doesn’t complain about any configuration errors and doesn’t log anything. The DNS settings appear to be correct. Disabling the firewall doesn’t help. As usual when dealing with anything Linux-related, most of the available documentation is either woefully out of date or specific to a completely different distro. (Imagine a Linux in which all of the effort went into creating a handful of tools that worked properly, instead of a hundred crappy, semi-functional alternatives.)

I have no interest in server maintenance. None. I don’t want to run my own email server. It’s simply the downside of migrating to a VPS so that I can run some Go web apps. I’ve wasted an afternoon on something I actively dislike doing.

At this point I decided to look around for an alternative solution. I got nullmailer set up. It’s supposed to automatically redirect incoming messages to another email server, which would enable me to forward messages straight to my GMail account. Clever! Unfortunately, I couldn’t get it working. SSMTP does the same thing (though it’s no longer maintained) and this one did work, but Google detected its attempt at logging in and blocked it.

Now I need an alternative to my alternative. Is there some way I can point the MX records at a dedicated email service and have someone else handle all of this tedium?

I eventually settled on creating a Google Apps account. It took some setting up – account creation, MX records, domain verification, counter-intuitive web UI – but it was considerably less annoying than running my own server. On the downside, Google Apps accounts used to be free but now cost $5/month. On the upside, all of the email to my woopsi.org and simianzombie.com email addresses now appear in GMail.

In other news, woopsi.org is now hosted on my Digital Ocean VPS. I had to convert it from a PHP website back to static HTML pages, but as it wasn’t using PHP for much more than some simple templating this wasn’t a big problem.

Good grief.

A year or so ago I wrote my own blogging engine in C#, and then a while later pointed out some of the worst flaws in the system. The major one for me was the choice of platform:

…using C# means I have to maintain a Windows virtual machine, an installation of VMWare Fusion and a copy of Visual Studio. That’s far too much overhead for something that could be written in a few hundred lines of Go and deployed on any cheap Linux server.

I’ve rectified that with a new blogging engine called Gobble, written in Go. Although it blew away my initial estimate and ended up consisting of 6000 lines of code, CSS and HTML templates (etc), it fulfils my goal of being a blogging engine that:

• Consists of a single binary;
  
• Uses the file system for post storage instead of a database;
  
• Supports comments;
  
• Uses Akismet and reCAPTCHA for spam prevention;
  
• Uses Markdown for post/comment formatting;
  
• Can be deployed easily to a cheap Linux server/VPS;
  
• Has a search feature;
  
• Supports tags.
  

It has a lot of features that I like in WordPress that didn’t make it into BitBlogger, including search, self-hosted comments, syntax highlighting and more. It even retains the WordPress post ID-based URLs (as a secondary way to link to a post) to ensure that it’s a drop-in replacement as far as external links are concerned. It omits WordPress features that I don’t care about, such as categories, plugins, a fancy admin section and others.

Gobble uses a similar in-memory data storage system to BitBlogger, but this time there’s no enforced shutdown that kills the cache. I’d considered using MongoDB, CouchDB or RethinkDB for data storage, but in the end I realised that the entire textual content for this blog fits in 2MB. A database wouldn’t add anything but unnecessary complexity for managing that tiny amount of data. Posts are stored on disk on the same server as the Gobble executable. I can still use Mercurial and BitBucket to store the posts; I just need to SSH to the VPS in order to pull and update. In future I might add simple admin section that allows me to edit posts in a web browser and view new comments/spam.

I’ve got a complete copy of this blog running in Gobble on a VPS from Digital Ocean. It works well, but some of the formatting of the posts is a bit screwy (it got lost during translation from HTML to Markdown). Once I fix the formatting I’m intending to switch away from my WordPress installation to Gobble.

My only concern with migrating platforms is the fact that moving the simianzombie.com domain to the VPS means I’ll have to run my own mail server, which sounds thoroughly tedious.

More Thoughts on Go

In some ways I find Go a little awkward to write. Working with pointers is identical to working with values directly (unlike in C in which you have to manually dereference pointers) so it’s frequently hard to tell if you’ve got a pointer or a value. Sometimes it’s nice not to care; at other times it’s handy to know if you’re editing a copy of a struct or the original struct (via a pointer), and the language hides that information. Switching back to a more procedural rather than OO paradigm involves some extra thought in deciding how best to structure the app. I’m still of the opinion that Go’s naming conventions are ridiculously anachronistic, and some of the design decisions in the libraries are bloody stupid (sorting and comparison functions are defined on the container rather than the items being sorted and compared, for example).

At the same time, it has some very nice features. Compiling down to a single binary makes Go programs trivial to deploy. Defining structs with associated methods instead of objects makes a lot of sense, and hopefully it should prevent Java-esque program architectures. The concept of implicit interfaces is very reminiscent of Objective-C’s informal protocols and works well. I’ve found that I can write programs in Go extremely quickly, and I’ll get faster as I become more familiar with how to best structure programs and the available libraries.

iOS doesn’t have a standard, easy-to-use way of encoding UTF-8 strings into HTML entities (ie. from “>” into “>“). One library that can be used to achieve this is libXML, but its API is particularly unpleasant:

int htmlEncodeEntities (unsigned char * out, 
                        int * outlen, 
                        const unsigned char * in, 
                        int * inlen, 
                        int quoteChar)

out: a pointer to an array of bytes to store the result
outlen: the length of @out
in: a pointer to an array of UTF-8 chars
inlen: the length of @in
quoteChar: the quote character to escape (' or ") or zero.

Returns: 0 if success, -2 if the transcoding fails, or -1 otherwise
The value of @inlen after return is the number of octets consumed as
the return value is positive, else unpredictable. The value of
@outlen after return is the number of octets consumed.

What’s wrong with that function signature, you ask? Other than the quality of the documentation, that is? Consider this situation. You need to encode this string:

<p>

That will encode into this string:

<p>

Now consider this braindead catch-22 situation: You must allocate the memory for the “out” parameter before you call the function, but until you call the function you won’t know how much memory you need to allocate.

The raw string in the example above is 4 bytes long; the encoded version is 10 bytes (don’t forget that this is C and all strings are NULL-terminated). If you follow the advice from Stack Overflow and double the initial size you’ll end up with 7 bytes (1 terminator + (3 characters * 2)), which is still too short. Another alternative is to figure out the maximum amount of memory that an encoded string could possibly consume (9 bytes per character) and use that, but then you’ll potentially be wasting massive amounts of memory.

If I were writing the function I’d probably return a pointer to a block of memory allocated inside the function itself. It could re-allocate the memory as needed and users of the function wouldn’t need to guesstimate the buffer size. That would raise my favourite C question, though: Who owns this memory?

I couldn’t find any examples of how to use the httpEncodeEntities() method, so I came up with my own. This solution uses a loop and encodes the string in chunks. It uses an encoded buffer twice the size of the initial string, but will resize it if it finds that the buffer isn’t large enough for a single encoded character. It’s implemented as a category method on NSString.

#import <libxml/htmlparser.h>

@implementation NSString (ZOHTMLEncoding)

- (NSString *)stringByEncodingHTMLEntities {

    if (self.length == 0) return;
    
    NSData *data = [self dataUsingEncoding:NSUTF8StringEncoding];
    
    int remainingBytes = data.length;
    int bufferSize = (data.length * 2) + 1;
    const unsigned char *bytes = (const unsigned char *)[data bytes];
    
    // We have to add an extra byte on the end of the encoded string to enable
    // us to add a terminator character.
    unsigned char *buffer = malloc(bufferSize);
    buffer[bufferSize - 1] = '\0';
    
    NSMutableString *output = [NSMutableString stringWithCapacity:remainingBytes];
    
    do {
        
        int outLen = bufferSize - 1;
        int inLen = remainingBytes;
    
        int result = htmlEncodeEntities(buffer, &outLen, bytes, &inLen, '"');
        
        // libXML doesn't append a terminator to the string - presumably because
        // NSString doesn't include one - so we'll have to take care of that in
        // order to convert back to an NSString.  We only add this if we haven't
        // completely filled the buffer.  If we've filled it, we've already
        // added the terminator character.
        if ((NSUInteger)outLen < bufferSize - 1) {
            buffer[outLen] = '\0';
        }
        
        if (result == 0) {
            
            NSString *string = [NSString stringWithCString:(const char *)buffer encoding:NSUTF8StringEncoding];
            
            [output appendString:string];
            
            remainingBytes -= inLen;
            
            if (remainingBytes > 0 && inLen == 0) {
                
                // Oh no!  We've got characters left to encode but they aren't
                // encoding.  This happens if our buffer isn't big enough, so
                // we'll resize it.
                free(buffer);
                bufferSize = ((bufferSize - 1) * 2) + 1;
                buffer = malloc(bufferSize);
                buffer[bufferSize - 1] = '\0';
            }
        } else {
            
            // Something bad happened
            break;
        }

        bytes += inLen;
        
    } while(remainingBytes > 0);

    free(buffer);
        
    return output;
}

@end

…would be more enjoyable if it automatically filtered out any comments with the words “sigh” or “yawn” in them.

I’ve published an initial version of my US DL/ID barcode data parser library here:

• DLID
  

This is a library for Go that can parse the data extracted from the PDF417 barcode on the back of a US driving licence into a Go struct. It doesn’t handle many of the encoded fields yet but it’s a start. I’ve grumbled about the multiple problems with the spec and its various implementations before.

Now that fishfish is available via Homebrew, it’s time to clear out the installed package and let brew take care of it for you. But how do you delete it? Here’s what I’ve been using:

#!/bin/sh

rm -rf /usr/local/share/fish
rm -rf /usr/local/share/doc/fish
rm -rf /usr/local/etc/fish
rm /usr/local/bin/fish*
rm /usr/local/bin/set_color
rm /usr/local/bin/mimedb
rm /usr/local/share/man/man1/set_color.1
rm /usr/local/share/man/man1/mimedb.1
rm /usr/local/share/man/man1/fishd.1
rm /usr/local/share/man/man1/fish_pager.1
rm /usr/local/share/man/man1/fish_indent.1
rm /usr/local/share/man/man1/fish.1

Remember to use chsh -s /bin/bash to switch back to Bash first if you swapped your default shell.

I’ve spent far too long fighting with this bug today, mainly because Apple don’t appear to publish a list of the bugs they’ve fixed between iOS versions. And wow, their bug reporting tool is still styled after the 11 year old OSX Puma! There’s a design that says “we care about your bug reports”.

Anyway, here’s an example of a pathological Core Data database design that will crash iOS 5. It’s an obfuscated subset of the database I’m using in my current app:

Attempting to create an instance of “Entity3” and then set its “guid” attribute will crash iOS 5 every single time. What’s most annoying about this bug in the OS is that adding or removing an attribute or relationship anywhere in the object graph fixes the crash. The bug was silently fixed in iOS 6. There’s an example project here:

• CoreDataCrash
  

In the last SFE update post I described an approach to choosing eggs by a voting system rather than “a pseudorandom progression for the sequence of eggs”; however, as every developer should know, random number generators are almost all pseudorandom. If each client starts with the same seed and only uses the random number generator for generating new egg colours, all peers in the game should agree as to the order of eggs.

The real problem is that the rand() function might be called for egg generation and for other things, meaning the generators will be out of step. I’ve stripped out egg voting and replaced it with the MTRandom library, which wraps the Mersenne Twister random number generation algorithm in a class. All peers vote for the random seed at the start of each game (and each round) and the vote from the peer with the highest ID is chosen. The egg factory sets up an internal MTRandom instance and the egg sequences are synchronised automatically with no further network traffic.

The next problem to solve was the issue of minor timing differences between peers throwing out synchronisation. Suppose an egg lands on one player’s screen. The next egg appears at the top of the grid, and the other player throws across some garbage eggs. Those garbage eggs will have to wait until the current eggs land before they can be added to the grid. Meanwhile, on the networked view of that player on another iPad, the other player throws across some garbage before the egg lands. Network lag meant that the “drop egg” message was delayed by a fraction of a second. The egg lands, the garbage appears, and the two representations of that player’s grid disagree entirely about the current state of the game.

My fix for this was to hold up the appearance of the next egg until all of the networked representations of a grid are ready. Once the current egg pair lands the grid sends a “waiting” message and enters a wait state, in which they can receive garbage eggs, until all peers announce that they are waiting. At that point they all add new eggs and switch back to the active state.

The final problem is rather more tricky: sometimes egg movement messages can be processed out of sequence. All of the egg movement messages are sent by the player’s grid instance. Moving, rotating or dropping an egg results in a message being sent to all peers in the network. Once received, all of the messages are dispatched to the “network controller” instance (where they are queued in order) except for the “egg drop” message, which is sent directly to the appropriate grid. This means that an “egg drop” message could be sent after a “rotate” message, but would end up being processed before the “rotate” message. To fix this I’ll either have to send the “egg drop” message to the controller and treat it as a “down” button press, or I’ll have to scrap the entire concept of a networked controller and create a duplicate grid class that excises most of the game logic and relies on network messages instead. I’m hoping that the controller option works.

The to-do list isn’t growing any shorter. I still need to:

• Create a peer chooser (though it’s possible that I might be able to use an Apple-provided class to do that for me);
  
• Provide a way to quit the current game when paused;
  
• Replace the “press a key” bitmap with a “tap to continue” bitmap;
  
• Figure out why taps on the “press a key” screen are intermittently detected;
  
• Quit the game automatically when a networked peer disconnects.
  

Once all of that’s done I’ll sign up for the iOS App Store and get a release out. Next steps will be to merge the changes into the OSX version of the game (to allow for Mac/iPad networked games) and increase the number of players that can participate in a game from 2 to 8.