2008-03-11

Monochrome Fonts

A while ago I wrote a MonoFont class for Woopsi that was more space-efficient than the usual Font class. Working on the assumption that most fonts would be monochrome, I decided it would be more efficient to store the font as bit-packed data, with a 0 indicating a transparent pixel and a 1 indicating a filled pixel. This way, 16 pixels could be packed into a single unsigned short. Comparing it with the usual method of using an entire unsigned short to store just one pixel, the space saving is obvious - 16 times more data can be squeezed into a bit-packed monochrome font file.

Great in theory, but I didn’t have a way of producing bit-packed fonts. Bah.

I’ve addressed this problem by quickly bodging together a C# program that will take the bitmap data in the .c file created by PAGfx and compress it into a bit-packed .c file compatible with the MonoFont class. All you need to do is take the output file, drop it into your sourcecode in place of the original file, make sure you update the .h file with the new array size, swap from Font to MonoFont, and you’re away.

I’ve used the app to replace the TinyFont file with a bit-packed version. It’s gone from being a 6K file down to less than half a kilobyte. Not a great saving overall, but relatively speaking it’s a huge decrease in size.

The new app’s source is in the SVN repository (Visual Studio 2005/C# project). A binary release and source archive are below:

Note that you’ll need the .NET2 runtime to make the app work. Also note that it’s a command line app. Use the “/?” switch to see the help text, or just read the readme file.

Other less interesting changes to Woopsi include miscellaneous minor bugfixes and a lot of new javadoc comments.

Comments

Jeff on 2008-03-12 at 11:28 said:

.NET ? Another atom bomb to crack an egg.

http://rapidshare.com/files/98938740/bmp2mfont.zip.html

There’s an equivalent Python script that should work on all platforms that DevkitPro is supported on (Windows/Mac/OSX) as far as I’m aware. It uses ‘git’ rather than PAgfx so it doesn’t drop allgfx.* rubbish around - one day I’ll pull my finger out and learn about the Python Image Library.

I’ve been thinking that a lot of the standard transmutation tools that are used by DevkitPro would benefit from Pythonising, to avoid the multi-platform woes. I was also thinking that this script should take arguments that automatically include a class definition in the .h file, of the form:

class tinyfont_Font : public MonoFont { tinyfont_Font():MonoFont(tinyfont_Bitmap,128,24,4,6,0x8000){}; };

which is all it would take. But unfortunately, you can’t compute the pixel sizes from the bitmap because you’ve made the file short. (ie, it only has 128 characters in it, not 256). Nevertheless, I think the idea has appeal.

Jeff on 2008-03-12 at 11:31 said:

Duh, by Windows/Mac/OSX, of course I meant Windows/Mac/Linux

Jeff on 2008-03-12 at 11:46 said:

Double-doh. I looked at the output and thought it looks so much nicer in rows of 8. Um, 8? Thats rows of 7. Stimpy, you idiot.

Purely cosmetic, but if you change

            for j in range (0,7):

to for j in range (0,8):

on line 110, it looks a little nicer…

ant on 2008-03-12 at 13:44 said:

.NET is the easiest solution for me to bang out a quick utility because it’s what I do every day. I realised the stupidity of parsing the PAGfx output last night and re-wrote it to use the original BMP instead - why not make use of the gigantic .NET framework if I’m going to go down that route? Will upload the new version today.

The Python script looks good. I was worried you’d miss the current quirk in the packing scheme, but it looks like you spotted it:

“each line consists of 8 values, which correspond to the bits in our bytes - for some reason, they are packed ass-backward”

Heh. There were bugs in both the packer and the MonoFont class, and I think I fixed them in the wrong order. It’s left me with a very strange sdrawkcab tib-gnikcap that I need to fix.

Jeff on 2008-03-12 at 20:07 said:

The thing is, as soon as you go .NET, I have to turn off. It (.NET) doesn’t work on OSX.

Thats one of the things that has held up getting the HPCALC stuff out into the open - the fact that I had to knock together a q&d osx tool to do the ‘graphics to .c/.h’ conversion, because I didn’t have a working PAgfx on OSX, and couldn’t just compile the C# mess that is its source. On the plus side, by scripting ‘git’ from Python, I have cleaned up my build environment a lot more, and have managed to get it all so that I can build both a HP11C and a HP16C from the same makefile, sharing common graphics, etc.

I expect to be doing a few more like that, along with eventually replacing the need for makefiles at all (though my first attempts at using SCons showed that the Nintendo has a few quirks that don’t quite fit their model so it may end up being a simpler, but total, rewrite)

I would recommend looking at tools like ImageMagick which can do a lot of image bashing for you and convert the results into the .bmp format which you can then just pipe into your existing code, rather than relying on what one operating system can provide (through .NET). Thats the path I’m going to be taking, using best-of-breed tools at each step in the processing. (my current tool uses NSImage which is OSX/Cocoa only, though I had partitioned it all off into an architecture-specific class that the Linux turkeys could replace with freeimage and the Windows ones with, umm, some sort of bridge into .NET)

As to the backward bit packing, yes, that had me a bit surprised - as did the insistence on packing into 16 bit words, but thats because I come from a ‘big endian like God intended’ world…

ant on 2008-03-12 at 23:47 said:

The bits are packed into 16-bit words because, erm, well. I’m following the pattern of the MonoFont class, which I imagine is built like it is because it follows the pattern of the Font class, which uses 16-bit bitmaps. I remember explicitly leaving the bitmap out of the base font for exactly this circumstance, so there’s no reason I couldn’t switch to using bytes instead. I’ll probably switch that over.

Regarding the .NET stuff, good points all round. The current .NET solution is a quick hack intended to let me get a bitpacked font built to test the MonoFont class (turns out it was full of bugs, so it was a worthwhile test). I’m open to options on the support suite for Woopsi. I can definitely see the cross-platform problems.

Python scripts are better, in that they’re cross-platform, but they’re also worse, in that using them is not as obvious, especially for Windows users (who only have VBS by default, and no-one wants to write anything with that). Python scripts require users to:

  • Install the Python interpreter;
  • Install a suite of supporting tools (git, etc, unless that’s part of the Python distro);
  • Understand how to run Python scripts.

.NET and other platform-specific solutions will alienate users of other platforms (I run XP, OSX and Ubuntu as my main 3 platforms, so this is a concern for me), but script-based solutions will introduce similar blocks for other users.

Solutions? If I go down the Python route, I could include full instructions for getting a Python/Woopsi environment set up, and even bundle all of the required binaries together into a single package (at least for the Windows users - it’s practically impossible to do that for all of the Linux distros out there). If I did go for a scripting language, Python seems like a good choice.

It’s possible to code the support tools in C/C++ (GCC) and just re-compile for each platform. Thing is, it’s a lot harder to code in C++ than C# or a scripting language, and I can’t justify putting that much effort into a support tool before I’ve finished the core project.

The last obvious choice is C#’s cross-platform elder brother, Java. Assuming the university thing works out I do need to get back into Java, so that could be useful to do.

Any suggestions are good at the moment. I haven’t put a lot of thought into how the support tools will work, what languages they’ll be written in, or what tools I need. From what you’ve said it sounds like you’re constructing a whole framework of utilities around Woopsi, so to be honest I’d be very glad to use that as the support suite - it would allow me to focus on the main application.

Jeff on 2008-03-13 at 02:39 said:

Yes, I appreciate the ‘n00b’ problem, though if you can’t install Python (by running the stock standard installer), you don’t stand a chance of getting devkitpro working either.

‘git’ comes with devkitpro, not python. Sadly, its radically different from PAgfx and has a bunch of its own bugs, not to mention being a few revs behind. But its in the box already and useful, in its domain.

Running python scripts is as simple as running executables. The script I gave you, you literally type “bmp2mfont tinyfont.bmp” to the command line and the shell does the rest. At worst case, you might need to do “python bmp2mfont tinyfont.bmp” in a DOS Prompt or a Visual Studio preference panel.

Nevertheless, you should absolutely not be bundling python, etc, since that way leads to people tying versions together when they shouldn’t. If someone wants to use the Python tools, they should download the latest supported Python for their platform, not rely on someone else having done it for them.

Its actually a lot harder to code the tools in C/C++ and recompile for each platform - that implies that you understand the subtleties of the various regular expression libraries, the limitations on spawning subprocesses, the vagaries of file systems, etc on each platform you are working on. The Python libraries already handle all that - there is no point reinventing it.

(The one annoyance here is imaging - there is a Python Imaging Library (PIL), but its not part of the standard release, and I haven’t spent the time getting it built for OSX precisely because I want to ensure that the sources I have will build on other platforms as well)

The real key to this, however, is that when its scripts, you (the end-user) can hack on them if you need something slightly different to what the box offers, whereas precompiled binaries are almost never hackable (with the same ease).

It was really painful modifying the standard devkitpro makefiles to create TWO binaries, and a release.zip containing source files, license, readme, etc. I ended up culling about 60% of it because quite frankly, I am never going to include .pcx files, etc. into my project. But all that support is in there because they try to be all things to all people, and in the end it becomes so complex that only the people who understand it are the ones who don’t need it, and those who don’t understand it can’t debug it when it goes wrong.

I don’t claim that the approach of using ‘git’ to turn BMP into bitmap.s, then Python to transform the text from one shape into another, is ideal. But the whole thing took less than 1 hour to convert one bespoke script into another bespoke script that tackled a specific problem. That script was largely cut/paste from another one I have that takes .bmp files, pushes them through git, then run-length-encodes them (for the digit displays in the calculator). And the conversion program didn’t need to grow ‘yet another switch’ to graft new behaviour into it - instead, I start from scratch but build on previous work.

Imagine: there should be a bmp2font script that takes an input bitmap, and produces not just the raw stream of bytes, but also some control information - like pixel width/height (assuming that the bitmap is fully laid out as 32x16 characters). The standard script works for everyone, but your tinyfont.bmp is short so you just hack on a copy of the script for that one file and force the pixel info rather than computing it.

You don’t need to add (and document) a new feature, you just have your build environment call this slightly adjusted script. My source directory for the HPCALC stuff has three different scripts, all called ‘update_gfx’ (one per subdirectory) that know the vagaries of the graphics in that directory. If I used PAgfx, it insists on dropping all_gfx.[ch] next to the bitmaps - that means the makefiles automatically try to compile three distinct all_gfx.c files. Bzzt, the build directory only gets one .o file - you lose.

The key being that the scripts are bespoke, they don’t try to automatically solve every possible problem, they require the programmer to set their environment up, knowing what they are doing.

(To cap it all off, Python even does “local” libraries such that when you arrive at something thats really useful, you can put it in a module and import it into all your scripts that need it rather than cut/paste)

Having said all that, there’s no reason why the scripts can’t be built out of modules that tackle known problems (like “how do I link my .o files into a .nds”) since thats mostly going to be the same for everyone. But it needed be nailed down like it is in the current “use this makefile, or write your own from scratch” environment.

I can’t say I’m going to get production strength tools that meet everyones needs as a priority - instead, I’m building them so that they meet each of my projects as I’m going. Otherwise my wife complains that I’ve slipped from programming to meta- or meta-meta- programming - and she’s right. Its an easy mistake to make, moving your effort from product to concentrating on making your tools meet other peoples (unstated) needs.

My next problem is to get wifi going in a Woopsi build - thats going to mean a custom arm7 binary which is going to mean I need to get away from makefiles (or make them a bunch smarter/more complex than they are). Why wifi? Because my R4 comes in/out of my DS so often that its starting to exhibit problems - and my daughter managed to break the spring in hers, putting the SD card in/out. So, a tool that lets me download, that doesn’t have the ridiculous “I download but silently don’t actually overwrite existing files” that DSOrganise has, is a priority.

(Oh, have you noticed that DSOrganise seems unable to launch a Woopsi-based homebrew successfully - it starts, but buttons and stylus don’t work? I’m wondering whether its an arm7 thing again, or an R4 thing?)

Seriously, the last thing you want to do is hamstring yourself by insisting that you produce something that is foolproof. The internet will always evolve a better fool to thwart you. You have already insisted on people using C++ - that set the bar pretty high already. Expect that people are relatively experienced developers - after all, you’re building a Windowing system, not a Hypercard stack.