2010-01-03

UTF-8 and TrueType Fonts

If you haven’t been following this thread lately, you won’t be aware of Woopsi’s latest contributor, Lakedaemon. He’s doing a fantastic job of introducing two new, much sought-after features to Woopsi - unicode (UTF-8) and TrueType font support.

Both features seem to work well. Both of these features have pros and cons, which I’ve been contemplating a lot lately. Unicode support, for example, has a very negative impact upon random access within strings. Unicode is a variable width encoding system, so there is no way to locate a random character within a string without first parsing all of the bytes that come before the character. However, Woopsi doesn’t really do much random string access, so this is not a huge problem.

The real issue with supporting extra character sets is the amount of memory consumed by each string. As a variable width system, UTF-8 uses single bytes for the standard ASCII set and so the problem is a non-issue.

In order to have system-wide support for UTF-8, all gadgets that have previously worked with char arrays must be altered to work with the WoopsiString instead. This implies some overhead, which is unavoidable, but as support for non-ASCII character sets is one of the most requested features for Woopsi, I think we can live with it.

Supporting TrueType fonts is more complex. Lakedaemon has used the FreeType library. It looks like a solid project and is dual licenced under the GPL and MIT licences. The MIT licence is basically the BSD licence with a different name, so there are no legal problems with using it. However, it does make the build process more complicated, and it adds substantial bulk to the size of a Woopsi ROM file. The TrueType font must be loaded from the flash disk, so the ROM must be linked with libfat, further adding to the ROM size.

I haven’t tried out a Woopsi GUI that uses TrueType fonts yet, but I’m guessing that it will be slower than a simple bitmap font-based GUI. FreeType is designed for embedded devices, however, which may mean that the difference in performance isn’t noticeable.

Bearing these points in mind, I think I’ll add TrueType support as an optional extra that can be enabled in the makefile. I’m not sure how that’s going to work yet; it may be that I split Woopsi into two pre-compiled libraries, one of which is lightweight (no libfat/freetype) and one of which is heavyweight (libfat and freetype enabled).

Someone using a TrueType font as their system font will soon come across an interesting problem - the glyphs on their buttons (close gadgets, screen flip gadgets, radio buttons, etc) will be drawn as characters, not images. The reason for this is that the glyphs are stored in the extended ASCII portion of the bitmap fonts. At this point, it has clearly become a bad idea, and I’ll need to split the glyphs into a separate glyph font. It is unlikely that anyone will want two or more glyph fonts active at once, so I’ll make the glyph font a global variable initialised in the woopsifuncs files. It’ll be user-definable, just like the system font.

Aside from that, I’ve made a couple of other fixes. I’ve removed the “raisesEvents” flag from the Gadget class, as that is now handled by the GadgetEventHandlerList class. The setRaisesEvents() and raisesEvents() functions interact with the GadgetEventHandlerList instance correctly, which has fixed a glitch in the ScrollingPanel. The ScrollingPanel::raiseScrollEvent() method refers to the GadgetEventHandler’s isEnabled() function to see if events should be enabled or not.

Finally, the FontBase class now includes a getCharHeight() method, which should hopefully help out Lakedaemon’s desire to support variable-height rows of text.

Comments

Richard on 2010-01-03 at 22:02 said:

Great news! Not supporting UTF-8 was one of the big missing features that Woopsi has (or had).

As for True/FreeType, not sure what is planned, but perhaps an intermediate format would be worth considering? Something like a “mangled” TrueType font file but one that maps more closely to the single sized font used in Woopsi. Instead of just indices 0-255 it would cover the Unicode code points in the input font file. After all, Woopsi doesn’t take advantage of multiple font sizes in the TrueType fonts does it? (or maybe it does?)

Just learn from Okiwi if you do this: don’t have the DS do the font mangling! :-) Do it offline and then read just the intermediate format into Woopsi. Okiwi was infamous for its 2 minute+ startup time as it compacted the ttf file down into a consumable format.

Lakedaemon on 2010-01-04 at 10:56 said:

There is no penalty for starting time with freetype.

Woopsi doesn’t take advantage of multiple font sizes YET ^^

At some point, I might try to code a “kinda rich text” gadget…i.e. basically a multilinetextbox that displays/formats strings with different fonts/font sizes (think japanese, english, cursive, bold, italic, etc…) and other stuff (that have width, height, distance from top to baseline…) like images (png/svg) or who knows (gadgets ?) No promise given…but as I need this for my app, it might well happen… and the new getCharHeight function will be a handy addition when/if that happens

With some optimizations done in the present woopsi code, I think that Freetype may become as fast (or faster ?) than the present code as :

Freetype uses a (probably very efficient) cache for bitmaps (so it uses bitmaps too), whose size you choose And the freetype bitmaps are smaller (they are variable width/height) than packed fonts bitmaps (as you only need like 4 pixels to draw a point char… instead of width*height and each pixel is 1 byte instead of an u16).

Basically freetypes brings : 1) unicode fonts

2) antialisasing through 255 levels of gray (+alpha channel) = antialiasing with transparancy (you can write without overwriting the background…nice for buttons with images and text)

3) the user can choose what font to use (now, it’s the developper who chooses), this opens woopsi to new kind of Apps…

There is probably no need to have 2 libwoopsi libraries as the freetype code don’t get added to the rom if you don’t use it in your app.

Adding freetype as an extra is probably the way to go (as with your map/hash and linked lists extras… by the way, how big is the stl overhead of those structures compared to your implamentation ?)

Lakedaemon on 2010-01-04 at 11:10 said:

Technically speaking, it should be very easy to make woopsi use utf-16 or utf-32 instead of (or maybee in addition of) utf-8… as only a few methods of the woopsi string class would have to be modified.

As utf-32 are fixed length u32 this would increase storage size a lot…but speed up random read a lot (the developper that want that feature could only convert a few strings to utf-32)

Basically utf-8, utf-16 and utf-32 are equivallent but have different trade offs/advantages :

utf-8 is great for ascii char strings / adding unicode support to a legacy ascii system but uses 50% more space than u16 for chinese/japanese

utf-16 is great for chinese/japanese (1 u16 instead of 3 chars for each japanese char) but worse for ascii (1 u16 instead of 1 char)

utf-32 is great for random access but sucks for storage space.

karagh on 2010-01-06 at 09:33 said:

Nice to hear about unicode support. If freetype can be as fast as bitmap it will be well received too, because I’m a bit concerned about performance.

Lakedaemon on 2010-01-09 at 10:52 said:

Update : I have made some progress with the “finding the path to the rom” and “embbed data in the rom” problems thanks to NitroFS and Lady Erys http://blea.ch/wiki/index.php/Nitrofs

And as I have nearly finished porting/rewriting my little app to woopsi and as I’m starting implementing new features, I’m back to improving freetype support in woopsi.

First step : write some classes/clean up the code implementing basic support and release it.

Here are some thoughts on the subject :

/*
          Ok, I’m trying to implement gont support in woopsi…here are my thoughts about how it should be done :

      1) have a class (FontManager ?) that stores all the installed fonts, 
          basically as an array (we want fast random read, don't nead insert) of pointers to FontID struct

                     struct FontID{
                         char* path;
                         u8 index;
                     }

         (the array might be a dynamic one for people who code applications where you can add/remove fonts)

      2) have a class in woopsy (FontCache) that creates a freetype fontcache.

                a) users might want to share a font cache between fonts (less memory used)
                b) users might want to have a font that has it's own fontcache (faster, no switching between faces, cache more efficient)
                c) users might want to do both

      The way it is now, you can only use 1 font inside a gadget (multilinebox, textboxetc...) and you can only change the font at the gadget level.
      (In the very near future, I'll need/implement a RichTextBox that uses multiple fonts/faces/sizes inside... I'll have to do a lot of tinkering for that (or not))

      An implication of the present interface is that : 
                if you create 2 MultiLineTextBox and make them share the same font instance :

                a) when you change the size/face/index at the font level...it'll screw both gadgets
                b) when you change the size/face/index at the gadget level... it'll redraw/recompute one box _but not the other_ and we don't want that

                     0) I guess that you could change the size/face/index at the font level (or maybee at an higher level : FontManager) and message all gadgets that are using the font to redraw 

                                                 implement/trigger a face/index/size fontchange event might be the way to go


                     1)  You could share a fontcache but NOT share the font between gadgets (but that means having lot's of font instances, 1 for each gadget...we probably don't want that)
                     and change the font properties at the gadget level


      Well, got to ponder about that...in the mean time, I must write code to make stuff work...so I'll go for a basic model at first : 

      a) create a fontcache class, whose size you choose (default being 30k)
      b) create a fontclass that is linked to a fontcache (that way, you can share/or not a fontcache) and create a fontsubclass that contains it's own fontcache

     /*

Lakedaemon on 2010-01-10 at 16:50 said:

Ok, I have finished cleaning the freetype code,

we are now able to use 1+ fontcaches with 1+ freetype fonts (ttf, type1, opentype…), any color, any width or height (freetype scales the glyphs) in a woopsi application.

I’m posting the code and a little demo here : http://www.lakedaemon.org/FontClass.zip

It should be considered experimental/Beta code as it is working but there are a few caveats to using it : If you mess the paths or the index of your font, it displays nothing (i.e. I haven’t implemented tests to make sure the user uses this correctly) and may crash your app.

There are probably a few improvements to be made, but this works and is a good start.

please notice that the demo is still using my old code (I.e. the quadratic wrapping algorithm in the text class) that is clearly not optimised for speed…

Next I’m going to at last have a look at the svn/woopsi 0.43 code, to speed up the “render chars transparently” code and to try to find two remaining bugs in my code : 1) don’t display newlines 2) when a miltilinebox is dragged, redraw it correctly

ps : it would be nice if there was a background color for a few gadgets (like multiline boxes) as we could write white/gray chars on a black background.

WinterMute on 2010-01-23 at 01:55 said:

@Lakedaemon

Please don’t include finding the path to the ROM, use libfilesystem incuded with the devkitARM install and insist on launchers that pass argv. Having code around which uses random libraries and/or problematic methodology causes major support issues and encourages people to use inappropriate solutions for avoidable issues. Right now the only menus that support this are the homebrew menu in devkitPro svn and akaio.

It’s probably also worth noting that there are some prebuilt libraries from devkitPro, including libfreetype. See http://wiki.devkitpro.org/index.php/portlibs

Lakedaemon on 2010-01-23 at 17:21 said:

Don’t worry, I didn’t include the “finding path to the rom” code and I won’t now. (I’ll stilll use it for my own projects though because I (and the users of my little software) need it and as you pointed out, lots of flashcard haven’t properly implemented argc and argv support)

I only contributed code to allow freetype rendering inside woopsi.

I noticed too that some libraries had been prebuilt for the nintendo ds on sourceforge (thank you ! That’s nice, I’ll probably use those later on).

If you are interested, the cyassl library can be very easily ported to the nintendo ds (you only need to modify how the random number generator is initialized).

It’s a small, fast and very efficient ssl/cryptographic library I can send you my modifications if you are interested.

Using this ssl library and dswifi9, it is possible to connect to free hotspots that use (WEP/OPEN) and mac address identifiaction…like the freewifi network in france (which means that free users can connect with a nintendo ds from virtually anywhere in france) If you are interrested, I can contribute that code too.

ps : thanks for the great work on devkitpro (I love you).

Lakedaemon on 2010-01-23 at 17:29 said:

and many thanks for pointing me to AKAIO (my flashcard is an acekard 2i).

I’ll upgrade and make it so my software tries to use argv and argc first before falling back to the \finding the path to rom\ code.

ant.simianzombie.com » Woopsi 0.99 Released on 2010-05-20 at 17:03 said:

[…] obsolete as soon as it is written. Before I wrote this document, for example, I hadn’t added UTF-8 support to the WoopsiString class. Nor had I decided that WoopsiStrings, not char arrays, would be the […]