2011-10-25

Installing Hatta on Windows

I’ve been trying to find a replacement for a SharePoint 2007 wiki that was foisted on my team. If you’ve ever used SharePoint’s standard wiki system, you’ll know that it’s just a cruddy wiki-like front-end for a ropey CMS. Browsers other than IE have the WYSIWYG editor disabled and the user is left to edit raw HTML. Wiki markup? What’s that?

I’ve got a number of requirements for the replacement.

Must be cross-platform

As I find myself developing in both C# and Objective-C, I need a wiki for half a dozen trusted users that works in Windows and on the Mac.

Must use a standard wiki markup

I’d prefer Markdown or Creole. As long as I’m not forced to drop into HTML or use a WYSIWYG interface I’ll be reasonably happy.

Must be easy to set up

Preferably I just want to run an installer and have the wiki running. Every dependency I need to install and set up - web server, database server, scripting language - is a mark against it.

Must be simple

I neither need nor want authentication systems, online help, sandboxes, advanced formatting, styles, drawing APIs or file uploading. Just a way to store short, linked text documents.

Must use a DVCS as its storage system

If the wiki stores its data in a DVCS I can clone it and take it home with me.

What I really want, I realised, is an open-source version of BitBucket’s wiki. I’ve looked at dozens of systems. The most promising are MoinMoin version 2, which unfortunately isn’t finished yet, and Hatta.

Hatta is a fantastic little wiki. Its features include:

  • Cross-platform (written in Python)
  • Simple but attractive interface
  • Mercurial storage system
  • Creole syntax (for the most part)
  • Built-in web server

It comes in two forms. You can download it as a pre-compiled binary, with all dependencies included, for OSX and Windows. Alternatively, you can download it as a pure Python solution, though this entails setting everything up yourself.

I started out with the binary distribution, which was very easy to get up and running, but I quickly ran into problems. The binaries are now 4 minor revisions out of date, and according to the author the chances of them being updated are slim to none. Worse, the binary regularly freezes up. That’s no use as a centralised wiki.

Trying to get the plain old Python version working in Windows is a pain as there’s no documentation for it, so here’s how to do it.

Installing Hatta in Windows

Firstly, we need to get Python set up and install Hatta’s dependencies.

  • Download Python 2.7.2 from python.org. Hatta will not (currently) work with Python 3.
  • Install Python using all of the default settings (just keep hitting “Next”).
  • Download “setuptools” from pypi.python.org. Assuming you’re using a 64-bit version of Windows you’ll need to download the “ez_setup.py” script and run it like this:
C:\Python27\python.exe .\ez_setup.py
  • Open up a command line as admin and enter the following:
cd C:\Python27\Scripts
.\easy_install-2.7.exe werkzeug
.\easy_install-2.7.exe pygments

That’s Python installed and two Python packages that Hatta is dependent on. Next we’ll need to download Hatta itself and integrate it with Mercurial.

  • Download Hatta from hatta-wiki.org. You’ll need the zipped version.
  • Unzip Hatta to C:. This should give you the following paths:
C:\Hatta\doc
C:\Hatta\examples
C:\Hatta\locale

Etc. You don’t have to put Hatta in the root of C: but it simplifies this guide.

  • Download the Mercurial source code from mercurial.selenic.com. The binary builds are of no use to us.
  • Extract the tarball.
  • Open the extracted directory and locate the “mercurial” folder.
  • Copy the “mercurial” folder to C:\Hatta\ (giving you the folder C:\Hatta\mercurial\hgweb etc).
  • Copy the “hgext” folder to C:\Hatta
  • Open the “mercurial” folder in C:\Hatta.
  • Open the “pure” folder.
  • Copy the contents of “pure” into the “mercurial” folder (giving you C:\Hatta\mercurial\bdiff.py etc).

We’re done! What we’ve done is copy the parts of Mercurial required by Hatta into the Hatta directory. Yes, this is an ugly hack, but it’s the easiest way of getting Hatta up and running. When new versions of Mercurial get released, you’ll need to delete the “mercurial” and “hgext” folders and upgrade them with the latest source (remembering to fiddle about with the “pure” folder).

You can now test the installation by opening a command line and typing:

C:\Python27\python.exe C:\Hatta\hatta.py

Point your web browser at http://localhost:8080 and, if everything has gone to plan, you’ll see the Hatta editor page.

2011-03-12

BitSyncPy Updated

BitSyncPy, my Python 3 utility for pushing/pulling/cloning/etc a whole directory of Mercurial repositories with BitBucket, has had a few updates. It now works under OSX (and should therefore work with Linux). It can also force local repository names to be lower case when cloning.

2011-02-17

BitBug and BitBugPy

Another pair of C#/Python programs. This time it’s an idea branched from BugBare, my distributed bug tracker. The real problem with distributed bug trackers is their complete lack of utility and visibility for people who aren’t programmers on the project in question. BitBug dumps the distributed part and is really just a command line interface to BitBucket. All bugs are stored in BitBucket’s issues database. They obviously don’t work offline, but they’re as quick to use as BugBare and come with a ready-made web interface.

The two programs are here:

Of the two, BitBugPy is the tidier program. It uses the BitBucketAPI library for all BitBucket interaction, so the script itself is tiny. Again, it uses Python 3.2 (release candidate) so I’m sure 99% of the world won’t be able to use it. I’d look into making it a Mercurial extension, but - shock! - Mercurial is written in Python 2, not 3, and I’m not interesting in backporting it.

The C# version - BitBug - uses my MurkyCommando library as its command line UI. Since playing with the various command line argument parsers in Python I’m a lot less happy with it now.

They are both rather limited at the moment, mainly because the API offered by BitBucket is still a work in (very slow) progress.

2011-02-17

BitSyncPy

Combining Python, the best parts of BitSync and BitBucketAPI has produced BitSyncPy. This is a very short Python 3 script that will:

  • Connect to BitBucket’s API using a user-supplied username and password;
  • Get a list of the remote repositories belonging to a specified user;
  • Get a list of local repositories in a specific path;
  • Print the total number of remote and local repositories;
  • List any repositories that are local but not remote;
  • Clone all repositories that are remote but not local;
  • Pull/update all repositories that are both local and remote.

It’s essentially the same utility as the C# version of BitSync, but it’s a fraction of the size and should be cross-platform. Note, however, that it uses the new argparse module included in Python 3.2 (yep, I’m running the release candidate) and so won’t work without modifications in the stable version of Python 3.1. It won’t work in Python 2 either.

2011-02-17

The BitBucket API, Python and urllib

If you’ve ever tried googling for anything related to Python 3, you’ll have encountered difficulties. This is because “Python 3” is also known as both “Python 3000” and “Py3k”. Search for just one name and you’ll miss one third of the potential results. This is somewhat contrary to the Python philisophy:

There should be one-- and preferably only one --obvious way to do it.

We’re off to a good start already.

As discussed recently, BitBucket has an incomplete but promising REST API that allows for the automation of things like repository creation and issue tracking. I’ve written a messy C# program that takes advantage of some of the repository features, but wanted to try writing a complete API library in Python 3.

Python is currently having problems. Zed Shaw over at sheddingbikes.com has complained about the unwillingness of operating system providers to upgrade from ancient versions of Python. I personally can’t quite see why so much of Python 3 has been backported to Python 2, when encouraging people to upgrade where they need to by offering new features would be far more beneficial for the new version of the language.

I’ve been using Python’s urllib to access the BitBucket API. The BitBucket API requires the use of basic authentication in order to get access to the full feature set. Unfortunately, the “obvious way to do it” doesn’t work.

This StackOverflow question gives the “canonical” way of achieving basic authentication using urllib. I’ve ported it from Python 2 to Python 3:

import urllib.request

username = "username"
password = "password"
url = "https://api.bitbucket.org/1.0/users/ant512/"

passman = urllib.request.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, url, username, password)

authhandler = urllib.request.HTTPBasicAuthHandler(passman)
opener = urllib.request.build_opener(authhandler)

urllib.request.install_opener(opener)

pagehandle = urllib.request.urlopen(theurl)
print(pagehandle.read())

Looks great! Doesn’t work. The problem is that urllib expects this sequence of events to happen:

  • Client issues a request
  • Server issues a 401 Not Authenticated error
  • Client re-issues the request with authentication
  • Server sends data with a 200 response.

BitBucket doesn’t work like that, though. BitBucket does this if no credentials are supplied:

  • Client issues a request
  • Server sends data with a 200 response.

The data returned in this case is a list of public repositories.

When the initial request is sent with credentials, BitBucket does this:

  • Client issues a request with authentication
  • Server sends data with a 200 response.

In this case, the data returned is a list of public and private repositories to which the authenticated user has access. For this situation, urllib doesn’t work.

What we actually need to do is this:

import urllib.request
import base64

username = "username"
password = "password"
url = "https://api.bitbucket.org/1.0/users/ant512/"

credentials = base64.b64encode("{0}:{1}".format(username, password).encode()).decode("ascii")
headers = {'Authorization': "Basic " + credentials}
request = urllib.request.Request(url=url, headers=headers)

connection = urllib.request.urlopen(request)
content = connection.read()

Of course! Why didn’t I see it in the first place? All I have to do is ignore the library’s API and instead encode a username/password tuple into base 64, decode it back into ASCII, manually inject it into the request header, and then use urllib. Obvious. Yes.

At least we can now retrieve a list of a user’s repositories. Let’s make things a little more interesting and update an existing issue in the BitBucket issue tracker.

import urllib.request
import base64

username = "username"
password = "password"
url = "https://api.bitbucket.org/1.0/issues/ant512/woopsi/issues/10/"

credentials = base64.b64encode("{0}:{1}".format(username, password).encode()).decode("ascii")
headers = {'Authorization': "Basic " + credentials}
data = {"title": "Issue title", "content": "Issue content"}
request = urllib.request.Request(url=url, headers=headers, data=data)

connection = urllib.request.urlopen(request)
content = connection.read()

This won’t work. A REST API requires the use of 4 HTTP verbs:

  • GET
  • POST
  • PUT
  • DELETE

Out of the box, urllib only gives us GET and POST. It automatically determines whether to GET or POST depending on whether or not a request has a data payload. The urllib thus reveals itself to be an abstraction so leaky it should be on the bottom of the sea by now. It also explains why so many other people have written their own replacements for it.

What we need to do instead is subclass the Request object so that it is possible to specify which HTTP verb to use:

class RESTRequest(urllib.request.Request):
    __method = None

    def __init__(self, url, data=None, headers={}, origin_req_host=None, unverifiable=False, method=None):
        super().__init__(url, data, headers, origin_req_host, unverifiable)
        self.__method = method

    def get_method(self):

        # Uses the standard method choosing logic if no other method has been
        # explicitly set.

        if self.__method is None:
            return super().get_method()
        else:
            return self.__method

We can now alter our code so that we update issues in the issue tracker:

import urllib.request
import urllib.parse
import base64

username = "username"
password = "password"
url = "https://api.bitbucket.org/1.0/issues/ant512/woopsi/issues/10/"

credentials = base64.b64encode("{0}:{1}".format(username, password).encode()).decode("ascii")
headers = {'Authorization': "Basic " + credentials}
data = urllib.parse.urlencode({"title": "Issue title", "content": "Issue content"}).encode()
request = RESTRequest(url=url, headers=headers, data=data, method="PUT")

connection = urllib.request.urlopen(request)
content = connection.read()

Note that the data we send has to be in the format of a set of URL-encoded tuples.

Here’s one I haven’t been able to solve yet. Suppose we want to give bob@example.com write access to one of our repositories:

import urllib.request
import urllib.parse
import base64

username = "username"
password = "password"
url = "https://api.bitbucket.org/1.0/privileges/ant512/woopsi/bob@example.com"

credentials = base64.b64encode("{0}:{1}".format(username, password).encode()).decode("ascii")
headers = {'Authorization': "Basic " + credentials}
data = urllib.parse.urlencode({"data": "write"}).encode()
request = RESTRequest(url=url, headers=headers, data=data, method="PUT")

connection = urllib.request.urlopen(request)
content = connection.read()

The data we’re sending over consists of the following tuple:

{"data": "write"}

The BitBucket API says that the only data that should be sent as part of the request should be the words “read”, “write” or “admin”, depending on the level of access that should be given. We shouldn’t be sending a tuple at all. As expected, running this gives us a 400 “bad request” error. The problem we have is that the urllib will only allow us to send tuples. We can’t send a single word on its own. If we try it, Python throws a ValueError. The leaky abstraction comes back to bite us again.

Despite the obstacles put in the way, I’ve managed to get almost all of BitBucket’s API wrapped up in a Python 3 library:

The library probably doesn’t adhere amazingly well to Python’s coding standards (though there really is one-- and only one --obvious way to perform any action), but it does provide access to almost all of the functionality offered by BitBucket’s current API. The only exceptions are:

  • The grantPrivileges() method does not work (broken due to the limitations of urllib, but there might be a way to work around it)
  • The updateRepository() method currently doesn’t offer much of the possible functionality
  • The createWiki() method does not work (this could either be a bug in BitBucket’s API, a limitation of urllib or a bug in the library itself)

All data is returned as JSON objects. It is parsed from strings using the json library. The library can use basic authentication if a username/password combination is supplied. If not, it will interact with the server without authenticating where possible.

2010-07-05

More Mini Projects

I’ve been tinkering with a few mini projects lately, all of which are available from my BitBucket page:

www.bitbucket.org/ant512

Networked Pacman

Though I released this ages ago, I published it as a zip archive instead of hosting the code in a source code management system. I’ve rectified this and added it to BitBucket:

Networked Pacman

PyScrabble

This is a simple implementation of the logic of Scrabble, written in Python 3. It doesn’t have a UI, so it’s not terribly exciting for anyone but Python programmers, but perhaps someone will write a front-end for it. This was written in a couple of days and was an exercise in trying to think “Pythonically”. Whether or not I achieved this I’m not sure…

PyScrabble

WorkingWeek

Finally, this is a C# library that provides date functions that operate on a user-defined working week. For example, you can define a week of 9:30am to 5:30pm shifts (with time for lunch, natch) and use the functions provided by the library to determine whether or not a given date/time falls within a shift, or how many working hours there are between two dates, etc. It does some neat things with custom iterators and the yield statement to condense a lot of fiddly logic into a couple of very powerful methods.

WorkingWeek

2009-12-13

Creating Fonts and Bitmaps

More work on fonts. Jeff’s solution for creating PackedFont fonts is fantastic - instead of mucking around with u16 arrays, he subclasses the fonts and does all of the work for you. Instead of this:


PackedFont1* myFont = new PackedFont(somearray, somewidth, someheight, blah, blah);

…you have this:


NewTopaz* myFont = new NewTopaz();

So much easier.

I’ve shamelessly stolen this idea and reworked the existing “FontBitPacker” application so that it follows this pattern. I’ve heavily modified it so that the app will convert BMP files to either MonoFont, Font, PackedFont1 or PackedFont16 font classes. As long as they have version 3.5 of the .NET framework, Windows users no longer need to faff about with PAGfx or the bmp2font Python script. The new “bmp2font” .NET console application (which the FontBitPacker has become) will do all of the font conversion work needed for Woopsi.

This pattern seemed like such a good idea I’ve used it for bitmaps, too. Instead of converting BMP files with PAGfx, then using the “all_gfx” files to include the bitmap data before wrapping them up in a BitmapWrapper class, it is now possible to use a new “bmp2bitmap” .NET app to convert straight from a BMP file to a BitmapWrapper subclass.

These new programs have enabled me to delete the PAGfx binary and the files it converted from the Woopsi demo folder. I’ve replaced the converted files with classes converted with bmp2bitmap.

2009-12-06

Fixing the Fonts

The packed font system written by Jeff Laing has been broken since I refactored the bitmap system in Woopsi 0.40. I put together a new PackedFonts example project in order to identify and fix the problems, and the classes now work again. In addition, Woopsi now includes a “fonts” folder in its bonus directory, that includes 3 new fonts: Courier, CourierMono and Latha. Jeff wrote a demo with loads of fonts but I can’t seem to find the code, unfortunately, so I can’t add them to Woopsi.

I’ve also been hacking away at his bmp2font Python script. This is a Python 2 script that uses the devKitARM “grit” program to convert a BMP into a packed font class. It had a number of problems. Under Windows, it couldn’t find the grit program (the DEVKITARM environment variables use Linux-style paths, which don’t resolve properly). It didn’t put the resultant font classes into the WoopsiUI namespace (I must have introduced this after Jeff wrote the script), and it didn’t follow Woopsi naming conventions. All of these problems are now fixed.

In addition, there was still a minor clipping problem in the ListBox. This is fixed too.

2009-10-22

Woopsi, Bitmaps and Python 3

Back when I was programming mediocre games in AMOS, one of my friends decided he wanted to code too. I introduced him to the basics and he quickly advanced way beyond me. Whilst I was struggling to get simple things working in BASIC he was writing demo effects - fire, spinning 3D environments, toruses - in Pascal and C. One of the most impressive programs that he wrote, at least to me, was a BMP loader. I had no idea how to go about writing anything like that. I put it down to two things:

  • BASIC, and the way it hides all implementation details from the coder to the point that he finds himself doing nothing but bolting together existing library functions (ie. “loadBMP()”, “saveBMP()”, “makeAnAmazingGame()”);
  • The total lack of any useful books on coding where I grew up.

You could point out that if he could learn with no books, I could too, and he’s probably just smarter than me. That’s probably true.

Anyhoo, having just spent a month manually serialising objects for network transmission in C++ (exciting new project to be released soon), I decided it was time to write my own BMP loader for Woopsi.

Instead of leaping straight into the C++ code, I figured I’d make a prototype in Python first:

BitmapLoader

This is a Python 3 class that can load BMP files in either 1, 4, 8 or 24-bit colour depth. 16-bit support would be possible fairly simply, but I didn’t have a BMP in the right depth whilst writing this. It can only load uncompressed bitmaps. Finally, it supports all versions of the BMP DIB header, but it treats V4 and V5 as V3 and discards the extra data.

Once that was written, getting a C++ version created was simple. Woopsi now has two new classes in the “bonus” folder:

  • BinaryFile, which can read and write binary files (works with both the SDL and DS libfat versions, as libfat is POSIX compliant for files);
  • BitmapIO, which can load and save BMP files.

The Woopsi version is rather more limited than the Python version. It will only load and save 24-bit bitmaps. It can only deal with version 3 of the DIB header (the most popular BMP format, as far as I can tell). It will only load uncompressed bitmaps.

Usage is like so:

// Loading
Bitmap* bitmap = BitmapIO::loadBMP("/mypicture.bmp");

// Saving
BitmapIO::saveBMP("/myotherpicture.bmp");

Writing the BMP code highlighted a number of bugs in other parts of Woopsi, all of which are now fixed. The Bitmap::getPixel() method didn’t work for bitmaps larger than 65535 pixels. The Bitmap::drawBitmap() method wasn’t calling DC_FlushRange() before trying to DMA copy the source bitmap, leading to rows of black pixels. The SuperBitmap class has an overload for drawBitmap() that accepts a Bitmap object.

Lastly, there’s a new example program demonstrating the BitmapIO functionality.

2009-10-16

80 Character Displays

From the Python styleguide:

There are still many devices around that are limited to 80 character lines

I tend to stick with this limit. But what kind of device is still limited to 80 character lines? All of my computers have at least 1024 pixel wide displays, including my netbook. My Amiga has a wider display. My phone has a wider display. Who the hell is still using these ancient computers?