2013-01-06

DL/ID Parser Library For Go

I’ve published an initial version of my US DL/ID barcode data parser library here:

This is a library for Go that can parse the data extracted from the PDF417 barcode on the back of a US driving licence into a Go struct. It doesn’t handle many of the encoded fields yet but it’s a start. I’ve grumbled about the multiple problems with the spec and its various implementations before.

2011-02-17

The BitBucket API, Python and urllib

If you’ve ever tried googling for anything related to Python 3, you’ll have encountered difficulties. This is because “Python 3” is also known as both “Python 3000” and “Py3k”. Search for just one name and you’ll miss one third of the potential results. This is somewhat contrary to the Python philisophy:

There should be one-- and preferably only one --obvious way to do it.

We’re off to a good start already.

As discussed recently, BitBucket has an incomplete but promising REST API that allows for the automation of things like repository creation and issue tracking. I’ve written a messy C# program that takes advantage of some of the repository features, but wanted to try writing a complete API library in Python 3.

Python is currently having problems. Zed Shaw over at sheddingbikes.com has complained about the unwillingness of operating system providers to upgrade from ancient versions of Python. I personally can’t quite see why so much of Python 3 has been backported to Python 2, when encouraging people to upgrade where they need to by offering new features would be far more beneficial for the new version of the language.

I’ve been using Python’s urllib to access the BitBucket API. The BitBucket API requires the use of basic authentication in order to get access to the full feature set. Unfortunately, the “obvious way to do it” doesn’t work.

This StackOverflow question gives the “canonical” way of achieving basic authentication using urllib. I’ve ported it from Python 2 to Python 3:

import urllib.request

username = "username"
password = "password"
url = "https://api.bitbucket.org/1.0/users/ant512/"

passman = urllib.request.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, url, username, password)

authhandler = urllib.request.HTTPBasicAuthHandler(passman)
opener = urllib.request.build_opener(authhandler)

urllib.request.install_opener(opener)

pagehandle = urllib.request.urlopen(theurl)
print(pagehandle.read())

Looks great! Doesn’t work. The problem is that urllib expects this sequence of events to happen:

  • Client issues a request
  • Server issues a 401 Not Authenticated error
  • Client re-issues the request with authentication
  • Server sends data with a 200 response.

BitBucket doesn’t work like that, though. BitBucket does this if no credentials are supplied:

  • Client issues a request
  • Server sends data with a 200 response.

The data returned in this case is a list of public repositories.

When the initial request is sent with credentials, BitBucket does this:

  • Client issues a request with authentication
  • Server sends data with a 200 response.

In this case, the data returned is a list of public and private repositories to which the authenticated user has access. For this situation, urllib doesn’t work.

What we actually need to do is this:

import urllib.request
import base64

username = "username"
password = "password"
url = "https://api.bitbucket.org/1.0/users/ant512/"

credentials = base64.b64encode("{0}:{1}".format(username, password).encode()).decode("ascii")
headers = {'Authorization': "Basic " + credentials}
request = urllib.request.Request(url=url, headers=headers)

connection = urllib.request.urlopen(request)
content = connection.read()

Of course! Why didn’t I see it in the first place? All I have to do is ignore the library’s API and instead encode a username/password tuple into base 64, decode it back into ASCII, manually inject it into the request header, and then use urllib. Obvious. Yes.

At least we can now retrieve a list of a user’s repositories. Let’s make things a little more interesting and update an existing issue in the BitBucket issue tracker.

import urllib.request
import base64

username = "username"
password = "password"
url = "https://api.bitbucket.org/1.0/issues/ant512/woopsi/issues/10/"

credentials = base64.b64encode("{0}:{1}".format(username, password).encode()).decode("ascii")
headers = {'Authorization': "Basic " + credentials}
data = {"title": "Issue title", "content": "Issue content"}
request = urllib.request.Request(url=url, headers=headers, data=data)

connection = urllib.request.urlopen(request)
content = connection.read()

This won’t work. A REST API requires the use of 4 HTTP verbs:

  • GET
  • POST
  • PUT
  • DELETE

Out of the box, urllib only gives us GET and POST. It automatically determines whether to GET or POST depending on whether or not a request has a data payload. The urllib thus reveals itself to be an abstraction so leaky it should be on the bottom of the sea by now. It also explains why so many other people have written their own replacements for it.

What we need to do instead is subclass the Request object so that it is possible to specify which HTTP verb to use:

class RESTRequest(urllib.request.Request):
    __method = None

    def __init__(self, url, data=None, headers={}, origin_req_host=None, unverifiable=False, method=None):
        super().__init__(url, data, headers, origin_req_host, unverifiable)
        self.__method = method

    def get_method(self):

        # Uses the standard method choosing logic if no other method has been
        # explicitly set.

        if self.__method is None:
            return super().get_method()
        else:
            return self.__method

We can now alter our code so that we update issues in the issue tracker:

import urllib.request
import urllib.parse
import base64

username = "username"
password = "password"
url = "https://api.bitbucket.org/1.0/issues/ant512/woopsi/issues/10/"

credentials = base64.b64encode("{0}:{1}".format(username, password).encode()).decode("ascii")
headers = {'Authorization': "Basic " + credentials}
data = urllib.parse.urlencode({"title": "Issue title", "content": "Issue content"}).encode()
request = RESTRequest(url=url, headers=headers, data=data, method="PUT")

connection = urllib.request.urlopen(request)
content = connection.read()

Note that the data we send has to be in the format of a set of URL-encoded tuples.

Here’s one I haven’t been able to solve yet. Suppose we want to give bob@example.com write access to one of our repositories:

import urllib.request
import urllib.parse
import base64

username = "username"
password = "password"
url = "https://api.bitbucket.org/1.0/privileges/ant512/woopsi/bob@example.com"

credentials = base64.b64encode("{0}:{1}".format(username, password).encode()).decode("ascii")
headers = {'Authorization': "Basic " + credentials}
data = urllib.parse.urlencode({"data": "write"}).encode()
request = RESTRequest(url=url, headers=headers, data=data, method="PUT")

connection = urllib.request.urlopen(request)
content = connection.read()

The data we’re sending over consists of the following tuple:

{"data": "write"}

The BitBucket API says that the only data that should be sent as part of the request should be the words “read”, “write” or “admin”, depending on the level of access that should be given. We shouldn’t be sending a tuple at all. As expected, running this gives us a 400 “bad request” error. The problem we have is that the urllib will only allow us to send tuples. We can’t send a single word on its own. If we try it, Python throws a ValueError. The leaky abstraction comes back to bite us again.

Despite the obstacles put in the way, I’ve managed to get almost all of BitBucket’s API wrapped up in a Python 3 library:

The library probably doesn’t adhere amazingly well to Python’s coding standards (though there really is one-- and only one --obvious way to perform any action), but it does provide access to almost all of the functionality offered by BitBucket’s current API. The only exceptions are:

  • The grantPrivileges() method does not work (broken due to the limitations of urllib, but there might be a way to work around it)
  • The updateRepository() method currently doesn’t offer much of the possible functionality
  • The createWiki() method does not work (this could either be a bug in BitBucket’s API, a limitation of urllib or a bug in the library itself)

All data is returned as JSON objects. It is parsed from strings using the json library. The library can use basic authentication if a username/password combination is supplied. If not, it will interact with the server without authenticating where possible.

2010-07-28

WoopsiGfx - A 2D Graphics Library

WoopsiGfx is a C++ 2D graphics library for the Nintendo DS, derived from Woopsi. It allows developers to create and manipulate bitmaps using a comprehensive set of drawing tools. It includes an extensible font system for drawing text to bitmaps, and features support for packed monochrome and 16-bit fonts out of the box.

WoopsiGfx can be used to draw directly to the DS’ VRAM. This is useful when the DS is in MODE_FB0 or MODE_5_2D.

Features

  • Extensible font system that supports compressed proportional and fixed-width fonts (monochrome and 16-bit);
  • Animation class with support for variable framerates and standard/pingpong looping;
  • Bitmap class for 16-bit bitmap image manipulation;
  • Graphics class providing clipped, DMA-accelerated drawing functions;
  • Dynamic array container and iterator classes;
  • Object-orientated design for easy integration into other C++ software;
  • Simple API;
  • Unicode strings encoded with UTF-8;
  • Compatible with Woopsi font tools.

You can download a demo here:

http://bitbucket.org/ant512/woopsigfx/downloads/woopsigfx-demo-1.00.zip

The source is available as a zip here:

http://bitbucket.org/ant512/woopsigfx/downloads/woopsigfx-src-1.00.zip

Alternatively, you can pull down the Mercurial sourcecode repository from here:

http://bitbucket.org/ant512/woopsigfx

2009-10-27

Libraries, Upgrades and Fixes

Version 0.40 of Woopsi is well underway. There are a number of significant changes coming in this release.

Folder Structure

Woopsi was initially set up to be easy to edit using Visual Studio Express. I used its default folder structure. However, this is at odds with how a library should be organised. So, the Woopsi root directory no longer includes a “woopsi” folder with all sourcecode. It has a “libwoopsi” folder instead, which contains “include”, “src” and “lib” folders containing .h, .cpp and pre-built .a files respectively. It also contains a makefile that will create the library.

For the first time, Woopsi can now easily be used as a proper library. Projects can link against the existing library binary. Installation instructions are included in the archive, but it boils down to this:

  1. Copy the “libwoopsi” folder to the devkitPro directory.
  2. Include $(DEVKITPRO)/libwoopsi/include in your makefile’s include path.
  3. Include $(DEVKITPRO)/libwoopsi/lib in your makefile’s library path.
  4. Link with woopsi.

Version 0.40 includes a template program with the makefile ready-made.

Visual Studio Upgrade

The Woopsi VC++ project has been upgraded to use VC++ 2008 Express. After having performed a number of VS2003 to VS2005 upgrades for ASP.NET web projects, I was expecting the upgrade to be far more onerous than it eventually turned out to be. The upgrade did nothing more than change a couple of version strings in the VS project files.

PALib Support

As mentioned in the last post, PALib support has been dropped. This makes Woopsi slightly tidier and easier to maintain.

Alpha Release

Version 0.40 will be the first alpha version of Woopsi. I have already updated the SourceForge page. From this point on, I am mainly fixing bugs in the existing code.

Testing and Bugfixes

I’ve added three new gadget test projects to the Woopsi distribution - calendar, multiline textbox and standard textbox tests. These have illuminated yet more bugs.

The Calendar gadget’s getPreferredSize() method now returns correct values. Its layout code is now far more intelligent. Previously, the calendar only really looked right when created at certain sizes; at most sizes, the calendar left gaps on the right of the day selection buttons and between those buttons and the labels above them. This has now been addressed, and the calendar’s gadgets use all of the available space. Additionally, its resize() routine is magnitudes faster.

If a day in the calendar was selected and the month changed, if the same day was selected in the new month, the selection wasn’t remembered. For example, selecting “01/01/2009” and then choosing “01/02/2009” (dd/mm/yyyy) would result in the calendar ignoring the second selection.

The last calendar bug is a small one. The day labels are greyed out when the calendar is disabled. This fix has also been applied to the multiline and standard textboxes. The MultiLineTextBox::removeText() method correctly repositions the cursor.

There are a couple of fixes to more fundamental gadgets. The Gadget and ScrollingPanel classes only fire their drag and scrolled events if the stylus actually moves. Previously, just clicking and holding a gadget would cause the drag event to be fired continually.

Finally, the “scrolltest” example now compiles properly.