David Janes' Code Weblog

Djolt is a reimplementation of Django’s template language in Python. Why do this?

• I like the Django template language
• I wanted something that small and independent of Django
• I wanted something that will work with WORK paths (this was the real deal breaker for using Django)
• I wanted something that I could take and reimplement in Javascript and maybe Java too
• Some template engines, Cheetah for example, are far too heavy for the kind of light-weight applications I have in mind; note that I’ve had great success with Cheetah in the past
• Some template engines, such as that in Python 2.6, are for too underfeatured

However, if you’re really looking for the whole Django template experience and don’t want to use Djolt, just start here.

How do I get it?

Djolt is packaged as part of the pybm library.

How do I use it?

import djolt

t = djolt.Template("""
<ul>
{% for name in names %}
<li>{{ name }}</li>
{% endfor %}
</ul>
""")
print t.Render({
    "names" : [ "Johnny", "Jack", "Ray", "Mary & Sam", ]
})

Which gives the results:

<ul>
<li>Johnny</li>
<li>Jack</li>
<li>Ray</li>
<li>Mary &amp; Sam</li>
</ul>

Note the “autoescaping” of the & character.

What tags does it define?

• autoescape/endautoescape
• if/else/endif
• equal/endequal
• for/endfor
• notequal/notendequal

It does not implement blocks.

What filters does it define?

• add
• cut
• default (see otherwise below)
• default_if_none
• divisibleby
• first
• join
• last
• length
• length_is
• linebreaks
• lower
• pluralize
• random
• safe (respecting all the Django autoescape rules)
• slug
• upper

Unimplemented filters are due to laziness and will be done “on demand”. We also introduce a few new filters:

• jslug – like slug, but more Javascript friendly
• otherwise – like default, except the empty string/empty values trigger the filter also

Are their differences between Djolt and Django templates?

• Djolt tags suck up whitespace if they’re on a line by themselves
• If Djolt cannot resolve a variable, it resolves to the appropriate “empty” value (as opposed to failing). This is keeping in line with WORK philosophy

Beyond that you should be able to use most Django template examples (that don’t use block/implements) as-is.

Is it extensible?

Yes. You can add your own tags and filters by following the examples in code (djolt_nodes.py and djolt_filters.py respectively).

The normal way to load Python code is through the import statement:

import pprint
pprint.pprint('Hello, world.')

But what do you do if you want to dynamically load a module? A classic example of where you’d like to do this is adding ‘extensions’ to your application. Your application has no way of knowing the exact name of the module that it’s going to use; it only knows the filename(s). The way to do this is the imp module:

import md5
import os.path
import imp
import traceback

def load_module(code_path):
    try:
        try:
            code_dir = os.path.dirname(code_path)
            code_file = os.path.basename(code_path)

            fin = open(code_path, 'rb')

            return  imp.load_source(md5.new(code_path).hexdigest(), code_path, fin)
        finally:
            try: fin.close()
            except: pass
    except ImportError, x:
        traceback.print_exc(file = sys.stderr)
        raise
    except:
        traceback.print_exc(file = sys.stderr)
        raise

A few notes:

• call load_module with the path to a .py file that you want to load
• the md5.new generates a unique module identifier. If you don’t do this it’s difficult to import two modules in different directories with the same name!
• the different excepts are to give you a flavor of the issues you may see, ImportError is expected, the others are not

The return value is a module, which is a Python object that you can address in all the normal ways that you’d use a module. For example, if you have the following file extension.py:

def hello(x): print "Hello, %s" % x

You can use it as follows to get Hello, world.

m = load_module('extension.py')
m.hello("World")

Here’s a Python code pattern that I find myself falling into every once in awhile. If you’re a highly disciplined milspec-type non-pragmatic programmer, I suggest you stop reading here lest you burn your eyes.

The patterm useful in two situations:

• when you have an evolving superclass that may take new constructor (or method!) arguments in the future and you don’t want to have to recode your subclasses to reflect those changes
• you have a number of interchangeable subclasses that may or may not use certain arguments (say, because you’re constructing the object from a command line)

class Component:
    def __init__(self, a, b = None, *av, **ad):
        ...

class ComponentTemplate(Component):
    def __init__(self, *av, **ad):
        Component.__init__(self, *av, **ad)

a and b are two arguments that are being used by superclass. With this pattern you can add c to Component in the future without worrying about rewriting ComponentTemplate. Similarly, if an unexpected argument is passed down to Component, it will be silently ignored.

In case you’re wondering what *av and **ad are, they’re Python’s way of referring to arguments that have been passed in, by position and by name, but have not been explicitly listed in the method’s signature. The first is a list and the second a dictionary. If you’re a Python user and you’re not familiar with this, you can and should read more about this here.

Here’s a few things I was reading about over the weekend.

SQLAlchemy

SQLAlchemy is a full-featured Design Pattern-heavy pythonic database ORM. I am totally going to use this for my next Python SQL database project and may even do some playing with old datasets (using the reflection features, yum) soon. If you are considering doing SQL work on your next Python project, don’t even bother with the usual PEP 249 stuff, start with this.

Note that if you’re working with Django it handles the DB in its own way so SQLAlchemy may be of limited utility.

CouchDB

CouchDB “is a distributed, fault-tolerant and schema-free document-oriented database accessible via a RESTful HTTP/JSON API”. I couldn’t have written that more succently myself, so I didn’t. I qualified the paragraph above on SQLAlchemy that I’m going to use that for my next SQL project because I’m really biting at the bit to try CouchDB out. The CouchDB design philosophy – a REST API a returning lists of JSON-objects – reflects my current design paradigm very closely, and the only question I have is whether in practically scales to millions of rows.

A caveat that it’s written in the-cool-nerds-are-doing-it language Erlang, but because you don’t have to interact with that it should be OK for us mortals.

CouchDB is about to officially become a “top level” Apache project, though none of the documentation on the Apache.org site reflects this yet.

Virtuoso

Virtuoso is a “high-performance object-relational SQL database”. It apparently can perform well. As I came across through the Planet RDF aggregator, this may be something you want to look into if you’re working on an RDF/SPARQL project.

Amazon Web Services Hosted Data Sets

That’s a mouthfull, isn’t it? Amazon is offering to host public datasets on EC2 for free. What’s the catch? It will host the data, but you have to pay for the computing resources to use that data in the normal EC2 manner. Still, if you’re using a large public dataset and you’re already EC2-friendly, you might want to consider this program. An even more interesting thought occurs (though I’m not sure if it will fly): if you’re using large amounts of your own data on EC2, you may want to offer it up as a free resource.

There’s more on this on by Lidija Davis on Read/Write Web.

Here’s a little mashup I’ve been putting together for the last few days: Toronto Fires.

It’s taking the data listed here on the City of Toronto’s Fire Services “Active Accidents”, scraping it (by pretending HTML is XHTML and treating it as WORK objects), geocoding it (using our WORK Google API) and mapping it (using this information).

This is very much a work in progress, but here’s a few more things that are involved:

• we read body.table.tr.td[1].table.tr[1].td.table.tr as a list to get the rows in the table
• we map those rows into the Geocoder use a new magic technology we’ll be explaining in the next few days: Djolt – Django-like templates
• the output program is just one big Djolt template

I’m not quite satisfied with how the current page is constructed: I want the final result to be much more simple.

A WORK object is simply a way of looking of any JSON-like dictionary – it’s an “attitude”. The primary difference is in how we use that dictionary, especially in the context of using APIs. Here’s a rough overview of the thre

• the WORK is the interface – we don’t need to write specialized methods to deal with an API because we know what we’re looking for anyway. It’s not like we spend weeks looking at our API interface – typically, it’d be more minutes (once it’s coded) in a typically programming session
• what is in the WORK is defined by how we want to use it. For example, if we want an Integer, we ask for an integer for the WORK: it may be stored as a string, a boolean, an integer, or a float; it doesn’t matter. When using WORK objects we expect to the conversion to be done for us at runtime
• if you are looking for a list in the WORK and there’s another type of object, we pretend that it’s in a list of length 1
• if you are looking for an object (by key) and you find a list, look in the first object in the list

These rules are written from a pragmatic vision of using APIs: data is sometimes in lists, sometimes it isn’t. Sometimes we know the types being sent on the wire, sometimes it’s just strings.

In order to use WORK objects efficiently, we define a “dot-path” for accessing items that may be hierarchically nested. We’ll address the type coercion issue in another post. To illustrate our point, we’ll be working with the following WORK object

{
  "name" : "Sally Jones",
  "age" : 22,
  "hobbies" : [ "Skiing", "Windsurfing", "Chillaxing" ],
  "sites" : {
    "facebook" : "http://www.facebook.com",
    "gmail" : "http://gmail.com",
  },
  "address" : [
    {
      "street" : "1 Main Street",
      "city", "Toronto",
      "province" : "Ontario",
    },
    {
      "street" : "RR4",
      "city", "Bala",
      "province" : "Ontario",
    }
  ]
}

Here’s a few dot-paths and the value they’ll retrieve:

• path: name
  value: "Sally Jones"
• path: hobbies
  value: [ "Skiing", "Windsurfing", "Chillaxing" ]
• path: hobbies[1]
  value: "Windsurfing"
• path: address.street
  value: "1 Main Street"
  … this demonstrates seeing a list where we want a dictionary and just looking at the first object in the list
• path: sites[0].facebook
  value: "http://www.facebook.com"
  … this is an example of looking a list, not finding it and assuming there’s an list of length 1 there. sites[1].facebook would return null.

When developing Python code there’s a tendency to do add a __main__ section to test the code:

def add(a, b):
    return  a + b

if __name__ == '__main__':
    print add(3, 4)

Don’t. Python has a great little package called unittest that let’s you quickly frame functions in testcases.

If the example above is called add.py, I’ll generally make a subdirectory called tests and add a test program called test_add.py. This can be as simple as:

import unittest

class TestAdd(unittest.TestCase):
    def setUp(self):
        pass

    def test_1(self):
        self.assertEqual(add(3, 4), 7)
        self.assertEqual(add(4, 4), 8)
        self.assertEqual(add(4, -4), 0)

if __name__ == '__main__':
    unittest.main()

But I prefer to use the following pattern:

class TestAdd(unittest.TestCase):
    def test_add(self):
        checkds = [
            {
                "a" : 4,
                "b" : 3,
                "@result": 7
            },
            {
                "a" : 4,
                "b" : 4,
                "@result": 8
            },
            {
                "a" : 4,
                "b" : -4,
                "@result": 0
            },
        ]

        for checkd in checkds:
            expected_result = checkd.pop("@result")
            actual_result = add(**checkd)

            if expected_result == -1:
                print checkd, actual_result
                continue

            try:
                self.assertEqual(expected_result, actual_result)
            except:
                print checkd, actual_result
                raise

In particular:

• the individual tests are defined in the checkds list of dictionaries
• the bottom part (the for loop) is boilerplate
  • it removes the @result from the dictionary
  • it calls add with the remaining dictionary
  • and it then asserts that the actual_result was the same as the expected_result
• if the expected_result is -1, it doesn’t run the test, it just prints the actual_result. This is great for setting up your tests in the first place. Obviously you might way to change this marker for testing functions that can return -1, but you get the idea

The advantage of using unittest like is that you’re now not depending on visual inspection or remembering which files you put a __main__ in to test your code. As a secondary benefit, unittest helps you think about edge cases, how other people might call your code.

Just go to your test directory and run them all and you’ll be sure your libraries are behaving as designed.

Here’s an example of implementing an API with many different endpoints. It’s the Google AJAX Search API which lets you access all of Google’s search engines programmatically! A few notes:

• In the Javascript API Google provides “branding” functions to make sure search results are properly attributed. There doesn’t seem to be a corresponding AJAX call — that is, it’s probably implemented directly in the Javascript — but I’d still like to provide a corresponding function. It would be nice if API providers actually gave a branding end-point
• The code doesn’t support (yet) multi-page results: coming soon
• The clever bit is in _item_path, which describes how to pull WORK result objects out of the AJAX result
• all this code is actually available right now, via SVN: the instructions are here. This library is standalone (and is in fact the basis for many of the other projects I have on Google code)
• The Google API requires a _http_referer: the URL of the site that’s using the results
• The Google API does not require an API key, but you can pass one (in the constructor or in individual search calls) under the key api_key. You can use the same API key that you’ve created for Google Maps.

Here’s the Google API class: quite simple. I’ll probably extend each individual search function to provide all the known parameters by name, rather than passing in a **ad catch-all.

class Google(bm_api.API):
    _base_query = {
        "v" : "1.0",
    }

    _item_path = "responseData.results"
    _meta_path = "responseData.cursor"
    _convert2work = bm_work.JSON2WORK()

    def __init__(self, _http_referer, **ad):
        bm_api.API.__init__(self, _http_referer = _http_referer, **ad)

    def WebSearch(self, q, **ad):
        self._uri_base = "http://ajax.googleapis.com/ajax/services/search/web"
        self.SearchOn(q = q, **ad)

    def LocalSearch(self, q, **ad):
        self._uri_base = "http://ajax.googleapis.com/ajax/services/search/local"
        self.SearchOn(q = q, **ad)

    def VideoSearch(self, q, **ad):
        self._uri_base = "http://ajax.googleapis.com/ajax/services/search/video"
        self.SearchOn(q = q, **ad)

    def BlogSearch(self, q, **ad):
        self._uri_base = "http://ajax.googleapis.com/ajax/services/search/blogs"
        self.SearchOn(q = q, **ad)

    def NewsSearch(self, q, **ad):
        self._uri_base = "http://ajax.googleapis.com/ajax/services/search/news"
        self.SearchOn(q = q, **ad)

    def BookSearch(self, q, **ad):
        self._uri_base = "http://ajax.googleapis.com/ajax/services/search/books"
        self.SearchOn(q = q, **ad)

    def ImageSearch(self, q, **ad):
        self._uri_base = "http://ajax.googleapis.com/ajax/services/search/images"
        self.SearchOn(q = q, **ad)

    def PatentSearch(self, q, **ad):
        self._uri_base = "http://ajax.googleapis.com/ajax/services/search/patentNew"
        self.SearchOn(q = q, **ad)

Here’s how you use it:

api_key = os.environ["GMAPS_APIKEY"]
referer = "http://code.davidjanes.com"
query = "Paris Hilton"

api = Google(key = api_key, _http_referer = referer)
api.VideoSearch(query)

for item in api.IterItems():
    pprint.pprint(item)

Here’s an example of a results, searching for “Paris Hilton” in Videos. I tried searching in Patents without luck.

{'@Index': 0,
 '@Page': 1,
 u'GsearchResultClass': u'GvideoSearch',
 u'content': u"Paris Hilton's new video clip for 'Nothing In This World'",
 u'duration': u'204',
 u'playUrl': u'http://www.youtube.com/v/...',
 u'published': u'Thu, 12 Oct 2006 09:33:23 PDT',
 u'publisher': u'www.youtube.com',
 u'rating': u'4.52872',
 u'tbHeight': u'240',
 u'tbUrl': u'http://0.gvt0.com/vi/Ki2M3-2W-cQ/0.jpg',
 u'tbWidth': u'320',
 u'title': u'Paris Hilton - Nothing In This World',
 u'titleNoFormatting': u'Paris Hilton - Nothing In This World',
 u'url': u'http://www.google.com/url?q=...',
 u'videoType': u'YouTube'}

Here’s the following changes I made to this blog to make it readable on an iPhone

Add iPhone directives to header

• the META tag informas the iPhone about how wide we want the page to look – i.e. the width of the iPhone
• the LINK tag loads our iPhone specific CSS (tip from here)
• this should be after your normal CSS LINK (or whatever) directive

<meta name="viewport" content="width=320" />
<link rel="stylesheet" type="text/css"
  media="only screen and (max-device-width: 480px)"
  href="/blog/wp-content/themes/davidjanes/iphone.css" />

Define iPhone specific CSS

• this will obviously vary depending on what your current CSS does – that still gets loaded
• most of this was pragmatically discovered
• I hide several sections which I don’t think the user wants to see on an iPhone; I’ll probably play with this more
• the PRE directive doesn’t use line breaking, so we just clip the examples, after making sure the font is small enough to get enough on a single line

body {
    padding: 5px;
    width: 480px;
}
div#content {
    float: none;
}
div#menu {
    display: none;
}
p.credit {
    display: none;
}
pre {
    overflow: hidden;
    font-size: 10px !important;
}
h1, h1 * {
    font-size: 36px;
}

What still needs to be done

• we shouldn’t serve sections that users are not going to see – it’s a waste of bandwidth
• we shouldn’t serve more than 10 articles
• I’ll have to figure out how to do mobile browser detection on WordPress

All the major map APIs have the ability to zoom in and out if your pointer is over the map and you scroll the mouse wheel. Being able to disable this function if you’re working in a small popup form window is very important!

Google Maps

By default, this feature is disabled. To enable:

map.enableScrollWheelZoom();

To disable (again):

map.disableScrollWheelZoom();

Source

Yahoo Maps

By default, this feature is enabled. To disable:

map.disableKeyControls()

There doesn’t appear to be a way to re-enable afterward.

Source

Microsoft Virtual Earth

By default, this feature is enabled. To disable you have to capture the event:

trap = function() { return true; }
map.AttachEvent("onmousewheel", trap);

To re-enable, you have to detach the exact same event (hence the trap function)

map.DetachEvent("onmousewheel", trap);

Source