David Janes' Code Weblog

Djolt is a reimplementation of Django’s template language in Python. Why do this?

• I like the Django template language
• I wanted something that small and independent of Django
• I wanted something that will work with WORK paths (this was the real deal breaker for using Django)
• I wanted something that I could take and reimplement in Javascript and maybe Java too
• Some template engines, Cheetah for example, are far too heavy for the kind of light-weight applications I have in mind; note that I’ve had great success with Cheetah in the past
• Some template engines, such as that in Python 2.6, are for too underfeatured

However, if you’re really looking for the whole Django template experience and don’t want to use Djolt, just start here.

How do I get it?

Djolt is packaged as part of the pybm library.

How do I use it?

import djolt

t = djolt.Template("""
<ul>
{% for name in names %}
<li>{{ name }}</li>
{% endfor %}
</ul>
""")
print t.Render({
    "names" : [ "Johnny", "Jack", "Ray", "Mary & Sam", ]
})

Which gives the results:

<ul>
<li>Johnny</li>
<li>Jack</li>
<li>Ray</li>
<li>Mary &amp; Sam</li>
</ul>

Note the “autoescaping” of the & character.

What tags does it define?

• autoescape/endautoescape
• if/else/endif
• equal/endequal
• for/endfor
• notequal/notendequal

It does not implement blocks.

What filters does it define?

• add
• cut
• default (see otherwise below)
• default_if_none
• divisibleby
• first
• join
• last
• length
• length_is
• linebreaks
• lower
• pluralize
• random
• safe (respecting all the Django autoescape rules)
• slug
• upper

Unimplemented filters are due to laziness and will be done “on demand”. We also introduce a few new filters:

• jslug – like slug, but more Javascript friendly
• otherwise – like default, except the empty string/empty values trigger the filter also

Are their differences between Djolt and Django templates?

• Djolt tags suck up whitespace if they’re on a line by themselves
• If Djolt cannot resolve a variable, it resolves to the appropriate “empty” value (as opposed to failing). This is keeping in line with WORK philosophy

Beyond that you should be able to use most Django template examples (that don’t use block/implements) as-is.

Is it extensible?

Yes. You can add your own tags and filters by following the examples in code (djolt_nodes.py and djolt_filters.py respectively).

The normal way to load Python code is through the import statement:

import pprint
pprint.pprint('Hello, world.')

But what do you do if you want to dynamically load a module? A classic example of where you’d like to do this is adding ‘extensions’ to your application. Your application has no way of knowing the exact name of the module that it’s going to use; it only knows the filename(s). The way to do this is the imp module:

import md5
import os.path
import imp
import traceback

def load_module(code_path):
    try:
        try:
            code_dir = os.path.dirname(code_path)
            code_file = os.path.basename(code_path)

            fin = open(code_path, 'rb')

            return  imp.load_source(md5.new(code_path).hexdigest(), code_path, fin)
        finally:
            try: fin.close()
            except: pass
    except ImportError, x:
        traceback.print_exc(file = sys.stderr)
        raise
    except:
        traceback.print_exc(file = sys.stderr)
        raise

A few notes:

• call load_module with the path to a .py file that you want to load
• the md5.new generates a unique module identifier. If you don’t do this it’s difficult to import two modules in different directories with the same name!
• the different excepts are to give you a flavor of the issues you may see, ImportError is expected, the others are not

The return value is a module, which is a Python object that you can address in all the normal ways that you’d use a module. For example, if you have the following file extension.py:

def hello(x): print "Hello, %s" % x

You can use it as follows to get Hello, world.

m = load_module('extension.py')
m.hello("World")

Here’s a Python code pattern that I find myself falling into every once in awhile. If you’re a highly disciplined milspec-type non-pragmatic programmer, I suggest you stop reading here lest you burn your eyes.

The patterm useful in two situations:

• when you have an evolving superclass that may take new constructor (or method!) arguments in the future and you don’t want to have to recode your subclasses to reflect those changes
• you have a number of interchangeable subclasses that may or may not use certain arguments (say, because you’re constructing the object from a command line)

class Component:
    def __init__(self, a, b = None, *av, **ad):
        ...

class ComponentTemplate(Component):
    def __init__(self, *av, **ad):
        Component.__init__(self, *av, **ad)

a and b are two arguments that are being used by superclass. With this pattern you can add c to Component in the future without worrying about rewriting ComponentTemplate. Similarly, if an unexpected argument is passed down to Component, it will be silently ignored.

In case you’re wondering what *av and **ad are, they’re Python’s way of referring to arguments that have been passed in, by position and by name, but have not been explicitly listed in the method’s signature. The first is a list and the second a dictionary. If you’re a Python user and you’re not familiar with this, you can and should read more about this here.

Here’s a few things I was reading about over the weekend.

SQLAlchemy

SQLAlchemy is a full-featured Design Pattern-heavy pythonic database ORM. I am totally going to use this for my next Python SQL database project and may even do some playing with old datasets (using the reflection features, yum) soon. If you are considering doing SQL work on your next Python project, don’t even bother with the usual PEP 249 stuff, start with this.

Note that if you’re working with Django it handles the DB in its own way so SQLAlchemy may be of limited utility.

CouchDB

CouchDB “is a distributed, fault-tolerant and schema-free document-oriented database accessible via a RESTful HTTP/JSON API”. I couldn’t have written that more succently myself, so I didn’t. I qualified the paragraph above on SQLAlchemy that I’m going to use that for my next SQL project because I’m really biting at the bit to try CouchDB out. The CouchDB design philosophy – a REST API a returning lists of JSON-objects – reflects my current design paradigm very closely, and the only question I have is whether in practically scales to millions of rows.

A caveat that it’s written in the-cool-nerds-are-doing-it language Erlang, but because you don’t have to interact with that it should be OK for us mortals.

CouchDB is about to officially become a “top level” Apache project, though none of the documentation on the Apache.org site reflects this yet.

Virtuoso

Virtuoso is a “high-performance object-relational SQL database”. It apparently can perform well. As I came across through the Planet RDF aggregator, this may be something you want to look into if you’re working on an RDF/SPARQL project.

Amazon Web Services Hosted Data Sets

That’s a mouthfull, isn’t it? Amazon is offering to host public datasets on EC2 for free. What’s the catch? It will host the data, but you have to pay for the computing resources to use that data in the normal EC2 manner. Still, if you’re using a large public dataset and you’re already EC2-friendly, you might want to consider this program. An even more interesting thought occurs (though I’m not sure if it will fly): if you’re using large amounts of your own data on EC2, you may want to offer it up as a free resource.

There’s more on this on by Lidija Davis on Read/Write Web.

Here’s a little mashup I’ve been putting together for the last few days: Toronto Fires.

It’s taking the data listed here on the City of Toronto’s Fire Services “Active Accidents”, scraping it (by pretending HTML is XHTML and treating it as WORK objects), geocoding it (using our WORK Google API) and mapping it (using this information).

This is very much a work in progress, but here’s a few more things that are involved:

• we read body.table.tr.td[1].table.tr[1].td.table.tr as a list to get the rows in the table
• we map those rows into the Geocoder use a new magic technology we’ll be explaining in the next few days: Djolt – Django-like templates
• the output program is just one big Djolt template

I’m not quite satisfied with how the current page is constructed: I want the final result to be much more simple.

A WORK object is simply a way of looking of any JSON-like dictionary – it’s an “attitude”. The primary difference is in how we use that dictionary, especially in the context of using APIs. Here’s a rough overview of the thre

• the WORK is the interface – we don’t need to write specialized methods to deal with an API because we know what we’re looking for anyway. It’s not like we spend weeks looking at our API interface – typically, it’d be more minutes (once it’s coded) in a typically programming session
• what is in the WORK is defined by how we want to use it. For example, if we want an Integer, we ask for an integer for the WORK: it may be stored as a string, a boolean, an integer, or a float; it doesn’t matter. When using WORK objects we expect to the conversion to be done for us at runtime
• if you are looking for a list in the WORK and there’s another type of object, we pretend that it’s in a list of length 1
• if you are looking for an object (by key) and you find a list, look in the first object in the list

These rules are written from a pragmatic vision of using APIs: data is sometimes in lists, sometimes it isn’t. Sometimes we know the types being sent on the wire, sometimes it’s just strings.

In order to use WORK objects efficiently, we define a “dot-path” for accessing items that may be hierarchically nested. We’ll address the type coercion issue in another post. To illustrate our point, we’ll be working with the following WORK object

{
  "name" : "Sally Jones",
  "age" : 22,
  "hobbies" : [ "Skiing", "Windsurfing", "Chillaxing" ],
  "sites" : {
    "facebook" : "http://www.facebook.com",
    "gmail" : "http://gmail.com",
  },
  "address" : [
    {
      "street" : "1 Main Street",
      "city", "Toronto",
      "province" : "Ontario",
    },
    {
      "street" : "RR4",
      "city", "Bala",
      "province" : "Ontario",
    }
  ]
}

Here’s a few dot-paths and the value they’ll retrieve:

• path: name
  value: "Sally Jones"
• path: hobbies
  value: [ "Skiing", "Windsurfing", "Chillaxing" ]
• path: hobbies[1]
  value: "Windsurfing"
• path: address.street
  value: "1 Main Street"
  … this demonstrates seeing a list where we want a dictionary and just looking at the first object in the list
• path: sites[0].facebook
  value: "http://www.facebook.com"
  … this is an example of looking a list, not finding it and assuming there’s an list of length 1 there. sites[1].facebook would return null.

When developing Python code there’s a tendency to do add a __main__ section to test the code:

def add(a, b):
    return  a + b

if __name__ == '__main__':
    print add(3, 4)

Don’t. Python has a great little package called unittest that let’s you quickly frame functions in testcases.

If the example above is called add.py, I’ll generally make a subdirectory called tests and add a test program called test_add.py. This can be as simple as:

import unittest

class TestAdd(unittest.TestCase):
    def setUp(self):
        pass    

    def test_1(self):
        self.assertEqual(add(3, 4), 7)
        self.assertEqual(add(4, 4), 8)
        self.assertEqual(add(4, -4), 0)

if __name__ == '__main__':
    unittest.main()

But I prefer to use the following pattern:

class TestAdd(unittest.TestCase):
    def test_add(self):
        checkds = [
            {
                "a" : 4,
                "b" : 3,
                "@result": 7
            },
            {
                "a" : 4,
                "b" : 4,
                "@result": 8
            },
            {
                "a" : 4,
                "b" : -4,
                "@result": 0
            },
        ]

        for checkd in checkds:
            expected_result = checkd.pop("@result")
            actual_result = add(**checkd)

            if expected_result == -1:
                print checkd, actual_result
                continue

            try:
                self.assertEqual(expected_result, actual_result)
            except:
                print checkd, actual_result
                raise

In particular:

• the individual tests are defined in the checkds list of dictionaries
• the bottom part (the for loop) is boilerplate
  • it removes the @result from the dictionary
  • it calls add with the remaining dictionary
  • and it then asserts that the actual_result was the same as the expected_result
• if the expected_result is -1, it doesn’t run the test, it just prints the actual_result. This is great for setting up your tests in the first place. Obviously you might way to change this marker for testing functions that can return -1, but you get the idea

The advantage of using unittest like is that you’re now not depending on visual inspection or remembering which files you put a __main__ in to test your code. As a secondary benefit, unittest helps you think about edge cases, how other people might call your code.

Just go to your test directory and run them all and you’ll be sure your libraries are behaving as designed.

Here’s an example of implementing an API with many different endpoints. It’s the Google AJAX Search API which lets you access all of Google’s search engines programmatically! A few notes:

• In the Javascript API Google provides “branding” functions to make sure search results are properly attributed. There doesn’t seem to be a corresponding AJAX call — that is, it’s probably implemented directly in the Javascript — but I’d still like to provide a corresponding function. It would be nice if API providers actually gave a branding end-point
• The code doesn’t support (yet) multi-page results: coming soon
• The clever bit is in _item_path, which describes how to pull WORK result objects out of the AJAX result
• all this code is actually available right now, via SVN: the instructions are here. This library is standalone (and is in fact the basis for many of the other projects I have on Google code)
• The Google API requires a _http_referer: the URL of the site that’s using the results
• The Google API does not require an API key, but you can pass one (in the constructor or in individual search calls) under the key api_key. You can use the same API key that you’ve created for Google Maps.

Here’s the Google API class: quite simple. I’ll probably extend each individual search function to provide all the known parameters by name, rather than passing in a **ad catch-all.

class Google(bm_api.API):
    _base_query = {
        "v" : "1.0",
    }

    _item_path = "responseData.results"
    _meta_path = "responseData.cursor"
    _convert2work = bm_work.JSON2WORK()

    def __init__(self, _http_referer, **ad):
        bm_api.API.__init__(self, _http_referer = _http_referer, **ad)

    def WebSearch(self, q, **ad):
        self._uri_base = "http://ajax.googleapis.com/ajax/services/search/web"
        self.SearchOn(q = q, **ad)

    def LocalSearch(self, q, **ad):
        self._uri_base = "http://ajax.googleapis.com/ajax/services/search/local"
        self.SearchOn(q = q, **ad)

    def VideoSearch(self, q, **ad):
        self._uri_base = "http://ajax.googleapis.com/ajax/services/search/video"
        self.SearchOn(q = q, **ad)

    def BlogSearch(self, q, **ad):
        self._uri_base = "http://ajax.googleapis.com/ajax/services/search/blogs"
        self.SearchOn(q = q, **ad)

    def NewsSearch(self, q, **ad):
        self._uri_base = "http://ajax.googleapis.com/ajax/services/search/news"
        self.SearchOn(q = q, **ad)

    def BookSearch(self, q, **ad):
        self._uri_base = "http://ajax.googleapis.com/ajax/services/search/books"
        self.SearchOn(q = q, **ad)

    def ImageSearch(self, q, **ad):
        self._uri_base = "http://ajax.googleapis.com/ajax/services/search/images"
        self.SearchOn(q = q, **ad)

    def PatentSearch(self, q, **ad):
        self._uri_base = "http://ajax.googleapis.com/ajax/services/search/patentNew"
        self.SearchOn(q = q, **ad)

Here’s how you use it:

api_key = os.environ["GMAPS_APIKEY"]
referer = "http://code.davidjanes.com"
query = "Paris Hilton"

api = Google(key = api_key, _http_referer = referer)
api.VideoSearch(query)

for item in api.IterItems():
    pprint.pprint(item)

Here’s an example of a results, searching for “Paris Hilton” in Videos. I tried searching in Patents without luck.

{'@Index': 0,
 '@Page': 1,
 u'GsearchResultClass': u'GvideoSearch',
 u'content': u"Paris Hilton's new video clip for 'Nothing In This World'",
 u'duration': u'204',
 u'playUrl': u'http://www.youtube.com/v/...',
 u'published': u'Thu, 12 Oct 2006 09:33:23 PDT',
 u'publisher': u'www.youtube.com',
 u'rating': u'4.52872',
 u'tbHeight': u'240',
 u'tbUrl': u'http://0.gvt0.com/vi/Ki2M3-2W-cQ/0.jpg',
 u'tbWidth': u'320',
 u'title': u'Paris Hilton - Nothing In This World',
 u'titleNoFormatting': u'Paris Hilton - Nothing In This World',
 u'url': u'http://www.google.com/url?q=...',
 u'videoType': u'YouTube'}

Here’s the following changes I made to this blog to make it readable on an iPhone

Add iPhone directives to header

• the META tag informas the iPhone about how wide we want the page to look – i.e. the width of the iPhone
• the LINK tag loads our iPhone specific CSS (tip from here)
• this should be after your normal CSS LINK (or whatever) directive

<meta name="viewport" content="width=320" />
<link rel="stylesheet" type="text/css"
  media="only screen and (max-device-width: 480px)"
  href="/blog/wp-content/themes/davidjanes/iphone.css" />

Define iPhone specific CSS

• this will obviously vary depending on what your current CSS does – that still gets loaded
• most of this was pragmatically discovered
• I hide several sections which I don’t think the user wants to see on an iPhone; I’ll probably play with this more
• the PRE directive doesn’t use line breaking, so we just clip the examples, after making sure the font is small enough to get enough on a single line

body {
    padding: 5px;
    width: 480px;
}
div#content {
    float: none;
}
div#menu {
    display: none;
}
p.credit {
    display: none;
}
pre {
    overflow: hidden;
    font-size: 10px !important;
}
h1, h1 * {
    font-size: 36px;
}

What still needs to be done

• we shouldn’t serve sections that users are not going to see – it’s a waste of bandwidth
• we shouldn’t serve more than 10 articles
• I’ll have to figure out how to do mobile browser detection on WordPress

All the major map APIs have the ability to zoom in and out if your pointer is over the map and you scroll the mouse wheel. Being able to disable this function if you’re working in a small popup form window is very important!

Google Maps

By default, this feature is disabled. To enable:

map.enableScrollWheelZoom();

To disable (again):

map.disableScrollWheelZoom();

Source

Yahoo Maps

By default, this feature is enabled. To disable:

map.disableKeyControls()

There doesn’t appear to be a way to re-enable afterward.

Source

Microsoft Virtual Earth

By default, this feature is enabled. To disable you have to capture the event:

trap = function() { return true; }
map.AttachEvent("onmousewheel", trap);

To re-enable, you have to detach the exact same event (hence the trap function)

map.DetachEvent("onmousewheel", trap);

Source

Implementing a merchant search using the Praized API took about 10 minutes (mainly finding the right documentation), using my WORK framework:

class PraizedMerchants(bm_api.API):
    """See: http://code.google.com/p/praized/wiki/A_Second_Tutorial_Search"""

    _uri_base = "http://api.praized.com/apitribe/merchants.xml"
    _meta_path = "community"
    _item_path = "merchants.merchant"
    _page_max_path = 'pagination.page_count'
    _page_max = -1

    def __init__(self, api_key, slug = "apitribe", **ad):
        bm_api.API.__init__(self, api_key = api_key, **ad)

        self._uri_base = "http://api.praized.com/%s/merchants.xml" % slug

    def CustomizePageURI(self, page_index):
        if page_index > 1:
            return  "page=%s" % page_index

Partially hardcoding ‘apitribe’ as a ‘community slug’ is probably a bad idea. Anyhoo, here’s how you call it…

api_key = os.environ["PRAIZED_APIKEY"]
api = PraizedMerchants(api_key = api_key, slug = "david-janess-code")
api.SearchOn(
    q = "Bistro",
    l = "Toronto",
)
for item in api.IterItems():
    print json.dumps(item, indent = 1)

… and a set if results, somewhat edited below. I’ll have to figure out what that “permalink” is all about (I’ve edited it to shorten it)  … it could be something neat, but I haven’t quite grasped all the ins and outs of what Praized wants to accomplish as a business.

{
 "@Index": 0,
 "@Page": 1,
 "short_url": "http://przd.com/zAU-7",
 "pid": "af5bebd604f3d1517a8113e0a2e8cc58",
 "updated_at": "2008-10-04T20:49:34Z",
 "phone": "(416) 585-7896",
 "permalink":
   ".../praized/places/ca/ontario/toronto/coffee-supreme-bistro?l=Toronto&q=Bistro",
 "name": "Coffee Supreme Bistro",
 "created_at": "2008-10-04T20:49:34Z",
 "location": {
  "city": {
   "name": "Toronto"
  },
  "country": {
   "code": "CA",
   "name_fr": "Canada",
   "name": "Canada"
  },
  "longitude": "-79.384071",
  "regions": {
   "province": "Ontario"
  },
  "postal_code": "M5J 1T1",
  "latitude": "43.646347",
  "street_address": "40 University Avenue"
 }
}

Following from the concepts I wrote about yesterday, here’s two examples of API parsers using a WORK model.

RSS 2.0

Class definition – that’s the whole thing there!:

class RSS20(API):
    _item_path = "channel.item"
    _meta_path = "channel"

    def __init__(self, uri):
        API.__init__(self) 

        self._uri_base = uri

Using it:

api = RSS20(uri = 'http://feeds.feedburner.com/DavidJanesCode')
for item in api.IterItems():
    print "-", item['title']

Results:

- WORK - Web Object Records
- Syntax Error on Line 1
- Adding MapField to inputEx
- Switching between mapping APIs and universal zoom levels
- How to dynamically load map APIs
- How to use the Google Maps API
- How to use the Microsoft Virtual Earth API
- Tip - how to get your browser’s User Agent
- How to use the MapQuest API
- How to use the Yahoo Maps Service AJAX API
- How to detect internal link jumps
- GenX - first public demonstration
- Amazon’s OpenSearch: mostly useless
- More style updates
- How to do multi-column multilingual full text searching in Oracle
- Tip - fixing broken menus over form on IE6 and IE7
- New style for this weblog
- AUMFP - Demo
- Tip - use mod_rewrite to redirect to subdirectory
- AUMFP - The Almost Universal Microformats Parser

Amazon ECS

This will probably end up replacing PyECS!

Class definition:

class AmazonECS(API):
    _base_query = {
        "Sort" : "relevancerank",
        "Operation" : "ItemSearch",
        "Version" : "2008-08-19",
        "ResponseGroup" : [ "Small", ],
    }
    _uri_base = "http://ecs.amazonaws.com/onca/xml"
    _meta_path = "Items.Request"
    _item_path = "Items.Item"
    _page_max_path = 'Items.TotalPages'
    _item_max_path = 'Items.TotalResults'
    _page_max = -1

    def __init__(self, **ad):
        API.__init__(self, **ad)

    def CustomizePageURI(self, page_index):
        if page_index == 1:
            return

        return  "%s=%s" % ( "ItemPage", page_index )

Using it:

api = AmazonECS(AWSAccessKeyId = os.environ["AWS_ECS_ACCESSKEYID"])
api.SearchOn(
    Keywords = "Larry Niven",
    SearchIndex = "Books",
    Condition = "New",
)
for item in api.IterItems():
    print "-", item['ItemAttributes.Title']

Results … note that this fetching many pages of results:

- Fleet of Worlds
- Juggler of Worlds
- Escape from Hell
- Inferno
- N-Space
- The Ringworld Engineers (Ringworld)
- The Draco Tavern
- Legacy of Heorot: Legacy of Heorot
- Footfall
- A WORLD OUT OF TIME (ORBIT BOOKS)
- The Burning City (Hardback)
- Protector
- Burning Tower
- Three Books of Known Space
- Ringworld Throne
- Tales of Known Space: The Universe of Larry Niven
- Scatterbrain
- Ringworld
- Lucifer's Hammer
... (continues) ...

Introduction

As technologists, we’re all familiar with REST – Representational State Transfer:

Representational state transfer (REST) is a style of software architecture for distributed hypermedia systems such as the World Wide Web. As such, it is not strictly a method for building what are sometimes called “web services.” The terms “representational state transfer” and “REST” were introduced in 2000 in the doctoral dissertation of Roy Fielding, one of the principal authors of the Hypertext Transfer Protocol (HTTP) specification.

REST talks about how we address and use information on the World Wide Web. I’d like to introduce the concept of WORK -  Web Object Records – which defines how we think about data being transmitted across the web.

WORK is not a descriptive standard – it is not telling you what to do, it’s describing what you are doing. The hope is that by having a delineated description of what we are doing, we can then write tools to cut through the babel of API standards being currently promulgated by a multitude of vendors; we can standardize the unstandarded.

Defintion

A WORK item:

• is conceptually a JSON-like dictionary, consisting of string keys and object values
• each value in the dictionary is a (usually-) shallow JSON-like object, that is:
  • a dictionary, list or basic value type
• the basic value types are Unicode strings, floating point numbers, integers and booleans
• the difference between strings and other basic value types is fuzzy (data encoded in XML, HTML form data)
• null/None is rarely explicitly sent, instead it is the absence of a value being defined
• the difference between a list of objects and a single object is fuzzy and fluid (XML children)
• the data model defined implicitly by “what you see” is as useful as formal definition elsewhere
• there are no cycles or explicit ways of cross referencing within a WORK item
• WORK items can – and often are – nested within another WORK item, but only one level deep

Benefits

Because we technologists inherently use a WORK model of data, it explains:

• why we prefer XML over CSV – because we like to store more that a single atomic value in a “cell”
• why we prefer JSON to XML – because we think about data as JSON-like WORK objects, not as nested text constructs
• why we don’t adopt RDF (in it’s variants) for transmitting data, implementing APIs and so forth – because we don’t think in graphs
• why we find it easier to work with web data in Python and Ruby than in Java – because those languages explicitly use the same model for storing data as we think about the data

Examples

Here are a few examples of how one can view common API / feed results as WORK items.

RSS feeds

RSS is defined by a two level WORK hierarchy. The first level is:

{
  "channel" : CHANNEL-WORK,
  "item" : [ ITEM-WORK, ITEM-WORK, ... ]
}

A ITEM-WORK looks like:

{
  "title" : STRING,
  "link" : STRING,
  "description" : STRING
}

If you look at at the XML for a RSS feed with only 1 ITEM, there’s no way to tell without reading the spec than ITEM repeats. This is what we mean by saying that the difference between a single object and a list is sometimes fuzzy.

White Pages API

The White Pages API is also a two level WORK hierarchy (this pattern is very very common). Here’s the first level, slightly more complicated than RSS due to the XML serialization:

{
 "meta" : META-WORK,
 "listings" : {
   "listing" : [ LISTING-WORK, LISTING-WORK, ... ]
 }
}

A LISTING-WORK looks like:

{
  "geodata" : OBJECT,
  "phonenumbers" : OBJECT,
  "business" : { "businessname" : "Fred's Pizza" },
  "address" : OBJECT
}

The OBJECTs above in the White Pages API are somewhat complicated, but tractable (as we shall see in another post)

Amazon AWS API

The Amazon Associates Web Service allows one to retrieve information about Amazon products via XML responses. The response is a little convoluted but still recognizable:

{
 "Items" : {
   "RequestHeader" : REQUEST-HEADER-WORK,
   "Item" : [ ITEM-WORK, ITEM-WORK, ... ]
 },
 "OperationRequest" : { ... }

The individual ITEM-WORK describe products:

{
 "ASIN" : STRING,
 "ImageSets": {
   "ImageSet": {
    "LargeImage": {
     "URL": "http://ecx.images-amazon.com/images/I/31e55zf53VL.jpg",
     "Width": "300",
     "Height": "300"
   },
  },
 "ItemAttributes": {
   "Title": "Under a Blood Red Sky - Deluxe Edition CD/DVD",
   "Manufacturer": "Island",
   "ProductGroup": "Music",
   "Artist": "U2"
 }
}

Google search result

We can also look at HTML pages as if they’re returning data as WORK items. This could be explicit if rules such as microformats or RDFa were used,  or once again it could be just a convenient way of modeling the data. Here’s a hypothetical WORK item for a single result returned from a Google:

{
 "title" : "Bombardier Inc. - Bombardier - Home",
 "url " : "http://www.bombardier.com/",
 "description" : "Manufacturers of a large range of regional...",
 "links" : [
  {
   "title" : "Careers",
   "url" : "...",
  },
  {
   "title" : "Business Aircraft",
   "url" : "...",
  },
  ...
 ]
}

Conclusion

WORK gives us a powerful way of looking at – at simplifying – data that’s retrieved over the Internet via REST calls. If we can view API results as being made up of standardized components – WORK items – then the amount of work we need to do to work with new APIs can be absolutely minimized.

Designing and writing some of these tools is my next task.

Here’s a nasty little error that I tracked down on IE6 this morning. The error message (click on the icon in the lower left of the browser) is something like Error: Syntax Error; Code 0; Line: 1.

Of course, when you go to look for the error in your Javascript, nothing seems to be wrong.  One explanation I found for this was that a BODY onload tag is broken. In our case however this wasn’t the issue.

Instead it was really something quite simple: we were using a SCRIPT tag to download a Javascript file that no longer existed and was 302 redirecting to an HTML page! The IE6 Javascript engine then attempted to execute the HTML and of course immediately failed. In our case, this happened because we were getting a partner widget and they had changed the URL without notifying us. A 404 error would have been more appropriate, though I have not tested to see whether the same error would then occur.

One further note: if you’re debugging scripts on IE6:

• download this Microsoft tool
• enable it (Tools > Internet Options | Advanced; uncheck Disable Script Debugging)
• restart IE

InputEx is a neat Javascript library by Eric Abouaf and other contributors to dynamically create forms. It’s best shown by example, so:

• detailed examples of how to create all individual fields
• a form builder in Javascript (click on the pencil to close popups or try my version – see the notes below)

I’m working on a project called GenX that allows people to dynamically lookup, edit and create data in editors (such as in WordPress). One of the features I want is the ability to add maps so after a week of research, I’ve created a MapField for inputEx:

• see the MapField example

Notes

• this is just a preliminary version; there’s probably some code style and idiom issues that need to be worked out
• I’m going to ask the InputEx group to pick this up; I reserve no rights
• it’s not fully integrated into the Builder example yet
• it works with Google Maps, Yahoo Maps and Microsoft’s Virtual Earth. You will need a key for your domain for Google Maps, see the example above about how to insert your key
• it introduces a concept of a Globals for MapField, as there are certain settings (i.e. your mapping keys) that really belong at “class” level rather than instantiated object level
• Mapping APIs are dynamically loaded
• Zoom levels are returned in native format and “universal format“
• I’ve done all my testing of FF3, so there may be browser issues I need to look at

Code changes

• the only files needed are
  • examples/map_field.html
  • js/fields/MapField.js

Where this is going

• I’ve spent the last week working on maps; I need to get back to using this code rather than improving and documenting it
• I want to add a geocoder
• I want the geocoder to be able to be sensitive to other form fields, so for example if you’re entering an address it can actually use those as an input

One final mod to inputEx

I’ve modified the builder to make the pencil icon more obviously become the close button when you’ve opened a subpanel. You can see the demo here. Here’s the change set:

In images:

curl --location http://www.famfamfam.com/lab/icons/silk/icons/pencil_delete.png \
 > pencil_delete.png

In inputex/inputExBuilder/css/inputExBuilder.css add:

div.inputEx-TypeField-EditButton.opened {
   background-image: url(../../images/pencil_delete.png);
}

In inputex/js/fields/TypeField.js replace onTogglePropertiesPanel with:

   onTogglePropertiesPanel: function() {
      if (this.propertyPanel.style.display == 'none') {
         this.propertyPanel.style.display = '';
         YAHOO.util.Dom.addClass(this.button, "opened");
      } else {
         this.propertyPanel.style.display = 'none';
         YAHOO.util.Dom.removeClass(this.button, "opened");
      }
   },

Did you know that Google Maps, Yahoo Maps and Virtual Earth all use the same map tile resolutions? That is, you can actually seamlessly switch between mapping systems and have everything line up exactly the same way. If you’d like to play with this, look at our map examples:

• Google Maps example
• Virtual Earth example
• Yahoo Maps example (change the zoom level to 5)

But it’s even easier to show this using a graphic (thank you to Alistair Morton of Peapod Studios for compositing):

Table of map zoom level equivalents

Google Maps and Virtual Earth the same zoom levels, excepting that Google has a level 0. Bigger numbers are closer to the ground, smaller numbers are up into space. Yahoo Maps has fewer levels to work with – it doesn’t let you drill down below the block level. Yahoo numbers its zoom levels the opposite of Google and Virtual Earth, small numbers being closer to the ground and so on and so forth.

To allow seamless switching of data between APIs, I propose here universal zoom levels, which can be used as-is on any of these mapping APIs. To avoid confusion of which type of zoom level one is using, we use the negative number range for universal zoom levels. As Google and Microsoft are using the same zoom levels, I’ve decided that the universal zoom levels should simply be the inverse of their zoom levels, bounded by (-1, -19) inclusive.

Conversion Formulae

Here’s Javascript to convert between your favorite mapping API’s zoom levels and universal zoom levels. Note that all to_native functions accept positive numbers as-is.

zoom = {
  google : {
    to_univeral : function(native) {
      return  Math.min(-1, -native);
    },  

    to_native : function(universal) {
      return  universal > 0 ? universal : -universal;
    }
  },

  virtual_earth : {
    to_univeral : function(native) {
      return  Math.min(-1, -native);
    },  

    to_native : function(universal) {
      return  universal > 0 ? universal : -universal;
    }
  },

  yahoo : {
    to_univeral : function(native) {
      return  Math.min(-1, native - 18);
    },  

    to_native : function(universal) {
      return  universal > 0 ? universal : Math.max(universal + 18, 1);
    }
  }
};

End Notes

A truly universal zoom level would just use altitude; dealing with a simple integer is easier and less complication prone though. MapQuest is not included in this post as its tiles do not line up with the other APIs.

When using various map APIs it’s often useful for performance reasons not to actually load the API until we’re sure we’re going to need it. This is called dynamic loading or lazy loading. The post explains how to do this on the three major mapping APIs.

Note about the examples

These have been excerpted from a project I am currently working on, and so don’t work as-is. The basic calling structure (not shown) is as follows:

• assume each function is being called from CALLER.
• If true is returned we allow the dynamic loader to do it’s work; when that is complete it calls CALLER.CREATE_MAP.
• If true isn’t returned, we assume everything is good to go and the caller directly calls CALLER.CREATE_MAP – this will typically happen when the Map API is detected in Javascript!

There’s also a global context in GLOBALS used for keeping track of state. In particular, we assume the user may be trying to create multiple maps so we have to make sure all the CREATE_MAP callbacks are triggered once all the Javascript has been loaded!

In the Google Maps and Virtual Earth we see the Javascript-with-callback pattern:

• a callback function is passed in the URL as a parameter
• the downloaded Javascript document is dynamically modified so that it calls that function. Hence, you know the Javascript has been loaded.

If you’re a Javascript noob, scripts are dynamically loaded into your HTML document by creating a SCRIPT element and attaching it to the document HEAD.

Google Maps API

Google Maps API uses a multi-phase loading scheme:

• get the generic loader, giving it a callback function to invoke when it’s completely loaded
• when that’s loaded (into google)…
• call the loader to get the maps API
• when that’s loaded, create the map

This feature is documented here.

google_preload : function(CALLER) {
    if (window.GMap2) {
        return;
    }

    var api_key = YOUR_GOOGLE_API_KEY_FOR_YOUR_DOMAIN;

    var preloader = 'MapGooglePreloader_' + GLOBALS.MapFieldsNumber;
    GLOBALS[preloader] = function() {
        google.load("maps", "2", {
            "callback" : function() {
                CALLER.CREATE_MAP();
            }
        });
    }

    if (window.google) {
        GLOBALS[preloader]();
    } else {
        var script = document.createElement("script");
        script.src = "http://www.google.com/jsapi?key=" + api_key +
        "&callback=GLOBALS." + preloader;
        script.type = "text/javascript";

        document.getElementsByTagName("head")[0].appendChild(script);
    }

    return  true;
}

Microsoft Virtual Earth

The Virtual Earth API is easy to dynamically load also. As with Google this feature is directly supported through a Javascript callback mechanism; unlike Google it’s essentially a single phase design.

One issue I ran into was the error message p_elSource.attachEvent is not a function error. Although a Google search for a solution mostly turns up a blank, a tip of the hat to Soul Solutions in Australia for finding a work-around (and have similarly solved dynamic API loading at the link).

virtualearth_preload : function(CALLER) {
    if (window.VEMap) {
        return;
    }

    var preloader = 'MapVEPreloader_' + GLOBALS.MapFieldsNumber;
    GLOBALS[preloader] = function() {
        CALLER.CREATE_MAP();
    }

    if (!window.attachEvent) {
        var script = document.createElement("script");
        script.src = "http://dev.virtualearth.net/mapcontrol/v6.2/js/atlascompat.js";
        script.type = "text/javascript";

        document.getElementsByTagName("head")[0].appendChild(script);
    }

    var script = document.createElement("script");
    script.src = "http://dev.virtualearth.net/mapcontrol/mapcontrol.ashx?v=6.2" +
      "&onScriptLoad=GLOBALS." + preloader;
    script.type = "text/javascript";

    document.getElementsByTagName("head")[0].appendChild(script);

    return  true;
}

Yahoo AJAX Map API

The Yahoo AJAX Map API does not directly support dynamic API loading. However, just loading the Javascript and running a timer that waits to see whether the script has loaded seems to do the trick. If you’re new-ish to Javascript, remember that it’s not multi-threaded so we can always be sure that when a Javascript file is loaded, it is completely loaded (excepting errors, of course).

yahoo_preload : function(CALLER) {
    if (window.YMap) {
        return;
    }

    var preloader = 'MapYahooPreloader_' + GLOBALS.MapFieldsNumber;
    if (!GLOBALS[preloader]) {
        GLOBALS[preloader] = 1;

        window.YMAPPID = YOUR_YAHOO_API_KEY;

        var script = document.createElement("script");
        script.src = "http://us.js2.yimg.com/us.js.yimg.com/lib/map/js/api/ymapapi_3_8_0_7.js";
        script.type = "text/javascript";

        document.getElementsByTagName("head")[0].appendChild(script);
    }

    window.setTimeout(function() {
        if (window.YMap) {
            CALLER.CREATE_MAP();
        } else {
            CALLER.yahoo_preload(CALLER);
        }
    }, 0.1);

    return  true;
}

The Google Maps API is the great granddaddy of mapping API. Originally, the GMap API was an internal private Google API until reverse engineered by Adrian Holovaty to create the first great mapping mashup, Chicago Crime (now folded into Every Block). The API is well documented, well understood and easy-to-use but suffers from the look getting a little long in the tooth and the API key design imposing a higher administrative burden than the competing packages.

The way we set up the events in Google Maps is a little different than we did in MapQuest, Yahoo Maps and Visual Earth. GMaps uses a two phase creation scheme, using one call to create the map object and a second call to actually display it somewhere. I like this and it would have got around the “jumping map” issue we saw in Virtual Earth. Because we want to capture the load event – to get the initial state of the map – we have to put it between these two calls otherwise we’ll miss the load event (it would be nice if Google just called your “onload” callback if the map was already loaded).

My least favorite part of the GMap API is that you need a per-domain API key. This is the most restrictive licensing of any of the mapping packages. Amongst the problems this will cause you:

• if you’re working in development, staging, deployment environments you’ll have to drive your map from dynamic HTML, as you’ll need to switch in the correct key
• http://user.example.com, it will mean that all your users will have to individually sign up for API keys — e.g. uptake will be slow. update: see the comments

The documentation for the GMap API is by far the best of any of the mapping package. Everything is cleanly laid out and there’s no question how you get from concepts, to examples, to the API spec.

Here’s our example map application and the source below. You’ll have to get your own API key and substitute it in if you want to start using this example. Geocoding is almost as easy as in the Yahoo API, with the caveat that you really want to set the baseCountryCode, as it won’t make a guess for the country on your behalf. Unlike all the other applications, the GMap API will not take strings where it expects numbers, hence all the parseFloats in the example below.

<html>
<head><script
    type="text/javascript"
    src="http://maps.google.com/maps?file=api&amp;v=2&amp;key=YOURKEY"></script>
<style type="text/css">
#id_map {
    height: 75%;
    width: 100%;
}
</style>
</head>
<body>
<div id="id_map"></div>
<p>
Lat: <input type="text" id="id_lat" onchange="map_js.onupdate_ll()" />
Lon: <input type="text" id="id_lon" onchange="map_js.onupdate_ll()" />
Zoom: <input type="text" id="id_zoom" onchange="map_js.onupdate_zoom()" />
</p>
<script type="text/javascript">
map_js = {
    e_lat : null,
    e_lon : null,
    e_map : null,
    g_map : null,
    g_geocoder : null,

    create : function() {
        map_js.e_map = document.getElementById("id_map");
        map_js.e_lat = document.getElementById("id_lat");
        map_js.e_lon = document.getElementById("id_lon");
        map_js.e_zoom = document.getElementById("id_zoom");

        map_js.g_map = new GMap2(map_js.e_map);
        map_js.g_geocoder = new GClientGeocoder();
        map_js.g_geocoder.setBaseCountryCode("ca")

        GEvent.addListener(map_js.g_map, "load", map_js.onposition);
        GEvent.addListener(map_js.g_map, "moveend", map_js.onposition);
        GEvent.addListener(map_js.g_map, "zoomend", map_js.onposition);

        map_js.g_map.addControl(new GSmallMapControl());
        map_js.g_map.addControl(new GMapTypeControl());

        map_js.g_geocoder.getLatLng("Toronto, ON", map_js.ongeocoded);
        // map_js.g_map.setCenter(new GLatLng(43.648565, -79.385329), 13);
    },

    ongeocoded : function(point) {
        if (!point) {
            alert("not found");
        } else {
            map_js.g_map.setCenter(point, 13);
        }
    },

    onposition : function() {
        var c = map_js.g_map.getCenter();
        map_js.e_lat.value = c.lat();
        map_js.e_lon.value = c.lng();

        var z = map_js.g_map.getZoom();
        map_js.e_zoom.value = z;
    },

    onupdate_ll : function() {
        map_js.g_map.panTo(
            new GLatLng(parseFloat(map_js.e_lat.value), parseFloat(map_js.e_lon.value)));
    },

    onupdate_zoom : function() {
        map_js.g_map.setZoom(parseInt(map_js.e_zoom.value));
    },

    end : 0
};
map_js.create();
</script>

Next up in our tour of Javascript mapping APIs is Microsoft’s Virtual Earth. VE is a full-featured mapping API with a visually appealing look, but with a number of gotchas what watchfors which I’ll list here.

VE is rather picky about the CSS and styles it sees (not nearly so much as MapQuest though!). In particular:

• make sure you add a position: relative or position: absolute to the CSS definitions for the map’s DIV, otherwise it will go crazy on at least Firefox. I added position: relative and it worked fine
• contrary to the documentation, VE wants to see the style rules directly on the element rather than inherited from CSS, otherwise it seems to default to the 600px by 400px default. Fortunately, once you’ve done this it does understand width: 100% and similar.

Unlike MapQuest and Yahoo Maps, VE comes with a rather comprehensive map control installed. The downside of this is that if you want a simpler interface you’re going to have to code it yourself. In particular, there’s an option for a “Bird’s Eye” view of the map you’re looking at. Once you click on this, concepts like the map having a latitude and longitude go out the window (for reasons I don’t grasp). You can see in the example below that we actually test to see if Lat/Lon is available before attempting to display it. Also note that the documentation for VEMap.GetCenter is wrong — a null is not returned; instead Latitude and Longitude are null.

Microsoft really needs to adopt a more modern view of Javascript. One obvious thing would be to start using namespaces to isolate code — use VE.Map rather than VEMap. The VEMap constructor doesn’t accept a (DOM) element as an argument, you have to pass the ID. Another ugliness is the use of lengthy optional argument lists, such as this atrocity:

VEMap.Find(what, where, findType, shapeLayer, startIndex,
  numberOfResults, showResults, createResults, useDefaultDisambiguation,
  setBestMapView, callback);

This was understandable in the Win32 C-code days but not so much when passing a dictionary would accomplish much of the same in a simpler manner.

VE includes a rather straightforward geocoding call – despite what the documentation above implies. Just do:

map_js.ve_map.Find(null, "Toronto, ON");

Unfortunately, this has a two shortcomings:

• you cannot “Geocode at setup time” like you can in Yahoo Maps, so you have to display one location, then jump to the one you really want
• the Geocoder changes you’re zoom level, whether you want it to or not. From a mapping point-of-view I could see how this makes sense (e.g. show all of Toronto), but it would be nice if it was optional

We’ve got the geocoder in the example below commented out, but you may find it useful. Just remember that it’s asynchronous, so the actual map jump will not happen until after your current JS terminates.

Finally, to end on high note, VE doesn’t appear to need a developer or application key meaning you can just take this example to modify and run as you please. Here’s our example map application and the source below:

<html>
<head>
<script charset="UTF-8" type="text/javascript"
  src="http://dev.virtualearth.net/mapcontrol/mapcontrol.ashx?v=6.2&mkt=en-us"></script>
<style type="text/css">
#id_map{
position: relative;
}
</style>
</head>
<body>
<div id="id_map" style="width: 100%; height: 75%;">&nbsp;</div>
<br />
<p>
Lat: <input type="text" id="id_lat" onchange="map_js.onupdate_ll()" />
Lon: <input type="text" id="id_lon" onchange="map_js.onupdate_ll()" />
Zoom: <input type="text" id="id_zoom" onchange="map_js.onupdate_zoom()" />

</p>
<script type="text/javascript">
map_js = {
	e_lat : null,
	e_lon : null,
	e_map : null,
	ve_map : null,

	create : function() {
		map_js.e_lat = document.getElementById("id_lat");
		map_js.e_lon = document.getElementById("id_lon");
		map_js.e_zoom = document.getElementById("id_zoom");

		map_js.ve_map = new VEMap("id_map");
		map_js.ve_map.LoadMap(new VELatLong(43.648565, -79.385329), 13,
		  VEMapStyle.Road, false, VEMapMode.Mode2D, true, 1);
		// map_js.ve_map.Find(null, "Toronto, ON");

		map_js.ve_map.AttachEvent("onendzoom", map_js.onposition);
		map_js.ve_map.AttachEvent("onendpan", map_js.onposition); 

		map_js.onposition();
	},

	onposition : function() {
		var c = map_js.ve_map.GetCenter();
		if (!c || !c.Latitude) {
			return;
		}

		map_js.e_lat.value = c.Latitude;
		map_js.e_lon.value = c.Longitude;

		var z = map_js.ve_map.GetZoomLevel();
		map_js.e_zoom.value = z;
	},

	onupdate_ll : function() {
		map_js.ve_map.PanToLatLong(new VELatLong(map_js.e_lat.value, map_js.e_lon.value));
	},

	onupdate_zoom : function() {
		map_js.ve_map.SetZoomLevel(map_js.e_zoom.value);
	},

	end : 0
};
map_js.create();
</script>
</body>
</html>

Every browser normally sends a User-Agent header in its HTTP request. You can use this header to determine what browser and version a website is using, i.e. Internet Explorer, Firefox, whether it’s a mobile browser, an iPhone, etc..

So, what’s your browser’s User-Agent? Just type this in the address bar:

javascript:alert(navigator.userAgent)

Source