David Janes’ Code Weblog

The Boy Genius wrote a blog post on the state of RIM BlackBerry application development this morning that’s getting more than a little traction:

One word is where RIM fails so miserably it isn’t even imaginable: software.

You have to look at the big picture here… for what RIM is working with (an incredibly miserable Java OS with so much security and encryption and smoke-blowing APIs) they’ve hit the jackpot. Their OS architecture is fantastic, their use of security is what makes them so trustworthy. But, as each handset release comes closer and closer, people start to see the bigger picture. And that’s the fact that RIM’s OS is more than antiquated, it’s borderline laughable. But it works, you’re thinking, so what’s wrong? I’ve been saying this for years, but it wasn’t designed to do anything the BlackBerry does now. Imagine scotch taping car parts to a 200hp engine and see how far that gets you. Obviously, it’s just a viciously rough metaphor, but we believe a correct one.

There’s so many limitations to RIM’s OS, and even RIM’s data network that it offsets all the wonderful things they’ve managed to accomplish. Remember when people were so excited over leaked shots of OS 4.6 and I said somewhere it was just a theme? Well, was I wrong? Oh, look! OS 5.0! What changed? 99% nothing. Some functionality is added here and there, but the mobile phone landscape has changed so drastically in the last two years, that RIM, admittedly known to planning “three years out” looks to be unable to see the proper direction to head.

You can throw $1,000,000,000 at developers but you won’t get any if your OS, tools, and documentation are so bad, and that’s really in the end a lot of what I’m getting at. I was laying in bed at around 3AM early one morning recently, looking through the iPhone App Store and I came across EA’s Tiger Woods Golf. $4.99, why not? Wait, it’s 150MB? Wow, it must be good. I clicked purchase and literally 4 minutes later, Tiger Woods was installed and up on my screen. Granted I was on a high-speed Wi-Fi connection, but it made me realize more than ever that RIM has the most uphill battle of their lifetimes. When a BlackBerry application over 500k is considered “large”, something’s wrong. When TweetGenius is one of the first BlackBerry applications to do fun, unique things like transparent overlays, consistent shortcuts, and a straight forward UI, something is wrong.

The reason why this is so frustrating to me and I’m guessing many is because RIM literally almost has it all. They’ve got it! They are 90% there but that last 10% has become the most important. If you take Apple for example, and see their shortcomings, and then what they’ve done to fix them, it’s remarkable. It’s a completely different DNA than RIM’s but it’s working. In two years Apple has practically matched Research In Motion in almost every consumer area while having the most advanced mobile operating system with the most advanced mobile SDK on the planet. If Apple can do this in just two years and RIM has stood still, no one thinks that’s a problem?

Larry Dignan of ZDNet chimes in (I suspect he was going to the iPhone anyway):

The BlackBerry operating system issue struck home for me while on vacation. My Storm was my only ramp to the Web. Under stress and heavy usage the BlackBerry OS was clearly struggling. You could almost feel it choke when switching between apps, browsing and handling basic tasks. The interface was fine, but under the hood something is off.

Will these OS limitations affect my next phone choice when my Verizon Wireless contract is up? Possibly. The device is only 25 percent of the smartphone game these days. The operating system is everything. Apple gets it. Palm gets it. Google gets it. I’m not sure that Microsoft gets it. For RIM, it remains to be seen if the company gets OS religion.

For my next phone, I’ll be buying and OS instead of a device. If the iPhone comes to Verizon Wireless it’s most likely a no-brainer for me. I have more than a year to see if Palm’s Pre and WebOS is the real deal. Even the Motorola Android devices may hold promise for me. My last phone choice was basically an escape from Windows Mobile. If RIM doesn’t get its OS strategy together I may ditch the BlackBerry too. All I really need is to browse and tether to my laptop as a wireless card on a good network.

The real power behind the RIM brand is all the relationships they’ve built with communication providers and lock-in to the IT departments in business and government around the world. That said, there is a medium term danger that RIM is going to be undermined from the bottom by consumer-friendly devices such as the iPhone, in the same way that the PC undermined big iron’s lock on the enterprise market in the 80’s and 90’s.

I’m actively developing for the BlackBerry right now and I have to say it’s a struggle:

• setup of the development environment is non-obvious and fragmented
• it requires Windows; the cool kids are using Linux & Mac
• documentation is wildly incomplete and basically incorrect, since there’s so many references to obsolete technology
• how to transfer “normal” programming skills into the RIM environment is entirely non-obvious; it’s difficult even to figure out whether one can talk about there being a filesystem on the device
• Java ME is garbage. Yes, I went there.

My current application approach is to create an entirely self contained browser application and bypass almost all RIM UI features. We should know soon whether this is a sane approach (I may open source this code if there’s interest).

Update: Alec Saunders of Calliflower adds comments about Nokia.

Alec Saunders has a seriously cool article on his blog about how your iPhone knows where it is. I got to experience how well this works yesterday when we used TimmyMe in the McCowan Road/Ellesmere Avenue neighborhood to grab a coffee pre-meeting:

Many folks assume that iPhone’s location services are GPS based. In fact, they’re not. [...]

Skyhook’s innovation is to augment that with WiFi hotspot locations. Why WiFi? It can be wonderfully accurate, to within 30 to 60 feet, or 10 to 20m which is roughly the same as GPS. WiFi can establish a fix within less than a second. And, WiFi is not vulnerable to overhead obstructions, so it can be used indoors.

Skyhook’s technology relies on a database of WiFi access points in over 2,000 cities (and growing). Much like Google, they use vehicles to drive cities, and using a laptop in the vehicle, they chart the location of WiFi access points. In addition, on iPhone as users use the mapping application, new locations and newly discovered WiFi access points are automatically added to the database. It’s a kind of automated crowdsourcing, based on usage.

My emphasis on that last sentence. Very cool.

OK, it’s time to officially unveil my new company: Discover Anywhere Mobile. We’re creating mobile guides for the tourism and hospitality sector, and similar products for advanced store locators, conference guides, etc.. Platform-wise, we’re creating iPhone & BlackBerry applications, and then a mobile web browser version to catch other devices. Our belief is that over the next three years this is going to be the way tourists and travelers consume information - they want everything in the palm of their hands. Why would you want to carry a tour guide book with you anymore?

Note, and this is important, that we are not a content company. Our target clients are Destination Marketing Organizations (DMOs), festival & event organizers, large corporations, and so forth. We will write apps on their behalf using their data and the app will be released under their names. Discover Anywhere Mobile will handle all the bolts and nuts of making this happen: importing & managing data, creating & customizing apps, updating user’s apps, look & feel, etc.. DMOs provide the marketing and data, we provide the infrastucture and services to make it all happen.

Here’s the start of our official blurb:

Discovery Anywhere Mobile creates customizable mobile travel guide applications for Destination Management Organizations. Discover Anywhere Mobile can quickly and cost effectively compile your DMO’s event and listing information into mobile applications for the iPhone, BlackBerry and mobile web browsers. Applications created with Discover Anywhere Mobile’s services will meet traveler’s need for concise, relevant, timely destination information at the exact time and place they need it most – at your destination.

Go to our website and read more, and if you have your own site, please provide a link!

Sorry for the lack of posting - I’m busy applying my ideas rather than coming up with new ones ;-)

We’ll be holding the next Coffee & iPhone tomorrow between 4 and 6p again at the Dark Horse Cafe, 215 Spadina Avenue for Coffee and iPhone.

• What: Coffee & iPhone
• When: Thursday, April 23 from 4 - 6p
• Where: Dark Horse Cafe, 215 Spadina Avenue (at Sullivan Street)

Join myself, Blake, Wayne (and more!) tomorrow between 4 and 6p at the Dark Horse Cafe, 215 Spadina Avenue (just north of Queen) for Coffee and iPhone. Bring your MacBook, your iPod, your iPod Touch or just yourself and we’ll talk all things iPhone and mobile.

• What: Coffee & iPhone
• When: Thursday, April 16 from 4 - 6p
• Where: Dark Horse Cafe, 215 Spadina Avenue (at Sullivan Street)

At the last Coffee & iPhone we had the following attendees:

• David Janes @dpjanes
• Jason Moore @xinsight
• wayne a. lee (bunnyhero) @bunnyhero
• Lin Zhou @remixnine
• Darren Baptiste @darrenbaptiste
• Blake Winton @bwinton

Note that this is the day before Joey’s Coffee & Code, at the same location!

Join myself, Blake, Wayne and more tomorrow between 1 and 3pm at the Lettieri Cafe at 581 Bloor West (at the corner with Bathurst) for “Coffee and iPhone”. Bring your MacBook and we’ll talk all things iPhone development.

View Larger Map

This code could be very useful for iPhone developers:

The name of the new project is Three20, after the 320-pixel-wide screen of the iPhone. The code is all hosted on github for your cloning pleasure. There is an excellent sample app called TTCatalog which lets you play with all of the various UI components. Documentation? Well… there are instructions for how to add Three20 to your project, but I am still working on comprehensive documentation for each of the classes. For now, the sample app and the code itself are your documentation.

The projects are:

• Photo viewer
• Message composer
• Web image views
• Internet-aware table view
• Better text fields (including type-ahead)
• HTTP disk cache
• URL-based navigation (this could be interesting)

The source base is under the Apache license.

I always seem to forget this:

$ mysql -u root -p
Enter password: enter mysql-root-user passwd
mysql> CREATE DATABASE DATABASENAME
mysql> GRANT ALL PRIVILEGES ON DBNAME.* to "DBUSER"@"localhost" IDENTIFIED BY "DBPASSWORD";
mysql> FLUSH PRIVILEGES;
mysql> EXIT
$

If you have not created a XIB, do the following:

• select Resources in the Interface Building
• press ⌘N
• select User Interfaces in iPhone OS
• select View XIB … and create in the normal way

To create the Simple Table in the Interface Builder

• open the XIB in Resources
• drag the Table View from the Library to the View … it should resize to the full extent of the View
• select the Table View object and press ⌘2
• from the Outlets section, drag
  • dataSource to File’s Owner
  • delegate to File’s Owner
• save

Make your Controller.h a data source and delegate for the table:

@interface ThemesController : UIViewController
<UITableViewDelegate, UITableViewDataSource>
{
}

Add the following basic code to the Controller.m file:

#pragma mark -
#pragma mark Table View Data Source Methods

- (NSInteger) tableView:(UITableView*) tableView numberOfRowsInSection:(NSInteger) section
{
  return  ... the number of rows ...;
}

- (UITableViewCell*) tableView:(UITableView*) tableView
  cellForRowAtIndexPath:(NSIndexPath*) indexPath
{
  static NSString *SimpleTableIdentifier = @"SimpleTableIdentifier";

  UITableViewCell* cell = [tableView dequeueReusableCellWithIdentifier:SimpleTableIdentifier];
  if (cell == nil) {
    cell = [[[UITableViewCell alloc] initWithFrame:CGRectZero
      reuseIdentifier: SimpleTableIdentifier] autorelease];
  } 

  NSUInteger row = [indexPath row];
  cell.text = ... an NSString containing what to display for row row ...;
  // cell.image = [UIImage imageNamed:@"star.png"];

  return  cell;
}

#pragma mark -
#pragma mark Table Delegate Methods

- (void) tableView:(UITableView*)tableView didSelectRowAtIndexPath:(NSIndexPath*)indexPath
{
}

Compile and run.

Property Lists - plists - are a standard way of storing data in iPhone and MacOS programs. Here’s how you use them on the iPhone.

Add this code to Controller.m to populate member listData with the data from plist dwarves.plist:

- (void) viewDidLoad
{
	NSBundle* bundle = [NSBundle mainBundle];
	NSString* plistPath = [bundle pathForResource:@"dwarves" ofType:@"plist"];

	NSArray* dwarves = [[NSArray alloc] initWithContentsOfFile:plistPath];
	self.listData = dwarves;
	[dwarves release];	

	[super viewDidLoad];
}

To create the plist:

• open and select the Resources folder in Interface Builder
• right-click and select Add > New File…
• select Other in the popup (at the bottom)
• select the Property List icon and then press the Next button
• choose your name and Finish

To edit the plist:

• select it in the Resources Folder
• right-click and select Open with Finder

Make sure:

• the name of the plist in the code (dwarves) is the same as the plist you created
• the type of the plist in the code (NSArray) is the same as the plist you created - by default it is actually NSDictionary, which is the other common option

Geographic location is a little more problematic to encode than other data types we’ve looked at for the Almost Universal API. The issue is that standards for adding geographic information are either overly complicated or they encode information in a hard-to-use-from-JSON manner.

Using GeoRSS

GeoRSS provides two ways (well, way more I’m sure) of adding geographic point information to a blog post (which is essentially the same as AUAPI item). From the webpage:

<georss:point>45.256 -71.92</georss:point>

which would be encoded as:

{
 "georss:point" : "45.256 -71.92"
}

and

<georss:where>
   <gml:Point>
      <gml:pos>45.256 -71.92</gml:pos>
   </gml:Point>
</georss:where>

which would be encoded as:

{
 "georss:where" : {
  "gml:Point" : {
   "gml:pos" : "45.256 -71.92",
  }
 }
}

Neither of these look particularly satisfying because the lat/lon is encoded as a string. We could encode as an array of numbers, but then we have to write a custom transcriber for converting to XML.

Using hCard

Another option would be to use the geo.latitude, geo.longitude attributes in hCard. Unfortunately, hCard requires elements such as FN and in general imply that we are talking about a person or organization, so this isn’t really satisfying either.

How AUAPI recommends geographic information should be encoded

'geo': {
   'latitude': 34.743763,
   'longitude': -86.572568
}

This in fact, is exactly the microformat’s geo standard.

Thanks to Kevin Marks, I’ve just stumble onto the Portable Contacts effort. They seem to have thought quite a bit about the hCard/vCard serialization issue. I’ve added my two bits, in particular making the claim that they should consider making there proposal more compatible with existing consumers and infrastructure by piggybacking on top of the Atom syndication format, rather than requiring PC consumers to redevelop all this infrastructure around their format. This, in essence, is what the Almost Universal API idea is about.

---

I’ve been thinking and writing recently about how APIs can be made compatible with each other - that is, the same consumers can be used with the results from different APIs - and Kevin Marks pointed me this way, as I had written about hCard serialization.

However, it occurs to me that PC could be made to overlay the Atom syndication standard with very few changes (by the looks of it, you’re fairly late into the design cycle though). For example, this:

{
  "id": "703887",
  "displayName": "Mork Hashimoto",
  "name": {
    "familyName": "Hashimoto",
    "givenName": "Mork"
  },
  "birthday": "0000-01-16",
  "gender": "male",
  "drinker": "heavily",
  "tags": [
    "plaxo guy",
    "favorite"
  ],
}

Could be encoded like:

{
  "id": "703887",
  "title": "Mork Hashimoto",
  "updated" : "2008-...",
  "published" : "2007-...",
  "category" : [
   { "term" : "plaxo guy", },
   { "term" : "favorite", },
  ],
  "contact" : {
    "name": {
      "familyName": "Hashimoto",
      "givenName": "Mork"
    },
    "birthday": "0000-01-16",
    "gender": "male",
    "drinker": "heavily",
  }
}

Which is almost the same, except now that your XML serialization can be Atom (noting of course there will be changes for the paging elements, etc. that you’ve defined). The immediate implication of this is that you’re working within a large existing infrastructure that knows about update notification, has tools for display, and so forth.

MediaRSS is the way to encode images in the Almost Universal API (AUAPI). Because MediaRSS encodes its values in attributes, we use the @ symbol to prefix keys.

Example 1 - a single image

'media:content': {
    '@medium': 'image',
    '@type': 'image/jpeg',
    '@url': u'http://farm3.static.flickr.com/2165/2271778128_b59c01a695_m.jpg'
}

Example 2 - thumbnail

'media:thumbnail': {
    '@url': u'http://farm3.static.flickr.com/2165/2271778128_b59c01a695_t.jpg'
}

Example 3 - multiple images in different sizes

'media:group': {
    'media:content': [
        {'@medium': 'image',
        '@type': 'image/jpeg',
        '@url': u'http://farm3.static.flickr.com/2165/2271778128_b59c01a695_m.jpg'},
        {'@medium': 'image',
        '@type': 'image/jpeg',
        '@url': u'http://farm3.static.flickr.com/2165/2271778128_b59c01a695_s.jpg'},
        {'@medium': 'image',
        '@type': 'image/jpeg',
        '@url': u'http://farm3.static.flickr.com/2165/2271778128_b59c01a695_b.jpg'}
    ],
    'media:thumbnail': {
        '@url': u'http://farm3.static.flickr.com/2165/2271778128_b59c01a695_t.jpg'
    }
}

The best model for describing people is the vCard standard, RFC 2425 and RFC 2426. The microformats community has adapted the vCard standard for serialization into HTML using hCard. In the Almost Universal API (AUAPI), people and organizations should almost always be described using a JSON-encoded hCard.

It is difficult to describe, without going into great minutiae, what the difficulties are in transforming the hCard and vCard standards into a pleasant looking and more importantly an easy-to-use hierarchy: there are certainly a number of edge cases that one would have to deal with it! There’s certainly an argument for just encoding hCard/vCards as a straight vCard serialization - at least in terms of simplicity of encoding. The issue is that the end consumer (which I believe should be the strongest focus) really has to do the dirty work in grouping everything together themselves.

Algorithm

This algorithm is destructive to the data structure it works upon, so generally you’ll be make a copy first.

• note that though we reference to all upper, mixed case, camel case and so forth hCard attributes, all attributes are actually physically encoded in lower case with “-” separators
• let the “groupers” be ADR, GEO, N, ORG, TEL. Groupers group together attributes that are related (such as FirstName and LastName)
• let the “narrowers” be Home, Work, Parcel, Postal (and no-narrower). Narrowers assign a specific meaning to a value, i.e. this a Work phone number.
• assume each value is described by a number of attributes, i.e. “416-515-5555″ can be described by ( TEL, Work, Mobile )

Then:

• for Narrower, then for each Grouper
  • create a dictionary ’subd’
  • for each values that is described by the ( Narrower, Grouper )
    • for each remaining attribute (besides Narrower and Grouper), add to subd
    • if the value was fully described by ( Narrower, Grouper ), add to subd under the key ‘@’
  • for key, value in subd
    • add to the final result
    • if narrower is not ‘no-narrower’, add ‘@narrower = narrower’
  • add subd to the result under the key Grouper
• add all remaining values from the original hCard to the result, noting that
  • if the value is described by a Narrower, we encoded it as a dictionary with ‘@narrower = narrower’

Clear? Well, the examples below will help. We the “416-515-5555″ above we would get:

{
 "hcard:hcard" : {
  'tel' : {
   '@work' : 'work',
   'mobile' : '416-515-5555',
  }
 }
}

Code

The source code for this algorithm is in the AUMFP tree, in file vcard.py function decompose (see around line 1083)

Namespace

All JSON encoded hCards are in the namespace hcard:. In the AUAPI serialization, this namespace should only be on the enclosing element, all children will be assumed to be in the namespace. I am currently using the URI http://purl.org/uF/hCard/1.0/ for this namespace (when XML serializing); this may change in the future.

Example 1 - home phone number from whitepages.com

{
 'hcard:hcard': {'adr': {'country-name': u'United States',
                         'locality': u'Huntsville',
                         'postal-code': '35801-2908',
                         'region': 'Alabama',
                         'street-address': u'1114 Humes Avenue NE'},
                 'fn': u'Jack Smith',
                 'geo': {'latitude': 34.743763000000001,
                         'longitude': -86.572568000000004},
                 'n': {'family-name': u'Smith', 'given-name': u'Jack'},
                 'tel': {'voice': u'256-539-8788'}},
}

Example 2 - work phone number from whitepages.com

{ 'hcard:hcard': {'adr': {'country-name': u'United States',
                         'locality': u'Gurley',
                         'postal-code': '35748-8715',
                         'region': 'Alabama',
                         'street-address': u'148 Little Cove Road'},
                 'fn': u'Jack Smith',
                 'geo': {'latitude': 34.698258000000003,
                         'longitude': -86.383027999999996},
                 'n': {'family-name': u'Smith', 'given-name': u'Jack'},
                 'org': {'organization-name': u'Alldyne Powder Technoliges'},
                 'tel': {'@work': 'work', 'voice': u'256-776-1238'}},
}

Example 3 - hCard directly to JSON

{ 'hcard:hcard': {
                 'adr': {u'country-name': u'United States of America',
                         u'locality': u'San Francisco',
                         u'region': u'CA'},
                 u'fn': u'Tantek \xc7elik',
                 u'logo': u'icon-2007-128px.png',
                 'n': {'family-name': u'\xc7elik',
                       'given-name': u'Tantek'},
                 u'photo': u'http://tantek.com/icon-2007-128px.png',
                 u'url': u'http://feeds.technorati.com/contact/tantek.com/#hcard'},
}

Here is a brief outline of how one would “naively” transform Almost Universal API’s (AUAPI) JSON into XML. We say “naive” because in general one wants to make a transformation into a specific XML application: Atom, RSS, OPML, KML, etc.. In those cases, one has to rename and rework certain elements first for standards compliance, then complete the naive transformation for remaining elements.

Walking

Walking JSON objects is done depth first. Most of the complexity involved is in handling dictionaries, which can be valued as being comprised of ( key, value ) pairs. For each dictionary, we are creating an XML node whose properties are defined as follows:

• keys beginning with @@ are ignored
• the key @ means “the text” of the  node (the examples will make this more clear)
• other keys beginning with @ are attributes of the node
• all other keys are defining children of the node

There are number of complexities that have to be addressed; for this I suggest looking at the examples or source code.

Namespace handling

• collect all the namespaces used in the JSON and add to the root XML node
• if any JSON element has a namespace, assume that namespace is inherited by its children

Code

You can see the code for this in the AUAPI source base in api.py in XMLAPIWriter.TranscribeNode.

Example 1

{
    "numbers" : [ 1, -0.23, ],
    "strings" : [ "bob", "caf\xe9", ],
    "booleanish" : [ True, False, None, ],
}

<root>
    <numbers>1</numbers>
    <numbers>-0.23</numbers>
    <booleanish>True</booleanish>
    <booleanish>False</booleanish>
    <booleanish />
    <strings>bob</strings>
    <strings>caf\xc3\xa9</strings>
</root>

Example 2

{
    "a1" : {
        "b1" : 1,
        "b2" : 2,
    },
    "a2" : {
        "b3" : "hi",
        "b4" : "there",
    },
}

<root>
    <a1>
        <b1>1</b1>
        <b2>2</b2>
    </a1>
    <a2>
        <b4>there</b4>
        <b3>hi</b3>
    </a2>
</root>

Example 3

{
    "@attribute" : "hello",
    "@bttribute" : "there",
    "a" : "some string",
},

<root attribute="hello" bttribute="there">
    <a>some string</a>
</root>

To quickly review, the AUAPI sees API results as composing of two parts. A “response” (formerly the “meta”) and “items”, composed of one or more items. This division is documented in Work Object Records.

Here is how we use Atom to encode API items in the AUAPI (we will document “response” in a different post). This sticks fairly closely to the Atom standard, the differences being in how each term is serialized into JSON for convenience and ease of use - and that there’s no real required elements. Don’t forget that each “item” is a discrete unit returned from an API and represents “whatever” - a Flickr photo, an Amazon product

• title (string, plain text) - the title; this is a plain text string, i.e. no entities or HTML allowed
• id (string, plain text) - a unique ID (in the context of the API being called) for this result
• content (string, HTML) - the complete text; this is HTML
• summary (string, HTML) - the summary text; this is HTML.
• link (string, URL) - this is the “main” link of whatever the item represents, a page on Amazon, a blog post’s original HTML and so forth
• links (array of dictionary) - these are other links related to the item; the format is documented below
• category (array of dictionary) - tags/categories; the format is documented below
• author (a string or dictionary) - if a dictionary, the format is documented below
• updated (string, Atom datetime format) - when the item was last updated
• posted (string, Atom datetime format) - when the item was originally posted; if using updated and/or posted, always use updated and then use posted only if you have a meaningful difference.

Links Format

links are results related to the item. The most important link should be encoded in the link item. links are a list of dictionary, each dictionary containing (further documentation):

• href (string, URI)
• rel (string, enumeration)
• type (string, MIME type)
• hreflang (string, language code)
• title (string, plain text)
• length (string, integer)

Category Format

category are the tags for the item, and are a list of a dictionary each dictionary containing (further documentation):

• @ (string, plain text) - the tag
• rel (string, from enumeration)
• scheme (string, URI)

Author Format

The author can be encoded as a string (containing the author’s name) or as a dictionary, or as a list of dictionaries. If a dictionary, this is how it should be encoded (further documentation)

• @ (string, plain text) - the author’s name
• uri (string, URI)
• email (string, email address)

Example

This is a API result from Whitepages.com, encoded as AUAPI. I have removed non-Atom items and shortened long text for clarity.

{'content': "<div class="vcard">\n<div class="fn">Jack Smith</div>...\n</div>\n",
 'id': '40b296d1a95f3b379a8108b27daf009c',
 'links': [{'href': 'http://www.whitepages.com/16176/t...',
            'rel': 'related',
            'title': 'Find Neighbours',
            'type': 'text/html'},
           {'href': 'http://www.whitepages.com/16176/track...',
            'rel': 'related',
            'title': 'Map',
            'type': 'text/html'},
           {'href': 'http://www.whitepages.com/16...',
            'rel': 'related',
            'title': 'Map',
            'type': 'text/html'},
           {'href': 'http://www.whitepages.com/16176/track/102...',
            'rel': 'alternate',
            'title': 'Whitepages.com',
            'type': 'text/html'}],
 'title': u'Jack Smith'}

The Almost Universal API is a culmination - or at least a local maxima - of several projects I’ve been working on for the last few months: in particular, Web Object Records, Pipe Cleaner and PyBM. The AUAPI is:

• a way of presenting results returned from many popular APIs
• a Python library to actual do this

I’ll be making several posts about how to use the AUAPI, including installation instructions. The plan is to make an easy_install version, but initially this will be a SVN from Google Code thing.

The AUAPI is mainly about how to present results returned from APIs, not how to send data to APIs nor how to encode requests. The encoding is designed to “look good” in JSON and be easily and algorithmically encoded into XML. The AUAPI data model is based on:

• Atom, the “core” vocabulary, particularly providing title, content, summary, updated, category, link and links
• MediaRSS, for encoding images
• hCard, for encoding information about people
• hCalendar, for encoding information about events

There are several “maybe” standards too:

• hProduct, for encoding information about things
• Google’s SGN URLs, for providing a universal way of talking about accounts

I have already worked a fair number of APIs into the AUAPI. These are documented on the Mashematica Wiki:

• api_csv - download and interpret CSV documents
• api_delicious - get information from Delicious.com API
• api_download - download HTML, XML and JSON documents
• api_feed - download Atom and RSS feeds
• api_feeds - find feeds on a page
• api_foaf - read FOAF files
• api_flickr - Flickr API
• api_geocode - geocode
• api_google - get information from Google AJAX Search API
• api_lastfm - return results from Last.fm
• api_microformat - get microformatted information on a page
• api_opml - download OPML documents
• api_whitepages - look up phone numbers

This post was originally sent to the ChangeCamp mailing list in response to a question about “what framework should we use for public APIs?“.

The core “frameworks” are POSH, REST and JSON. POSH is “Plain Old Semantic HTML”, meaning websites should be developed using modern web standards, pages should validate and use HTML elements correctly, and presentation is coded using CSS. REST can have deeper implications, but amongst the simplest is that pages can be returned using simple GET statements against well known URLs. JSON has emerged as the defacto standard for returning API results, amongst the reasons for is simplicity of creating mashups and embedability.

Atom and/or RSS provide the framework for update notifications. There are emerging technologies for real-time delivery, but it’s too early to worry about that.

Microformats provide a framework for embedding well-understood objects in HTML, are based on popular and well-understood standards, are easy(-ish) to implement, and a “consumer” ecosystem exists. In particular, people can be represented by hCard, events by hCalendar, tagged data by rel-tag and microcontent (articles within a page) by hAtom. Note that no parallel infrastructure need exist to do microformats: they are served within HTML pages.

Identify should use OAuth and OpenID; pragmatism says Facebook Connect and Google Friend Connect should be in the mix too, though I have a number of reservations about those.

I am very non-bullish about RDF, particularly as a model for delivering data of well-defined formats. IMHO it has missed almost the entirely the mashup wave of the last few years, and successes seem to be scattered at best. RDFa is competing in microformat’s “space” and may see success yet if it starts proving concrete solutions rather than “here’s a format that can do anything”, especially given microformat’s process issues.

Here’s a Pipe Cleaner script to convert this text list of Science Journals and converts it an OPML subscription list (here)

import module:api_csv.CSV;

CSV uri:'http://www.tictocs.ac.uk/text.php' delimeter:'\t';

items := map value:$items map:{
    "title" : "{{ C1 }}",
    "links" : {
        "href" : "{{ C2 }}",
        "type" : "text/xml",
        "rel" : "alternate",
    }
};

I decided not to use the “header name” feature of the CSV command because I had to remap anyway to create the links object. This has to be run with the following command (or from the web UI):

pc --format opml science-journals

Of course, this is a little unwieldy in size so maybe you only want journals with “Astrophysics” in their title:

import module:api_csv.CSV;

CSV uri:'http://www.tictocs.ac.uk/text.php' delimeter:'\t';

items := map value:$items map:{
    "title" : "{{ C1 }}",
    "links" : {
        "href" : "{{ C2 }}",
        "type" : "text/xml",
        "rel" : "alternate",
    }
};

items := search value:$items for:"Astrophysics";

Cool, eh? Not only this, this can be run entirely from the Web Interface with selectable strings so (theoretically) a Pipe Cleaner user would have an API to this data.