David Janes' Code Weblog

Foursquare has revved their API, but there’s no Python Library to support it. Here’s how to use it from Python

1. Get OAuth info from foursquare — the callback URL does not matter _but_ you must record here. https://foursquare.com/oauth/register. Fill in the results into CLIENT_ID, CLIENT_SECRET and CALLBACK_URL
2. Run this, enter the URL in browser, get code as FS_ACCESS_CODE, add back here
3. Run this, fill in FS_ACCESS_TOKEN
4. Run this, you’re authorized: enjoy!

Notes

1. Python OAuth2 from here:  https://github.com/rodbegbie/python-oauth2
2. This is entirely standing on the shoulder’s of Rod Begbie’s work.

Code:

import oauth2
import json
import pprint

OAUTH_BASE = "https://foursquare.com/oauth2/authenticate"

CLIENT_ID = ""
CLIENT_SECRET = ""
CALLBACK_URL = ""

FS_ACCESS_CODE = ""
FS_ACCESS_TOKEN = ""

client = oauth2.Client2(CLIENT_ID, CLIENT_SECRET, OAUTH_BASE)

if not CLIENT_ID:
  print "=== Register your Foursquare data here"
  print "=== Copy back CLIENT_ID, CLIENT_SECRET and CALLBACK_URL here"
  print "https://foursquare.com/oauth/"
elif FS_ACCESS_TOKEN:
  headers, content = client.request("https://api.foursquare.com/v2/users/self", 
   access_token = FS_ACCESS_TOKEN)

  pprint.pprint({
    "headers" : headers,
    "content" : json.loads(content),
  })
elif not FS_ACCESS_CODE:
  print "=== Enter this URL in your browser."
  print "=== Copy the CODE in the URL that results into FS_ACCESS_CODE"
  print client.authorization_url(redirect_uri = CALLBACK_URL, 
    endpoint='authenticate')
else:
  print "=== Here is your Access Token"
  print "=== Copy into FS_ACCESS_TOKEN as start using Foursquare"
  print client.access_token(FS_ACCESS_CODE, redirect_uri= CALLBACK_URL,
    grant_type='authorization_code')["access_token"]

The other day I twittered Chris Messina about OAuth:

@factoryjoe #OAuth is an incomprehensible mess. Programming a Python client to connect to a service has never been so hard

This is the details of my experience, plus suggestions about how to fix the problems I’ve encountered.

The username/password gold standard

To interact with a service like Twitter’s API, you need three pieces of information: a username,  password and an API endpoint I want to use. Once I have this information I can use a standard library in almost any language to start using the Twitter API – I am up and running within 45 seconds. Now, don’t mistake that I think giving up your username and password to a third party service is a good idea: it’s horrible. However, the other part – to be up and running within a few minutes – is critical from a programming usability point of view. No bucks, no Buck Rogers; no API users, no API usage.

It took me a day and half (albeit of scattered hours) to get OAuth to work for me. To put this in perspective, I had Google’s authentication system – including recoding urllib2 to deal with PUT and 301/302 errors – usable in about 2 hours.

What I’ve discovered is that OAuth is as almost as easy to use as HTTP Basic Authentication (the username / password scheme above); the issue is the confusing way OAuth is currently presenting information to developers. I have documented my coding experiences here and the code is freely available for use and perusal here (though you’ll really be buying into a web resources model that might not be your thing if you do).

The informational issues I had – with suggested fixes – are documented below.

Critical OAuth information is poorly packaged

This is what you need to know to access an OAuth API:

• A “consumer key”
• A “consumer secret”
• An “authorization URL”
• A “token URL”
• An “access token URL”

Not to mention a list of API URLs that are the API end points. Note that all of these items are defined by terminology unique to OAuth and thus unfamiliar to the new developer. Now, try going to Fire Eagle and getting all of that information, and when you’re finished that head on over to PostRank to do the same. If you’re clever, it’ll take you 5 minutes but more likely (especially if you’ve never seen OAuth before) it’ll take you about 10 minutes and you’re likely to have got something wrong. Did you catch that Fire Eagle has similar looking but not quite the same URLs? Does PostRank use “http://” or “https://” for “standard request paths“? Did you know that PostRank also has two entirely different hostnames for URLs in its APIs? If you didn’t, well, you’ll probably be revisiting your lists.

Here’s what I suggest: OAuth should recommend that every OAuth Service Provider return the following JSON dictionary (in a TEXTAREA or PRE) in the place where they’re currently returning the consumer key & secret:

{
    "api_uri": [
        "http://www.postrank.com/myfeeds/",
        "http://www.postrank.com/user/",
        "http://api.postrank.com/"
    ],
    "oauth_access_token_url": "http://www.postrank.com/oauth/access_token",
    "oauth_authorization_url": "http://www.postrank.com/oauth/authorize",
    "oauth_token_url": "http://www.postrank.com/oauth/request_token",
    "oauth_consumer_key": "XXXXXXXXXXXXXXXXXXXXXX",
    "oauth_consumer_secret": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
    "oauth_signature_method": "HMAC-SHA1",
    "oauth_version": "1.0"
}

That’s it: one single piece of information that can be read without difficultly by every single language modern developers use and can be copied and pasted in a single operation by the developer.

I’d almost like to note that Fire Eagle has the most useful OAuth developer documentation I’ve seen and the OAuth team should consider adopting it wholesale as their own.

The OAuth website is confusing

The front page of the OAuth website is very promising. There’s a big round button like area that says “For Consumer developers…” and another that says “For Service Provider developers…”. From there things go rapidly downhill. Neither of these items are buttons. Instead, one clicks on the “Get Started…” link, from there you examine a list of other links and then start reading about how OAuth got it’s name after the sound Blaine Cook’s college roommate’s brother’s cat used to make horking up furballs or whatever. Honestly: no one cares at this stage. What I’d like to do is click on “Consumer developers” link and start seeing a concrete example of what I need to do interact with an OAuth enable service. All the other stuff is filler.

One further point: it’d be nice to see a proper logo.

The OAuth website needs a API playground

My final recommendation is that OAuth provide on their website a live sample API Service Provider than all the libraries interact with out of the box. It’s difficult enough to get an API working without wondering whether the problem is on my side or on their side.

Here’s a neat API I completed this morning, called api_feeds. It takes a URL (or a list of them) and transforms them into:

• the home page associated with the URL
• the feed(s) for the URL
• the name of the home page

If you’re following along at home, this is essentially the information needed for a single outline in an OPML subscription list.

Here’s a simple python example:

api = api_feeds.OneFeed()
api.request = {
    "uri" : "http://code.davidjanes.com/blog/2009/01/23/transparently-working-with-oauath/",
}

pprint.pprint(api.response, width = 1)

And here’s what the output looks like:

{'link': u'http://code.davidjanes.com/blog',
 'links': [{'href': u'http://feeds.feedburner.com/DavidJanesCode',
            'rel': 'alternate',
            'type': u'application/rss+xml'}],
 'title': u"David Janes' Code Weblog"}

There’s actually quite a bit going on here behind the scenes, most of it using code I didn’t initially write but have quite heavily hacked: the Universal Feed Parser and the Feed Finder.

What becomes really interesting what happens when we combine this with other modules. Here’s an example of how we can build an OPML subscription list from all the posts I’ve tagged “python” and “django” in del.icio.us. The code looks up each link I’ve bookmarked, does the feed discovery above, filters out items that don’t have feeds, and outputs as OPML. Note the neat pipeline type aspect to the code:

api_delicious = api_delicious.PostsList(tag = "python django")
api_many = api_feeds.ManyFeeds(require_feed = True)
api_opml = api_opml.OPMLWriter()

api_many.items = api_delicious.items
api_opml.items = api_many.items

print api_opml.Produce()

Producing the following OPML:

<opml encoding="utf-8" version="2.0">
  <head>
    <title>[Untitled]</title>
  </head>
  <body>
    <outline htmlUrl="http://push.cx"
      rssUrl="http://push.cx/feed"
      text="Push cx"
      type="rss"/>
    <outline htmlUrl="http://crankycoder.com"
      rssUrl="http://crankycoder.com/feed/"
      text="crankycoder.com"
      type="rss"/>
    <outline htmlUrl="http://blog.dowski.com"
      rssUrl="http://blog.dowski.com/feed/"
      text="the occasional occurrence"
      type="rss"/>
    <outline htmlUrl="http://www.b-list.org/feeds/entries/"
      rssUrl="http://feeds2.feedburner.com/b-list-entries"
      text="The B-List: Latest entries"
      type="rss"/>
    <outline htmlUrl="http://blog.thescoop.org"
      rssUrl="http://blog.thescoop.org/feed/"
      text="The Scoop"
      type="rss"/>
    <outline htmlUrl="http://effbot.org"
      rssUrl="http://effbot.org/zone/rss.xml"
      text="effbot.org"
      type="rss"/>
    <outline htmlUrl="http://blog.disqus.net"
      rssUrl="http://feeds.feedburner.com/BigHeadLabs"
      text="Disqus"
      type="rss"/>
    <outline htmlUrl="http://blog.ianbicking.org"
      rssUrl="http://blog.ianbicking.org/feed/atom/"
      text="Ian Bicking: a blog"
      type="rss"/>
    <outline htmlUrl="http://antoniocangiano.com"
      rssUrl="http://feeds.feedburner.com/ZenAndTheArtOfRubyProgramming"
      text="Zen and the Art of Programming"
      type="rss"/>
    <outline htmlUrl="http://www.carthage.edu/webdev"
      rssUrl="http://www.carthage.edu/webdev/?feed=rss2"
      text="carthage webdev"
      type="rss"/>
    <outline htmlUrl="http://www.eweek.com"
      rssUrl="http://www.eweek.com/rss-feeds-13.xml"
      text="Application Development - RSS Feeds"
      type="rss"/>
    <outline htmlUrl="http://jeffcroft.com/"
      rssUrl="http://feeds.feedburner.com/jeffcroft/blog"
      text="JeffCroft.com: Latest blog entries"
      type="rss"/>
  </body>
</opml>

This will be just as terse (terser, probably) when written as a Pipe Cleaner script; I’m just struggling over how to introduce the authentication code gracefully into the scripts.

This is part one of two posts I’m going to write about OAuth; the second will be somewhat more critical in tone. Before I criticize – and I know it’s hard to put together technologically things like OAuth – I want to actually accomplish something with it, so I at least I appear that I have somewhat of a clue about it. This is a report of what I’ve done.

bm_uri is a libary and tool I’ve written for working with URIs, and in particular http:// and https:// URLs. Here are some of the advantages of using bm_uri over all the normal Python urllib and urllib2 methods:

• downloads are cached; if a URL is temporarily not available, bm_uri will return the cached version, likewise if it has been downloaded in the near past, the cached version will be returned rather than hitting the net again
• downloads can be cooked, meaning converted into a more useful form such as TIDY-cleaned up HTML, JSON, Unicode text and so forth
• bm_uri handles all the protocol stuff for you (such as User-Agent, Last-Modified and so forth) so you don’t have to
• authentication is handled “invisibly” as possible for you … at least after the initial setup

Here is an example of accessing a OAuth resource using bm_uri returning my current location from Fire Eagle as a Python object. From a programming point of a view, I believe I have reduced this to close to the minimum number of steps possible. Here’s the setup phase:

import bm_uri
import bm_oauth
import pprint

bm_cfg.cfg.initialize()

bm_oauth.OAuth(service_name = "fireeagle")

Here’s using it in code – note how there’s no reference to OAuth here whatsoever.

loader = bm_uri.JSONLoader('https://fireeagle.yahooapis.com/api/0.1/user.json?format=json')
loader.Load()

pprint.pprint(loader.GetCooked())

And here’s the output of the program:

{u'stat': u'ok',
 u'user': {u'location_hierarchy': [{u'best_guess': True,
         u'geometry': {u'coordinates': [-79.418426513699998,
                   43.731891632100002],
              u'type': u'Point'},
         u'id': 572261,
         u'label': None,
         u'level': 1,
         u'level_name': u'postal',
         u'located_at': u'2008-03-19T04:09:30-07:00',
...
         u'name': u'Canada',
         u'normal_name': None,
         u'place_id': u'EESRy8qbApgaeIkbsA',
         u'woeid': 23424775}],
     u'readable': True,
     u'writable': False}}

Gather information

The devil is in the details, obviously and with OAuth, the little satan is doing the initial setup. Here’s how I did this for Fire Eagle – there’ll be something analogous for whatever service you are using:

• Log in or sign up (obviously)
• Go to the Developers’ Page
• Click on Create a New App
• Copy the “Consumer Key” and the “Consumer Secret” … these will be long-ish strings of nonsense
• Find out the Request Token URL, the Access Token URL, and the Authorization URL. These are public knowledge and for Fire Eagle are:
  • https://fireeagle.yahooapis.com/oauth/request_token
  • https://fireeagle.yahooapis.com/oauth/access_token
  • http://fireeagle.yahoo.net/oauth/authorize

Note how Yahoo has conveniently made that last URL similar looking to the others, but not quite the same. Thanks!

However you implement OAuth, you’re probably going to need to be able to persist information to disk or database. As documented here several weeks ago, we already have that covered with our bm_cfg module. In ~/.cfg/fireeagle.json, create the following JSON format file:

{
 "fireeagle": {
  "api_uri" : "https://fireeagle.yahooapis.com/",
  "oauth_access_token_url": "https://fireeagle.yahooapis.com/oauth/access_token",
  "oauth_authorization_url": "http://fireeagle.yahoo.net/oauth/authorize",
  "oauth_consumer_key": "ABCDEFGHIJKL",
  "oauth_consumer_secret": "ABCDEFGHIJKLMNOPQRSTUVWXYZ012345",
  "oauth_token_url": "https://fireeagle.yahooapis.com/oauth/request_token",
 }
}

The only new item here is the api_uri: that’s the prefix of URLs that bm_uri will use OAuth with.

Set it up

Next you have to do all sorts of OAuth stuff to actually work with OAuth. If the why interests you, please go read the spec! I’m more of how person myself, and this is what we need to do:

• run: python bm_uri.py --service fireeagle --authorize
• this will pop up a browser window; grant your application access and then…
• run: python bm_uri.py --service fireeagle --exchange

And that’s it – you should now be able to just work with the Fire Eagle API in bm_uri without even having to know OAuth is there!

End notes

• the current implementation only works with HTTP/REST GET; POST to come soon, DELETE and PUT as needed
• bm_uri, bm_config and the rest of the code is freely licensed and available here. It is a constantly changing product, albeit converging on perfection in my own mind