David Janes' Code Weblog

March 14, 2011

“Have Startups Become a Fetish?”

startups,sxsw,toronto · admin · 5:36 am ·

I’m totally loving this Gigaom article by Stacey Higgenbotham:

Since the magical breakout of Twitter in 2007 and the Foursquare success of 2009, SXSW has become more and more cluttered with startups trying to break out. It has also become a celebration of startups in general. However, that celebration has turned into a fetish — placing the act of creating a startup on a pedestal without casting any sort of critical eye on the quality or likelihood of that startup or idea succeeding.

[...] But amid the hundreds of startups launching, pitching, forming or otherwise trying to break out at SXSW, how many of them are real businesses? How many of them are thought out beyond a scrawl on a napkin, or a quick debate ahead of a startup weekend?

[...] The thousands of startups today that are pitching themselves at app competitions or in industry conferences all seem to think being a startup is enough. That daring to come up with some idea, any idea, and build a beta site is enough. That the users will come and then the business model will come and then the money will come. Google, Facebook and Twitter are their icons. Somehow the act of creating a startup has become the goal instead of the building of a business.

[...] My issue is less with those littering the web with launch pages — if people want to take some time to test out a web site idea in their spare time, that’s far better than watching Two and Half Men reruns — but with the media, the venture firms and the ecosystem that has been built up to worship this idea of a startup. Maybe a little less fawning in the coverage and a little more skepticism is needed.

I quoted a little more than I wanted to here but there’s so many good lines – read the whole thing. An “the emperor has no clothes moment”? Remember, the characters in Neil Stephenson novels — if that’s the life you’re trying to recreate — actually had business plans and actually tried to execute them.

 

February 28, 2011

Foursquare, Python & OAuth2

authentication,code fragments,python · admin · 11:21 am ·

Foursquare has revved their API, but there’s no Python Library to support it. Here’s how to use it from Python

  1. Get OAuth info from foursquare — the callback URL does not matter _but_ you must record here. https://foursquare.com/oauth/register. Fill in the results into CLIENT_ID, CLIENT_SECRET and CALLBACK_URL
  2. Run this, enter the URL in browser, get code as FS_ACCESS_CODE, add back here
  3. Run this, fill in FS_ACCESS_TOKEN
  4. Run this, you’re authorized: enjoy!

Notes

  1. Python OAuth2 from here:  https://github.com/rodbegbie/python-oauth2
  2. This is entirely standing on the shoulder’s of Rod Begbie’s work.

Code:

import oauth2
import json
import pprint

OAUTH_BASE = "https://foursquare.com/oauth2/authenticate"

CLIENT_ID = ""
CLIENT_SECRET = ""
CALLBACK_URL = ""

FS_ACCESS_CODE = ""
FS_ACCESS_TOKEN = ""

client = oauth2.Client2(CLIENT_ID, CLIENT_SECRET, OAUTH_BASE)

if not CLIENT_ID:
  print "=== Register your Foursquare data here"
  print "=== Copy back CLIENT_ID, CLIENT_SECRET and CALLBACK_URL here"
  print "https://foursquare.com/oauth/"
elif FS_ACCESS_TOKEN:
  headers, content = client.request("https://api.foursquare.com/v2/users/self", 
   access_token = FS_ACCESS_TOKEN)

  pprint.pprint({
    "headers" : headers,
    "content" : json.loads(content),
  })
elif not FS_ACCESS_CODE:
  print "=== Enter this URL in your browser."
  print "=== Copy the CODE in the URL that results into FS_ACCESS_CODE"
  print client.authorization_url(redirect_uri = CALLBACK_URL, 
    endpoint='authenticate')
else:
  print "=== Here is your Access Token"
  print "=== Copy into FS_ACCESS_TOKEN as start using Foursquare"
  print client.access_token(FS_ACCESS_CODE, redirect_uri= CALLBACK_URL,
    grant_type='authorization_code')["access_token"]

December 28, 2010

Display only one jPicker at a time

code fragments,javascript,jquery · David Janes · 9:26 am ·

I’ve been playing with jQuery (in general) and jPicker (in particular) over the last few days. I’m using jPicker because it’s not only a color picker but it also supports alpha-levels.

One – major – issue I’ve had is that jPicker doesn’t make it very easy to display only one control at a time, making the screen a bit of a visual mess when multiple color fields are being used.

Here’s how I fixed this.

$(document).ready(function() {
    $('body').click(function(evt) {
        if ($('div.jPicker.Container:visible').length > 0) {
            if ($(evt.target).parents('span.jPicker').length > 0) {
                $('div.jPicker.Container').each(function() {
                    if ($(this).css('z-index') == '10') {
                        $(this).hide('fast');
                    }
                });
            } else if ($(evt.target).parents('div.jPicker.Container').length == 0) {
                $('div.jPicker.Container').hide('fast');
            }
        }
    });
});

Notes:

  • this will dismiss the current jPicker if you click outside of one
  • the first if makes sure there’s something to do first
  • the second and third ifs take care of the case where the user clicks on the control to popup a jPicker. This takes advantage of the fact that the current jPicker is shown at z-index = 20 (which makes me wonder if we could do a lot of this with just a CSS selector for z-index = 10)
  • the final if takes care of the case where we’ve clicked outside of all jPicker controls

December 24, 2010

Python recipe – sorting strings with numbers in them

code fragments,python · David Janes · 9:16 am ·

Here’s a quick Christmas Eve recipe for sorting lists of strings that have numbers in them at some place.

import re
import string

numeric_rex = re.compile('(\d+)')

def numeric_sorter(a, b):
    av = filter(None, numeric_rex.split(a))
    bv = filter(None, numeric_rex.split(b))

    while av and bv:
        af = av.pop(0)
        bf = bv.pop(0)

        if af[0] in string.digits and bf[0] in string.digits:
            r = cmp(int(af), int(bf))
        else:
            r = cmp(af, bf)

        if r != 0:
            return  r

    if av:
        return  1
    elif bv:
        return  -1
    else:
        return  0

Here’s an example of it being used:

>>> items = [ "Item1", "Item10", "Item", "Item9", "Item100", "ItemLogo", ]
>>> items.sort()
>>> pprint.pprint(items)
['Item', 'Item1', 'Item10', 'Item100', 'Item9', 'ItemLogo']
>>> items.sort(numeric_sorter)
>>> pprint.pprint(items)
['Item', 'Item1', 'Item9', 'Item10', 'Item100', 'ItemLogo']

December 22, 2010

Installing Drupal on a Macbook Air

drupal,macintosh · David Janes · 1:58 pm ·

This documents how I installed Drupal on a brand new Macbook Air running Snow Leopard 10.6.5.

Web/Desktop/Shell: Install MySQL

  • Go here (“Mysql Community Server”) and download the 64 bit version
  • Double click on the DMG and Install
  • Drag the .PREF file in the DMG to the System Preferences

Note – I’m still having trouble starting MySQL from the command line. Right now I’m doing:

  • cd /usr/local/mysql/bin
  • nohup ./mysqld_safe &
  • disown %1

If you have advice, I’d appreciate it.

Web/Shell: Install Drupal

  • Download Drupal
  • Untar Drupal in ~/Sites/
  • In Apple > System Preferences > Sharing, select Web Sharing
  • In your browser go to http://localhost/~USERNAME/drupal-6.20/

In case it’s not clear what happened here:

  • Snow Leopard has an Apache Server built in
  • each user account gets it’s HTTP files in ~/Sites/
  • you turn on Apache by turning on Web Sharing
  • you browse by going to ~USERNAME/

You’ll notice at this stage we’re just seeing a directory listing, or if you’re a little more ambitious and click on “index.php”, PHP code. If you’re actually seeing a Drupal setup page, skip the next section.

Shell: Setup PHP

This is just following the excellent instructions from here.

  • sudo vim /etc/apache2/httpd.conf
  • uncomment LoadModule php5_module
  • bash (to start a subshell)
  • cd /etc
  • sudo cp php.ini.default php.ini
  • sudo chmod 666 php.ini
  • sudo vi /etc/php.ini
  • search for ;date.timezone
  • uncomment and change to:
    date.timezone=America/Toronto
  • restart apache: sudo apachectl restart
  • end the subshell: exit

Note – if you see the error “/usr/sbin/apachectl: line 82: ulimit: open files: cannot modify limit: Invalid argument“, follow the instructions here:

  • sudo vim /usr/sbin/apachectl
  • comment out ULIMIT_MAX_FILES at line 64

Local Web/Shell: Set up Drupal

  • go to the webpage install.php (on the URL figured out way above)
  • click on Install Drupal in English
  • if you get an error message about default.settings.php
    • cd ~/Sites/drupal-6.20/sites/default
    • cp default.settings.php settings.php
    • chmod a+w settings.php
  • if you get an error message about sites/defaults/files
    • cd ~/Sites/drupal-6.20/sites/default
    • mkdir files
    • chmod 777 files

At this point you should be brought to a “Database Configuration Page” and we have to leave the browser and go back to the command line.

Shell: MySQL Database Configuration

These instructions create a MySQL database called “drupal_db” with access by “drupal_user” using the password “drupal_password“.

  • /usr/local/mysql/bin/mysql -u root
    • create database drupal_db;
    • grant all privileges ON drupal_db.* TO "drupal_user"@"localhost" identified by "drupal_password";
    • grant all privileges ON drupal_db.* TO "drupal_user"@"127.0.0.1" identified by "drupal_password";
    • flush privileges;
    • exit;

Local Web: Enter MySQL information into Drupal

Back in the browser you should be on the Drupal Database Configuration page. Enter drupal_db, drupal_user and drupal_password (the values you created in the previous step)

Submit the form. If you get the following error message:

Failed to connect to your MySQL database server. MySQL reports the following message: No such file or directory.

Make sure you typed everything correctly. If necessary, click on Advanced Options and change the hostname from localhost to 127.0.0.1.

Local Web: Configure Site page for Drupal

  • Site Information:
    • Site Name: localhost (why not)
    • Site e-mail address: your@email.com
  • Administrator account:
    • Username: admin (why not)
    • E-mail address: your@email.com
    • Password:/Confirm password: your-favorite-password

Local Web: Drupal installation complete

After you hit submit, you should be finished installing Drupal, except for the next little cleanup step.

Shell: Clean up files

Drupal doesn’t like you leaving settings.php alterable by the web server around:

  • cd ~/Sites/drupal-6.20/sites/default
  • chmod go-w settings.php

The directory files must be left writable.

November 29, 2010

Good-bye XML, SOAP & SOA

ideas,javascript,xml · David Janes · 7:17 am ·

In my callow youth, I remember being so excited about XML. Ah, how wrong I was. Two articles of interest:

The End Of The Road For Web Services

Web services as a strategy was fundamentally flawed in my view because it was so un-web. It took an idea that hardly worked on an Intranet – remote manipulation of tightly-specified objects – and tried to make it work on the Internet. It led to software applications that by default were complex, brittle and heavy. Although I know many brilliant software engineers who worked unexpected miracles with Web Services, implementation by the common corporate programmer was stodgy in every case I heard about. In the end Web Services became an intranet tool for most uses, rendering the “W” incorrect even if WS* will be with enterprise developers for years to come as a kind of architectural COBOL.

and

XML vs the Web

It’s “Yay”, because for important use cases JSON is dramatically better than XML. In particular, JSON shines as a programming language-independent representation of typical programming language data structures. This is an incredibly important use case and it would be hard to overstate how appallingly bad XML is for this. The fundamental problem is the mismatch between programming language data structures and the XML element/attribute data model of elements. This leaves the developer with three choices, all unappetising

[...]

So what’s the way forward? I think the Web community has spoken, and it’s clear that what it wants is HTML5, JavaScript and JSON. XML isn’t going away but I see it being less and less a Web technology; it won’t be something that you send over the wire on the public Web, but just one of many technologies that are used on the server to manage and generate what you do send over the wire.

In the short-term, I think the challenge is how to make HTML5 play more nicely with XML. In the longer term, I think the challenge is how to use our collective experience from building the XML stack to create technologies that work natively with HTML, JSON and JavaScript, and that bring to the broader Web developer community some of the good aspects of the modern XML development experience

Having in recent memory been involved in a SOA deathmarch project (still in the process of tumbling down to inevitable failure), and dealing daily with the issues of transferring data using XML, I can firmly say: good riddance.

What do we do now? We do JSON. Why do we do JSON? Because the JSON object model (and the implicit RESTful JSON API model) is almost an exact match for how programmers – especially those of languages such as Python, Ruby & JavaScript – think about data .

November 7, 2010

Twitter4J – working with the source in Android

android,java · David Janes · 8:58 am ·

Twitter4J is a is highly spoken of Java library for working with the Twitter API. We needed to implement a simple List Widget that would display Twitter statuses based on a Twitter List, but unfortunately at this time Twitter4J doesn’t supply a method to easily do this (and the one that does do it requires authentication).

Here’s how we did it. First, to the Android project, we:

  • removed the Twitter4J JAR file from libs
  • added twitter4j directory (and subdirectories) to src – this is the directory with the .java files in it
  • added the following libraries to the project (in libs):
    • commons-logging-1.1.1.jar (from here)
    • log4j-1.2.16.jar (from here)
    • slf4j-api-1.6.1.jar
    • slf4j-simple-1.6.1.jar (from here)

I’m a little scared by the number of libraries we had to add just to handle logging (on a limited mobile phone!), but what can you do?

Then we added one new method (sigh) to the code:

public ResponseList getUserListStatuses(String listOwnerScreenName, String id, Paging paging)
  throws TwitterException
{
  return StatusJSONImpl.createStatusList(
   http.get(
    conf.getRestBaseURL() + listOwnerScreenName +"/lists/" + id + "/statuses.json",
    paging.asPostParameterArray(Paging.SMCP, Paging.PER_PAGE), auth));
}

The only issue is we’re getting a lot of weird warnings, but it seems to work well. We’ll probably take this out if the Twitter4J folks add the method to the regular releases.

April 9, 2010

Useful Android Command Line Commands

android · David Janes · 9:13 am ·

This post is basically an FYI for me.

Running an Android emulator

$ emulator -avd david_6 &

or if you’re getting Out of Disk Space errors

$ emulator -avd david_6 -wipe-data &

Seeing the console output

$ adb logcat

Seeing what AVDs are available

$ android &

Djolt-obj: Django-like templates for Objective-C

code fragments,djolt,iphone · David Janes · 7:51 am ·

My company, Discover Anywhere Mobile, has open sourced a software component that many iPhone developers may be interested in: Djolt (Django-like Templates) for Objective C. The objective of this project is to be able to take NSObjects – particularly NSDictionary, NSArray, NSString and NSNumber – and be able to feed them to a text template, Django style, and get text output. This will be fantastic for working with APIs that deliver data via JSON or XML.

It’s dependent on RegexKit library for now, though if the iPhone OS 4 NSRegularExpression is powerful enough we may transition to that.

The code is being tested against the Django template test cases, with the goal of having as complete as possible (given different language idioms) compatibility. We only cover a small subset of Django yet – the stuff we need – but if you’re interested in working on the project, please by all means join in.

Code

NSDictionary* d = [[NSData dataWithContentsOfFile:@"test.json"] yajl_JSON];

DjoltTemplate* t = [[DjoltTemplate alloc] initWithTemplateFile:@"test.djolt"];
NSString* r = [t render:d];

test.json

{
 "title" : "Test 1",
 "items" : [
  {
   "href" : "http://www.example.com/1",
   "title" : "Example 1"
  },
  {
   "href" : "http://www.example.com/2",
   "title" : "Example 2 & Example \"2.5\""
  },
  {
   "href" : "http://www.example.com/3",
   "title" : "Example 3"
  }
 ]
}

test.djolt

<html>
<head>{{ title }}</head>
<body>
{% if fred or items or bleck %}
<ul>
{% for itemd in items %}
<li><a href='{{ itemd.href }}'>{{ itemd.title|safe }}</a> ({{ forloop.counter }})</li>
{% endfor %}
</ul>
{% endif %}
<p>
{{ nothing|default:"something" }}
</p>
</body>
</html>

Output

<html>
<head>Test 1</head>
<body>

<ul>

<li><a href='http://www.example.com/1'>Example 1</a> (1)</li>

<li><a href='http://www.example.com/2'>Example 2 & Example "2.5"</a> (2)</li>

<li><a href='http://www.example.com/3'>Example 3</a> (3)</li>

</ul>

<p>
something
</p>
</body>
</html>

April 3, 2010

Optimizing JSON database loading on an iPod Touch

iphone · David Janes · 3:13 pm ·

This is the story of how we sped up loading of our application’s database on a 1 generation iPod Touch from 19.28 seconds to 8.75 seconds. We test all our applications of the oldest hardware in use out there to get a sense of what our “floor” performance will be – generally if it’s bearable there, it’ll be better to awesome on a newer iPhone.

The Battle Plan

Our current database schema/code is called the “B” database. It loads from a JSON file (actually several, but only one counts) which is a list of dictionary. Each dictionary represents a location, listing, event, coupon or whatever in our application.

To speed up loading times, we decided to create a “C” database with the following characteristics:

  • still load from a JSON file
  • empty values would be removed from the database
  • default values – e.g. “country_name” : “Canada”, where the country is always Canada – would be removed
  • we would “lazy” load most of the data
    • the database would only contain the ID for each item, the lat/lon and a few other essential items
    • a separate JSON file for each location would be made also, containing all the data
    • because we well encapsulate access to our data objects (no peeking inside of the dictionary that holds all the data!), when the lazy data was needed we’d load it on the spot

We ended up only doing the the first two of these options. We’re likely to do the lazy loading in the future, but because this would require implementing a proper search (because search looks at every record) we decided to defer to another release. Our testing shows that this will shave about another 25% of load time, which at this point wasn’t worth the trouble but may be later when we start getting into 2500+ items to load.

Initially we planned also to do the following, but after learning that most of the work was in object creation and not in data loading, we decided not to pursue this

  • well known keys would be replaced by a single letter (i.e. “title” with “T”)
  • hash keys, represented as a 32 byte hex string, would be replaced by 20 character base64 strings

Because it turns out that object allocation and messing around with dictionaries is relatively expensive, and loading strings into memory not so much, we probably will never do these.

Performance Numbers

All times in seconds, measured from the start of loading

B database starting point
  • loading data from disk: 0.41
  • finishing converting JSON data to NSObjects: 11.34
  • finished wrapping NSObjects into Location objects: 19.28

We learned here that we have two obvious problems: converting JSON and creating the Location objects.

Use C database

The C database JSON file is about 20% of the size of the B database.

  • loading data from disk: 0.08
  • finishing converting JSON data to NSObjects: 5.56
  • finished wrapping NSObjects into Location objects: 13.30

Already we’ve seen a big improvement of almost 6 seconds.

Turn off slideshow

Here we just turned off the slideshow to see what would happen

  • loading data from disk:0.05
  • finishing converting JSON data to NSObjects: 4.42
  • finished wrapping NSObjects into Location objects: 11.57

Almost two seconds saved! Our new code now doesn’t start the animation (or in fact, any other non-critical background task) until the DB is loaded. For the most we can say that 4.42s is the floor for loading a database from disk using JSON – it really isn’t going to get smaller than this.

Slightly optimized B database, better Location constructor

There’s a lot of steps I’ve left out here – I just turned off the Location “init” function altogether, found that things worked at almost the speed shown below. Looking at the code, I discovered a totally unnecessary dictionary copy, a couple of computations that could be done lazily, and some unnecessary initializations. After that was all done, we’ve got the the Location creating step down to about 1.1 seconds – originally it was almost 8!

  • loading data from disk: 0.40
  • finishing converting JSON data to NSObjects: 7.6
  • finished wrapping NSObjects into Location objects: 8.76

Note that we we lost 3 seconds (4.42 to 7.6) going back to the B database. However, this saved us another long programming and testing stint, so we’ll let this ride for now. However, we know that we can get this down to about 25% of the original speed when we need to.

Conclusions

  • Look for things inside of loops – the costs add up quickly
  • Object creation isn’t free, and object copying is expensive
  • Don’t make assumptions whether you’re CPU bound or IO bound – test and find out
  • Where you’re CPU bound – don’t do other stuff (e.g. animation cuteness) that’s competing for the CPU!
  • Where you’re IO bound – make things smaller (duh)
  • we’re likely to do some experimenting with SQLdb in the future, but given that loading isn’t really the problem, it probably won’t make much difference
  • CoreData didn’t exist when we started this application, so we’re not even looking at that
  • On an iPhone 3GS the complete database loads in about 3.5 seconds, which is sweet enough

« Newer Posts · Older Posts »

Powered by WordPress

Switch to our mobile site