David Janes' Code Weblog » djolt

Djolt-obj: Django-like templates for Objective-C

David Janes — Fri, 09 Apr 2010 12:51:26 +0000

My company, Discover Anywhere Mobile, has open sourced a software component that many iPhone developers may be interested in: Djolt (Django-like Templates) for Objective C. The objective of this project is to be able to take NSObjects – particularly NSDictionary, NSArray, NSString and NSNumber – and be able to feed them to a text template, Django style, and get text output. This will be fantastic for working with APIs that deliver data via JSON or XML.

It’s dependent on RegexKit library for now, though if the iPhone OS 4 NSRegularExpression is powerful enough we may transition to that.

The code is being tested against the Django template test cases, with the goal of having as complete as possible (given different language idioms) compatibility. We only cover a small subset of Django yet – the stuff we need – but if you’re interested in working on the project, please by all means join in.

Code

NSDictionary* d = [[NSData dataWithContentsOfFile:@"test.json"] yajl_JSON];

DjoltTemplate* t = [[DjoltTemplate alloc] initWithTemplateFile:@"test.djolt"];
NSString* r = [t render:d];

test.json

{
 "title" : "Test 1",
 "items" : [
  {
   "href" : "http://www.example.com/1",
   "title" : "Example 1"
  },
  {
   "href" : "http://www.example.com/2",
   "title" : "Example 2 & Example \"2.5\""
  },
  {
   "href" : "http://www.example.com/3",
   "title" : "Example 3"
  }
 ]
}

test.djolt


{{ title }}

{% if fred or items or bleck %}

{% for itemd in items %}
{{ itemd.title|safe }} ({{ forloop.counter }})
{% endfor %}

{% endif %}

{{ nothing|default:"something" }}

Output


Test 1




Example 1 (1)

Example 2 & Example "2.5" (2)

Example 3 (3)




something

Pipe Cleaner

David Janes — Thu, 18 Dec 2008 23:38:21 +0000

I’ve been working (in my decreasing available spare time) on a project to pull together into a project called “Pipe Cleaner” all the various concepts I’ve been mentioning on this blog: Web Object Records (WORK) for API Access and object manipulation, Djolt for generating text from templates, Data/Query/Transform/Template (DQT) for transforming data and JD for scripting these elements together. The pieces came together this morning enough to put a demo together and here it is – the Toronto Fires Pt II Demo.

How, you may ask, does this differ from the original Toronto Fires Demo? The answer is how it is put together, which we describe here.

Index.dj

This is the Djolt template that generates the output. The data fed to this template is generate by the JD script, described in the next section.



    
    {{ gmaps.js|safe }}



    
        {{ gmaps.html|safe }}
    
    
{% for incident in incidents %}
    
        {{ incident.body_sb|safe }}
    
{% endfor %}

Quite simple … as you can see, most of the data is being pulled in from elsewhere. The elsewhere is provided by the script described in the next section.

Index.jd

This is the script that pull all the pieces together. Note that I’m not 100% happy with the way the data is imported, I would like the geocoding to become part of this data flow too. In the next release perhaps.

First we pull in the “fire” module that we wrote in the previous Map examples. This is doing exactly what you think: importing a Python module. We may have to increase the security or restrict this to working with an API for general purpose use.

import module:"fire";

Next we define two headers – one that is going to appear in the Google Maps popup, the next that is going to appear in the sidebar. They need to be different as they refer to themselves. Note that the sidebar header “breaks” the encapsulation of Google Maps – this seems to be unavoidable. The to:"fitem.head.map" and to:"fitem.head.sb" are manipulating a WORK dictionary to store values.

Note also here that we’ve extended JD to accept Python multiline strings – this was unavoidable if JD was to be useful to me.

set to:"fitem.head.map" value:"""

{{ AlarmLevel}}: {{ IncidentType }} on {{ RawStreet }}

""";

set to:"fitem.head.sb" value:"""

{% if latitude and longitude %}
*
{% endif %}
{{ AlarmLevel}}: {{ IncidentType }} on {{ RawStreet }}

""";

The next block defines the text of the body used to describe a fire incident. It follows much the same pattern as the previous block.

set to:"fitem.body" value:"""

Alarm Level: {{ AlarmLevel }}


Incident Type: {{ IncidentType }}


City: {{ City }}


Street: {{ Street }} ({{ CrossStreet }})


Units: {{ Units }}

""";

This is a map: it is translating the values in fire.GetGeocodeIncidents into a new format and storing that in incidents. The format that we were are storing it in is understood by the Google Maps generating module.

We may rename this translate, as the word map is somewhat overloaded.

map from:"fire.GetGeocodedIncidents" to:"incidents" map:{
    "latitude" : "{{ latitude }}",
    "longitude" : "{{ longitude }}",
    "title" : "{{ AlarmLevel}}: {{ IncidentType }} on {{ RawStreet }}",
    "uri" : "{{ HOME_URI }}#{{ IncidentNumber }}",
    "body" : "{{ *fitem.head.map|safe }}{{ *fitem.body|safe }}",
    "body_sb" : "{{ *fitem.head.sb|safe }}{{ *fitem.body|safe }}",
    "IncidentNumber" : "{{ IncidentNumber }}"
};

Next we set up the “meta” (see WORK meta description if you’re not following along) for the maps. The render_value:true declaration makes PC interpret the templates in strings). We then call our Google Maps generating code (which are actually more Pipe Cleaners) and that gets fed to the Djolt template we first showed you. Clear? Maybe not, we’ll have more examples coming…

set to:"map_meta" render_value:true value:{
    "id" : "maps",
    "latitude" : 43.67,
    "longitude" : -79.38,
    "uzoom" : -13,
    "gzoom" : 13,
    "api_key" : "{{ cfg.gmaps.api_key|otherwise:'...mykey...' }}",
    "html" : {
        "width" : "1024px",
        "height" : "800px"
    }
};

load template:"gmaps.js" items:"incidents" meta:"map_meta";
load template:"gmaps.html" items:"incidents" meta:"map_meta";

A brief survey of Yahoo Pipes as a DQT

David Janes — Thu, 11 Dec 2008 12:19:55 +0000

Yahoo Pipes is a visual editor of mashups, allowing you to take data from sources on the net, transform them in various interesting ways and output the result as Atom, RSS or JSON. The primary downside Pipes of course is that you’re totally dependent on Yahoo for the infrastructure: it runs at Yahoo pulling feeds that have to be accessable through the public Internet.

It’s easy to use Pipes: just go to this page and start working with the sample example Pipe. You’ll need a Yahoo login ID, but most of us have that anyway. I’ve created an example that uses Yahoo Pipes to feed a Djolt template which you can see here.

We can analyze Pipes in the terms of the DQT paradigm we’ve outlined in the previous post.

Data Sources and Queries

Sources and Queries are merged (quite logically) in the Pipes interface. You can read in depth documentation here.

Fetch CSV
Feed Autodiscovery – outputs syndication feeds found on a page (RSS feeds on a CBC page)
Fetch Feed
Fetch Page – will read a page and parse the contents with a reg
Fetch Site Feed – this is the logical combination of Fetch Feed and Fetch Autodiscovery
Flickr – find images by tag near a location (photos of cats in Toronto)
Google Base – look up information in Google Base
Item Builder – a way of building new items from existing items
Yahoo Local
Yahoo Search

Transforms

The operator documentation can be read here.

Count
Filter
Location Extractor – a geocoder that magically looks for locations
Loop
Regex
Rename
Reverse
Sort
Split
Sub-element – pulls a particular sub-element of an item and makes that the item. This is very much like WORK path manipulation
Tail
Truncate
Union
Unique
Web Service

Plus a number of specialized data services, for dealing with elements such as dates.

Templates

Pipes does not provide an arbitrary Djolt-like template producing HTML. Instead, they provide a number of pre-made code templates that output well known data types, including RSS, JSON and Atom (and some stranger choices, like PHP).

Coding backwards for simplicity

David Janes — Mon, 08 Dec 2008 21:58:57 +0000

I haven’t been posting as much as I like here for the last three weeks, not because of lack of ideas but because I haven’t been able to consolidate what I’ve been working on into a coherent thought. I’m trying to come up with a overreaching conceptual arch that covers WORK, Djolt and the various API interfaces I’ve been coded. Tentatively and horribly, I’m calling this Data/Query/Transform/Template right now though I’m expecting this to change.

The first demo of this … without further explanation … can be seen here. More details about what this is actually demonstrating (besides formatting this blog) will be forthcoming.

What I want to draw attention to in this post is how I coded this. What I’ve been doing for the last several weeks is coding backwards: I start with what I want the final code to look like and then figure out all the libraries, little languages and so forth that would be needed to code that. After several false starts, my conceptual logjam broke about a week ago and code started radically simplifying.

The ideal code, in my mind, is almost entirely static declarations: no loops, no if statements, no while statements, no goto-type statements (god help us). We simply specify how the parts are connected, and hope that we can abstract the complexity into the libraries that make this all happen. The code that you see below is actually post all my conceptualizing: I just wanted to write some code and since I had almost all the parts together it fell together quite nicely:

import bm_wsgi
import bm_io

import djolt
import api_feed

from bm_log import Log

class Application(bm_wsgi.SimpleWrapper):
    def __init__(self, *av, **ad):
        bm_wsgi.SimpleWrapper.__init__(self, *av, **ad)

    def CustomizeSetup(self):
        self.html_template_src = bm_io.readfile("index.dj")
        self.html_template = djolt.Template(self.html_template_src)

        self.context = djolt.Context()
        self.context["paramd"] = {
            "feed" : "http://feeds.feedburner.com/DavidJanesCode",
            "template" : """\

{% for item in data.items %}
	{{ item.title }}
{% endfor %}
""",
        }
        self.context.Push()
        self.context["paramd"] = self.paramd
        self.context["data"] = api_feed.RSS20(self.context.as_string("paramd.feed"))

    def CustomizeContent(self):
        yield   self.html_template.Render(self.context)

if __name__ == '__main__':
    Application.RunCGI()

There’s almost nothing there! In particular, note:

bm_wsgi.SimpleWrapper handles all the WSGI interface work, including determining when to output HTML headers, error trapping, and Unicode to UTF-8 encoding
the most complicated part of the application is setting up the Context. In particular, note that self.paramd is automatically populated by the QUERY_STRING passed to the application, and the double setting we do here allows us to have default values.
If you want to see the HTML template that drives the application it is here. Note two variations from Django templates: the {% asis %} block which doesn’t intrepret it’s content as Djolt code and the {{ *paramd.template|safe }} variable which interprets the variable’s contents as a template.
Methods called Customize-something are my convention for framework functions, i.e. methods that will be called for us rather than methods we call.

Djolt Indirection

David Janes — Thu, 04 Dec 2008 11:05:28 +0000

I’ve been working through a sticky problem with Djolt, trying to implement my Toronto Fires example in as few lines as possible. As part of this, I’ve come up with the idea of adding indirection to Djolt templates:

import djolt

d = {
    "a" : "It says: {{ b }}",
    "b" : "Hello, World"
}

t = djolt.Template("""
a: {{ a }}
b: {{ b }}
*a: {{ *a }}
""")

print t.Render(d)
""")

print t.Render(d)

Which yields:

a: It says: {{ b }}
b: Hello, World
*a: It says: Hello, World

This is significantly updated from the original version I posted here an hour ago. The indirection now makes the variable read as a template. This is a much more powerful concept.

Djolt – Django-like Templates

David Janes — Fri, 28 Nov 2008 21:34:39 +0000

Djolt is a reimplementation of Django’s template language in Python. Why do this?

I like the Django template language
I wanted something that small and independent of Django
I wanted something that will work with WORK paths (this was the real deal breaker for using Django)
I wanted something that I could take and reimplement in Javascript and maybe Java too
Some template engines, Cheetah for example, are far too heavy for the kind of light-weight applications I have in mind; note that I’ve had great success with Cheetah in the past
Some template engines, such as that in Python 2.6, are for too underfeatured

However, if you’re really looking for the whole Django template experience and don’t want to use Djolt, just start here.

How do I get it?

Djolt is packaged as part of the pybm library.

How do I use it?

import djolt

t = djolt.Template("""

{% for name in names %}
{{ name }}
{% endfor %}

""")
print t.Render({
    "names" : [ "Johnny", "Jack", "Ray", "Mary & Sam", ]
})

Which gives the results:


Johnny
Jack
Ray
Mary & Sam

Note the “autoescaping” of the & character.

What tags does it define?

autoescape/endautoescape
if/else/endif
equal/endequal
for/endfor
notequal/notendequal

It does not implement blocks.

What filters does it define?

add
cut
default (see otherwise below)
default_if_none
divisibleby
first
join
last
length
length_is
linebreaks
lower
pluralize
random
safe (respecting all the Django autoescape rules)
slug
upper

Unimplemented filters are due to laziness and will be done “on demand”. We also introduce a few new filters:

jslug – like slug, but more Javascript friendly
otherwise – like default, except the empty string/empty values trigger the filter also

Are their differences between Djolt and Django templates?

Djolt tags suck up whitespace if they’re on a line by themselves
If Djolt cannot resolve a variable, it resolves to the appropriate “empty” value (as opposed to failing). This is keeping in line with WORK philosophy

Beyond that you should be able to use most Django template examples (that don’t use block/implements) as-is.

Is it extensible?

Yes. You can add your own tags and filters by following the examples in code (djolt_nodes.py and djolt_filters.py respectively).

Toronto Fires

David Janes — Sat, 22 Nov 2008 21:19:58 +0000

Here’s a little mashup I’ve been putting together for the last few days: Toronto Fires.

It’s taking the data listed here on the City of Toronto’s Fire Services “Active Accidents”, scraping it (by pretending HTML is XHTML and treating it as WORK objects), geocoding it (using our WORK Google API) and mapping it (using this information).

This is very much a work in progress, but here’s a few more things that are involved:

we read body.table.tr.td[1].table.tr[1].td.table.tr as a list to get the rows in the table
we map those rows into the Geocoder use a new magic technology we’ll be explaining in the next few days: Djolt – Django-like templates
the output program is just one big Djolt template

I’m not quite satisfied with how the current page is constructed: I want the final result to be much more simple.