Happy New Year, everyone. I\u2019ve been busy at paying work recently, plus cleaning up and testing existing code I\u2019ve been discussing here over the last few months. At work I\u2019ve been developing in WebObjects, which though a lovely platform is not the way of the future so I\u2019m not documenting many of my experiences here.
\n
The applications I\u2019ve been working on recently, Pipe Cleaner and GenX, need - like most applications - configuration. This will store information which can be safely exposed to the public, such as my Google Maps API key, and information that I need to keep private within the application, such as my Freebase username and password (cf. however the password anti-pattern). Furthermore, though the code I\u2019m writing is in Python it is possible that the code that provides the UI will be written in another language, such as PHP inside of WordPress.
\n
Given these considerations, here\u2019s my design choices:
\n
\n
configuration files are stored as multiple individual files inside a directory (or directories)
\n
configuration files are in JSON, and contain a dictionary of dictionaries (see below)
\n
configuration files can be marked as private or public
\n
the same logical configuration (say for Amazon, which has both public and private information) can be in a public and private file
\n
the configuration is global, but is accessed through setter/getter properties
\n
non-global versions of the configuration can be made
As an aside, I\u2019m not 100% sure about Python decorators and wonder if my favorite language is being turned into a C++ like mess.
\n
Next, the \u2018add\u2019 function that adds information to the configuration ensuring private and public are handled correctly. Note that there can be multiple dictionaries inside of \u2018d\u2019, but \u2018d\u2019 is either all Public or not.
\n
def add(self, d):\n if type(d) != types.DictType:\n raise TypeError(\"only dictionaries can be added\")\n\n if d.get('@Public'):\n #\n # Public definitions never overwrite private definitions\n #\n for key, value in d.iteritems():\n if type(value) != types.DictType:\n continue\n\n if not self._cfg_private.has_key(key):\n self._cfg_private[key] = value\n\n self._cfg_public[key] = value\n else:\n self._cfg_private.update(d)
\n
And finally the loader, which gets everything in a directory or one level down. Note the \u2018exception\u2019 parameter which makes me a bad person, but I don\u2019t like code failing unless I tell it to.
\n
def load(self, path, exception = False, depth = 0):\n try:\n if os.path.isdir(path) and depth < 2:\n for file in os.listdir(path):\n self.load(os.path.join(path, file))\n elif os.path.isfile(path):\n if path.endswith(\".json\"):\n self.add(json.loads(bm_io.readfile(path)))\n\n except:\n if exception:\n raise\n\n Log(\"ignoring exception\", exception = True, path = path)
\n
And one more thing: make the global configuation:
\n
cfg = Cfg()
\n
Here\u2019s how you use it:
\n
import bm_cfg\n\n# setup ... on a per-file or directory basis\nfor file in sys.argv[1:]:\n bm_cfg.cfg.load(file)\n\n# use it\npprint.pprint({\n \"private\" : bm_cfg.cfg.private,\n \"public\" : bm_cfg.cfg.public,\n}, width = 1)
\n
Here\u2019s what my configuration directory looks like:
Note that if the private version of the Amazon file wasn\u2019t available, the public version would also be in the private one. I.e. the private configuration basically is \u201ceverything\u201d (noting possibly exceptions above in the code).
",
"guidislink": false,
"id": "http://code.davidjanes.com/blog/?p=406",
"link": "http://code.davidjanes.com/blog/2009/01/09/thinking-about-configuration/",
"links": [
{
"href": "http://code.davidjanes.com/blog/2009/01/09/thinking-about-configuration/",
"rel": "alternate",
"type": "text/html"
}
],
"summary": "Happy New Year, everyone. I\u2019ve been busy at paying work recently, plus cleaning up and testing existing code I\u2019ve been discussing here over the last few months. At work I\u2019ve been developing in WebObjects, which though a lovely platform is not the way of the future so I\u2019m not documenting many of my experiences here.\nThe [...]",
"title": "Thinking about Configuration",
"updated": "2009-01-09T12:20:02+00:00",
"wfw:commentrss": "http://code.davidjanes.com/blog/2009/01/09/thinking-about-configuration/feed/"
},
{
"author": {
"@": "David Janes",
"uri": "http://code.davidjanes.com/blog"
},
"category": [
{
"label": null,
"scheme": null,
"term": "db"
},
{
"label": null,
"scheme": null,
"term": "ideas"
},
{
"label": null,
"scheme": null,
"term": "semantic web"
}
],
"comments": "http://code.davidjanes.com/blog/2008/12/29/links-from-the-last-month/#comments",
"content": "
\n
Aspen - a web server for highly extensible Python-based publication, application, and hybrid websites. As a potential alternative to Python\u2019s builtin HTTPServer. MIT license.
\n
V8 - V8 is Google\u2019s open source JavaScript engine; written in C++; can run standalone, or can be embedded into any C++ application. I am very excited by this, as allowing users to send code to the server to execute Javascript is an amazingly powerful idea. If anyone knows of a Python wrapper, let me know please. New BSD license.
\n
KomodoEdit (a testimonial) - I am going to try this out, though vi/vim will always be my first love (JJ also has an article on using ctags).
\n
Virtuoso - an innovative Universal Server platform that delivers an enterprise level Data Integration and Management solution for SQL, RDF, XML, Web Services, and Business Processes. There\u2019s way to much bla bla bla in that sentence, but apparently this is really sweet at handling SPARQL/RDF triples. Kingsley Idehen writes extensively about this on his blog (e.g.).
\n
Drizzle - a database optimized for Cloud and Net applications. Way too early to commit to this yet. See The New MySQL Landscape for more interesting going ons.
\n
AuthKit - authentication and authorization toolkit for WSGI applications and frameworks.
\n
Geodjango - a world-class geographic web framework. Lots of great ideas and pointers to libraries in here, even if you\u2019re not planning to use this itself.
\n
Disco - an open-source implementation of the Map-Reduce framework for distributed computing. The Disco core is written in Erlang, a functional language that is designed for building robust fault-tolerant distributed applications. Users of Disco typically write jobs in Python, which makes it possible to express even complex algorithms or data processing tasks often only in tens of lines of code. Here\u2019s a blog post about the same, with references to vs. Hadoop.
",
"guidislink": false,
"id": "http://code.davidjanes.com/blog/?p=401",
"link": "http://code.davidjanes.com/blog/2008/12/29/links-from-the-last-month/",
"links": [
{
"href": "http://code.davidjanes.com/blog/2008/12/29/links-from-the-last-month/",
"rel": "alternate",
"type": "text/html"
}
],
"summary": "Aspen - a web server for highly extensible Python-based publication, application, and hybrid websites. As a potential alternative to Python\u2019s builtin HTTPServer. MIT license.\nV8 - V8 is Google\u2019s open source JavaScript engine; written in C++; can run standalone, or can be embedded into any C++ application. I am very excited by this, as allowing users [...]",
"title": "Interesting links from the last month",
"updated": "2008-12-29T14:29:25+00:00",
"wfw:commentrss": "http://code.davidjanes.com/blog/2008/12/29/links-from-the-last-month/feed/"
},
{
"author": {
"@": "David Janes",
"uri": "http://code.davidjanes.com/blog"
},
"category": [
{
"label": null,
"scheme": null,
"term": "macintosh"
},
{
"label": null,
"scheme": null,
"term": "tips"
}
],
"comments": "http://code.davidjanes.com/blog/2008/12/23/interact-with-applications-from-the-command-line-on-the-mac/#comments",
"content": "
I\u2019m a command line guy - I spend 90% of my non-blog reading day in Terminal, working on Python apps on my Mac or SSHed into work and working on Java and Javascript applications. I do realize the benefit of \u201creal\u201d applications, for image editing, for advanced text processing and so forth. On the Mac you can send files to the default application easily:
\n
\nopen \"Madeline Doll House.jpg\"\n
\n
(Don\u2019t ask). If the Mac doesn\u2019t know how to deal with the file type, or you want to specify a particular app, that\u2019s cool too:
",
"guidislink": false,
"id": "http://code.davidjanes.com/blog/?p=396",
"link": "http://code.davidjanes.com/blog/2008/12/23/interact-with-applications-from-the-command-line-on-the-mac/",
"links": [
{
"href": "http://code.davidjanes.com/blog/2008/12/23/interact-with-applications-from-the-command-line-on-the-mac/",
"rel": "alternate",
"type": "text/html"
}
],
"summary": "I\u2019m a command line guy - I spend 90% of my non-blog reading day in Terminal, working on Python apps on my Mac or SSHed into work and working on Java and Javascript applications. I do realize the benefit of \u201creal\u201d applications, for image editing, for advanced text processing and so forth. On the Mac [...]",
"title": "Interact with applications from the command line on the Mac",
"updated": "2008-12-23T11:50:38+00:00",
"wfw:commentrss": "http://code.davidjanes.com/blog/2008/12/23/interact-with-applications-from-the-command-line-on-the-mac/feed/"
},
{
"author": {
"@": "David Janes",
"uri": "http://code.davidjanes.com/blog"
},
"category": [
{
"label": null,
"scheme": null,
"term": "demo"
},
{
"label": null,
"scheme": null,
"term": "maps"
},
{
"label": null,
"scheme": null,
"term": "pipe cleaner"
}
],
"comments": "http://code.davidjanes.com/blog/2008/12/23/pipe-cleaner-ii/#comments",
"content": "
Here\u2019s the latest evolution of Pipe Cleaner, mainly recorded here for historical interest. The big change is that there isn\u2019t a separate outside template - everything is in the one index.jd file. The new directive is template, which can read and execute an outside module or actually produce the final output (as we see in the very last directive). I have not put this up as an independent demo.
\n
\n#\n#\tImport the Python fire module\n#\t- used in: map from:\"fire.GetGeocodedIncidents\" to:\"incidents\"\n#\nimport module:\"fire\";\n\n#\n#\tHeader for Google Maps popup\n#\t- used in: map from:\"fire.GetGeocodedIncidents\" to:\"incidents\"\n#\n#\nset to:\"fitem.head.map\" value:\"\"\"\n
\n\"\"\";\n\n#\n#\tHeader for the sidebar\n#\t- used in: map from:\"fire.GetGeocodedIncidents\" to:\"incidents\"\n#\nset to:\"fitem.head.sb\" value:\"\"\"\n
\n\"\"\";\n\n#\n#\tBody for the Google Maps pop and the sidebar\n#\t- used in: map from:\"fire.GetGeocodedIncidents\" to:\"incidents\"\n#\nset to:\"fitem.body\" value:\"\"\"\n
\nAlarm Level: {{ AlarmLevel }}\n \nIncident Type: {{ IncidentType }}\n \nCity: {{ City }}\n \nStreet: {{ Street }} ({{ CrossStreet }})\n \nUnits: {{ Units }}\n
\n\"\"\";\n\n#\n#\tConvert all the incidents from the fire module\n#\tto the path 'incidents' using the mapping rules defined above\n#\n#\t- incidents are used in \"gmaps.js\" and \"gmaps.html\"\n#\nmap from:\"fire.GetGeocodedIncidents\" to:\"incidents\" map:{\n\t\"latitude\" : \"{{ latitude }}\",\n\t\"longitude\" : \"{{ longitude }}\",\n\t\"title\" : \"{{ AlarmLevel}}: {{ IncidentType }} on {{ RawStreet }}\",\n\t\"uri\" : \"{{ HOME_URI }}#{{ IncidentNumber }}\",\n\t\"body\" : \"{{ *fitem.head.map|safe }}{{ *fitem.body|safe }}\",\n\t\"body_sb\" : \"{{ *fitem.head.sb|safe }}{{ *fitem.body|safe }}\",\n\t\"IncidentNumber\" : \"{{ IncidentNumber }}\"\n};\n\n#\n#\tLoad the 'gmaps' templates (for arbitrary geo-mapping),\n#\tusing the 'incidents' for its items and the specified meta.\n#\n#\t- used in in \"gmaps.js\" and \"gmaps.html\"\n#\nset to:\"map_meta\" value_render:true value:{\n\t\"id\" : \"maps\",\n\t\"latitude\" : 43.67,\n\t\"longitude\" : -79.38,\n\t\"uzoom\" : -13,\n\t\"gzoom\" : 13,\n\t\"api_key\" : \"{{ cfg.gmaps.api_key|otherwise:'ABQIAAA...pIxzZQ' }}\",\n\t\"html\" : {\n\t\t\"width\" : \"1024px\",\n\t\t\"height\" : \"800px\"\n\t}\n};\n\n#\n#\tProduce GMaps\n#\ntemplate script:\"gmaps\" items:\"incidents\" meta:\"map_meta\";\n\n#\n#\tProduce the final output\n#\ntemplate value:\"\"\"\n\n\n \n\t{{ gmaps.js|safe }}\n\n\n
\n\t
\n\t\t{{ gmaps.html|safe }}\n\t
\n\t
\n{% for incident in incidents %}\n\t
\n\t\t{{ incident.body_sb|safe }}\n\t
\n{% endfor %}\n
\n\n\n\"\"\";\n\n
The gmaps.jd (imported in the second last directive) looks like as follows (there will not be a test). It\u2019s designed to be a universal \u201cshow a map and plot points on in it\u201d inclusion. I\u2019ve added a few line breaks so the PRE box doesn\u2019t break.
",
"guidislink": false,
"id": "http://code.davidjanes.com/blog/?p=388",
"link": "http://code.davidjanes.com/blog/2008/12/23/pipe-cleaner-ii/",
"links": [
{
"href": "http://code.davidjanes.com/blog/2008/12/23/pipe-cleaner-ii/",
"rel": "alternate",
"type": "text/html"
}
],
"summary": "Here\u2019s the latest evolution of Pipe Cleaner, mainly recorded here for historical interest. The big change is that there isn\u2019t a separate outside template - everything is in the one index.jd file. The new directive is template, which can read and execute an outside module or actually produce the final output (as we see in [...]",
"title": "Pipe Cleaner (II)",
"updated": "2008-12-23T11:37:58+00:00",
"wfw:commentrss": "http://code.davidjanes.com/blog/2008/12/23/pipe-cleaner-ii/feed/"
},
{
"author": {
"@": "David Janes",
"uri": "http://code.davidjanes.com/blog"
},
"category": [
{
"label": null,
"scheme": null,
"term": "demo"
},
{
"label": null,
"scheme": null,
"term": "python"
}
],
"comments": "http://code.davidjanes.com/blog/2008/12/22/pytz-utcoffset/#comments",
"content": "
In the previous entry, we talked about the difficultly in finding out the delta from UTC for a timezone returned from the pytz module. In particular, consider the offset for St. John\u2019s, Newfoundland which should be at -3:30.
message: using datetime.utcoffset\n offset: -4:29 (-12660)
\n
I did a fair bit of Google searching for an answer without finding a satisfactory result, so I did further research on my own. To find the correct offset value, I found that this works:
message: using delta to UTC\n offset: 03:30 (12600)
\n
Note that if you\u2019re going to use the above method for finding deltas, you\u2019re going to have to take Daylight Savings Time into consideration also. I have not done this here, as I\u2019m a little pressed for time and just want to illustrate the problem.
\n
The issue seems to be with the way that pytz uses the Olson database entry (from here) for St. John\u2019s - and all other locations. It appears that pytz is using the first rule it sees, from 1884, rather than the rule for the date that was passed in. I think this is a bug.
\n
#\n# St John's has an apostrophe, but Posix file names can't have apostrophes.\n# Zone NAME GMTOFF RULES FORMAT [UNTIL]\nZone America/St_Johns -3:30:52 - LMT 1884\n -3:30:52 StJohns N%sT 1918\n -3:30:52 Canada N%sT 1919\n -3:30:52 StJohns N%sT 1935 Mar 30\n -3:30 StJohns N%sT 1942 May 11\n -3:30 Canada N%sT 1946\n -3:30 StJohns N%sT
",
"guidislink": false,
"id": "http://code.davidjanes.com/blog/?p=384",
"link": "http://code.davidjanes.com/blog/2008/12/22/pytz-utcoffset/",
"links": [
{
"href": "http://code.davidjanes.com/blog/2008/12/22/pytz-utcoffset/",
"rel": "alternate",
"type": "text/html"
}
],
"summary": "In the previous entry, we talked about the difficultly in finding out the delta from UTC for a timezone returned from the pytz module. In particular, consider the offset for St. John\u2019s, Newfoundland which should be at -3:30.\ndt_now = datetime.datetime.now()\ntz = pytz.timezone('America/St_Johns')\n\noffset = tz.utcoffset(dt_now)\n\nLog(\n \"using datetime.utcoffset\",\n offset = format(offset),\n)\nWith [...]",
"title": "Issues with utcoffset and pytz",
"updated": "2008-12-22T15:14:45+00:00",
"wfw:commentrss": "http://code.davidjanes.com/blog/2008/12/22/pytz-utcoffset/feed/"
},
{
"author": {
"@": "David Janes",
"uri": "http://code.davidjanes.com/blog"
},
"category": [
{
"label": null,
"scheme": null,
"term": "demo"
},
{
"label": null,
"scheme": null,
"term": "python"
}
],
"comments": "http://code.davidjanes.com/blog/2008/12/22/working-with-dates-times-and-timezones-in-python/#comments",
"content": "
Here\u2019s a few examples of working with dates, times and timezones in Python. We are using the following packages:
\n
\n
datetime (part of the standard Python distribution)
\n
dateutil - for date parsing, though there\u2019s a lot more depth to this package that I\u2019m not touching here
There\u2019s a lot of complexity to working with datetimes in any language; I\u2019m not going to get into that but would prefer instead to show a few practical examples. Keep the following in mind:
\n
\n
datetimes may or may not have timezones associated with them. If they do not, they are called \u201cnaive\u201d and their meaning is effectively defined by the program. In general, you want to work with non-naive datetimes. Generally the assumption would be that the naive datetime is in the application\u2019s current timezone or the user\u2019s preferred timezone
\n
when working with datetimes, consider the strategy of converting everything to the universal UTC timezone, then converting back to the user\u2019s timezone only when you need to display that to the user
\n
if you are rolling your own code for handling dates, times and timezones and you haven\u2019t done a lot of research, your implementation is garbage. Do yourself and everyone else a favor and use a library.
\n
\n
Our standard imports. Log is from the pybm library and it\u2019s purpose is rather obvious.
\n
from bm_log import Log\nimport dateutil.parser\nimport pytz\nimport datetime
\n
Here\u2019s an example of parsing the an e-mail or RSS type date using dateutil.
\n
dts = \"Thu, 13 Nov 2008 05:41:35 +0000\"\ndt = dateutil.parser.parse(dts)\n\nLog(\n \"Parsing an RFC type date\",\n src = dts,\n dt = dt,\n iso = dt.isoformat(),\n)
\n
\n message: Parsing an RFC type date\n dt: 2008-11-13 05:41:35+00:00\n iso: 2008-11-13T05:41:35+00:00\n src: Thu, 13 Nov 2008 05:41:35 +0000\n
\n
Here\u2019s an example of parsing an ISO Datetime
\n
dts = '2008-11-13T05:41:35-0400'\ndt = dateutil.parser.parse(dts)\n\nLog(\n \"Parsing an ISO Date with Timezone\",\n src = dts,\n dt = dt,\n iso = dt.isoformat(),\n)
\n
\n message: Parsing an ISO Date with Timezone\n dt: 2008-11-13 05:41:35-04:00\n iso: 2008-11-13T05:41:35-04:00\n src: 2008-11-13T05:41:35-0400\n
\n
Here\u2019s an example of parsing a naive timezone.
\n
dts = '2008-11-13T05:41:35'\ndt = dateutil.parser.parse(dts)\n\nLog(\n \"Parsing an ISO Date without a Timezone\",\n src = dts,\n dt = dt,\n iso = dt.isoformat(),\n)
\n
\n message: Parsing an ISO Date without a Timezone\n dt: 2008-11-13 05:41:35\n iso: 2008-11-13T05:41:35\n src: 2008-11-13T05:41:35\n
\n
Here\u2019s are two similar example, showing how to force the timezone if it\u2019s not present. This will happen in the first part, but not the second.
\n
tz = pytz.timezone('America/Toronto')\ndts = '2008-11-13T05:41:35'\ndt = dateutil.parser.parse(dts)\nif dt.tzinfo == None:\n dt = dt.replace(tzinfo = tz)\n\nLog(\n \"Parsing an ISO Date without a Timezone BUT specifying default TZ\",\n src = dts,\n dt = dt,\n iso = dt.isoformat(),\n tz = tz,\n)\n\ntz = pytz.timezone('America/Toronto')\ndts = '2008-11-13T05:41:35-0400'\ndt = dateutil.parser.parse(dts)\nif dt.tzinfo == None:\n dt = dt.replace(tzinfo = tz)\n\nLog(\n \"Parsing an ISO Date with a Timezone AND specifying default TZ\",\n src = dts,\n dt = dt,\n iso = dt.isoformat(),\n tz = tz,\n)
\n
\n message: Parsing an ISO Date without a Timezone BUT specifying default TZ\n dt: 2008-11-13 05:41:35-05:00\n iso: 2008-11-13T05:41:35-05:00\n src: 2008-11-13T05:41:35\n tz: America/Toronto\n\n message: Parsing an ISO Date with a Timezone AND specifying default TZ\n dt: 2008-11-13 05:41:35-04:00\n iso: 2008-11-13T05:41:35-04:00\n src: 2008-11-13T05:41:35-0400\n tz: America/Toronto\n
\n
Update: here\u2019s an example of moving datetimes to UTC and then to a different Timezone. Remember: you want your backend code to work with UTC datetimes for simplicity and correctness:
\n
\ndts = '2008-11-13T05:41:35-0400'\ndt_orig = dateutil.parser.parse(dts)\ndt_utc = dt.astimezone(pytz.UTC)\n\nLog(\n \"Changing a datetime to UTC\",\n src = dts,\n dt_orig = dt_orig,\n dt_utc = dt_utc,\n)\n\ntz_vancouver = pytz.timezone('America/Vancouver')\ndt_vancouver = dt_utc.astimezone(tz_vancouver)\n\nLog(\n \"Changing UTC datetime to a different timezone\",\n dt_vancouver = dt_vancouver,\n dt_utc = dt_utc,\n)\n
\n
\n message: Changing a datetime to UTC\n dt_orig: 2008-11-13 05:41:35-04:00\n dt_utc: 2008-11-13 09:41:35+00:00\n src: 2008-11-13T05:41:35-0400\n\n message: Changing UTC datetime to a different timezone\n dt_utc: 2008-11-13 09:41:35+00:00\n dt_vancouver: 2008-11-13 01:41:35-08:00\n
\n
Here is an example of listing all \u201ccommon\u201d timezones using pytz. Note that \u201cAmerica\u201d refers to the two continents, not the Irish word for the United States. Printing the actual timezone offset turned out to be a surprisingly complex task, which I will outline in a different blog post. For now let it suffice that with pytz try not to depend on utcoffset.
",
"guidislink": false,
"id": "http://code.davidjanes.com/blog/?p=377",
"link": "http://code.davidjanes.com/blog/2008/12/22/working-with-dates-times-and-timezones-in-python/",
"links": [
{
"href": "http://code.davidjanes.com/blog/2008/12/22/working-with-dates-times-and-timezones-in-python/",
"rel": "alternate",
"type": "text/html"
}
],
"summary": "Here\u2019s a few examples of working with dates, times and timezones in Python. We are using the following packages:\n\ndatetime (part of the standard Python distribution)\ndateutil - for date parsing, though there\u2019s a lot more depth to this package that I\u2019m not touching here\npytz - for timezone handling, and specifically making available the Olson timezone database [...]",
"title": "Working with dates, times and timezones in Python",
"updated": "2008-12-22T12:37:56+00:00",
"wfw:commentrss": "http://code.davidjanes.com/blog/2008/12/22/working-with-dates-times-and-timezones-in-python/feed/"
},
{
"author": {
"@": "David Janes",
"uri": "http://code.davidjanes.com/blog"
},
"category": [
{
"label": null,
"scheme": null,
"term": "demo"
},
{
"label": null,
"scheme": null,
"term": "djolt"
},
{
"label": null,
"scheme": null,
"term": "dqt"
},
{
"label": null,
"scheme": null,
"term": "html / javascript"
},
{
"label": null,
"scheme": null,
"term": "ideas"
},
{
"label": null,
"scheme": null,
"term": "jd"
},
{
"label": null,
"scheme": null,
"term": "maps"
},
{
"label": null,
"scheme": null,
"term": "pipe cleaner"
},
{
"label": null,
"scheme": null,
"term": "pybm"
},
{
"label": null,
"scheme": null,
"term": "work"
}
],
"comments": "http://code.davidjanes.com/blog/2008/12/18/pipe-cleaner/#comments",
"content": "
I\u2019ve been working (in my decreasing available spare time) on a project to pull together into a project called \u201cPipe Cleaner\u201d all the various concepts I\u2019ve been mentioning on this blog: Web Object Records (WORK) for API Access and object manipulation, Djolt for generating text from templates, Data/Query/Transform/Template (DQT) for transforming data and JD for scripting these elements together. The pieces came together this morning enough to put a demo together and here it is - the Toronto Fires Pt II Demo.
\n
How, you may ask, does this differ from the original Toronto Fires Demo? The answer is how it is put together, which we describe here.
\n
Index.dj
\n
This is the Djolt template that generates the output. The data fed to this template is generate by the JD script, described in the next section.
\n
\n\n \n {{ gmaps.js|safe }}\n\n\n
\n
\n {{ gmaps.html|safe }}\n
\n
\n{% for incident in incidents %}\n
\n {{ incident.body_sb|safe }}\n
\n{% endfor %}\n
\n\n\n
Quite simple \u2026 as you can see, most of the data is being pulled in from elsewhere. The elsewhere is provided by the script described in the next section.
\n
Index.jd
\n
This is the script that pull all the pieces together. Note that I\u2019m not 100% happy with the way the data is imported, I would like the geocoding to become part of this data flow too. In the next release perhaps.
\n
First we pull in the \u201cfire\u201d module that we wrote in the previous Map examples. This is doing exactly what you think: importing a Python module. We may have to increase the security or restrict this to working with an API for general purpose use.
\n
import module:\"fire\";
\n
Next we define two headers - one that is going to appear in the Google Maps popup, the next that is going to appear in the sidebar. They need to be different as they refer to themselves. Note that the sidebar header \u201cbreaks\u201d the encapsulation of Google Maps - this seems to be unavoidable. The to:\"fitem.head.map\" and to:\"fitem.head.sb\" are manipulating a WORK dictionary to store values.
\n
Note also here that we\u2019ve extended JD to accept Python multiline strings - this was unavoidable if JD was to be useful to me.
The next block defines the text of the body used to describe a fire incident. It follows much the same pattern as the previous block.
\n
set to:\"fitem.body\" value:\"\"\"\n
\nAlarm Level: {{ AlarmLevel }}\n \nIncident Type: {{ IncidentType }}\n \nCity: {{ City }}\n \nStreet: {{ Street }} ({{ CrossStreet }})\n \nUnits: {{ Units }}\n
\n\"\"\";
\n
This is a map: it is translating the values in fire.GetGeocodeIncidents into a new format and storing that in incidents. The format that we were are storing it in is understood by the Google Maps generating module.
\n
We may rename this translate, as the word map is somewhat overloaded.
Next we set up the \u201cmeta\u201d (see WORK meta description if you\u2019re not following along) for the maps. The render_value:true declaration makes PC interpret the templates in strings). We then call our Google Maps generating code (which are actually more Pipe Cleaners) and that gets fed to the Djolt template we first showed you. Clear? Maybe not, we\u2019ll have more examples coming\u2026
",
"guidislink": false,
"id": "http://code.davidjanes.com/blog/?p=372",
"link": "http://code.davidjanes.com/blog/2008/12/18/pipe-cleaner/",
"links": [
{
"href": "http://code.davidjanes.com/blog/2008/12/18/pipe-cleaner/",
"rel": "alternate",
"type": "text/html"
}
],
"summary": "I\u2019ve been working (in my decreasing available spare time) on a project to pull together into a project called \u201cPipe Cleaner\u201d all the various concepts I\u2019ve been mentioning on this blog: Web Object Records (WORK) for API Access and object manipulation, Djolt for generating text from templates, Data/Query/Transform/Template (DQT) for transforming data and JD for [...]",
"title": "Pipe Cleaner",
"updated": "2008-12-18T23:38:21+00:00",
"wfw:commentrss": "http://code.davidjanes.com/blog/2008/12/18/pipe-cleaner/feed/"
},
{
"author": {
"@": "David Janes",
"uri": "http://code.davidjanes.com/blog"
},
"category": [
{
"label": null,
"scheme": null,
"term": "administrivia"
}
],
"comments": "http://code.davidjanes.com/blog/2008/12/16/woah/#comments",
"content": "
SIMILE Timeline is \u201cthe Google Maps for time based events\u201d. It used to be housed at MIT but now it\u2019s graduated to Google Code. I\u2019ve created an example application showing this year\u2019s Oscar awards and a number of movies that are, umm, are not all likely to be nominated.
\n
\n
the application source can be seen here; it is based on this demo and some work I had done previously. Note:\n
\n
multiple scrolling bands linked together
\n
custom icons
\n
custom colors
\n
\n
\n
we demonstrate populating the timeline widget using JSON data coded in the application; the most difficult part of putting this demo together was cutting and pasting all this data, a task we hope to make easier with our DQT code
\n
the documentation for Timeline is starting to diverge from the source code; showEventText is now replaced by overview in the band creation code
\n
there\u2019s a large number of weaknesses (still) with Timeline for dealing with arbitrary data, these may be corrected in the Javascript code but I don\u2019t have time to go through all this\n
\n
note the incorrect placement of custom icons; using Firebug to inspect the HTML, I discovered that unfortunately everything is placed using style tags so it\u2019s difficult to correct using CSS. Ideally I would like to be able to assign classes and ID tags to everything
\n
there doesn\u2019t seem to be an obvious way to control the widths of the labels; in fact, if I reduce the band spacing in the top band, the text starts to overlap in a horrible manner
\n
popup information boxes get confused when there is too little space to display information
\n
when I don\u2019t add a description to events, it uses \u201cundefined\u201d (see the Oscar Nominations Period)
\n
date display functions are inferring more resolution (i.e. an actual time as opposed to the just the date) that I\u2019m giving it
\n
\n
\n
\n
If anyone has corrections for me I\u2019ll update the demo
",
"guidislink": false,
"id": "http://code.davidjanes.com/blog/?p=345",
"link": "http://code.davidjanes.com/blog/2008/12/13/brief-notes-on-simile-timeline/",
"links": [
{
"href": "http://code.davidjanes.com/blog/2008/12/13/brief-notes-on-simile-timeline/",
"rel": "alternate",
"type": "text/html"
}
],
"summary": "SIMILE Timeline is \u201cthe Google Maps for time based events\u201d. It used to be housed at MIT but now it\u2019s graduated to Google Code. I\u2019ve created an example application showing this year\u2019s Oscar awards and a number of movies that are, umm, are not all likely to be nominated.\n\nthe application source can be seen here; [...]",
"title": "Brief notes on SIMILE Timeline",
"updated": "2008-12-13T13:01:01+00:00",
"wfw:commentrss": "http://code.davidjanes.com/blog/2008/12/13/brief-notes-on-simile-timeline/feed/"
},
{
"author": {
"@": "David Janes",
"uri": "http://code.davidjanes.com/blog"
},
"category": [
{
"label": null,
"scheme": null,
"term": "ideas"
},
{
"label": null,
"scheme": null,
"term": "jd"
}
],
"comments": "http://code.davidjanes.com/blog/2008/12/12/jd-json-declaration-language/#comments",
"content": "
I\u2019ve just added a new module called bm_jd to the pybm project. It implements a \u201clittle language\u201d for declaring information, like a configuration file, when the details are all specified in JSON.
\n
The language is very simple, consisting of semi-colon terminated statements; each statement having a command and zero or more arguments. Each argument may or may not have JSON data - if it does, it will be set off with a colon.
::= *\n ::= ( | : )*\n | ::= [a-zA-Z0-9_]\n ::= ... any valid JSON data ...
\n
You can use the pybm JD parser in several ways:
\n
\n
implement a subclass of JDParser, defining CustomizeProduce; or
\n
implement a subclass of DispatchJDParser, defining a call_ method for each command you plan to allow
\n
\n
In either case, you call a method FeedString to get the parser rolling.
\n
There\u2019s also a LogJDParser, which just dumps parsing results. Here\u2019s an example of a JD document. Don\u2019t worry about Djolt code in the JSON, that\u2019s just text as far as this example is concerned:
",
"guidislink": false,
"id": "http://code.davidjanes.com/blog/?p=342",
"link": "http://code.davidjanes.com/blog/2008/12/12/jd-json-declaration-language/",
"links": [
{
"href": "http://code.davidjanes.com/blog/2008/12/12/jd-json-declaration-language/",
"rel": "alternate",
"type": "text/html"
}
],
"summary": "I\u2019ve just added a new module called bm_jd to the pybm project. It implements a \u201clittle language\u201d for declaring information, like a configuration file, when the details are all specified in JSON.\nThe language is very simple, consisting of semi-colon terminated statements; each statement having a command and zero or more arguments. Each argument may or [...]",
"title": "JD - JSON Declaration Language",
"updated": "2008-12-12T22:55:28+00:00",
"wfw:commentrss": "http://code.davidjanes.com/blog/2008/12/12/jd-json-declaration-language/feed/"
},
{
"author": {
"@": "David Janes",
"uri": "http://code.davidjanes.com/blog"
},
"category": [
{
"label": null,
"scheme": null,
"term": "macintosh"
},
{
"label": null,
"scheme": null,
"term": "tips"
}
],
"comments": "http://code.davidjanes.com/blog/2008/12/11/mount-ntfs-and-remote-filesystems-using-macfuse/#comments",
"content": "
Earlier this week I bought a LaCie 500Gb USB drive so I could bring VMWare images between work and home. When I went to copy the image, the copy failed with no meaningful error message (Error 0, I believe). Trying the copy on the command line was a little more informative: as it turns out, the LaCie drive ships with a FAT-32 file system which can only handle files up to 4Gb in size. As the image I was trying to copy had a 8Gb file in it, this was a no go.
\n
My initial thought was to use the UNIX commands tar and split to break the files into individual smaller chunks, but this is hardly a satisfactory answer. If I formatted the drive to the Mac filesystem, the Windows machines would not be able to read it at all. If I formatted the drive the \u201cnew\u201d NTFS filesystem, Windows can read and write just fine but the Macintosh wouldn\u2019t be able to write to it.
\n
Fortunately, there\u2019s an excellent install for the Mac called MacFUSE that allows access to all sorts of filesystem types not natively supported by the Macintosh, include NTFS. Here\u2019s how I set up MacFUSE.
Installation by itself does nothing except set you up for the next stage: installing drivers for particular file systems.
\n
NTFS
\n
You have to search through the documentation for a bit to figure out where to get NTFS to with Windows filesystems. It actually turns out to be rather easy:
click on the link, download and install the package
\n
reboot
\n
\n
You can now write to NTFS drives. It\u2019s a little slow - it\u2019s taking me about 2 hours to copy 8Gb to the La Cie drive, but that\u2019s better than not being able to do it at all. You wouldn\u2019t want to work live off the drive however, and it may be worth investigating commercial NTFS compatibility applications if you need to do this.
\n
To reformat your La Cie drive plotline, use Applications > Disk Utility to erase and install an empty NTFS file system.
\n
SSHFS
\n
SSHFS lets you see remote filesystems through SSH.
download the version appropriate to your Mac; you can store this in your home directory or if you\u2019re a little more organized about your path, a directory link ~/bin
\n
make a mount point - this is just a directory on your Mac that is needed by MacFUSE; it can be hidden as Mac OS will show you the mounted drive on your desktop and in /Volumes. For example, on the command line run mkdir -p ~/.Volumes/Remote.
\n
run the mount command; you\u2019ll be prompted for your remote system password
\n
\n
You\u2019ll see the drive appearing on your desktop. I\u2019ve actually created a shell alias to do the mounting for me called \u201cmount-xxx\u201d. If you don\u2019t know how to do this, it\u2019s probably too much to go into right now.
\n
The nice thing about SSHFS is that I could see being able to run an entire Mac desktop development shop with all the backend computing running Linux, all being accessed nicely through SSHFS.
",
"guidislink": false,
"id": "http://code.davidjanes.com/blog/?p=331",
"link": "http://code.davidjanes.com/blog/2008/12/11/mount-ntfs-and-remote-filesystems-using-macfuse/",
"links": [
{
"href": "http://code.davidjanes.com/blog/2008/12/11/mount-ntfs-and-remote-filesystems-using-macfuse/",
"rel": "alternate",
"type": "text/html"
}
],
"summary": "Earlier this week I bought a LaCie 500Gb USB drive so I could bring VMWare images between work and home. When I went to copy the image, the copy failed with no meaningful error message (Error 0, I believe). Trying the copy on the command line was a little more informative: as it turns out, [...]",
"title": "Mount NTFS and remote filesystems using MacFUSE",
"updated": "2008-12-11T16:56:24+00:00",
"wfw:commentrss": "http://code.davidjanes.com/blog/2008/12/11/mount-ntfs-and-remote-filesystems-using-macfuse/feed/"
},
{
"author": {
"@": "David Janes",
"uri": "http://code.davidjanes.com/blog"
},
"category": [
{
"label": null,
"scheme": null,
"term": "demo"
},
{
"label": null,
"scheme": null,
"term": "djolt"
},
{
"label": null,
"scheme": null,
"term": "dqt"
},
{
"label": null,
"scheme": null,
"term": "ideas"
},
{
"label": null,
"scheme": null,
"term": "semantic web"
},
{
"label": null,
"scheme": null,
"term": "work"
}
],
"comments": "http://code.davidjanes.com/blog/2008/12/11/brief-survey-of-yahoo-pipes/#comments",
"content": "
Yahoo Pipes is a visual editor of mashups, allowing you to take data from sources on the net, transform them in various interesting ways and output the result as Atom, RSS or JSON. The primary downside Pipes of course is that you\u2019re totally dependent on Yahoo for the infrastructure: it runs at Yahoo pulling feeds that have to be accessable through the public Internet.
\n
It\u2019s easy to use Pipes: just go to this page and start working with the sample example Pipe. You\u2019ll need a Yahoo login ID, but most of us have that anyway. I\u2019ve created an example that uses Yahoo Pipes to feed a Djolt template which you can see here.
Location Extractor - a geocoder that magically looks for locations
\n
Loop
\n
Regex
\n
Rename
\n
Reverse
\n
Sort
\n
Split
\n
Sub-element - pulls a particular sub-element of an item and makes that the item. This is very much like WORK path manipulation
\n
Tail
\n
Truncate
\n
Union
\n
Unique
\n
Web Service
\n
\n
Plus a number of specialized data services, for dealing with elements such as dates.
\n
Templates
\n
Pipes does not provide an arbitrary Djolt-like template producing HTML. Instead, they provide a number of pre-made code templates that output well known data types, including RSS, JSON and Atom (and some stranger choices, like PHP).
",
"guidislink": false,
"id": "http://code.davidjanes.com/blog/?p=325",
"link": "http://code.davidjanes.com/blog/2008/12/11/brief-survey-of-yahoo-pipes/",
"links": [
{
"href": "http://code.davidjanes.com/blog/2008/12/11/brief-survey-of-yahoo-pipes/",
"rel": "alternate",
"type": "text/html"
}
],
"summary": "Yahoo Pipes is a visual editor of mashups, allowing you to take data from sources on the net, transform them in various interesting ways and output the result as Atom, RSS or JSON. The primary downside Pipes of course is that you\u2019re totally dependent on Yahoo for the infrastructure: it runs at Yahoo pulling feeds [...]",
"title": "A brief survey of Yahoo Pipes as a DQT",
"updated": "2008-12-11T12:19:55+00:00",
"wfw:commentrss": "http://code.davidjanes.com/blog/2008/12/11/brief-survey-of-yahoo-pipes/feed/"
},
{
"author": {
"@": "David Janes",
"uri": "http://code.davidjanes.com/blog"
},
"category": [
{
"label": null,
"scheme": null,
"term": "dqt"
},
{
"label": null,
"scheme": null,
"term": "ideas"
}
],
"comments": "http://code.davidjanes.com/blog/2008/12/09/introducing-dqt-dataquerytransformtemplate/#comments",
"content": "
Data/Query/Transform/Template - DQT, dropping the final T - is a commonly used pattern for displaying data on a website. The elements of this pattern are:
\n
\n
a Data source, such a blog database, an e-mail store, the Internet as a whole, a MySQL, and in particular the results of an API call.
\n
a Query, which is a way of selecting a particular subset or slice of the data (typically homogeneous)
\n
Transform rules, which can make the data look different by renaming fields, enhancing data using tools such as geolocation, filtering records out, merging multiple data sources and so forth.
\n
a Template, which is a way of converting to a useful end-user format, such as HTML, JSON or XML
\n
\n
In the particular context of what I\u2019m writing about, we can assume that we\u2019re manipulating WORK items - that is, an an API returns a \u201cMeta\u201d block of information and a stream of \u201cItems\u201d, each in turn which are WORK items also. By identifying common patterns of dynamic page construction, my hope is that we can simplify page and mashup creation.
\n
You\u2019ve seen this pattern many times. My plan is that be describing it properly, we can make it easier to do.
\n
Wordpress Blogs
\n
\n
The Data source is the MySQL table with blog posts, plus ancillary information pulled from other tables.
\n
The Query is some combination of ( page number, post path, category, tag ). Not all combinations are legal obviously, but this is the information that can be encoded in a URL request. The Data source and the Query result a number of posts being made available for further processing
\n
The Template is the PHP code that converts the individual database items into HTML for display
BTW: Don\u2019t take this as a how-to guide for Wordpress. I\u2019m trying to look at this from a high-level conceptual point-of-view.
\n
Google Mail
\n
\n
the Data source is all the e-mails in the Google database, probably billions or trillions of messages
\n
the Query is some combination of ( userid, page number, search ). The userid is not encoded in the URL, it is known be
\n
the Template is the Javascript code that
\n
\n
That\u2019s a really high level view: in fact, Google Mail does this DQT twice: the first time around to select JSON or XML data to be transmitted to the user\u2019s browser; the second time around to locally on the user\u2019s browser select and display items.
\n
Yahoo News
\n
\n
the Data source is Yahoo\u2019s news database
\n
the Query is a category, or not category at all (poorly encoded the URL, I may add)
\n
the Transform groups news into like categories (play along with me here)
\n
the Template is the Yahoo\u2019s HTML generator, whatever that may be
The Data and the Query are substantially similar to the Wordpress Blog example, but:
\n
\n
we Transform the fields into a format that can be understood by an RSS generator
\n
the Template is a specialized object that converts WORK items into RSS entries, that is, we don\u2019t (or shouldn\u2019t) use a Djolt-like template to generate XML.
\n
\n
This is obviously a somewhat of a hypothetical example, but reflects my recent ideas about how machine readable data should be generated.
",
"guidislink": false,
"id": "http://code.davidjanes.com/blog/?p=318",
"link": "http://code.davidjanes.com/blog/2008/12/09/introducing-dqt-dataquerytransformtemplate/",
"links": [
{
"href": "http://code.davidjanes.com/blog/2008/12/09/introducing-dqt-dataquerytransformtemplate/",
"rel": "alternate",
"type": "text/html"
}
],
"summary": "Data/Query/Transform/Template - DQT, dropping the final T - is a commonly used pattern for displaying data on a website. The elements of this pattern are:\n\na Data source, such a blog database, an e-mail store, the Internet as a whole, a MySQL, and in particular the results of an API call.\na Query, which is a way [...]",
"title": "Introducing DQT - Data/Query/Transform/Template",
"updated": "2008-12-09T21:05:44+00:00",
"wfw:commentrss": "http://code.davidjanes.com/blog/2008/12/09/introducing-dqt-dataquerytransformtemplate/feed/"
},
{
"author": {
"@": "David Janes",
"uri": "http://code.davidjanes.com/blog"
},
"category": [
{
"label": null,
"scheme": null,
"term": "djolt"
},
{
"label": null,
"scheme": null,
"term": "dqt"
},
{
"label": null,
"scheme": null,
"term": "ideas"
},
{
"label": null,
"scheme": null,
"term": "pybm"
},
{
"label": null,
"scheme": null,
"term": "python"
},
{
"label": null,
"scheme": null,
"term": "work"
}
],
"comments": "http://code.davidjanes.com/blog/2008/12/08/coding-backwards-for-simplicity/#comments",
"content": "
I haven\u2019t been posting as much as I like here for the last three weeks, not because of lack of ideas but because I haven\u2019t been able to consolidate what I\u2019ve been working on into a coherent thought. I\u2019m trying to come up with a overreaching conceptual arch that covers WORK, Djolt and the various API interfaces I\u2019ve been coded. Tentatively and horribly, I\u2019m calling this Data/Query/Transform/Template right now though I\u2019m expecting this to change.
\n
The first demo of this \u2026 without further explanation \u2026 can be seen here. More details about what this is actually demonstrating (besides formatting this blog) will be forthcoming.
\n
What I want to draw attention to in this post is how I coded this. What I\u2019ve been doing for the last several weeks is coding backwards: I start with what I want the final code to look like and then figure out all the libraries, little languages and so forth that would be needed to code that. After several false starts, my conceptual logjam broke about a week ago and code started radically simplifying.
\n
The ideal code, in my mind, is almost entirely static declarations: no loops, no if statements, no while statements, no goto-type statements (god help us). We simply specify how the parts are connected, and hope that we can abstract the complexity into the libraries that make this all happen. The code that you see below is actually post all my conceptualizing: I just wanted to write some code and since I had almost all the parts together it fell together quite nicely:
There\u2019s almost nothing there! In particular, note:
\n
\n
bm_wsgi.SimpleWrapper handles all the WSGI interface work, including determining when to output HTML headers, error trapping, and Unicode to UTF-8 encoding
\n
the most complicated part of the application is setting up the Context. In particular, note that self.paramd is automatically populated by the QUERY_STRING passed to the application, and the double setting we do here allows us to have default values.
\n
If you want to see the HTML template that drives the application it is here. Note two variations from Django templates: the {% asis %} block which doesn\u2019t intrepret it\u2019s content as Djolt code and the {{ *paramd.template|safe }} variable whichinterprets the variable\u2019s contents as a template.
\n
Methods called Customize-something are my convention for framework functions, i.e. methods that will be called for us rather than methods we call.
\n
",
"guidislink": false,
"id": "http://code.davidjanes.com/blog/?p=311",
"link": "http://code.davidjanes.com/blog/2008/12/08/coding-backwards-for-simplicity/",
"links": [
{
"href": "http://code.davidjanes.com/blog/2008/12/08/coding-backwards-for-simplicity/",
"rel": "alternate",
"type": "text/html"
}
],
"summary": "I haven\u2019t been posting as much as I like here for the last three weeks, not because of lack of ideas but because I haven\u2019t been able to consolidate what I\u2019ve been working on into a coherent thought. I\u2019m trying to come up with a overreaching conceptual arch that covers WORK, Djolt and the various [...]",
"title": "Coding backwards for simplicity",
"updated": "2008-12-08T21:58:57+00:00",
"wfw:commentrss": "http://code.davidjanes.com/blog/2008/12/08/coding-backwards-for-simplicity/feed/"
},
{
"author": {
"@": "David Janes",
"uri": "http://code.davidjanes.com/blog"
},
"category": [
{
"label": null,
"scheme": null,
"term": "ideas"
},
{
"label": null,
"scheme": null,
"term": "python"
}
],
"comments": "http://code.davidjanes.com/blog/2008/12/08/json-encode-iterators/#comments",
"content": "
As part of my recent explorations, I\u2019ve been playing a lot with Python iterators/generators. The key efficiency of iterators is that when working with lengthy list-like objects, you need only create the part that\u2019s being looked at. It\u2019s just-in-time objects.
\n
If you attempt to JSON serialize an object with an iterator/generator object in it, the json module throws a cog: it doesn\u2019t know how to serialize these types of objects. The json module is extensible and the documentation makes a suggestion how to do this:
This seems somewhat ugly to me. In particular, lots of objects can be wrapped by the iter function that don\u2019t need to be, plus lots of objects will cause that TypeError to be thrown which seems to be rather a bit of waste. Here\u2019s the solution I came up with:
\n
class IterEncoder(json.JSONEncoder):\n def default(self, o):\n try:\n return json.JSONEncoder.default(self, o)\n except TypeError, x:\n try:\n return list(o)\n except:\n return x
\n
This tries to encode the object the normal way. Only if that doesn\u2019t work do we try to turn the object into a list. If that\u2019s not convertible (i.e. the list object constructor fails) we go back and throw the original exception provided by JSONEncoder - we\u2019ve really failed.
Don\u2019t be overly tempted to check the type of o: it may be types.GeneratorType or types.XRangeType or perhaps even something else that I haven\u2019t found out yet.
",
"guidislink": false,
"id": "http://code.davidjanes.com/blog/?p=308",
"link": "http://code.davidjanes.com/blog/2008/12/08/json-encode-iterators/",
"links": [
{
"href": "http://code.davidjanes.com/blog/2008/12/08/json-encode-iterators/",
"rel": "alternate",
"type": "text/html"
}
],
"summary": "As part of my recent explorations, I\u2019ve been playing a lot with Python iterators/generators. The key efficiency of iterators is that when working with lengthy list-like objects, you need only create the part that\u2019s being looked at. It\u2019s just-in-time objects.\nIf you attempt to JSON serialize an object with an iterator/generator object in it, the json [...]",
"title": "How to JSON encode iterators",
"updated": "2008-12-08T19:32:03+00:00",
"wfw:commentrss": "http://code.davidjanes.com/blog/2008/12/08/json-encode-iterators/feed/"
},
{
"author": {
"@": "David Janes",
"uri": "http://code.davidjanes.com/blog"
},
"category": [
{
"label": null,
"scheme": null,
"term": "maps"
}
],
"comments": "http://code.davidjanes.com/blog/2008/12/07/once-more-with-the-maps/#comments",
"content": "
With a back-to-the-future technology called JavaFX to be launched Thursday, Sun Microsystems hopes to attract a new class of developer while building a much-needed new revenue source.
\n
JavaFX 1.0 returns to the sales pitch that Sun used during Java\u2019s launch more than 13 years ago: a foundation for software on a wide variety of computing \u201cclients\u201d such as desktop computers or mobile phones. JavaFX builds on current Java technology but adds two major pieces.
\n
First is a new software foundation designed to run so-called rich Internet applications\u2013network-enabled programs with lush user interfaces. Second is a new programming language called JavaFX Script that\u2019s intended to be easier to use than traditional Java.
\n
The benefit for me (and maybe you) is that most web shops have three different classes of developers: backend programmers, HTML/CSS coders, and The Flash Guy. The problem with The Flash Guy is that they have to be involved to make any changes to Flash components; JavaFX instead provides components that can easily be manipulated by coders and by web designers (with a little training I think).
",
"guidislink": false,
"id": "http://code.davidjanes.com/blog/?p=301",
"link": "http://code.davidjanes.com/blog/2008/12/06/javafx-10-released/",
"links": [
{
"href": "http://code.davidjanes.com/blog/2008/12/06/javafx-10-released/",
"rel": "alternate",
"type": "text/html"
}
],
"summary": "CNET has a very detailed article on JavaFX, Sun\u2019s better-late-than-never-maybe-hopefully answer to Adobe\u2019s Flash / AIR and Microsoft\u2019s Silverlight:\nWith a back-to-the-future technology called JavaFX to be launched Thursday, Sun Microsystems hopes to attract a new class of developer while building a much-needed new revenue source.\nJavaFX 1.0 returns to the sales pitch that Sun used during [...]",
"title": "JavaFX 1.0 released",
"updated": "2008-12-06T11:54:34+00:00",
"wfw:commentrss": "http://code.davidjanes.com/blog/2008/12/06/javafx-10-released/feed/"
},
{
"author": {
"@": "David Janes",
"uri": "http://code.davidjanes.com/blog"
},
"category": [
{
"label": null,
"scheme": null,
"term": "db"
},
{
"label": null,
"scheme": null,
"term": "freebase"
},
{
"label": null,
"scheme": null,
"term": "semantic web"
}
],
"comments": "http://code.davidjanes.com/blog/2008/12/06/all-your-base/#comments",
"content": "
Freebase is a user-editable, user-extensible structured database, a sort of one-stop shop semantic web/Wikipedia application. I started playing with Freebase about a year ago and the application has made significant strides over that period, especially in the usability department. Freebase also provides a very nice API which I\u2019m using in GenX, with the caveat that it\u2019s currently almost useless because of query timeouts.
\n
I just came across the following page on Freebase: http://vancouver.freebase.com/. This page is what Freebase calls a Base, which is a collection of Tables/Views, which are things like \u201cVancouver Bloggers\u201c, \u201cMayoral Candidates 2008\u201d and so forth. A Table/View is a list of Topics, which are basically the equivalent of a Wikipedia page. Get all that? It makes sense after a while
\n
A few observations:
\n
\n
Why have I written Table/View above? Because in some places it\u2019s called a Table and other places it\u2019s called a View. Which is it? I\u2019m guessing View but it\u2019s still not 100% clear.
\n
I decided to create our own Toronto Base especially for the TorCamp community. Given that you get your own top-level domain name there\u2019s somewhat of an incentive to be a first-mover on this
\n
When you create a Base, it provides a list of suggested Views that can be added. Nice. Unfortunately, it added each View twice. I then had to go delete the duplicate View manually. Not so nice. And then even though I\u2019ve deleted the View it still shows up on a detail page. Sigh.
\n
On thus plus side, this is all done in a nice-Ajaxy way
My initial opinion was that Views seem to be copies, not references: this turns out to be a wrong assumption on my part. Views are in fact (if I got this right) the results of a query on the Freebase db. This means that as more Topics match the View query, they\u2019ll automatically show up. The query is a copy, not a reference, but this is a good thing.
\n
The implication is that it\u2019s difficult to create a View that is an arbitrary \u201cbag\u201d of topics. For example, if I want to create a Toronto Bloggers View, I have to actually make sure that all the Topics that will show up are marked with some attribute that can be matched to give them a Toronto-bloggerness quality.
\n
",
"guidislink": false,
"id": "http://code.davidjanes.com/blog/?p=295",
"link": "http://code.davidjanes.com/blog/2008/12/06/all-your-base/",
"links": [
{
"href": "http://code.davidjanes.com/blog/2008/12/06/all-your-base/",
"rel": "alternate",
"type": "text/html"
}
],
"summary": "Freebase is a user-editable, user-extensible structured database, a sort of one-stop shop semantic web/Wikipedia application. I started playing with Freebase about a year ago and the application has made significant strides over that period, especially in the usability department. Freebase also provides a very nice API which I\u2019m using in GenX, with the caveat that [...]",
"title": "All your Base are belong to us",
"updated": "2008-12-06T11:26:49+00:00",
"wfw:commentrss": "http://code.davidjanes.com/blog/2008/12/06/all-your-base/feed/"
},
{
"author": {
"@": "David Janes",
"uri": "http://code.davidjanes.com/blog"
},
"category": [
{
"label": null,
"scheme": null,
"term": "demo"
},
{
"label": null,
"scheme": null,
"term": "djolt"
},
{
"label": null,
"scheme": null,
"term": "ideas"
},
{
"label": null,
"scheme": null,
"term": "python"
}
],
"comments": "http://code.davidjanes.com/blog/2008/12/04/djolt-indirection/#comments",
"content": "
I\u2019ve been working through a sticky problem with Djolt, trying to implement my Toronto Fires example in as few lines as possible. As part of this, I\u2019ve come up with the idea of adding indirection to Djolt templates:
\n
import djolt\n\nd = {\n \"a\" : \"It says: {{ b }}\",\n \"b\" : \"Hello, World\"\n}\n\nt = djolt.Template(\"\"\"\na: {{ a }}\nb: {{ b }}\n*a: {{ *a }}\n\"\"\")\n\nprint t.Render(d)\n\"\"\")\n\nprint t.Render(d)
\n
Which yields:
\n
a: It says: {{ b }}\nb: Hello, World\n*a: It says: Hello, World\n
\n
This is significantly updated from the original version I posted here an hour ago. The indirection now makes the variable read as a template. This is a much more powerful concept.
",
"guidislink": false,
"id": "http://code.davidjanes.com/blog/?p=289",
"link": "http://code.davidjanes.com/blog/2008/12/04/djolt-indirection/",
"links": [
{
"href": "http://code.davidjanes.com/blog/2008/12/04/djolt-indirection/",
"rel": "alternate",
"type": "text/html"
}
],
"summary": "I\u2019ve been working through a sticky problem with Djolt, trying to implement my Toronto Fires example in as few lines as possible. As part of this, I\u2019ve come up with the idea of adding indirection to Djolt templates:\nimport djolt\n\nd = {\n \"a\" : \"It says: {{ b }}\",\n \"b\" [...]",
"title": "Djolt Indirection",
"updated": "2008-12-04T11:05:28+00:00",
"wfw:commentrss": "http://code.davidjanes.com/blog/2008/12/04/djolt-indirection/feed/"
},
{
"author": {
"@": "David Janes",
"uri": "http://code.davidjanes.com/blog"
},
"category": [
{
"label": null,
"scheme": null,
"term": "uncategorized"
}
],
"comments": "http://code.davidjanes.com/blog/2008/12/04/python-3000-we-are-no-longer-flying/#comments",
"content": "
Python 3000 - the next generation, backwards incompatiable with all previous versions of Python - is now available. I think I\u2019ll sit this one out for a while.
\n
",
"guidislink": false,
"id": "http://code.davidjanes.com/blog/?p=286",
"link": "http://code.davidjanes.com/blog/2008/12/04/python-3000-we-are-no-longer-flying/",
"links": [
{
"href": "http://code.davidjanes.com/blog/2008/12/04/python-3000-we-are-no-longer-flying/",
"rel": "alternate",
"type": "text/html"
}
],
"summary": "Python 3000 - the next generation, backwards incompatiable with all previous versions of Python - is now available. I think I\u2019ll sit this one out for a while.",
"title": "Python 3000 - we are no longer flying",
"updated": "2008-12-04T10:45:48+00:00",
"wfw:commentrss": "http://code.davidjanes.com/blog/2008/12/04/python-3000-we-are-no-longer-flying/feed/"
}
]
![](\"http://code.davidjanes.com/resources/Terminal%20Replacement%20Icons/Terminal%20Replacement_128x128.png\")
I\u2019m a command line guy - I spend 90% of my non-blog reading day in Terminal, working on Python apps on my Mac or SSHed into work and working on Java and Javascript applications. I do realize the benefit of \u201creal\u201d applications, for image editing, for advanced text processing and so forth. On the Mac you can send files to the default application easily:![\"Wordpress\"](\"http://code.davidjanes.com/resources/wordpress-100.png\")
This site has been upgraded to ![\"\"](\"http://code.davidjanes.com/examples/2008-12-13/timeline/timeline.png\" "\\"Timeline\\"")
![\"MacFUSE\"](\"http://code.davidjanes.com/examples/2008-12-13/macfuse-icon.png\"){kind=icon}
Earlier this week I bought a LaCie 500Gb USB drive so I could bring VMWare images between work and home. When I went to copy the image, the copy failed with no meaningful error message (Error 0, I believe). Trying the copy on the command line was a little more informative: as it turns out, the LaCie drive ships with a FAT-32 file system which can only handle files up to 4Gb in size. As the image I was trying to copy had a 8Gb file in it, this was a no go.![\"MacFUSE\"](\"http://code.davidjanes.com/examples/2008-12-13/pipes.png\")
![\"\"](\"http://imgs.xkcd.com/comics/python.png\")