David Janes' Code Weblog

December 29, 2008

Interesting links from the last month

db,ideas,semantic web · David Janes · 9:29 am ·
  • Aspena web server for highly extensible Python-based publication, application, and hybrid websites. As a potential alternative to Python’s builtin HTTPServer. MIT license.
  • V8V8 is Google’s open source JavaScript engine; written in C++; can run standalone, or can be embedded into any C++ application. I am very excited by this, as allowing users to send code to the server to execute Javascript is an amazingly powerful idea. If anyone knows of a Python wrapper, let me know please. New BSD license.
  • KomodoEdit (a testimonial) – I am going to try this out, though vi/vim will always be my first love (JJ also has an article on using ctags).
  • Virtuoso - an innovative Universal Server platform that delivers an enterprise level Data Integration and Management solution for SQL, RDF, XML, Web Services, and Business Processes. There’s way to much bla bla bla in that sentence, but apparently this is really sweet at handling SPARQL/RDF triples. Kingsley Idehen writes extensively about this on his blog (e.g.).
  • Drizzlea database optimized for Cloud and Net applications. Way too early to commit to this yet. See The New MySQL Landscape for more interesting going ons.
  • AuthKitauthentication and authorization toolkit for WSGI applications and frameworks.
  • Geodjangoa world-class geographic web framework. Lots of great ideas and pointers to libraries in here, even if you’re not planning to use this itself.
  • Disco – an open-source implementation of the Map-Reduce framework for distributed computing. The Disco core is written in Erlang, a functional language that is designed for building robust fault-tolerant distributed applications. Users of Disco typically write jobs in Python, which makes it possible to express even complex algorithms or data processing tasks often only in tens of lines of code. Here’s a blog post about the same, with references to vs. Hadoop.
  • On (Python) packaging. Debating distutil, easy_install and pip.

December 23, 2008

Interact with applications from the command line on the Mac

macintosh,tips · David Janes · 6:50 am ·

I’m a command line guy – I spend 90% of my non-blog reading day in Terminal, working on Python apps on my Mac or SSHed into work and working on Java and Javascript applications. I do realize the benefit of “real” applications, for image editing, for advanced text processing and so forth. On the Mac you can send files to the default application easily:

open "Madeline Doll House.jpg"

(Don’t ask). If the Mac doesn’t know how to deal with the file type, or you want to specify a particular app, that’s cool too:

open -a smultron index.jd

Note that it doesn’t matter that I’m SSHed into a work computer – we got around that issue last week using MacFUSE.

Pipe Cleaner (II)

demo,maps,pipe cleaner · David Janes · 6:37 am ·

Here’s the latest evolution of Pipe Cleaner, mainly recorded here for historical interest. The big change is that there isn’t a separate outside template – everything is in the one index.jd file. The new directive is template, which can read and execute an outside module or actually produce the final output (as we see in the very last directive). I have not put this up as an independent demo.

#
#	Import the Python fire module
#	- used in: map from:"fire.GetGeocodedIncidents" to:"incidents"
#
import module:"fire";

#
#	Header for Google Maps popup
#	- used in: map from:"fire.GetGeocodedIncidents" to:"incidents"
#
#
set to:"fitem.head.map" value:"""
<h3>
<a href="#{{ IncidentNumber }}">{{ AlarmLevel}}: {{ IncidentType }} on {{ RawStreet }}</a>
</h3>
""";

#
#	Header for the sidebar
#	- used in: map from:"fire.GetGeocodedIncidents" to:"incidents"
#
set to:"fitem.head.sb" value:"""
<h3>
{% if latitude and longitude %}
<a href="javascript:js_maps.map.panTo(new GLatLng({{ latitude }}, {{ longitude }}))">*</a>
{% endif %}
<a href="#{{ IncidentNumber }}">{{ AlarmLevel}}: {{ IncidentType }} on {{ RawStreet }}</a>
</h3>
""";

#
#	Body for the Google Maps pop and the sidebar
#	- used in: map from:"fire.GetGeocodedIncidents" to:"incidents"
#
set to:"fitem.body" value:"""
<p>
Alarm Level: {{ AlarmLevel }}
<br />
Incident Type: {{ IncidentType }}
<br />
City: {{ City }}
<br />
Street: {{ Street }} ({{ CrossStreet }})
<br />
Units: {{ Units }}
</p>
""";

#
#	Convert all the incidents from the fire module
#	to the path 'incidents' using the mapping rules defined above
#
#	- incidents are used in "gmaps.js" and "gmaps.html"
#
map from:"fire.GetGeocodedIncidents" to:"incidents" map:{
	"latitude" : "{{ latitude }}",
	"longitude" : "{{ longitude }}",
	"title" : "{{ AlarmLevel}}: {{ IncidentType }} on {{ RawStreet }}",
	"uri" : "{{ HOME_URI }}#{{ IncidentNumber }}",
	"body" : "{{ *fitem.head.map|safe }}{{ *fitem.body|safe }}",
	"body_sb" : "{{ *fitem.head.sb|safe }}{{ *fitem.body|safe }}",
	"IncidentNumber" : "{{ IncidentNumber }}"
};

#
#	Load the 'gmaps' templates (for arbitrary geo-mapping),
#	using the 'incidents' for its items and the specified meta.
#
#	- used in in "gmaps.js" and "gmaps.html"
#
set to:"map_meta" value_render:true value:{
	"id" : "maps",
	"latitude" : 43.67,
	"longitude" : -79.38,
	"uzoom" : -13,
	"gzoom" : 13,
	"api_key" : "{{ cfg.gmaps.api_key|otherwise:'ABQIAAA...pIxzZQ' }}",
	"html" : {
		"width" : "1024px",
		"height" : "800px"
	}
};

#
#	Produce GMaps
#
template script:"gmaps" items:"incidents" meta:"map_meta";

#
#	Produce the final output
#
template value:"""
<html>
<head>
    <link rel="stylesheet" type="text/css" href="css.css" />
	{{ gmaps.js|safe }}
</head>
<body>
<div id="content_wrapper">
	<div id="map_wrapper">
		{{ gmaps.html|safe }}
	</div>
	<div id="text_wrapper">
{% for incident in incidents %}
	<div id="{{ incident.IncidentNumber }}">
		{{ incident.body_sb|safe }}
	</div>
{% endfor %}
</div>
</body>
</html>
""";

The gmaps.jd (imported in the second last directive) looks like as follows (there will not be a test). It’s designed to be a universal “show a map and plot points on in it” inclusion. I’ve added a few line breaks so the PRE box doesn’t break.

#
#
#
template to:"html" value:"""
<div id="id_{{ meta.id|jslug }}"
style="{% if meta.html.width %}width: {{ meta.html.width }};{% endif %}
{% if meta.html.height %} height: {{ meta.html.height }};{% endif %}
{% if meta.html.style %} style: {{ meta.html.style }};{% endif %}"
{% if meta.html.class %} class="{{ meta.html.class }}"{% endif %}
></div>
<script type="text/javascript">
js_{{ meta.id|jslug }}.onload();
</script>
""";

#
#
#
template to:"js" value:"""
<script
 type="text/javascript"
 src="http://maps.google.com/maps?file=api&v=2&key={{meta.api_key}}">
</script>
<script type="text/javascript">
js_{{ meta.id|jslug }} = {
 onload : function() {
  js_{{ meta.id|jslug }}.map = new GMap2(document.getElementById("id_{{ meta.id|jslug }}"));
  m = js_{{ meta.id|jslug }}.map;
  m.setCenter(new GLatLng({{ meta.latitude }}, {{ meta.longitude }}), {{ meta.gzoom }});

  // {{ items|length }} items follow
{% for itemd in items %}
{% if itemd.latitude and itemd.longitude %}

  // {{ itemd.title }}
  var ll = new GLatLng({{ itemd.latitude }}, {{ itemd.longitude }});
  var marker = js_{{ meta.id|jslug }}.make_marker(m, ll, "{{ itemd.body|safe|escapejs }}");
  m.addOverlay(marker);
{% else %}
  // an item is missing latitude or longitude
{% endif %}
{% endfor %}
 },

 make_marker : function(m, ll, html) {
  var marker = new GMarker(ll);
  GEvent.addListener(marker, "click", function() {
   m.openInfoWindowHtml(ll, html);
  });

  return marker;
 },

 end : 0
}
</script>
""";

December 22, 2008

Issues with utcoffset and pytz

demo,python · David Janes · 10:14 am ·

In the previous entry, we talked about the difficultly in finding out the delta from UTC for a timezone returned from the pytz module. In particular, consider the offset for St. John’s, Newfoundland which should be at -3:30.

dt_now = datetime.datetime.now()
tz = pytz.timezone('America/St_Johns')

offset = tz.utcoffset(dt_now)

Log(
    "using datetime.utcoffset",
    offset = format(offset),
)

With the unexpected result:

  message: using datetime.utcoffset
  offset: -4:29 (-12660)

I did a fair bit of Google searching for an answer without finding a satisfactory result, so I did further research on my own. To find the correct offset value, I found that this works:

dt_sj = tz.localize(dt_now)
offset = dt_sj - pytz.UTC.localize(dt_now)

Log(
    "using delta to UTC",
    offset = format(offset),
)

Which yields the correct:

  message: using delta to UTC
  offset: 03:30 (12600)

Note that if you’re going to use the above method for finding deltas, you’re going to have to take Daylight Savings Time into consideration also. I have not done this here, as I’m a little pressed for time and just want to illustrate the problem.

The issue seems to be with the way that pytz uses the Olson database entry (from here) for St. John’s – and all other locations. It appears that pytz is using the first rule it sees, from 1884, rather than the rule for the date that was passed in. I think this is a bug.

#
# St John's has an apostrophe, but Posix file names can't have apostrophes.
# Zone  NAME        GMTOFF  RULES   FORMAT  [UNTIL]
Zone America/St_Johns   -3:30:52 -  LMT 1884
            -3:30:52 StJohns N%sT   1918
            -3:30:52 Canada N%sT    1919
            -3:30:52 StJohns N%sT   1935 Mar 30
            -3:30   StJohns N%sT    1942 May 11
            -3:30   Canada  N%sT    1946
            -3:30   StJohns N%sT

The setup code for the examples above is:

from bm_log import Log
import dateutil.parser
import pytz
import datetime

def format(td):
    seconds = td.seconds + td.days * ( 24 * 3600 )
    return  "%02d:%02d (%s)" % ( seconds // 3600, seconds % 3600 // 60, seconds, )

Update 2010-03-09: This has been fixed in the code base and (presumably) will be in the next upcoming release.

Working with dates, times and timezones in Python

demo,python · David Janes · 7:37 am ·

Here’s a few examples of working with dates, times and timezones in Python. We are using the following packages:

  • datetime (part of the standard Python distribution)
  • dateutil – for date parsing, though there’s a lot more depth to this package that I’m not touching here
  • pytz – for timezone handling, and specifically making available the Olson timezone database to Python

There’s a lot of complexity to working with datetimes in any language; I’m not going to get into that but would prefer instead to show a few practical examples. Keep the following in mind:

  • datetimes may or may not have timezones associated with them. If they do not, they are called “naive” and their meaning is effectively defined by the program. In general, you want to work with non-naive datetimes. Generally the assumption would be that the naive datetime is in the application’s current timezone or the user’s preferred timezone
  • when working with datetimes, consider the strategy of converting everything to the universal UTC timezone, then converting back to the user’s timezone only when you need to display that to the user
  • if you are rolling your own code for handling dates, times and timezones and you haven’t done a lot of research, your implementation is garbage. Do yourself and everyone else a favor and use a library.

Our standard imports. Log is from the pybm library and it’s purpose is rather obvious.

from bm_log import Log
import dateutil.parser
import pytz
import datetime

Here’s an example of parsing the an e-mail or RSS type date using dateutil.

dts = "Thu, 13 Nov 2008 05:41:35 +0000"
dt = dateutil.parser.parse(dts)

Log(
    "Parsing an RFC type date",
    src = dts,
    dt = dt,
    iso = dt.isoformat(),
)
  message: Parsing an RFC type date
  dt: 2008-11-13 05:41:35+00:00
  iso: 2008-11-13T05:41:35+00:00
  src: Thu, 13 Nov 2008 05:41:35 +0000

Here’s an example of parsing an ISO Datetime

dts = '2008-11-13T05:41:35-0400'
dt = dateutil.parser.parse(dts)

Log(
    "Parsing an ISO Date with Timezone",
    src = dts,
    dt = dt,
    iso = dt.isoformat(),
)
  message: Parsing an ISO Date with Timezone
  dt: 2008-11-13 05:41:35-04:00
  iso: 2008-11-13T05:41:35-04:00
  src: 2008-11-13T05:41:35-0400

Here’s an example of parsing a naive timezone.

dts = '2008-11-13T05:41:35'
dt = dateutil.parser.parse(dts)

Log(
    "Parsing an ISO Date without a Timezone",
    src = dts,
    dt = dt,
    iso = dt.isoformat(),
)
  message: Parsing an ISO Date without a Timezone
  dt: 2008-11-13 05:41:35
  iso: 2008-11-13T05:41:35
  src: 2008-11-13T05:41:35

Here’s are two similar example, showing how to force the timezone if it’s not present. This will happen in the first part, but not the second.

tz = pytz.timezone('America/Toronto')
dts = '2008-11-13T05:41:35'
dt = dateutil.parser.parse(dts)
if dt.tzinfo == None:
    dt = dt.replace(tzinfo = tz)

Log(
    "Parsing an ISO Date without a Timezone BUT specifying default TZ",
    src = dts,
    dt = dt,
    iso = dt.isoformat(),
    tz = tz,
)

tz = pytz.timezone('America/Toronto')
dts = '2008-11-13T05:41:35-0400'
dt = dateutil.parser.parse(dts)
if dt.tzinfo == None:
    dt = dt.replace(tzinfo = tz)

Log(
    "Parsing an ISO Date with a Timezone AND specifying default TZ",
    src = dts,
    dt = dt,
    iso = dt.isoformat(),
    tz = tz,
)
  message: Parsing an ISO Date without a Timezone BUT specifying default TZ
  dt: 2008-11-13 05:41:35-05:00
  iso: 2008-11-13T05:41:35-05:00
  src: 2008-11-13T05:41:35
  tz: America/Toronto

  message: Parsing an ISO Date with a Timezone AND specifying default TZ
  dt: 2008-11-13 05:41:35-04:00
  iso: 2008-11-13T05:41:35-04:00
  src: 2008-11-13T05:41:35-0400
  tz: America/Toronto

Update: here’s an example of moving datetimes to UTC and then to a different Timezone. Remember: you want your backend code to work with UTC datetimes for simplicity and correctness:

dts = '2008-11-13T05:41:35-0400'
dt_orig = dateutil.parser.parse(dts)
dt_utc = dt.astimezone(pytz.UTC)

Log(
    "Changing a datetime to UTC",
    src = dts,
    dt_orig = dt_orig,
    dt_utc = dt_utc,
)

tz_vancouver = pytz.timezone('America/Vancouver')
dt_vancouver = dt_utc.astimezone(tz_vancouver)

Log(
    "Changing UTC datetime to a different timezone",
    dt_vancouver = dt_vancouver,
    dt_utc = dt_utc,
)
  message: Changing a datetime to UTC
  dt_orig: 2008-11-13 05:41:35-04:00
  dt_utc: 2008-11-13 09:41:35+00:00
  src: 2008-11-13T05:41:35-0400

  message: Changing UTC datetime to a different timezone
  dt_utc: 2008-11-13 09:41:35+00:00
  dt_vancouver: 2008-11-13 01:41:35-08:00

Here is an example of listing all “common” timezones using pytz. Note that “America” refers to the two continents, not the Irish word for the United States. Printing the actual timezone offset turned out to be a surprisingly complex task, which I will outline in a different blog post. For now let it suffice that with pytz try not to depend on utcoffset.

dt_now = datetime.datetime.now()

def tzname2offset(tzname):
    dt_in_utc = pytz.UTC.localize(dt_now)
    dt_in_tz = pytz.timezone(tzname).localize(dt_now)

    offset = dt_in_utc - dt_in_tz
    seconds = offset.seconds + offset.days * ( 24 * 3600 )

    return  "%02d:%02d" % ( seconds // 3600, seconds % 3600 // 60, )

Log(
    "Olsen (pytz) common timezones and their UTC offsets",
    timezones = map(
        lambda tzname: ( tzname, tzname2offset(tzname), ),
        pytz.common_timezones,
    )
)
  message: Olsen (pytz) common timezones and their UTC offsets
  timezones:
    [('Africa/Abidjan', '00:00'),
     ('Africa/Accra', '00:00'),
     ('Africa/Addis_Ababa', '03:00'),
     ('Africa/Algiers', '01:00'),
     ('Africa/Asmara', '03:00'),
...
     ('Pacific/Wake', '12:00'),
     ('Pacific/Wallis', '12:00'),
     ('US/Alaska', '-9:00'),
     ('US/Arizona', '-7:00'),
     ('US/Central', '-6:00'),
     ('US/Eastern', '-5:00'),
     ('US/Hawaii', '-10:00'),
     ('US/Mountain', '-7:00'),
     ('US/Pacific', '-8:00'),
     ('UTC', '00:00')]

December 18, 2008

Pipe Cleaner

demo,djolt,dqt,html / javascript,ideas,jd,maps,pipe cleaner,pybm,work · David Janes · 6:38 pm ·

I’ve been working (in my decreasing available spare time) on a project to pull together into a project called “Pipe Cleaner” all the various concepts I’ve been mentioning on this blog: Web Object Records (WORK) for API Access and object manipulation, Djolt for generating text from templates, Data/Query/Transform/Template (DQT) for transforming data and JD for scripting these elements together. The pieces came together this morning enough to put a demo together and here it is – the Toronto Fires Pt II Demo.

How, you may ask, does this differ from the original Toronto Fires Demo? The answer is how it is put together, which we describe here.

Index.dj

This is the Djolt template that generates the output. The data fed to this template is generate by the JD script, described in the next section.

<html>
<head>
    <link rel="stylesheet" type="text/css" href="css.css" />
    {{ gmaps.js|safe }}
</head>
<body>
<div id="content_wrapper">
    <div id="map_wrapper">
        {{ gmaps.html|safe }}
    </div>
    <div id="text_wrapper">
{% for incident in incidents %}
    <div id="{{ incident.IncidentNumber }}">
        {{ incident.body_sb|safe }}
    </div>
{% endfor %}
</div>
</body>
</html>

Quite simple … as you can see, most of the data is being pulled in from elsewhere. The elsewhere is provided by the script described in the next section.

Index.jd

This is the script that pull all the pieces together. Note that I’m not 100% happy with the way the data is imported, I would like the geocoding to become part of this data flow too. In the next release perhaps.

First we pull in the “fire” module that we wrote in the previous Map examples. This is doing exactly what you think: importing a Python module. We may have to increase the security or restrict this to working with an API for general purpose use.

import module:"fire";

Next we define two headers – one that is going to appear in the Google Maps popup, the next that is going to appear in the sidebar. They need to be different as they refer to themselves. Note that the sidebar header “breaks” the encapsulation of Google Maps – this seems to be unavoidable. The to:"fitem.head.map" and to:"fitem.head.sb" are manipulating a WORK dictionary to store values.

Note also here that we’ve extended JD to accept Python multiline strings – this was unavoidable if JD was to be useful to me.

set to:"fitem.head.map" value:"""
<h3>
<a href="#{{ IncidentNumber }}">{{ AlarmLevel}}: {{ IncidentType }} on {{ RawStreet }}
</h3>
""";

set to:"fitem.head.sb" value:"""
<h3>
{% if latitude and longitude %}
<a href="javascript:js_maps.map.panTo(new GLatLng({{ latitude }}, {{ longitude }}))">*
{% endif %}
<a href="#{{ IncidentNumber }}">{{ AlarmLevel}}: {{ IncidentType }} on {{ RawStreet }}
</h3>
""";

The next block defines the text of the body used to describe a fire incident. It follows much the same pattern as the previous block.

set to:"fitem.body" value:"""
<p>
Alarm Level: {{ AlarmLevel }}
<br />
Incident Type: {{ IncidentType }}
<br />
City: {{ City }}
<br />
Street: {{ Street }} ({{ CrossStreet }})
<br />
Units: {{ Units }}
</p>
""";

This is a map: it is translating the values in fire.GetGeocodeIncidents into a new format and storing that in incidents. The format that we were are storing it in is understood by the Google Maps generating module.

We may rename this translate, as the word map is somewhat overloaded.

map from:"fire.GetGeocodedIncidents" to:"incidents" map:{
    "latitude" : "{{ latitude }}",
    "longitude" : "{{ longitude }}",
    "title" : "{{ AlarmLevel}}: {{ IncidentType }} on {{ RawStreet }}",
    "uri" : "{{ HOME_URI }}#{{ IncidentNumber }}",
    "body" : "{{ *fitem.head.map|safe }}{{ *fitem.body|safe }}",
    "body_sb" : "{{ *fitem.head.sb|safe }}{{ *fitem.body|safe }}",
    "IncidentNumber" : "{{ IncidentNumber }}"
};

Next we set up the “meta” (see WORK meta description if you’re not following along) for the maps. The render_value:true declaration makes PC interpret the templates in strings). We then call our Google Maps generating code (which are actually more Pipe Cleaners) and that gets fed to the Djolt template we first showed you. Clear? Maybe not, we’ll have more examples coming…

set to:"map_meta" render_value:true value:{
    "id" : "maps",
    "latitude" : 43.67,
    "longitude" : -79.38,
    "uzoom" : -13,
    "gzoom" : 13,
    "api_key" : "{{ cfg.gmaps.api_key|otherwise:'...mykey...' }}",
    "html" : {
        "width" : "1024px",
        "height" : "800px"
    }
};

load template:"gmaps.js" items:"incidents" meta:"map_meta";
load template:"gmaps.html" items:"incidents" meta:"map_meta";

December 16, 2008

Woah!

administrivia · David Janes · 8:00 am ·

WordpressThis site has been upgraded to WordPress 2.7.

December 13, 2008

Brief notes on SIMILE Timeline

demo,html / javascript · David Janes · 8:01 am ·

SIMILE Timeline is “the Google Maps for time based events”. It used to be housed at MIT but now it’s graduated to Google Code. I’ve created an example application showing this year’s Oscar awards and a number of movies that are, umm, are not all likely to be nominated.

  • the application source can be seen here; it is based on this demo and some work I had done previously. Note:
    • multiple scrolling bands linked together
    • custom icons
    • custom colors
  • we demonstrate populating the timeline widget using JSON data coded in the application; the most difficult part of putting this demo together was cutting and pasting all this data, a task we hope to make easier with our DQT code
  • the documentation for Timeline is starting to diverge from the source code; showEventText is now replaced by overview in the band creation code
  • there’s a large number of weaknesses (still) with Timeline for dealing with arbitrary data, these may be corrected in the Javascript code but I don’t have time to go through all this
    • note the incorrect placement of custom icons; using Firebug to inspect the HTML, I discovered that unfortunately everything is placed using style tags so it’s difficult to correct using CSS. Ideally I would like to be able to assign classes and ID tags to everything
    • there doesn’t seem to be an obvious way to control the widths of the labels; in fact, if I reduce the band spacing in the top band, the text starts to overlap in a horrible manner
    • popup information boxes get confused when there is too little space to display information
    • when I don’t add a description to events, it uses “undefined” (see the Oscar Nominations Period)
    • date display functions are inferring more resolution (i.e. an actual time as opposed to the just the date) that I’m giving it

If anyone has corrections for me I’ll update the demo

December 12, 2008

JD – JSON Declaration Language

ideas,jd · David Janes · 5:55 pm ·

I’ve just added a new module called bm_jd to the pybm project. It implements a “little language” for declaring information, like a configuration file, when the details are all specified in JSON.

The language is very simple, consisting of semi-colon terminated statements; each statement having a command and zero or more arguments. Each argument may or may not have JSON data – if it does, it will be set off with a colon.

The BNF looks like this:

    <document> ::= <statement>*
    <statement> ::= <command> ( <word> | <word>:<json> )*
    <command>|<word> ::= [a-zA-Z0-9_]
    <json> ::= ... any valid JSON data ...

You can use the pybm JD parser in several ways:

  • implement a subclass of JDParser, defining CustomizeProduce; or
  • implement a subclass of DispatchJDParser, defining a call_<command> method for each command you plan to allow

In either case, you call a method FeedString to get the parser rolling.

There’s also a LogJDParser, which just dumps parsing results. Here’s an example of a JD document. Don’t worry about Djolt code in the JSON, that’s just text as far as this example is concerned:

read_template from:"fire_body" render:false;
map from:"fire.GetGeocodeIndidents" to:"incidents" map:{
    "latitude" : "{{ latitude }}",
    "longitude" : "{{ longitude }}",
    "title" : "{{ AlarmLevel}}: {{ IncidentType }} on {{ RawStreet }}",
    "uri" : "{{ HOME_URI }}#{{ IncidentNumber }}",
    "body" : "{{ *fire_body|safe }}",
    "IncidentNumber" : "{{ IncidentNumber }}"
};
read_template from:"gmaps" items:"incidents" meta:{
    "id" : "maps",
    "latitude" : 43.67,
    "longitude" : -79.38,
    "uzoom" : -13,
    "gzoom" : 13,
    "api_key" : "{{ cfg.gmaps.api_key|otherwise:'_' }}",
    "html" : {
        "width" : "1024px",
        "height" : "800px"
    }
};

December 11, 2008

Mount NTFS and remote filesystems using MacFUSE

macintosh,tips · David Janes · 11:56 am ·

MacFUSEEarlier this week I bought a LaCie 500Gb USB drive so I could bring VMWare images between work and home. When I went to copy the image, the copy failed with no meaningful error message (Error 0, I believe). Trying the copy on the command line was a little more informative: as it turns out, the LaCie drive ships with a FAT-32 file system which can only handle files up to 4Gb in size. As the image I was trying to copy had a 8Gb file in it, this was a no go.

My initial thought was to use the UNIX commands tar and split to break the files into individual smaller chunks, but this is hardly a satisfactory answer. If I formatted the drive to the Mac filesystem, the Windows machines would not be able to read it at all. If I formatted the drive the “new” NTFS filesystem, Windows can read and write just fine but the Macintosh wouldn’t be able to write to it.

Fortunately, there’s an excellent install for the Mac called MacFUSE that allows access to all sorts of filesystem types not natively supported by the Macintosh, include NTFS. Here’s how I set up MacFUSE.

MacFUSE Installation

Installation by itself does nothing except set you up for the next stage: installing drivers for particular file systems.

NTFS

You have to search through the documentation for a bit to figure out where to get NTFS to with Windows filesystems. It actually turns out to be rather easy:

You can now write to NTFS drives. It’s a little slow – it’s taking me about 2 hours to copy 8Gb to the La Cie drive, but that’s better than not being able to do it at all. You wouldn’t want to work live off the drive however, and it may be worth investigating commercial NTFS compatibility applications if you need to do this.

To reformat your La Cie drive plotline, use Applications > Disk Utility to erase and install an empty NTFS file system.

SSHFS

SSHFS lets you see remote filesystems through SSH.

  • go to http://code.google.com/p/macfuse/wiki/MACFUSE_FS_SSHFS
  • download the version appropriate to your Mac; you can store this in your home directory or if you’re a little more organized about your path, a directory link ~/bin
  • make a mount point – this is just a directory on your Mac that is needed by MacFUSE; it can be hidden as Mac OS will show you the mounted drive on your desktop and in /Volumes. For example, on the command line run mkdir -p ~/.Volumes/Remote.
  • run the mount command; you’ll be prompted for your remote system password

You’ll see the drive appearing on your desktop. I’ve actually created a shell alias to do the mounting for me called “mount-xxx”. If you don’t know how to do this, it’s probably too much to go into right now.

The nice thing about SSHFS is that I could see being able to run an entire Mac desktop development shop with all the backend computing running Linux, all being accessed nicely through SSHFS.

Older Posts »

Powered by WordPress

Switch to our mobile site