David Janes' Code Weblog

Data/Query/Transform/Template – DQT, dropping the final T – is a commonly used pattern for displaying data on a website. The elements of this pattern are:

• a Data source, such a blog database, an e-mail store, the Internet as a whole, a MySQL, and in particular the results of an API call.
• a Query, which is a way of selecting a particular subset or slice of the data (typically homogeneous)
• Transform rules, which can make the data look different by renaming fields, enhancing data using tools such as geolocation, filtering records out, merging multiple data sources and so forth.
• a Template, which is a way of converting to a useful end-user format, such as HTML, JSON or XML

In the particular context of what I’m writing about, we can assume that we’re manipulating WORK items – that is, an an API returns a “Meta” block of information and a stream of “Items”, each in turn which are WORK items also. By identifying common patterns of dynamic page construction, my hope is that we can simplify page and mashup creation.

You’ve seen this pattern many times. My plan is that be describing it properly, we can make it easier to do.

Wordpress Blogs

• The Data source is the MySQL table with blog posts, plus ancillary information pulled from other tables.
• The Query is some combination of ( page number, post path, category, tag ). Not all combinations are legal obviously, but this is the information that can be encoded in a URL request. The Data source and the Query result a number of posts being made available for further processing
• The Template is the PHP code that converts the individual database items into HTML for display

There is no Transform in this example. See it here ;-).

BTW: Don’t take this as a how-to guide for Wordpress. I’m trying to look at this from a high-level conceptual point-of-view.

Google Mail

• the Data source is all the e-mails in the Google database, probably billions or trillions of messages
• the Query is some combination of ( userid, page number, search ). The userid is not encoded in the URL, it is known be
• the Template is the Javascript code that

That’s a really high level view: in fact, Google Mail does this DQT twice: the first time around to select JSON or XML data to be transmitted to the user’s browser; the second time around to locally on the user’s browser select and display items.

Yahoo News

• the Data source is Yahoo’s news database
• the Query is a category, or not category at all (poorly encoded the URL, I may add)
• the Transform groups news into like categories (play along with me here)
• the Template is the Yahoo’s HTML generator, whatever that may be

See it here.

An RSS feed

The Data and the Query are substantially similar to the Wordpress Blog example, but:

• we Transform the fields into a format that can be understood by an RSS generator
• the Template is a specialized object that converts WORK items into RSS entries, that is, we don’t (or shouldn’t) use a Djolt-like template to generate XML.

This is obviously a somewhat of a hypothetical example, but reflects my recent ideas about how machine readable data should be generated.