Daytime Running Lights

I don’t have to tell you that in the long run, open source software beats closed source software, in any domain where the software is broadly useful, or in those domains where users are generally technical enough to hack their own gear. Proprietary web-servers and filesystems just can’t keep up with the free competition. As programming gets easier, the open-source advantage is moving up the stack. Microsoft burns out generations of programmers trying to keep up with Rails and Django, and they still don’t have the other advantages of freedom.

I also don’t have to tell you that web development is where all the new developers are cutting their teeth these days. Middle-schoolers the world over are learning to hack by hitting “view source” and having that a-ha moment when they realize those angle brackets correspond to the web apps they use everyday.

The road from kludging together HTML and stolen JavaScript to piloting high-level frameworks like Rails and Django is arduous and fraught with dead ends and distractions, but enough of us make it here to have created this vibrant community of enlightened hackers. We learn from each other’s code, but we still haven’t found a compelling way to share the whole stack. There are a few open-source web application success stories, like WordPress and Trac, but most of us work on applications that will die when the original business case goes away. We certainly haven’t seen an explosion of end-user friendly open-source web apps, to rival the proliferation of free development tools, frameworks, and even end-user software like browsers and media tools.

We Don’t Need No Affero

You only have to Google AGPL to get an idea of the tensions seething under the surface of the Free Software movement. The rub: companies have grown fat and happy using proprietary forks of open-source software to power web services. Because they aren’t distributing the software, just distributing the ability to use the software, they aren’t required to share their code under traditional licenses. It can be discouraging to pour energy in to free software, only to see big companies turn a handsome profit with it, and fail to contribute back to the community.

Some say the solution is to require source distribution to accompany usage distribution. I say the solution is to out-compete the applications that don’t distribute their source. If we build apps that derive a lot of value from the fact that they can run locally or over a network, and we provide an easy way for users to share the applications, I think we can give people a reason to prefer freedom over convenience. Once users expect the benefits that come from taking ownership of their data and their application code, they’ll look at closed-source web apps as dinosaurs.

CouchDB Rocks, Duh

I’m not saying it’s the only way, but I will say that CouchDB is uniquely positioned to open up this new space. The affordances it creates around sharing apps and data make the opportunity cost of not sharing your applications and data higher than the costs of doing so. If you’re interested in my thoughts about the technical ins and outs of shareable apps, I just posted today on the topic of peer-to-peer apps on the CouchDB mailing list.

Let’s Do It!

If you just want to get down and dirty, and have a copy of the latest CouchDB trunk, let’s install my CouchDB Twitter client. Once you’ve got CouchDB installed and running on your local machine, installing it should take about a minute.

Let’s do this step by step – with screenshots!

First, go to your local machine’s CouchDB futon and create a database.

Now go to the Replicator, where we’ll enter the information we need to copy the CouchDB Twitter Client from my public database.

The remote database is http://jchris.mfdz.com:5984/twitter-client-design (I created a special database with just the application code, because in this case the Twitter client tends to accumulate quite a lot of information over time.) There’s a sad bug that was introduced for a few revs of CouchDB’s trunk, having to do with URL escaping. It’s fixed now so if replicating doesn’t work, try running svn up and rebuilding before you retry replication.

Now you have the app! Lets use it. Go back to the “Overview” page (the one you started on) and click into the twitter-client db. Now choose “Design documents” from the drop down menu.

Now click the _design/twitter-client document, and you’ll be happy you did because boy is it awesome! All it takes to enter the application is to click index.html.

And this is what it should look like, only with your friends’ updates instead of mine. If you don’t have a Twitter account, there won’t be anything to see here. There’s also a strange failure mode for some users where it just seems totally broken. This might be due to too many Twitter API accesses, so if you’re gung-ho, try quitting your other Twitter clients and give it an hour or so. If it still doesn’t work, well, you’ve got the source code all in that design document!

Note that you can add standing searches to your timeline by click the star in the lower-left.

Enjoy!

Update: I’ve relaunched the example blog on this blog’s host machine It should stay up for a while. Please write a post or two!

Over the last few days I’ve polished up my notion that CouchDB can be a perfectly viable application host, all on its own, without any 3rd tier between database and client. That is, CouchDB is capable of serving standalone applications. These standalone CouchDB applications can be deployed to any working CouchDB node and used from any browser.

Encapsulating business logic in a Ruby or Python middleware framework like Merb or Django is no longer necessary. Between CouchDB’s views, its security model and validations, and the new _external server, developers will have everything they need to present fully functional, responsive, and dare I say scalable, web applications.

Full disclosure: I’m an Apache CouchDB committer, but these views are not a result of that work, rather they are the reason I was driven to join the community in the first place.

Of course, in this dream world of logarithmically scalable JSON and Ajax web applications, all users have to bring is a browser. However, those users who run local CouchDB nodes can stay connected to the mothership via replication, empowering not only offline use, but an explosion in open source web applications. Standalone CouchDB applications travel with the data they manage, and data can be replicated between peer nodes. Users will be free to modify and share the tools they use to manage their data, as well as the data itself.

Thinking of peer-based application replication takes me back to Austin’s science magnet, LBJ High School. As freshman, my friends and I would share little video games between the TI-85 graphing calculators we were required to carry. Two calculators could connect by one of those smaller-than-headphone minijack cables, and we’d share Physics cheat sheets, Hangman, some text-based adventures, and at the height of our powers, I believe there may have been a Doom clone running.

The TI-85 programs were in Basic, so everyone was always hacking each others hacks. Perhaps the most ridiculous program was a version of Spy Hunter that you controlled with your mind. The idea was if you could influence the pseudo random number generator by concentrating hard enough, you’d be able to control the car. Didn’t work. Anyway, the point is that when you give the kids control of the source code, there’s no telling what will happen.

CouchDB’s replication facility has the potential to bring an explosion in innovation, as kids realize that they can hack and share the apps they use everyday. And duh, if you wanna be with it, you gotta keep up with what the kids are doing. ;) But seriously, the freedom we’ll see when people control their data, and the applications they use to access it is what keeps me up at night working on this.

To participate in this Cambrian explosion you’ll be required to write your application by new rules. At first blush, pure CouchDB apps follow more restrictive constraints than even Google App Engine. However, I’ll argue that pure CouchDB apps can offer more room for innovation. This is because with a CouchDB app, you control the whole stack from root to fruit, and can deploy your applications on your own terms, from small scale in-house use to datacenter-level distributed systems.

CouchDB already offers a built in HTTP server with attachment support and file uploads, as well as a rich set of APIs for storing and retrieving JSON records. By taking advantage of these characteristics pure-CouchDB applications can reach near parity with traditional server-heavy approaches. Notable exceptions include the ability to render dynamic text that isn’t JSON, or return results that require multiple queries against CouchDB.

With the recent addition of _external support to CouchDB, now it is possible to craft dynamic XML responses (or anything else you can do from a JavaScript function) directly from CouchDB’s Spidermonkey script runner. In this article I’ll show some code that renders a CouchDB view as an Atom feed. Everything’s still very rough, but I’m encouraging people to hack on it and share. There’s a framework (or maybe a whole new way of working) emerging, and I want you to be a part of it.

What does a Pure CouchDB app look like?

I assume you already know about CouchDB’s database capabilities. To refresh: it’s a RESTy document store with incremental map reduce queries. The documents are JSON and the map reduce queries are written in JavaScript, or your language of choice, via a stdio socket that speaks a simple JSON protocol.

Among CouchDB’s various documents, are those who’s _id starts with “_design/”. These documents contain the map reduce views, and according to Damien Katz, they should correspond 1:1 with the applications that run across a dataset. One application equals one design document.

The example application in this post (and the script I’ve written to bootstrap it into CouchDB from the filesystem) is designed around this principle of one design document per application. I’m especially keen on the innovation that we’ll see when people start to write distinct and competing applications, that run across the same dataset.

CouchDB documents can hold attachments, which are served up as the mime/type they were stored with. This makes it easy to store and serve HTML, CSS, Javascript, and even image files directly from CouchDB. By attaching a few HTML files and some JavaScript behavior to the design document, you can have a working blog, as I showed in the demo portion of my Arc90 talk.

As a proof of concept, look no further than CouchDB’s built-in administrative interface. Futon, as it’s called, is just a collection of HTML, CSS, and Javascript files, and it is a fully functional database browser and JSON document editor, as well as the runner for CouchDB’s functional test suite. The test suite is written in Javascript and runs from the browser. What else could more strongly indicate that Ajax and CouchDB go hand in hand?

JavaScript, static HTML, and CouchDB doesn’t buy you Atom or RSS feeds, however. And who’d seriously create blogging software without feeds? The _external server system to the rescue! Basically, CouchDB parses and forwards HTTP requests as JSON object to an external script. The external script can then process the JSON, make additional queries, and return whatever response it chooses, including redirects, HTTP status codes, plain HTML, JSON, or other content types.

IMHO, building raw Ajax + JSON apps is the fastest, simplest way to get a new idea in front of other people. I’ve done it a few times in various contexts. Grabb.it is just some Ajax on top of a JSON API (hosted on Rails and PostgreSQL). Grabb.it had to be Ajax-heavy, because changing pages would make the music stop. TracksPress (Grabb.it’s new skunkworks product) has a pay area, which is just Ajax on top of a Merb/CouchDB app. For me, pure Ajax applications have a winning track record.

An Ajax app is simple: just get a shell of an HTML page up, throw on a little CSS and some jQuery, and you’re up and running. Back that with the power of CouchDB and suddenly you’re within striking distance of a real web app. Add CouchDB’s `_external` handler and you can do anything another service would do, from rendering rss feeds to dropping an upload into a processing queue.

Still on the Edge (installing)

Installing couchdb-example-blog isn’t nearly as easy as it should be. First of all couchapp, the installer script, requires Ruby, the CouchRest gem, and all of it’s dependencies. I’m considering porting couchapp to a leaner environment for convenience.

Secondly, couchdb-example-blog requires a non-standard branch of CouchDB itself. This should be simplified in the future. The CouchDB team still needs to decide on a build process for plugins. The _external process manager I’ve been helping davisp with, which powers action servers, as well as the Full Text Search interface, may make a model plugin, although it may end up included by default.

For detailed installation directions, see the couchdb-example-blog README and it will step you through everything.

No Joke (Security Model)

CouchDB will have a comprehensive security and data validation module. The work may have already begun. It looks like Damien is targeting this as a 1.0 feature. The upshot is that the security model is not yet available, but there is a description of it. (see Security and Validation)

Basically, any changes, such as document POST, PUT and DELETE, are validated by a JavaScript function, which is supplied the currently saved version of the document, the proposed version of the document, and some information about the request (including the user or other authentication parameters.) The function may return false to indicate an invalid operation, or true to let the operation proceed. There’s talk of extending the specification to allow validation functions to set values on the to-be-saved document, like timestamps or user information. With such this security model, it is possible to implement author-only saves, data format checks, and probably most of what application programmers require.

E4X: ECMAScript for XML

Spidermonkey includes the E4X extensions, which give it the ability to manipulate XML with a native API. You get to use JavaScript “dot notation” to mess with XML. Most articles and tutorials out there are tailored towards parsing XML. That may be important to some folks (if so check out the links in the previous sentence.)

Instead I’ll show how I generated an Atom feed using Spidermonkey’s JavaScript interpreter. It’s deceptively simple. I’m sure there are better ways to do this with E4X, but hey, this one’s not that bad.

First we load the design document from CouchDB, and use it’s blog attribute to fill out some template values in the XML feed header.

  var blog = db.open('_design/couchdb-example-blog').blog;

  var feed = <feed xmlns="http://www.w3.org/2005/Atom">
    <title>{blog.title}</title>
    <link href={blog.url}/>
    <updated>2003-12-13T18:30:02Z</updated>
    <author>
      <name>{blog.author.name}</name>
    </author>
    <id>{blog.url}</id>
  </feed>

If we were to call `feed.toXMLString()` now we’d get just what it seems like we’d get, with everything all escaped and quoted correctly. We still need to add the entries to the feed, so we’ll fetch a view and iterate over its posts, copying them to the XML. Check out how easy the dot notation makes this. (Note that it looks like attribute access, but really it’s a big nasty XML operation, so don’t do more than you need.)

  var view = db.view('couchdb-example-blog/recent',{count:10,descending:true});

  for (r in view.rows) {
    var row = view.rows[r];
    var post = row.value;
    var entry = <entry/>;
    entry.id = blog.url + '/' + row.id;
    entry.title = post.title;
    entry.updated = post.date;
    entry.content = post.body;
    feed.entry += entry;    
  }

  return {body: feed.toXMLString()};

Now our example blog has a feed, so it’s a real blog. Of course it’s barely a real blog, but at this point features like Gravatar integration and comment moderation are just window dressing.

Data Portability

Why does it matter that CouchDB can serve standalone apps? Why should developers go through the trouble to learn a new paradigm?

I think any platform that gets out of the way enough to let you build Ajax apps with only the bare minimum server logic is a win in terms of developer productivity. But if you are hesitant on that front, consider the potential for portability.

When all developers need to do, to run their own instance of an application, is install CouchDB, suddenly we’re talking about a platform even more simple than PHP plus MySQL, which has become the gold standard for ease of installation. Imagine if the process for installing a new CMS, or a new ticket tracker, or any other browser based application was a simple as replicating a _design document from the application’s root repository.

What’s more, pure-CouchDB apps don’t care if they are hosted on a local laptop, on a school or company intranet, or open on the web for all to use. The same code base can work in all those contexts, and the same set of data can be replicated across them. One of CouchDB’s sweet-spots is offline replication. This means you could host an application at a school, allow students to replicate it to their laptops, work on it at home, and turn their work in by replicating back when they return to school. Data can be distributed without complex access controls, as would be required if the students accessed the school’s network from the public internet.

I look forward to seeing some of the prototypical Web 2.0 use cases migrate to this user-controlled future. What will photo-sharing look like when you can send your photo album, along with its user-modifiable browsing application, directly to friends? The web has shown that the power to view source is a catalyst for learning and innovation. With CouchDB apps, view-source exposes the whole application. I’d really like to see some public-data projects like Public.Resource.Org and The Sunlight Foundation experiment with user-owned applicatios.

Low-Hanging Fruit

Here are just a few ideas for standalone CouchDB application that I’ve come up with. I’d love to see any of them shared around, preferable under a free-license.

Feed Reader: CouchDB should be able to provide a user experience similar to Google’s Reader. Using their API you could probably even interact with your friend’s shared items like you’re used to. A nice point about action server modules, is that they can be easily generalized – eg a function that’s used by one app to fetch feeds can be reused in another to power an image downloader or even a web spider.

Ticket Tracker: Imagine Trac as a distributed Ajax application. Now remove all the features you never use. Working offline has proven productive in the mass exodus to git and other dvcs. Managing project information with offline clients may prove to have some of the same efficiencies.

Wikis: The appeal of a CouchDB wiki is that it’d be easy to keep relatively private. A prototypical use case is in the classroom: a group wiki which students are able to edit at home even without network access.

The Future

This essay has been largely speculative. However, it is backed up by running code. I consider it a call to arms. If you’re inspired by CouchDB, or by the potential freedom that we’ll win by open-sourcing all the web apps, I encourage you to start working now.

We could really use someone to package CouchDB as a browser plugin. Our license is compatible with the Gears license—all we need is someone to tackle the build.

Also, write applications, share applications, and fork them. I hope to be running this blog from CouchDB as soon as the security features have been implemented. What else is ready for prime time?

I’m very pleased to announce that I’m lucky enough to be co-authoring the forthcoming O’Reilly book: Relax With CouchDB

My co-authors are Jan Lehnardt and Noah Slater. They’ve already proven to be the right guys to write this book with.

We’re releasing the book online under a free license, and we hope to integrate your feedback.

CouchDB can be used in a lot of different ways, so ensuring that we’re covering the topics that you’re interested in is a top priority.

There is a Google Group for feedback—let us know what you’d like to see!

We’ll be soliciting reader feedback as we develop the book and API documentation. The content is freely licensed, so we’ll make it available as we write. Expect to see a few chapters and a some API docs up by the end of the year.

Writing a book takes time.

with my crazy overdubs =)

For those who are interested here is the slide deck (in pdf) corresponding to my talk video

And the slideshare embed:

Thanks Avi and Arc90 for hosting the event and providing the video.

The code for the example blog is on Github

Also, see this post for slides.

Andy Baio and Joshua Schachter released a cool little Firefox user-script today that tells you, for a big bunch of political blogs, which way they lean (left or right) by coloring the link backgrounds red or blue, respectively. See the official release, and find out more about how they used Singular Value Decomposition, web spiders, and cleverness to make this data available, at Andy’s blog

Seeing that the plugin only worked in Firefox, and being the great fan of bookmarklets that I am, I’ve cloned the data into a bookmarklet. To see it in effect in non-Firefox browsers, drag this link to your bookmarks toolbar: Political Colors

Now visit Memeorandum – when you get there, click your new bookmarklet, and you should see something like this:

Not only did I make this work in Safari, but another slight modification I made, allows it to provide meaningful colors on many other sites. Here’s our good friend Google News, showing its political colors:

The colors will show on any links to sites in Joshua and Andy’s dataset. Have fun!

There are a few ActiveRecord-style wrappers for CouchDB out there these days, but many of them have fallen behind as CouchDB’s powers have increased. I’ve taken an interest in writing regular old web apps again – so now is the time when CouchRest grows wings and flies to the vaulted heights of DataMapper and ActiveRecord. (I could have written a DataMapper adapter for CouchDB, but much of DataMapper’s code is based around SQL-like problems that CouchDB just doesn’t have.) CouchRest::Model provides the ease-of-use Rubyists have come to expect from their ORMs, and it does it in CouchDB style.

Getting Started

Here is an example (taken from the bright shiny new documentation) of what your Models can look like. If you click through to the actual blog post, there is also a longer example pasted as a gist embed which details the capabilities of the view-generation system even more.

class Article <  CouchRest::Model

  use_database CouchRest.database!('http://localhost:5984/couchrest-model-test')
  unique_id :slug

  view_by :date, :descending => true
  view_by :user_id, :date

  view_by :tags,
    :map =>
      "function(doc) {
        if (doc.type == 'Article' && doc.tags) {
          doc.tags.forEach(function(tag){
            emit(tag, 1);
          });
        }
      }",
    :reduce =>
      "function(keys, values, rereduce) {
        return sum(values);
      }" 

  key_reader :slug, :created_at, :updated_at
  key_accessor :title, :tags

  # You can make dates work nice without magic or special fields
  key_writer :date 
  def date
    Time.parse(@doc['date'])
  end

  timestamps!

  before(:create, :generate_slug_from_title)
  def generate_slug_from_title
    doc['slug'] = title.downcase.gsub(/[^a-z0-9]/,'-').squeeze('-').gsub(/^\-|\-$/,'')
  end
end

You can do the standard CRUD:

@post = Post.get(params[:id])
@post.title = "New Title" 
@post.save

View Generation

The view generation system is what I’m proudest of. It’s designed to be performant, and it strikes a balance between magic and usability. Simple views are as easy to declare as:

  view_by :date, :descending => true

They can be queried easily like this:

# get the 10 most recent posts
@posts = Post.by_date :count => 10

# get the raw view
@view = Post.by_date :count => 10, :raw => true

Views defined in this way take all the options of a CouchRest::Database#view call, and those options can be provided either as defaults at definition time, or overriden at query time. There is an additional option, :raw, which defaults to false. When :raw is true, you get back just what CouchDB sends. When :raw is false, CouchRest::Model::MagicViews requests the associated document for each view row, and gives you an array of documents. W00t!

CouchRest::Model views use _design documents, not temp views, so they take advantage of all the performance CouchDB has to offer.

Views can also be manually composed, if you want to write something that’s more complex than the built-in helpers can provide.

If you’re at the permalink page for this post, you’ll see a Gist embed here with an even longer example of CouchRest::Model in action.

Enjoy!

UPDATE 2: Thing are moving very quickly in Merb land this week. The best reference is Timothy Bennett’s summary so please go there if you are serious about trying out Merb in the few days that remain between now and 1.0. (I’ll leave the rest of this post up as historical oddity.)

I’ve decided to smooth the way for future Merbists to use CouchDB. This means getting into the edgy-edgy Merb land inhabited only by the most intrepid Rubyists. Merb will be 1.0 soon, and I hope some of that project’s maintainers take a look at my hoop-jumping, because they’d do well to make this all brain-dead. Or maybe they don’t want the brain-dead to use their framework. :)

First let’s get the latest versions of the source code. These are the incantations I provided. If you do them out of order, it could get icky.

UPDATE: this turned out not to work with edge, so I added lines to checkout the v0.9.7 tag.

# there are a few git repos we'll want to grab, let's keep them together
mkdir merbzone
cd merbzone

# undocumented dependency - extlib
git clone git://github.com/sam/extlib.git
cd extlib
rake install

# the main merb source code
cd ..
git clone git://github.com/wycats/merb-core.git
git checkout v0.9.7
cd merb-core
rake install

# another dependency problem
sudo gem install templater

# merb-more has all the goodies that (optionally) make merb fun
cd ..
git clone git://github.com/wycats/merb-more.git
cd merb-more
git checkout v0.9.7
rake install

OK now lets generate an application, and then I’ll shove you into Merbunity’s excellent and up-to-date tutorials,

merb-gen app MyApp --testing-framework=rspec --template-engine=erb --orm=datamapper

So there you go, you have an app (I hope) and now you can learn to add junk to it the right way. My new mini-project is to get comfortable in the crazy Merb world, so I can start to use it as a basis for developing CouchDB-backed web apps. One step at a time…

I’m happy to announce two talks on CouchDB I’m giving next week in New York City. They are both open to the public, so please RSVP at the links below if you plan to attend either.

I’ll be introducing CouchDB, showing off a tiny example app (which will make you never want to go back to SQL databases again), and giving a quick tour of CouchDB’s Erlang internals in the hopes of enticing new hackers into the fold.

The first talk will be at Arc60 (map) at 6pm on Tuesday (the 16th). Follow this link for the address, more information and to RSVP.

The second talk will be Wednesday (The 17th) at 6pm at Columbia University. If you don’t have a Columbia ID you must RSVP to jonah at ccnmtl dot columbia dot edu by Tuesday September 16th. The talk will be in room 523 Butler Library Take the 1 train to 116th street (NOT the 2/3 express trains). (map)