Every once in a while, something happens that makes me completely reassess my take on "what's next" in the technology sector. The original Macintosh was one such epiphany, Apple's Hypercard was another (Hypercard is a much-unappreciated program - had Apple added network links to it we'd have had a writeable, scriptable WWW in 1989). Netscape's first browser was a third. On Thursday, the release of Yahoo! Pipes shook my world.

The Pipes slogan is "Rewire the web", and that's what it purports to do - as a visual editor for RSS feeds that dramatically lowers the time and expertise needed to create a "mashup". Pipes lets you create your own personal web channels, and Gina Trapani at Lifehacker has a great example of how to do this.

And so, if your interest is, say, the NFL draft, it's pretty easy to wire up all the draft-tracking sites so that if a potential 7th round pick out in East Treestump gets a hangnail, you'll know about it. But the implications go so much further...

In a REST-ian world, it's pretty easy, once you've modeled your world as a set of resources, to create RSS feeds that track the changes in those resources. With an application like Pipes, it's really easy to tie those feeds together into new applications. This is the future of "SOA" and "Web Services."

Forget SOAP, and the growing pile of WS-* standards as a means of wiring up disparate applications. REST-ify the applications, use RSS to publish the changes, and a tool like Pipes (or one of the imitators that should be quick to follow) to wire the whole thing up. SOAPians - put that in your pipe and smoke it!

Seriously, though, while the Pipes idea is great, there are still some issues to work through. The intuitive purity of the Pipes concept was proven when everyone from the merest script-kiddy to the deepest hacker jumped on Yahoo! and bought the Pipes site Down! all day on launch day. Yahoo! knows a bit about keeping large sites up under volume, so we are surely talking about a big human wave here.

But that's not all, either. XML is wordy, and XML transformation is slow and computationally intensive. I'm curious to see if the sheer processing load leads to more restricted (e.g. JSON) or more efficient/binary formats as a wire protocol for Pipe-y services.

I'm convinced these things will be worked out, and Tim O'Reilly's comments about Pipes as a "milestone in the history of the internet" will prove to be true.

As such, let this be my 14th prediction for 2007: February 7, the Pipes release date, will be looked back on as the day SOAP died. Not that it's bad, but the future has been shown to lie elsewhere.

Rails 1.2 was recently released, and I've been updating and upgrading Pikaplanner to take advantage of some of the new features in 1.2.

The highlight of Rails 1.2 is built-in REST support, and I've been updating my controllers to provide what amounts to a standard, easy-to use, web-services API.

My first step in the process was to work through some of the examples in Rob Orsini's terrific developer volume Rails Cookbook. Diego Scataglini's examples in Rob's book provide a simple and quick intro to REST support in Rails 1.2.

REST is a great example of Clayton Christensen's concept of a disruptive technology. I've worked with SOAP and RPC in the past, and at first blush REST is less powerful than either. Still, with Rails, REST is easy and cheap to adopt at the service-provider end, and easy to experiment with at the receiver end.

For example, one of the first code areas I worked through was a simple product listing. The goal here is to provide both a web (HTML) and web services (XML) interface to our (sample) product information.

The REST support in Rails makes it easy, and all we need to do to turn our web pages into a web-services interface is tell (though the URL) our application that we want XML (instead of HTML) back.

So: to see the web page listing our products, we'd use "/products", as shown below:

So far, so good, so common—in Rails we've called up pages like this in the ":controller/:action/:id" form (that defaults to a GET on index.html here) zillions of times. What's new, with REST and Rails 1.2, is that all we have to do to have a web services interface is tell the application we want XML back, instead of HTML. So our URL is of the form ":controller/:action/:id.xml", as shown below:

and as you can see we get XML back - a web services interface practically for free (developer time-wise).

There is a lot of magic underlying this simple example, but a handful of key points stand out:

1) All we need for our "web services" interface is to tell our controller we want XML back.

2) We could have gotten JavaScript back (where that is appropriate) in the same manner

3) The key shift in development mindset is that the interface is not imperative (as SOAP and most web services are) but declarative—everything important in the system a) has its own URL for CRUD access, b) exists as a stateless resource, in which complex actions can be created by changes in state of one resource triggering actions to change the state of other, related resources, and 3) is capable of varying the format of responses, depending on what is requested.

That third point is a mouthful, and the core of stuff on which theses have been written.

Declarative programming is a significant mind-shift, one I'll be writing more about in the future.

One of the unsung facets of working reasonably up in the org structure of a large, successful tech company is that you get a unique view of how the tech media machine works.

Tech writing (like Tech Marketing) is hard, because 1) Any sufficiently advanced technology is indistinguishable from magic, 2) Only magicians really understand the magic, and 3) Magic skill != writing skill. So, back in those ol' days, I wasn't that surprised to find that media pieces often originated in the Marketing departments of big companies, but I am always surprised when media outlets appear to run the Marketing feed unedited.

As a sign that some of the old ways still prevail, today we have the following two, loosely-coupled headlines: Vista success hinges on developers and Developers take advantage of Vista. A nice two-headline tautology.

I'm sure both headlines are valid, but looking at things that shallowly completely misses what's going on in software right now. OEM Agreements guarantee Vista's success - things might be better if developers come aboard, but Vista should do just fine (on every new machine shipped) even if they don't.

This was probably true even back in 2001, when Microsoft CEO Steve Ballmer famously cried "Developers! Developers! Developers!", but it really doesn't bear up to scrutiny today. In 20-odd years of mass-market windowed software development, Zawinski's Law ("Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.") - has basically held, and now even the simplest "mainstream" apps offer a staggering amount of functionality.

In the desktop software world, probably 95% of all usage is performed using about a half-dozen applications (Word, Excel, PowerPoint, Outlook, Browser, Itunes, IM Client). Even if all the developers stay home, that still means that Vista will lose functionality only at the remaining 5% margin.

There are two things worth drawing from the articles: 1) We'll see better software on the Vista platform if developers flock to Vista, but Vista's success does not depend on that flocking, and 2) if you are writing software today you have to be mindful that most (95%, if you accept the guesstimated figure above) of the "horizontal software space" is already filled.

That's not to say that there's not room for great new software, only that great new software isn't going to take a piece of the current pie—the only way to create great new software is to make the pie bigger.

In a later post, I'll talk about models for doing just that.

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away. — Antoine de Saint-Exupery

I've always liked Clayton Christensen's book "The Innovator's Dilemma" as a metaphor to explain product innovation and succession. To vastly oversimplify the main idea, companies get so good at "hammering" that everything: screws, rivets, staples, welds, etc. is either treated as a "nail" or crushed in favor of "nails."

Software is an industry that produces natural monopolies, and so for software The Innovator's Dilemma applies, not just to individual companies, but to the entire industry.

This makes things really hard for software developers today. Microsoft, with Windows started out as a system for non-networked individual devices, and has tried to expand all the way up to clustered 'mainframe' solutions. Enterprise Java started out trying to boil the ocean, boiled it, only to find that ocean-boiling solutions aren't of much use when all you want is a cup of hot water for soup. In trying to solve the same universe of problems, both the Microsoft and Java camps have adopted the same set of ills. One size doesn't fit all.

In this case of two increasingly over-engineered systems, I prefer openness, not because of what you can add, but because of what you can take away. The key ratio for software managers to watch I call the "overhead ratio" which is (Time spent on the solution the customer will pay for) / (Total time spent). The ideal solution would have a quick installation, rapid configuration, and rapid user iterations to a state of solution-ness that is "just right." The ideal development effort will spend all its time on "value-add" components, and none on "configuration."

Both Microsoft and Java development have a pretty onerous overhead ratio - getting all the components right takes an increasingly long time. As I've previously noted, I like "virtual appliances", and the "Convention, not Configuration" mantra of Rails/Grails for just this reason.

But there's more to it that that, a notion that I'll posit as The Godfather's Law: "Technology components that reach a sufficient level of irritation are made to SIMPLY DISAPPEAR." Credit Mario Puzo with this: "As soon as the Corleone Family set up their usual business liaison with the local police force they were informed of all such complaints and all crimes by professional criminals. In less than a year Long Beach became the most crime-free town of its size in the United States. Professional stickup artists and strong-arms received one warning not to ply their trade in the town. They were allowed one offense. When they committed a second they simply disappeared."

Software licensing is a headache? BING - SaaS. MS Shop that doesn't run Linux? BANG - Virtual appliances. No (MS) or overcomplicated (EJB) ORM? - BOOM - Hibernate.

...and so on. As the best software people are "Lazy, Impatient, and Hubristic" (a description that also fit Sonny Corleone), no system of excessive overhead ratio will last long. One key to software success is to figure out when a paradigm is about to get "bumped off", and to move in on all the action when it happens.

There is a great opportunity for software development organizations who can surf the transition. The key is not in choosing the "next Java" or hot development environment, but in identifying all of the barriers to adoption and making them "simply disappear." Even if they might agree on nothing else, the two "C's" (Christensen and Corleone) would agree here.

Programming platforms are like holiday dinners - to succeed they have to offer something for everyone, and if they/you sample a bit of everything then they (like you) are likely to end up bloated.

Such is the case with Java, a pretty language with clean syntax that, in its full Enterprise-banquet form, has been compared with eating an elephant. Enterprise Java offers everything, and development teams have to make a lot of technical calls before the first line of code is ever written: EJB or JDO? bean-managed, container-managed, or framework-provided persistence? EJB 3.0 or JDO 2.0? Struts or JSF or Spring? The wrong call, of merely the passage of time can make for pretty hard-to-manage code.

Still, Java's been used in enterprise development for years now; it has a large library of code and many trained developers and all of this makes for a pretty appealing banquet-table. What is a development team to do?

A common answer lies in the realm of "opinionated code," in which the development team chooses a set of beliefs, and architecture follows. Ruby on Rails is a good example of opinionated code: everything, from default directory layouts to ORM framework (ActiveRecord) to dependency injection is baked in.

RoR is great, but it doesn't address the great banquet of Java development and platform that's evolved over the past nine years. Enter Ruby-on-Rails-for-the-Java-set: Groovy (the language) and Grails (the platform).

Groovy is a dynamic language based on Java and strongly influenced by Ruby, and Grails is a JEE-spiced framework modeled-on and very similar to Rails. Groovy reached it 1.0th birthday this week, and Grails is nearing it's 0.4 release, so the platform isn't particularly mature yet, but has solid financial backing and is already being explored in "mission-critical enterprise applications".

"So what?" you say—is this instant relief for enterprise development, or a "wafer-thin mint." My first experiences with GoG indicate the former. Setup is fast, and the initial development environment provides ORM (from Hibernate), inversion of control and MVC (from Spring), logging (log4j), and layouts (from SiteMesh). This is opinionated software, the decisions are made and many of the patterns that have made RoR popular (DRY, convention-over-configuration, etc.) are baked in to GoG as well.

Groovy and Grails may not be ready for full enterprise deployment, but the platform is easy to grasp for both Java and Ruby developers, provides much of the development speed and elegance of RoR, and embraces the libraries, seamless integration-to and feel-of JEE. It should be a great platform for prototypes and rapid JEE development, and 2007 should reveal whether it will mature to an enterprise-ready framework.

Tastes great, less filling...