29 September 2006

This post is in response to some comments in my previous post, which linked to Steve Yegge’s blog entry about Good Agile and Bad Agile.

One commenter, JH, makes a perfectly valid observation: criticizing Waterfall and Agile without offering an alternative isn’t very productive. So…

Actually, truth be known, I used Waterfall and several variations of it for many years - with success. We collected requirements, analyzed them, did our design, our development, testing - all the stuff you are supposed to do. And you know, the projects worked out pretty darn well.

Why? I don’t honestly think it was Waterfall that did it. It was the fact that I was working with a motivated team of reasonably talented developers who really cared about doing a good job. In order to do a good job building software it is so amazingly clear that you need to understand the need (requirements/analysis), translate them to software (design/development) and make sure the result actually works (testing).

MSF, by and large, is just Waterfall in a spiral, with some different documents and techniques. Agile is just Waterfall in random order, and with a few different techniques and tools. In the end it all comes down to gathering requirements, analyzing them, translating that into a design, doing some coding and some testing and away you go. You can rearrange the pieces on the board, but they are the same pieces.

Every consulting company I’ve worked for has had its own Methodology (or Framework, or whatever they chose to call it). Each of them was some merger of concepts, techniques and tools from Waterfall, MSF, RUP, Agile and various other Methodologies with which the creators were familiar. And each of them worked just fine.

The primary issue I have with Methodology (capital “M”) is that following the Methodology ceases being a means to an end, and becomes blind Doctrine. The steps in the process cease being tools used for a reason, and become mere Ritual.

“Why are you doing X?”

“Because the Methodology said to!”

“But there’s a better approach. If you just used this technique …”

“You aren’t questioning the infallibility of our Methodology are you? This Methodology was written by the Great PooBah, and has been handed down unchanged through time!”

“What? No, I was just suggesting…”

“HERETIC! Burn him at the stake!”

If you can somehow keep your Methodology from becoming Doctrine and Ritual, then you can be successful – with almost any Methodology, I think. Unfortunately it seems that human nature has an intrinsic desire to conform to Doctrine and Ritual – you can see it in the dogmatic positions of so many Methodologists out there.

As Clemens points out, within a given team, assuming reasonably sized teams, I think you can pick almost any Methodology and do quite well. With the right people, even the Cowboy approach works great in most cases, and it is quite clear that Agile can work fine too. So does Waterfall by the way.

But across teams? You must have some level of formal structure. Some level of formalized communication and process and coordination and authority. I know, that all sounds very ugly, and not at all fun, but it is unavoidable. Conflicting schedules, overlapping dependencies and competing political interests require a higher level of structure in these cases.

Traditional Waterfall presupposes lengthy blocks of time for each step, and isn’t practical in today’s world – even (especially?) across teams. However, a modified Waterfall, along the lines of the MSF spiral, works really well in my experience. Assembling a cross-project team, composed of representatives from the more tactical teams (regardless of their Methodology of choice) for the purposes of coordination and mediation is absolutely critical.

So my “Methodology” of choice? At a team level, pick the techniques and tools that work within your specific environment, with your specific team members – with all their varied talents and capabilities. Take the best from Waterfall, MSF, RUP, Agile and anything else you can find. Some techniques work in one setting and fail horribly in another.

At the level of a large project, or cross-project effort, some variation of Waterfall or one of its modern descendents really is, in my view, the right answer. Few other Methodologies provide sufficient structure and formalization of process to make such efforts manageable.