The Freak Parade

Summer is tumbling into fall and apparently it has stirred the technology muses because the last few days seem to have invoked a collective urge to wax philosophical about development practices in the small corner of the blogosphere that I follow. Jimmy Bogard started the ball rolling comparing  TDD to living a more healthy lifestyle, Ayende expressed his understanding of the true, sensitive core of Agile, Kyle Bailey wondered aloud if the ALT.NET horse is pulling too far ahead of the cart, Rod Cottingham of ReadWriteWeb briefly awakened from his Social Media junky trance to wonder if there wasn’t more to life before pushing in the plunger once more and sinking back into the carpet, and the usual band of roaming ALT.NET monks came to the vigorous defense of TDD, which was apparently issued a challenge on the ALT.NET mailing list. All this in addition to the usual, melodious gospel music emanating from Los Techies and CodeBetter.com.

For me, it was quite a treat. In addition to being a rabid technologist, I am also an armchair philosopher, so this kind of material is food for me. I will have more to say about the enormous question so fleetingly put in Rod Cottingham’s post another time, but today I want to think a little bit about the flurry of emotion regarding TDD, because while the passion its practitioners exhibit for the methodology is in itself compelling I don’t feel they are otherwise expressing its true value very effectively to those who I presume they are speaking, which is to say to me, the layman and the novice.

I myself have only just begun to experiment with Test Driven Development in particular and Agile methodologies in general, along with the rest of my small and equally inexperienced team. Speaking from such a demographic, I can confirm that TDD is a difficult discipline to become comfortable with. Speaking from a project management perspective, I can also confirm that starting from standing still getting up to cruising speed with TDD is, at least superficially, expensive from a productivity standpoint. I believe every member of my team is both talented and passionate about the mystical process of creating quality software, but taking up TDD is nevertheless quite challenging.

Even so, our limited and amateurish experience with TDD so far has deeply impressed all of us, so much so that the productivity cost of the learning curve has never once felt like anything but an investment. To wonder whether or not the cost is worth the benefit is perplexing to me, I can’t imagine having that emotion. Maybe I’m spoiled by the talent and passion of my colleagues, maybe if you find yourself alone amongst a crew of 9 – 5 clock punchers the row is much tougher to hoe, but you don’t have to scratch very deep into the surface of TDD to find its extraordinary value, even though it does require constant discipline (which often falters, by the way) to continue using it, at least at the early stages we are in.

What is this great value that outweighs the effort?

Well, there isn’t any point in my spelling out the tangible, objective benefits. That the TDD monks express quite well, and anyway I’m a layman, so I wouldn’t be able to do it effectively in any case. What I find strange, though, is how TDD seems to always be discussed outside of its Agile context – as if it were a side dish you could have with any kind of meal without it losing any of its overall value. Rather, I see TDD as the chorus to the Agile song – it may be beautiful to sing in isolation, but it grew out of, and belongs to, a larger melody, and to really understand its value an experience and understanding of the larger context is of the utmost importance.

The larger context? Iterative, outside in development, of course. Agile is a highly iterative development methodology that focuses first on the need and then on the implementation in short repetitive cycles. It has the potential to transform software development from a giant thought experiment in which we attempt to construct  tightly fitting, complicated wardrobes of clothes for our customers by asking them to describe the shapes of their bodies in perfect detail and list up front every social function they will ever need to attend, to one in which we take small, precise measurements, create one article of clothing at a time, try it on the customer, have them spin in front the mirror, and make adjustments accordingly: to the patterns, to the materials, to our tools and our processes.

The universe itself is rhythmic, cyclical and defined by nothing if not constant change, and so therefore are our lives, our work and the needs of our businesses. Software, unlike other more traditional engineering disciplines that deal with physical space and universal themes, must keep constant pace with the rhythm and change of human activity for it to be useful. Not only that, but the complex needs software aims to serve cannot simply be measured, like a plot of land or the gorge a bridge must cross, it must intersect usefully with myriad human minds rather than relatively predictable physical material, and the human mind is famous for not knowing itself very well.

So Agile attempts to adjust for this by addressing human need in an interactive way – assess the general outline of the need first, build a little something, see how it fits, make adjustments here and there, fill the general shape of the need, add the detail in layers as it is discovered. It brings to my mind an image of a potters wheel – the clay spins always, an iteration at a time, you shape a vessel first with your fingers, relying as much on tactile feedback as you do your logical understanding. Then you apply detail, little by little, and when it is finished you can fire it in the kiln and paint it, but not before. If you build the detail too soon, it will quickly get rubbed away as the true need emerges, or worse, it will remain because you’ve put the thing in the kiln too soon, but it won’t be appropriate to the piece you’re creating, so you cover it up with fresh clay and fire it again, and you end up with something structurally unsound, or ugly, or altogether useless.

What does all this melodrama have to do with TDD? TDD is your fingers in the clay. Code is clay, after all, not gears and pistons and fittings, it has constraints, of course, but is largely pliable imagination. TDD is what finds an ideal shape for the code that is elegant, appropriate and capable of realizing the emerging and actual (not theoretical) needs of your customers as they are incrementally uncovered by the larger Agile process. TDD feels out, does not deduce via thought experiment, an appropriate API. This has been my takeaway from our limited experiments with the methodology, at any rate. TDD as the smallest possible step, the smallest possible rhythm, red-green-refactor, red-green-refactor, you rely predominantly on a sense of practical aesthetic, an ability of the mind to find a natural shape for things, a process that is usually expressed in terms of “reducing friction” by the community of vocal practitioners. Of course this is a logical process too, of course, not just an artistic one, there are general rules and guidelines, it is a strange coupling of mathematics and logic with imagination and creativity and pseudo-tactile feedback. Your mind is clicking along in a scientific way, but a larger part of the brain is enlisted to participate, a part that, as engineers, we too often discount, more than just the cold eye of logic with too little information at its disposal and too high an opinion of itself. In the end its all very Taoist The Taoist butcher, as you may know, never has to sharpen his knives because he always cuts his meat precisely along the grain – he isn’t cutting at all, actually, but merely separating the material along its natural seams. TDD aims to find those seams in the seemingly solid material of a programming language as it applied to a complex human problem and the result is often of very high quality, both aesthetically and functionally, and the secret is both qualities are the same quality when approached in this way.

Anyway, that was a lot of grandiose hoo ha, and I’d be surprised if any of you left brained technologists have made it this far. Whether or not I’ve made any sense, my main point is that when practitioners are always saying TDD is about design and not about testing, I think this is in essence what they mean. Even if it isn’t, that distinction – the distinction between “design” and “testing” is tragically under-emphasized, or at least under-explained, and that distinction, in my mind, is the crux of the whole matter, is the first thing that should be said about TDD to anyone who is interested in listening. In that context, the question of “is it worth the cost” is almost nonsensical.

If by some miracle you have read this far, I’d love to hear what you think – what is your understanding of TDD? Or, do I simply need to lay off the hookah?

Replace “Toaist” with “Programmer” in that last sentence and you have the essence of our industry