Organizing Teams With Collective Fictions

There is often an observable difference between what is considered the right thing to do, and what actually is being done.

Terry Pratchett said it best when he made Death declare mercy and justice nonexistent: “TAKE THE UNIVERSE AND GRIND IT DOWN TO THE FINEST POWDER AND SIEVE IT THROUGH THE FINEST SIEVE AND THEN SHOW ME ONE ATOM OF JUSTICE, ONE MOLECULE OF MERCY.” (Note that Death is not shouting, he simply speaks upper case.)

We can’t measure justice and mercy. These are collective fictions — things we agree to believe to enable us to get along — and finding consensus on the immeasurable extends to political systems, religion, and most of economics. In a recent article [zwischenzugs] makes the point that methodologies in software development fall into the same category. Like collective societal fictions, methodologies tend to elicit strong emotional responses among those dealing with them.

A software development methodology is a playbook for getting from nothing to something. It’s a control system for how people working on the project spend their time. And there are a lot of these prescribed methods, from Agile to Waterfall, and any combination of letters is likely to turn an abbreviation for a methodology. An interesting game when hanging out in groups of software engineers is to start the “Have you ever tried the…” conversation. Just don’t expect to move to another topic anytime soon.

One disheartening aspect of methodologies is their resistance to scientific scrutiny. Two samples of development teams will differ wildly in so many characteristics that a meaningful comparison of the way they organize their work is not possible. Which will leaves us with anecdotes and opinions when discussing these things.

Current opinions regarding the impact of methodologies on the success of a project range from ‘marginal’ to ‘essential’. The latter position is mainly propagated by consultants selling agile certifications, so you may want to take it with a grain of salt. Whether a team adheres strongly to the methodology or adopts it in name only, it’s obvious they serve a purpose — but that purpose may not match the face value of the method.

Misused and Disliked

Methodologies are often introduced from outside of the team when things don’t go as expected. It is very likely that at this point the project has already failed due to circumstances outside the team’s control. In his classic book ‘The Mythical Man-Month‘, [Fred Brooks] recounts his struggles to ship software on schedule in a time when the volume of documentation for operating systems was measured in shelf-meters required to store the manuals.

The introduction of some methodology or another then is the attempt to regain control. Ironically the ways people resist such attempts seems to be better researched than the effectiveness of methodologies themselves. In a paper from 2003 Alistair Cockburn states:

“I was shocked to discover that people were not using even the simplest form of [Responsibility Driven Design], although it is a simple technique, they had just gone through a week-long course on the subject, and I was available on the project to answer questions. This awoke in me the awful possibility that the smallest effective methodology one could design might already be too much for practitioners to practice on a live project.”

Among developers, scenarios like these have resulted in the feeling that methodologies just don’t work. Lamenting their failings is the equivalent of gathering around the fire and bonding over war stories.

Why bother?

First, there is no way of escaping them if you are a software developer, so understanding what they are and how they work may help you. A lot of these processes are built from observations of how people get things done, so even if they fail to deliver as a whole on their promises, they often contain useful hints on how to improve your work.

[A typical programmer] argues that most methodologies are an attempt to magically inject ‘people skills’ to development teams. His takeaway is that developers should stop caring about processes and start talking to their peers. While good advice in itself, this ignores another feature of established processes. When performing well, they buy developers time to do their jobs by streamlining communication with other parts of their company. It is not uncommon for teams to adopt some methodology or another in name only to get management off their backs or to satisfy nervous inquiries about their process.

In the paper cited above, Cockburn proceeds to introduce his own concept that is typical for the current generation of methodologies. Instead of presenting us with a fixed process, he offers a set of tools that can be adjusted to the needs of each team and project. This mirrors my earlier point: you can’t directly compare one team to another and so the same application of one methodology will also not perform as well (or as poorly) from one team to the next.

So I suggest that you read up on what’s out there  — the comments to [Typical Programmer]’s article are a great starting point. Combine the pieces you like and give it a fancy name to sell it to your boss. That is exactly what the pros do.

38 thoughts on “Organizing Teams With Collective Fictions

  1. I was waiting to be exposed to some methodologies, and then nothing…. much like how most workplaces I’ve worked at actually function. You get told ‘We follow X here..’ and you think to yourself ‘Oh, it’ll be interesting to see how the scrumm meetings go’…. and then the time for the meeting comes around, someone yawns, another person says ‘shall we skip it today, too much on?’ and so on.

  2. “Like collective societal fictions, methodologies tend to elicit strong emotional responses among those dealing with them.”

    Which is more measurable, justice and mercy, or software methodologies? Results matter.

  3. I think the real problem lies in the fact that programming is an art. An art that happens to exist in the world of technology, but an art none-the-less. This is something that managers hate! In the art world you can teach someone how to hold a pencil, and how to draw a face, but they may never be good at it – Same goes with programming. And the sooner people realize that they are no good at something, the faster they can move on to something they are good at. Unfortunately the lack of artists in the programming field, coupled with the demand for the art, has led to a situation where vast amounts of money can be made and everyone wants a piece of it.

    That said, trying to regulate/schedule/dictate how art/programming is to be done, is a difficult task to say the least, and I have yet to see a fully successful implementation. I have seen a number of teams that work well, and an even higher number that are completely dysfunctional, and none if that has to do with process. Add in the fact that requirements are always unclear, open to interpretation, and non-static, and you’ve got yourself a real mess.

    1. Code Monkey get up get coffee
      Code Monkey go to job
      Code Monkey have boring meeting
      With boring manager Rob
      Rob say Code Monkey very diligent
      But his output stink
      His code not “functional” or “elegant”
      What do Code Monkey think?
      Code Monkey think maybe manager want to write god damned login page himself
      Code Monkey not say it out loud
      Code Monkey not crazy, just proud

      https://youtu.be/5W_wd9Qf0IE

    2. Designing a bridge isn’t art. Designing a bridge while making it look good can be, designing a bridge for a specific look and unlimited budget often isn’t. But doing it with a limited budget (time, money, resources) probably is.

      Programming should be more like engineering. It isn’t. But it isn’t an art either. While managing to make a good system given the limitations and expectations from customers and managers may be an art that isn’t directly related to programming, just as building a functional bridge that looks good isn’t directly related to engineering.

      Building designs that are designed for something more than just being functional requires artistic talent. But that often comes from people external from the engineering process itself (not always of course as engineers can be artists too).
      In the best cases it is a matter of cooperation between the engineers and the artists to try to make the building as close to the vision as physically possible – with proper safety standards applied of course.

      As long as programming isn’t seen as engineering we will have this ill notation that programming is an art, because the systems are frankly crap and programmers aren’t as much programming as creatively navigating the sea of crap to make something that at least appears to be working.

      Some are skilled in doing that and begin thinking it is the essence of programming rather than it being programming combined with a skill of making programming work in a very imperfect world.

      And the result are bugs. Many of them. Most programmers see them as something natural. It results in systems that actually doesn’t do what it appears to do (not talking about bugs here). And bloat.

      Just my ¢. Hateful messages below please.
      NB I often take the job as the devil’s advocate, pays well. ;)

      1. The beauty of a bridge comes partially from its utilitarian function aligning it as closely as possible with the platonic ideal of “bridge”.
        Some say this is not art, but it is in my book.
        Thanks for the soapbox!

        1. functionalism: anything that just works is beautiful

          Unfortunately, it doesn’t work for habitation, as homes are not actually “machines for living”. It just produces depressing and brutal concrete bunkers, like a computer generated joke.

          1. “there should be no features about a building which are not necessary for convenience, construction, or propriety” “all ornament should consist of enrichment of the essential construction of the building”

            So basically, if it doesn’t have a structural purpose, it’s out. Functionalist housing was then later “amended” by adding colorful panels or glass facades to parts of the buildings to make it more “fun”, but that only made it look like a soviet kindergarden.

      2. Ah, I believe you have the same misconception as bad programmers. “Creatively navigating the sea of crap” Why does this sea of crap exist? Bad programmers. There are plenty of bad artists, but fortunately, society weeds them out quickly. Bad Programmers on the other hand continue on for decades turning beautiful sculptures into piles of crap for a paycheck.

        Software is EXTREMELY over bloated. Why does it take multple gigs of memory to display the most basic web pages, when 20 years ago, it was done in less than to megabaytes. Why does the windows OS take up 30GB of storage today, when in 1995, it was less than half a GB. Don’t tell me it’s all features and improvements… It still does the same shit, run a mouse, manage windows, launch programs. What was once the windows API, is now the WinAPI, Win32api, directx, .net framework, and on and on. All to draw a few pixels on the screen? Insane.

        Learn to cut the crap, and move forward. Backwards compatibility broken? Sorry, don’t upgade until you NEED to. This is the crap that exists due to bad programmers building on other bad programmers.

        Cut and run. Hell, even arduino is bloatware compared to the simplicity of writing the C code to blink an LED. Lowering the barrier to entry makes the system even worse.

        IMHO

      3. Programming is a craft really, it has aspects in which it resembles art and aspects in which it resembles engineering, but on the whole I find myself, as a programmer, having more common ground with a plumber, machinist, or electrician than with artists (whose product, while worthwhile, doesn’t have the practical constraints of needing to successfully perform a utilitarian function when completed) or engineers designing mechanical or analog electonic systems (much of their efforts are concerned with analysis of the non-ideal properties that make real world materials and systems differ from their ideal models).
        Programming has a strong creative aspect, and loads of practical constraints, but our materials generally act in deterministic and fully specified ways and constraints then are more on resources, timing, and compatibility with other systems already deployed.
        In my mind the fusion of creativity, technical knowledge, and analytical thinking in the pursuit of designing an efficient functional product to meet a specific need is what really defines a craft. Engineering of many sorts can also be crafts. While some people get excited about the purity of math or art, I personally find the concreteness and pragmatism of craft-type disciplines more compelling.

      4. Programming *is* seen as engineering, when it’s called software engineering and done by trained and certified software engineers. The term software engineer dates back at least to the 1960s.

        Of course, anyone with a passing knowledge of BASIC, Python or Javascript can call themselves a programmer. It seems like plenty of mediocre programmers have gotten pretty good at using flashy buzzwords to sell themselves as a cheaper, better alternative to software engineers to managers who are paid well to know better.

        1. I don’t call myself a programmer just because i took one programming class for my degree.

          And that was long enough ago that I got lost after they ditched line numbers. Ok, I did take a Visual Basic class in the early 2000’s, but that atrophied from lack of use once I got one project completed (interfacing some HP signal generators to a PC for automated testing).

    3. An artist to an illustrator is like a hacker to a programmer.

      There exists this category of people who are well-versed in various techniques of art, have learned to work to a schedule and honed their skills, including people skills, to a consistency which enables them to be reliably hired to produce pictures, drawings, paintings, sculptures etc. to a specification and a need. They’re called illustrators.

      The artist is typically not consistent in their skill because they’re not interested in being consistent – they’re interested in being creative in a self-expressive and exploratory way. They don’t care that they have huge gaps in their knowledge and given a task requiring any systematic approach, they often produce poor results. They tend to prefer to get drunk and dribble paint over a canvas to produce interesting results, and they complain and act like princesses when someone demands them to replicate the results, fundamentally because they can’t.

      They name their lack of consistency, “lack of inspiration”, because they work to some extent more by accident and luck. Remember Bob Ross – “We don’t make mistakes, we make happy accidents.” – whose method was based exactly on dabbing your brush on the canvas and only /then/ deciding if it’s a bush or a rock. I’ve personally tried it, and it’s a feedback method where, if you try to control and design what appears on the canvas – if you try to paint instead of just letting painting happen – you’ll ruin it. It’s amazing what a complete amateur can produce, but you can never do the same painting twice.

      The artist is smart at coming up with new things, interesting things, clever things, but it takes the illustrator to come up with a technique to replicate the process and make more Jackson Pollock on command.

      And that’s why programming is not art. Programming is illustration – turning someone else’s ideas and needs into reality.

      1. True to every single letter. I could not put it any better.
        But this said, I also at times enjoy doing the other kind of programming, like firing up the good ol’ Common Lisp REPL and doing some hacking or thinking of new and interesting ways to have an Arduino blink some LEDs, but that’s recreation.
        I don’t get paid for art. I get paid for solutions. Which doesn’t mean that there is not art to be produced and that artists are not needed and that the artist and the illustrator cannot be one and the same person. Just not at the same time.

    4. Be careful calling it an “art”, or we’ll end up with STEAM education. Of course, welding is also an art, to a degree. And i’m speaking to honest-to-god bridge welder. Not just artist that weld sculptures.Heck, the guys that built motorcycles on TV while swearing were also artists.

  4. >>One disheartening aspect of methodologies is their resistance to scientific scrutiny. Two samples of development teams will differ wildly in so many characteristics that a meaningful comparison of the way they organize their work is not possible.

    Rather than compare the different teams, compare the projects. That should help. Surely a methodology designed to keep web pages up to date shouldn’t be compared to a method for writing missile firmware, under most circumstances. Tailor the method to the problem, not the team solving it.

    >>Among developers, scenarios like these have resulted in the feeling that methodologies just don’t work.

    Probably more related to their implementation than the actual methodology, barring the extreme example above.

    1. It’s the seperation between the physical world and the “metaphysical” world of ideas and ideals. Programming is simulating this fake “metaphysical” world in the “real and concrete” world of materials we interact with.

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.