What most companies get wrong about Agile?

Developers are often grumpy about numerous meetings every week. Retrospectives and multiple planning meetings take away from what most of them aim at - to do their job and build software. On the other hand, after all this planning, deadlines are often not met. Product and project managers as well as various stakeholders are understandably also upset by that. They want predictability because they care about business outcomes and want to keep on track with things like customer acquisition and revenue growth. Sometimes leaders try introducing more structure and more process to tackle that, perhaps in an attempt to make it into a science. Which means you then need more people to manage the process itself, only to add to the problem of too much process.

So why does this happen? These goals seem to be aligned and yet there's often this tension between developers and managers. Both want to establish a well oiled machine which efficiently produces software. What do most organizations do? They buy into the promises of…

Scrum

Scrum has become like a default in the software industry. Standups, planning meetings, sprints, a Jira board and you're good to go. You are now agile and ready to deliver amazing products on time and make the world a happier place.

Perhaps my social circle is biased but I don't think I've heard a software engineer friend or colleague who's said that he loves scrum. They often complain about too many meetings or problems with story point estimations. If the estimate is not correct, it's then used against them as to why a deadline wasn't reached. This then creates a really bad incentive of inflating the story point value of a task just to create this illusion that the project is on track to reach a deadline and then the deadline is also pushed back to accommodate for that inflation. And you know what happens? Tasks expand to fill the allocated time (Parkinson's law).

A friend recently said something which I kind of felt but hadn't articulated as well as he did in that moment - you can go as far as saying Scrum is not really agile as defined by the Agile Manifesto. The very first principle is 'Individuals and interactions over processes and tools'. Instead, Scrum gives you a specific set of processes. How often do people in your standups say something like 'yesterday I worked on the thing, today I'll continue working on the thing'? Is this really a useful meeting?

Scrum has spawned a whole industry of agile coaches and scrum masters that you can hire. What happens is the incentives they have are to sell you a coaching service to solve your problems. Why would they ever coach you to a place where you don't need them? It'll get them out of their job! What's the point of having all these roles that don't really know the domain of your product and neither work on implementing it? A scrum master can be anyone on a team - a product manager, tester or a developer. It's nothing more than moving tickets on a board.

I don't think Scrum is necessarily bad. Obviously the need for structure exists. People who become managers often have personality traits that make them more drawn to structure. The opposite of that would be complete chaos where nobody knows what needs to be done. So the question is what type and how much structure is needed?

The problem I'm trying to articulate is that Scrum is like an off the shelf solution which isn't tailored to your business. Of course it's easier that way but people forget about being agile about the agile process itself!

Context matters

It seems to me that many organisations talk about being agile and yet their delivery cycles are still in the order of months. I thought agile meant delivering working software every sprint? This easily leads to all the issues outlined above - deadlines are months in the future, stories get overestimated, tasks fill that expanded allocation in time and deadline gets pushed back because something new always comes up.

This seems to be especially true in B2B software where contracts are more often than not negotiated with a period of months or years. So if I know you're contractually obligated to pay me next month… I don't have an incentive to be in a hurry and deliver new features for you.

In the B2C world it's very different - a new competitor might pop up any minute with a better value proposition for your customers and they'll be out your door next month. You can see how monthly subscriptions incentivise you to move faster. That's not to say that all B2C companies are the beacon of agile, they can also fall victim to a lot of these issues.

Does that mean that B2B products are inherently not suitable for agile? I don't think so. It would just require a different approach to establishing processes in the organization. Too often in B2B software there's no mention of customer interviews. It's like there's a wall between two organizations - you pay me and I'll give you a software service without deeper analysis of customer needs. What happened to customer collaboration from the agile manifesto? Has anyone used HR software like Workday? The user experience on these tools is generally horrible in most cases.

The context of a business is what really matters - business model, ease of gathering data on customers and their behaviour, intricacies of your industry and the regulations that come with that. These and many other factors could influence the workflow. Scrum doesn't account for any of that.

So if Scrum is kind of agile but not really… what are we aiming at?

The Agile Mindset

Agile is NOT a set of processes and tools. It's not about Jira boards and tickets. Agile is a mindset and a set of values of the people in an organisation.

The word agile itself means to be able to move quickly and easily. Using processes is fine unless the process gets in the way. Not all processes are going to be applicable to every organisation. You need to carefully examine the context of your domain, business model and access to customers for collaboration. And even then, those things fluctuate so you need to reflect if a process you established two months ago still makes sense. It's a continuous process of reflecting and finding ways to improve not just the product but the workflow as well. The point is there isn't a one-size-fits-all solution for what structure a given business needs - that's the core idea of agile that so many companies often miss!

One of the best books on product development in my opinion is The Lean Startup by Eric Ries. It may sound like it's only relevant to small companies just starting out, but I believe large companies that grow and forget or ignore these ideas are the ones that start stagnating. The book manifests these agile ideas by defining a very high-level process called Build/Measure/Learn. The build part is self-explanatory, measure is about collecting data on whether the product is useful and does the job (could be customer interviews, usage data, etc) and finally learning from these data before building more. The real key to this being useful is making that feedback loop as quick as possible. That's agile. It's likely that in the process of building a product there will be mistakes so learning from them as quickly as possible is crucial. And yes, this means sometimes changing the process itself. That's why having a 6 month delivery cadence is too slow. You'd be half-way through your 12 month product roadmap and it gets much harder to correct course if something doesn't pan out. Even if the company has the money to afford that… it just leads to potentially wasted time and money that could have gone towards other product improvements or more innovation.

An analogy that illustrates this nicely is Tesla's manufacturing hell from a few years ago . There were a lot of inefficiencies in the factory which limited the output of cars. Yes, Tesla makes hardware which is different, but think of your company workflow and processes as the factory where the output is working software. By optimising the factory, you improve the product.