This blog entry is based on experiences from me and from other people from multiple companies and projects and does not refer to any one particular organization or situation.

Many people (including me) have at some point been amazed and bewildered at how much time and money large organizations spend on software projects. In addition, many developers have looked at such projects and thought they could be built in a fraction of the time and at a fraction of the cost. As far as I can tell this is often the case.

For instance, the Swedish police spent some 30 million Euro building a case management system that ended up being scrapped after three years in development. Or how about the 2.1 billion dollar price tag to develop the healthcare.gov website? There are many more examples of such spectacular budget overruns.

I'm sure there are many reasons why this happens, but I'd like to look at one that can be described a a rift between business and IT that can allow issues to remain undetected and may even exacerbate them. Let's look at how this might happen.

Let's say an organization with little or no experience building software needs something built. Perhaps they bring in a consultancy. The consultancy is optimistic about their ability to deliver the project and charge half as much as others. A good portion of the developers have senior in their title and everyone is happy to say yes to everything. At some point the expected delivery date approaches and everyone is optimistic that the delivery is just around the corner.

Then the delivery date arrives and it turns out there is a delay. There are a few bugs. Work is done to fix them but every time the known bugs are supposedly fixed, new ones are discovered. Sometimes bug fixes break existing features. In addition, some features are still missing and some that have been delivered don't work well. The cycle of listing things that need fixing, estimating the time required to fix them and then finding new things that need fixing keeps repeating itself.

Very likely, the truth is that the system just isn't good. It's held together with grit, spit, duct tape and a whole lot of optimism and denial.

How did things go this far without anyone noticing? Very likely many people did notice, but the not right people – or if they did know they didn't act, but that's another discussion. If the right people couldn't see where the project was headed, why not?

People in charge of other older knowledge intensive businesses like law firms and audit firms tend to be lawyers and auditors. They know the what and how of their respective businesses.

Looking at the most famous and successful tech companies, the same pattern emerges. Bill Gates (Microsoft) is or was a programmer, Steve Jobs (Apple) was well known for being hands on, Larry Page (Google) created the algorithm behind Google, and Elon Musk (SpaceX, Paypal and Tesla among others) is an engineer who takes an active role in product development.

However, in many companies there is a clear separation between IT and the business side. This rift continues all the way down to individual teams where a product owners represents what and developers represent how.

An end user can decide what they think of a product once it's usable, but what about while it's being built? What's the quality of the code? Do the technical choices make sense for the direction of the project? For that matter, in what direction is the project headed and how quickly? Answering such questions requires domain knowledge.

When the business side brings in developers, the business side also tends to end up in charge of the development process while unfortunately often lacking the knowledge and experience necessary to lead the project. Being in charge does not mean being in control.

All projects should have someone in a leadership position – regardless of what the title is called – who knows what to do and how to do it so that they can accurately gauge progress while providing purpose, direction and motivation.