Why do projects fail?

shutterstock_158722613Why do so many projects fail?  Because we make them fail!

There have been many studies that assess the failure rates of software and technology projects. One, the “McKinsey Oxford Reference Class Forecasting for IT Projects Study” indicated that 64% of projects experienced cost overruns and 78% experienced schedule overruns. This study was probably the most comprehensive, covering 3,607 projects worth a combined USD84 billon.

Why do projects fail?

Each of these studies, and thousands of books and online articles attempt to identify the various factors that cause these failures.

But I want to look at this from a different perspective. Rather than look at the individual factors, I want to ask “why do we still have project failures when we have so many reasons in place not to fail?”. In particular, why do we still have so many failures, even though we have so many methodologies, tools and techniques that are designed to prevent that failure?

These studies have been published since the late 1990’s, so we know a lot about the project failure landscape: it’s immense. So if we know that on day one the probability of impaired project delivery is so high, then why do we still allow it to happen?

“We know why projects fail; we know how to prevent their failure – so why do they still fail?” – Martin Cobb (CIO for the Secretariat of the Treasury Board of Canada in 1995) coining the so-called “Cobbs Paradox”

How can organisations resolve “Cobbs Paradox”?

Why projects should not to fail

The domain of project management has a multiplicity of tools, techniques and methodologies. Project managers have access to numerous service offerings for training, certification and consulting: there are courses of study in Project Management that confer Masters and Doctoral-level degrees.

Online, we have an endless number of blogs offering advice, templates, “how-to” guides, analysis and discussions via social media. And we’ve seen an explosion of variety and change in approach in the Agile era. There are some who have criticized Agile as being a consulting-led industry. There is certainly no shortage of Agile consultants.

Despite all this support and prescription on how to deliver projects, our projects still fail.

Even if projects don’t fail, they still experience “Project Entropy”, my term for all the negative project impacts such as design problems, technical defects, requirements changes, office bureaucracy, risk events, and all those events that arise from the human factor.

Common problems in failed projects

Common clusters of problems appear in projects and organisations where portfolios of projects suffer greatly. But organisations with larger portfolios typically have big investments in a particular methodological underpinning of their product delivery processes.

If you have managed more than a few projects, you will know the difference between the theory and the actual practise of methodology-based project management.

Traditional methodologies such as PMBOK, Prince2 and APM purport to be a “one stop” solution: that every project problem can be resolved through the exercise of a process or technique. And, overwhelmingly, the focus is on the control of Project Entropy, a negative approach that consumes a lot of energy with varying results. And their focus is realized into detailed flowcharts, recipes, and prescriptions.

Agile methodologies, despite their promise, all too often become rigid and process-focused, especially Scrum. Scrum teams, badly managed, can be reduced to a shell of ineffective rituals that mask confusion and lack of clarity on goals, roles and behavior. I’ve seen Scrum, poorly implemented, destroy more value than any so-called “Waterfall” process.

As a project management practitioner for over 30 years, I see the same mistake being made repeatedly.

This is the common denominator in many failed projects: letting the processes and tools drive the project and not the reverse. We forget that tools are the “micro-techniques” of a complex professional activity. They are not the drivers of behaviour and should be only employed when required.

Project management in a time of chaos

This focus on control has been in contrast to expanding variability and complexity in the project context.

For example, consider the market context for the project, the internal business context of an organisation and the rapidly evolving technology context inside and outside the organisation. There are multiple degrees of freedom in the evolution of complexity.

Businesses have become significantly more complicated in the past 15 years, and our problem domains have accelerated in their complexity as the rate of solution change increases. Boston Consulting Group researchers Yves Morieux and Peter Tollman estimate that business complexity has multiplied sixfold since 1955.

“over the past 15 years, the number of procedures, vertical layers, interface structures, coordination bodies, scorecards and decision approvals has increased dramatically: between 50% and 350% depending on the company” – Yves Morieux and Peter Tollman

In response to both observed project failure and the increases in complexity, “brand-name” traditional methodologies such as PMBOX or Price 2 expand over time to deal with more problematic scenarios. For example, APM now has 47 competency areas with a mass of detail. And at least a component of the additional organizational complexity has been in the portfolio management domain, with these organisations investing in more internal structures to support wider and deeper use of these methodologies

So we have more complex environments butting up against more complex methodologies, which produce more complexity in implementation.

But does this expanded complexity help?

Wicked Problems

There is a term that has been used for 30 years in social planning and government problem solving, called a “wicked problem”, a problem that isn’t tractable using standardized approaches. As Wikipedia describes it, a “wicked problem” is:

“difficult or impossible to solve because of incomplete, contradictory, and changing requirements that are often difficult to recognize. Moreover, because of complex interdependencies, the effort to solve one aspect of a wicked problem may reveal or create other problems” – Wikipedia

This term originated to describe societally complex problems such as global climate change, the AIDS epidemic and similar intractable problems.

But for me this definition could easily describe most of the projects I have worked in for the past 20 years. Do we think that problems have become more complex? Ohave organisations created or promoted the characteristics that lead to normal problems exhibiting the characteristics of “wickedness” in projects.

Most of the “branded” methodologies have their roots in practices that predate this rapid rise in complexity. They have roots in Taylor scientific management practices which emphasise process, structure, rules and controls.

So projects don’t have problems due to a lack of detail or rigour in the methodologies.

We make problems wicked

The question is: why do our projects appear to be trying to solve “wicked” problems? For example, I’m just building a website or a content publishing system or an app. How can this be so complicated?

The answer is that the problems are not ‘wicked’ in and of themselves. Instead we make them ‘wicked’ by how we go about framing or/or delivering them.

We need to understand why our requirements appear “incomplete, contradictory, and changing”, or why the interdependencies are complex.

  • Is it because we are unable to capture and reconcile requirements?
  • Is it that we create complex interdependencies through inappropriate design or implementation reasons?
  • Do the requirements change because we take so long to get the project complete?
  • Do the requirements change because the idea is bad?

Projects have problems because the methodologies are focused on the wrong thing: prescribed processes and controls.

If you are trying to improve the effectiveness of project delivery, the solution is not more tools and processes. It is also not more emphasis on execution effectiveness at the tool or process level.


They say that insanity is defined by repeating the same process many times, but expecting different results.

Instead of continually performing the same processes, we need to consider different approaches to defining and executing projects.

Agile, probably the biggest step-change in development thinking, showed promise, but has lost its way in its attempts to gain scale. There are other practices that contain elements that could be useful.

At the end of the day, it’s about groups of people making things. This has been a human practice for millennia. The situation is not new. There has to be a better way.

In the next article I will set out my thoughts on how to make this work better.

Additional Resources