Transformations Often Fail
It is an unfortunate, open, dirty secret in our industry that most agile transformations fail. Depending on how you define an agile transformation, nearly 70% (or even more) of these transformations fail to achieve their advertised goals[1][2] [3]. Why?
One root cause for transformation failure is they take a “…process and tools” over “…individuals and interactions” approach to the transformation. Agile transformations often focus on the technical system, introducing new frameworks, practices, and roles, all enforced by new tools. The expectation is for the social systems, that is, individuals, attitudes, skills, and values, to adapt to – even comply with – the technical transformation. However, entrenched hierarchies (e.g., silos), individual belief systems, social relationships, and loyalties often impede the adaptation of the social system to the new technical practices. There is a reason why the first value of the Agile Manifesto is “individuals and interactions over processes and tools” and not the other way around. Perhaps we can learn why such technically focused agile transformations have such a high failure rate from 1950s coal miners.
The Haigh Moor Coal Mines and Social Technical Systems
In the 1950s, post-war Britain’s economy was expanding, and so was the demand for coal to fuel that expansion. Yet, the newly nationalized coal industry was not doing well despite heavy capital investment in mechanization – processes and tools. Worker morale was poor, and turnover was high, along with high absenteeism rates. Except for the Haigh Moor seam in South Yorkshire. There, production was increasing, worker morale was good, and turnover was low. Researchers were dispatched to the Haigh Moor to learn why. They soon discovered that it was not just the introduction of new technology that led to substantial improvements. Instead, the dramatic improvement resulted from both the deployment of new technology and the mine management’s authentic delegation of authority to the miners to manage their way of working. Together, these changes enabled the miners to socially re-organize themselves into self-managing teams.
The researchers observed, “The improvements at the Haigh Moor mines are an example of a socio-technical system where an organization’s objectives are best met not by the optimization of the technical system and the adaptation of the social system to it, but by the joint optimization of the technical and social aspects, thus exploiting the adaptability and innovativeness of people in achieving goals instead of over determining the manner in which these goals should be achieved[4].”
As Agilists, the lesson we learn is the improvement in the work system’s outputs result from joint interactions between these two systems. Thus, any design or redesign of a work system must deal with both systems in an integrated form”[5]
But perhaps as you read this, you are thinking, “Well, duh.” We’re agile, and that is what we’re all about. We’re all about creating self-organizing, self-managing teams. Really? That is what we like to say and maybe even want to believe. But who really controls the team’s way of working? In classical Fredrick Taylor industrial era machine models, management owns the way of working with which workers have little to no control and must comply with. This is where agile transformations often start to fall short on individuals and interactions.
While we speak eloquently about retrospectives, learning, and continuous improvement, what authority over their way of working does the team doing the work actually have? What can they experiment with to learn, change, and improve? Can they change their story workflow? Can they change their meeting agendas and frequency? Can they change story formats? Can they change their definition of ready and done? Can they even delete a story from their backlog? Can they change their team dashboards? Can a team make these changes without drawing the wrath of their Agile coach or some centralized Agile consistency group? Can they go beyond pedantic agile and take ownership of their way of working? Or must they comply with a top-down imposition of their way of working?
Meet the New Boss, Same as the Old Boss[6]?
A “processes and tools over individuals and interactions” style transformation accomplishes nothing more than exchanging one Frederick Taylor-inspired machine organization model with another. If we believe an enterprise is a social-technical system, and if we are genuinely seeking transformation like that of Haigh Moor mines, then what does a holistic joint optimization of both the technical and social systems, a genuine “individuals and interactions over tools and processes” transformation look like? What can we learn from 1950s coal miners about agile?
In future blog posts, I will elaborate on how we can create a genuine “individuals and interactions over tools and processes” transformation. Those blog posts will explain how software development is primarily a social process and how social processes have the biggest influence on outcomes, often swamping all other cost drivers. Subsequent blogs will introduce five principles for a more biological perspective on transformation that creates consistency without improvement killing machine model compliance. You can preview some of this content by watching a recording of my Agile India presentation, “Don’t Bulldoze the Swamp for Agile.”
[1] N. Ramesh and D. Delen, “Digital Transformation: How to Beat the 90% Failure Rate?,” in IEEE Engineering Management Review, vol. 49, no. 3, pp. 22-25, 1 third quarter, Sept. 2021, doi: 10.1109/EMR.2021.3070139.
[2] https://www.forbes.com/sites/forbescoachescouncil/2022/03/16/12-reasons-your-digital-transformation-will-fail/?sh=5122b2351f1e
[3] https://www.mckinsey.com/capabilities/transformation/our-insights/why-do-most-transformations-fail-a-conversation-with-harry-robinson
[4] Cherns, A. (1976). The Principles of Sociotechnical Design. Human Relations 29(8), 783.
[5] Bostrom, R. P., & Heinen, J. S. (1977). MIS Problems and failures: a sociotechnical perspective part I: the cause. MIS Q., 1(3), 17-32.
[6] The Who, Won’t Get Fooled Again