End the User Story Tyranny

Somehow an agile myth has arisen that user stories are the only way for an agile team to capture needs. As a result, I have seen the crazy and inappropriate application of user stories create not only a lot of angst and confusion, but also a lot of resentment in agile teams. In my coaching practice, many of the issues I see with agile teams can be traced back to a low quality expression and understanding of needs. A common prescription for this is the rigid enforcement the user story format and the demand for precisely written acceptance criteria. Coaches and trainers love this because they can create the illusion of making a useful intervention. The team can now write textbook-perfect user stories. The problem is the original issues have not disappeared, because form is not a substitute for function. The team is still “agile in name” only.

Part of why Agile works is its focus on getting something done quickly. User stories are a really good way to quickly capture and create understanding about what valuable working software needs to do. User stories enable the rapid delivery of working software by capturing the desired behaviour of a small slice of the system. By delivering small slices of working software to our stakeholders we can quickly learn what the really need. So, what’s not to like about user stories? How can I assert that these beautifully intentioned and simple artifacts are imposing a tyrannical rule on a team, or even an entire program? When used appropriately, user stories are a wonderful and amazing tool. However, when we choose to disregard context and force all needs to be expressed as a user story, then they can become a tyranny imposed on the team or the program. Often the result is a frustrated team that believes agile does not work for them.

User stories were originally an XP artifact and were popularized with the publication of Mike Cohn’s “User Stories Applied” in 2004. Back in 2004, the scope of agile methodologies – at least for textbook writers – was a team of seven plus or minus two who worked directly with a stakeholder (e.g. the customer representative, or product owner). The systems they tended to build were customer-facing and had direct users. The world began and ended with the product owner or customer representative who co-wrote the user stories with the team. However, over the last 15 years we have dramatically stretched agile thinking from those small customer-facing teams to much bigger teams working on larger and more complex systems. User stories are not necessarily the most effective or appropriate tool for capturing in expressing needs in all domains.

Many of us tightly associate user stories with agile because when we were learning Scrum, our instructors dutifully took us through user story writing and judged us on our ability to express needs in the <As a> …<I want> …< So that> …format with precisely scoped acceptance criteria. But if you search the Scrum Guide you will not find a single reference to user stories because user stories are not a Scrum artifact. All Scrum declares is:

The Product Backlog lists all features, functions, requirements, enhancements, and fixes that constitute the changes to be made to the product in future releases. Product Backlog items have the attributes of a description, order, estimate, and value. Product Backlog items often include test descriptions that will prove its completeness when Done” – Scrum Guide

User stories are certainly a good way to satisfy this definition of a backlog item, but they are not the only way – and while user stories can make good backlog items, good backlog items are not necessarily user stories. Unfortunately, some agile methodologies and tools have bolstered the idea that user stories are the only true backlog item by defining their backlog items as “stories”.

This post is not an indictment of user stories. Rather it is about how dogmatically focusing on the user story format without consideration of context or understanding creates the illusion of agility – what is sometimes called cargo cult agile – while dooming the team to frustration and even failure. Agile teams work because they focus on getting something done, and the function of a user story is to create understanding about some small sliver of value the team can get done within a short timebox. Our job is to create a flow of value for our stake holders. If a user story can do that for you, then a user story is the right tool for the job. But user stories are just one “form” for performing this function. Do not let the user story tyranny stop you from doing what makes sense in your context and capturing stakeholder needs in whatever format works. After all, Mike Cohn is not going to come to your site and give you a gold star for beautifully written user stories. But your stake holders might for rapidly delivering a flow of value to them.

Be Honest: Are You Really the Product Owner?

Product Owner?

The product owner role is a manifestation of the agile manifesto’s values, particularly – “Customer interaction over contract negotiations” and  “Individuals and Interactions over processes and tools”.  The product owner provides delay busting  leadership and clarity for the team. The conversations with the product owner facilitated by user stories align the product owner and the team on a shared vision.  Rather than slow the development cycle and suppress learning through comprehensive documentation and change control boards, the product owner directly interacts with the team to learn what is really needed. From that learning the team and product owner refine, and reprioritize  the backlog to maximize value. The product owner is a value manager, exploiting fast learning cycles to re-prioritize the and revamp the backlog. Without an engaged product owner there is only at best incremental development. There is no agility.

One of the most common bad smells in many agile environments is the disengaged or unavailable product owner. However, there is a more insidious variant of this problem, one that can creates the appearance of an engaged product owner but in reality the product owner is dis-interested in and disengaged from the majority of the work team is responsible for.

In this variant of the dis-engaged product owner, the supposed PO is responsible for a specific project – or even a set of projects – that the team works on.  The problem is the team has other work in their backlog besides the PO’s pet projects. This may be more than the usual on-going maintenance, support, defect fixes, and the inevitable shoulder taps a development team must cope with because it may also include work on other projects that are not in their PO’s portfolio. The team is effectively left to fend for themselves on prioritizing and managing this “dark” or “off balance sheet” work.

i uses the term dark work as a metaphor for matter that is invisible but has significant affect. The problem with this dark work is that it is invisible, and therefore not managed. But like dark matter its effect on the team is significant, even destructive. Or as  my other metaphor “off balance sheet” work implies, dangerous liabilities that is not being accounted for.   It is critical that all team work is visible because if it is not visible then value cannot be managed. Very simple, if you are the product owner, then you are responsible for all work in the team’s backlog.

Bad smells that are symptoms of this problem include team members talking about their “day jobs” or a team with multiple backlogs. One backlog that is well managed by the PO and the other that is managed ad hoc by individuals of the team. This blindness lets us live for a while in fool’s paradise. For the product owner they are not forced to make hard choices between their pet project and all the other work the team needs to do to keep the lights on. And then we wonder why so many agile teams have lousy SAY/DO metrics.

Personally I think the term “product owner” is misleading because it implies responsibility for a specific product. In the modern agile context, a product may only be a subset of the teams overall work. The term product owner is a carryover from the early days of agile methodologies when the exemplar teams in the early textbooks were focused building a single product and the product owner represented the beginning and end of the value stream. Today most agile teams work in complex multi-system environments, and all of that work from multiple sources comes to the team and all of that work should be in one team backlog. The agile development team is one step in the value stream. So our model of the agile development team also needs to be updated.

I think we need to start thinking of the product owner as a “backlog owner” who is responsible for everything the team works on. Those who own a specific initiative or product are not “product” owners unless they can also take on the backlog owner role. The Scaled Agile Framework actually explicitly calls this out these different roles distinguishing between an “epic owner” role and a product owner role. While the same person may fulfill both roles, it is clearly states the interests of the EO may diverge from the PO role.

So be honest with yourself, are you really the product owner? Are you really interested and capable of working with the team to prioritize all the work in the backlog. You don’t have deep technical knowledge of everything in the backlog. The team can advise you and make recommendations. While the buck does stop with you, product ownership is still a negotiation, but it’s a negotiation about all the work, not just the bit you are interested in.

Put the Story Back Into User Stories

User Story – A Promise for a Conversation

User stories are a lovely manifestation of “individuals and interactions over processes and tools”. The team and the stake holder get together and have a conversation where a story about a need is told. The ensuing conversation helps create understanding and clarity. User stories facilitate collaboration, and create a shared clear understanding of what is really needed between customer representatives and development teams in a fast and economical way. Originally an artefact from eXtreme Programming (XP) user stories are now a part of most agile methodologies.

Unfortunately somewhere along our journey towards agile maturity we seem to have forgotten about the “story” part of user stories. Agile coaches love to teach the so called “3Cs” of user stories, Card, Conversation, and Confirmation and yet a common problem I have observed multiple times with many teams is that we have forgotten about the conversation. It seems the product owner writes the user stories and just drops them onto the team. There is very little of the collaborative writing and revision that fosters understanding. The promise for the conversation seems to have been broken.

This processes and tools approach to user stories is common when the product owner/customer representative is geographically separated from the team (e.g off shore development team). The lack of a conversation – the lack of interaction between individuals – now means heavyweight processes and tools are needed to communicate what could have been understood in a simple conversation. I have seen user stories where the acceptance criteria were 6 pages of psuedo code. All this did was push the cost of development up stream from the off-shore development team to the business analyst writing the user story. Worse, it dramatically slowed the decision making process and the cycle time. These are the real economic cost of trying to save money.

As agile coaches we have to share some responsibility for this. Many development problems are traceable to “poor” user stories and our typical solution as coaches is to emphasize well formatted user stories….those that meet the INVEST criteria, those written in the “As-a, I want, Such that” format popularized by Mike Cohn. We chastise those who do not write “high quality” acceptance criteria. However, high quality user stories are not a proxy for understanding. Are we failing to ask the questions, do these stories create a shared clear understanding of what the need is? Are they still a manifestation of “individuals and interactions” or have they become “processes and tools”?

As product owners, as business analysts, as developers and as coaches we need to put the story back into user story. Not break the “promise for a conversation” but embrace the individuals and interactions that co-write the user stories that creates the shared clear understanding of what the need really is. The first time a team sees a user story must not be the iteration or sprint planning meeting. For without the story, there is no user story.

Why Methodology Matters

Software methodology is easily most practitioners least favourite topic and yet I have devoted a better part of my career to studying and deploying software methodologies. I often like to explain why I have followed this path by citing a wonderful case study that comes from the auto industry, specifically the New Unified Motor Manufacturing Incorporate (NUMMI). This facility in Fremont California is now where Tesla manufactures its cars but it once was a General Motors facility. It was also reputed to have the worst workforces in the auto industry and the plant was closed in 1982. By 1984 it was re-opened as a joint venture between Toyota and GM, and had one of the highest quality ratings and productivity in the industry – with the same workforce. What changed was how people worked together to get the job done – their methodology.

When I use the term “methodology” I’m not talking about bureaucratic prescription of practices, roles and artifacts and then forcing everyone adapt their work practices to the methodology. Neither am  I talking about the puritanical enforcement of practices and narrow focus  often seen in so called agile methodologies. Rather I am talking about creating the shared clear understanding necessary for people to effectively work together and create value. I am talking about people knowing what they need to work on, who they need to work with, and how to effectively get the job done. Most importantly I am talking about people taking ownership of their methodology and not being forced to abdicate that responsibility to some obscure “center of excellence”.

When I reflect on my career and compare the more successful and more enjoyable projects to the ones we slogged through, and the projects that were outright disasters, the differentiator was not technical engineering talent, rather it was how well we worked together and how quickly we could learn and adapt. It didn’t matter if we assembled a team of superstars, if they could not work together to get the job done, then the job didn’t get done.

An interesting case study from my own personal experience, helps illustrate this point. I was part of a team parachuted in to a major industrial manufacturer to help recover a failing project. They were at the tail end of a long waterfall project and for a lack of better words, flailing badly. Defect opening rates were way higher than defect closing rates. People would work long hours on a critical defect, only to be interrupted and dragged off to a new higher priority crisis. The project deadline was looming and this was no mere presumed date on the calendar, failure to deliver by the deadline date, would mean total failure.

The project was shut down for two weeks to introduce a different methodology. During those two weeks the teams were re-organized and most importantly, trained together. Then in a mass “release planning” event, work was prioritized and collaboratively re-planned.  With the restart of the project, people’s attitude changed from “we are all doomed” to “we may have a fighting chance”. The project was delivered.  I will not claim any magic methodological silver bullet or personal brilliance for this. A lot of people worked massively hard and long hours to deliver. However, what I will claim is just like NUMMI, changing how people worked together, and how they think about their work – their methodology – created the opportunity for people to go from failing to delivering.

This is why methodology matters, why we need to think about it, and this is why I write about it.