The Backlog Is at the Center of the Team!

Placing the Backlog at the Center of a Scrum Team

Most models of agile software development portray the product backlog outside of the team. This is because, with the original small team focus of agile methodologies, the world began and ended with some kind of customer representative. For example, in Scrum, there is a Product Owner who is responsible for ensuring the team has a “ready” supply of backlog items and accepts the increment of working software delivered. What happens beyond the Product Owner is apparently not the team’s concern, and the team does not care how the Product Owner got the concept, how the analysis was performed, or how the design was created, just as long as they are not starved for stories.

Traditional Representation of Scrum with the “Product” Backlog Drawn Outside of the Sprint Cycle

From my point of view, this traditional view encourages a coding-centric view of agility and is misleading because it implies the purpose of the entire organizational ecosystem is to supply a software team with near “ready” backlog items which the team codes and tests into “working software”.

Coding is but one step in the creation of a valuable product, and to view agile teams strictly as just “design-build-test” teams is to completely devalue the work done by everyone else in the value stream. Nearly 40 years ago, in an article entitled “No Silver Bullet: Essence and Accidents of Software Engineering” (IEEE Computer, Vol. 20, No. 4. April 1987. pp. 10-19. Accessed 7 July 2019. https://www.researchgate.net/publication/220477127_No_Silver_Bullet_Essence_and_Accidents_of_Software_Engineering), Frederick Brooks wrote:

The hardest single part of building a software system is deciding precisely what to build.”

While there is clearly no valuable software product without “working software”, there is certainly no valuable product without the analysis and design to determine precisely what to build. It’s not enough to build the thing right, to create value we also have to build the right thing. It is wrong to offload and sweep under the carpet the hardest part of systems development to some undefined upstream process just to create the illusion of coding agility. Analysis and design still matter.

What if, rather than representing the backlog as a means for just “feeding” a development team, we call out ALL the work required to create a product or service and show EVERYTHING the team is fully responsible for. They are responsible for all the work which must be done to deliver each increment of working software –  not just coding and testing, but also analysis, design, and whatever other upstream and downstream activities are necessary to go from concept to cash. We have seen far too many organizations where analysis and design work is “invisible” or managed using a traditional stage gate model.

“Product” Backlog Drawn Inside the Scrum Sprint Cycle

Placing the backlog at the center of the team is an attempt to create a new visual where the whole team is responsible for all the steps required to create product, from ideation to delivery. The whole team not only builds the system, but is also involved in “…deciding precisely what to build”. The whole team performs the analysis, design, implementation, and deployment steps.

This means that everything we pull into planning will not necessarily be written as a traditional user story that yields some idealized slice of feature functionality. There are going to be backlog items representing analysis, design, packaging, and other work. For example, during a Scrum-style sprint planning meeting, a Business Analyst may take on the work to model some of the backlog items, perhaps to create a use case brief for a feature.

Don’t get me wrong, I am not advocating for traditional long drawn out analysis. Agility is about fast learning cycles, and our analysis and design needs to be part of the fast learning cycle. While the output may not be production ready “working software” – the output still needs to add verifiable knowledge that creates valuable learning. Pulling a line from the Scrum guide, any work we pull into a sprint from the backlog should still “… include test descriptions that will prove its completeness when done”. Just showing a “textbook-perfect” use case diagram at a sprint review is not useful.  Rather, if we pull a backlog item to “analyze and design,” there still has to be value generated in the form of validated learning. There has to be some demonstration, some proof that the model works – perhaps a code fragment implementing a slice of the model, an executable simulation, or even just a walk through. While we are arguing against a coding-centric view of agile teams, the intent behind the Agile Manifesto value statement “working software over comprehensive documentation”, that is getting something valuable done still applies.

Some may legitimately raise a red flag here suggesting that I am encouraging the waterfalling of sprints – a sprint to analyze, a sprint to code, a sprint to test. This can indeed happen, but I would prefer this problem be explicit and an issue the team can work to resolve in a retrospective, rather than sweeping it under the carpet and having the team believe they are truly agile and righteous. My argument is for making ALL the work visible. Placing the backlog at the center brings all those other people, who are traditionally thought of as being on the outside, and makes them visible and part of the team. It encourages us to apply agile thinking to analysis and design and whatever other processes are required to transform an idea into a valuable product or service.

End the User Story Tyranny

Somehow an agile myth has arisen that user stories are the only way for an agile team to capture needs. As a result, I have seen the crazy and inappropriate application of user stories create not only a lot of angst and confusion, but also a lot of resentment in agile teams. In my coaching practice, many of the issues I see with agile teams can be traced back to a low quality expression and understanding of needs. A common prescription for this is the rigid enforcement the user story format and the demand for precisely written acceptance criteria. Coaches and trainers love this because they can create the illusion of making a useful intervention. The team can now write textbook-perfect user stories. The problem is the original issues have not disappeared, because form is not a substitute for function. The team is still “agile in name” only.

Part of why Agile works is its focus on getting something done quickly. User stories are a really good way to quickly capture and create understanding about what valuable working software needs to do. User stories enable the rapid delivery of working software by capturing the desired behaviour of a small slice of the system. By delivering small slices of working software to our stakeholders we can quickly learn what the really need. So, what’s not to like about user stories? How can I assert that these beautifully intentioned and simple artifacts are imposing a tyrannical rule on a team, or even an entire program? When used appropriately, user stories are a wonderful and amazing tool. However, when we choose to disregard context and force all needs to be expressed as a user story, then they can become a tyranny imposed on the team or the program. Often the result is a frustrated team that believes agile does not work for them.

User stories were originally an XP artifact and were popularized with the publication of Mike Cohn’s “User Stories Applied” in 2004. Back in 2004, the scope of agile methodologies – at least for textbook writers – was a team of seven plus or minus two who worked directly with a stakeholder (e.g. the customer representative, or product owner). The systems they tended to build were customer-facing and had direct users. The world began and ended with the product owner or customer representative who co-wrote the user stories with the team. However, over the last 15 years we have dramatically stretched agile thinking from those small customer-facing teams to much bigger teams working on larger and more complex systems. User stories are not necessarily the most effective or appropriate tool for capturing in expressing needs in all domains.

Many of us tightly associate user stories with agile because when we were learning Scrum, our instructors dutifully took us through user story writing and judged us on our ability to express needs in the <As a> …<I want> …< So that> …format with precisely scoped acceptance criteria. But if you search the Scrum Guide you will not find a single reference to user stories because user stories are not a Scrum artifact. All Scrum declares is:

The Product Backlog lists all features, functions, requirements, enhancements, and fixes that constitute the changes to be made to the product in future releases. Product Backlog items have the attributes of a description, order, estimate, and value. Product Backlog items often include test descriptions that will prove its completeness when Done” – Scrum Guide

User stories are certainly a good way to satisfy this definition of a backlog item, but they are not the only way – and while user stories can make good backlog items, good backlog items are not necessarily user stories. Unfortunately, some agile methodologies and tools have bolstered the idea that user stories are the only true backlog item by defining their backlog items as “stories”.

This post is not an indictment of user stories. Rather it is about how dogmatically focusing on the user story format without consideration of context or understanding creates the illusion of agility – what is sometimes called cargo cult agile – while dooming the team to frustration and even failure. Agile teams work because they focus on getting something done, and the function of a user story is to create understanding about some small sliver of value the team can get done within a short timebox. Our job is to create a flow of value for our stake holders. If a user story can do that for you, then a user story is the right tool for the job. But user stories are just one “form” for performing this function. Do not let the user story tyranny stop you from doing what makes sense in your context and capturing stakeholder needs in whatever format works. After all, Mike Cohn is not going to come to your site and give you a gold star for beautifully written user stories. But your stake holders might for rapidly delivering a flow of value to them.

Be Honest: Are You Really the Product Owner?

Product Owner?

The product owner role is a manifestation of the agile manifesto’s values, particularly – “Customer interaction over contract negotiations” and  “Individuals and Interactions over processes and tools”.  The product owner provides delay busting  leadership and clarity for the team. The conversations with the product owner facilitated by user stories align the product owner and the team on a shared vision.  Rather than slow the development cycle and suppress learning through comprehensive documentation and change control boards, the product owner directly interacts with the team to learn what is really needed. From that learning the team and product owner refine, and reprioritize  the backlog to maximize value. The product owner is a value manager, exploiting fast learning cycles to re-prioritize the and revamp the backlog. Without an engaged product owner there is only at best incremental development. There is no agility.

One of the most common bad smells in many agile environments is the disengaged or unavailable product owner. However, there is a more insidious variant of this problem, one that can creates the appearance of an engaged product owner but in reality the product owner is dis-interested in and disengaged from the majority of the work team is responsible for.

In this variant of the dis-engaged product owner, the supposed PO is responsible for a specific project – or even a set of projects – that the team works on.  The problem is the team has other work in their backlog besides the PO’s pet projects. This may be more than the usual on-going maintenance, support, defect fixes, and the inevitable shoulder taps a development team must cope with because it may also include work on other projects that are not in their PO’s portfolio. The team is effectively left to fend for themselves on prioritizing and managing this “dark” or “off balance sheet” work.

i uses the term dark work as a metaphor for matter that is invisible but has significant affect. The problem with this dark work is that it is invisible, and therefore not managed. But like dark matter its effect on the team is significant, even destructive. Or as  my other metaphor “off balance sheet” work implies, dangerous liabilities that is not being accounted for.   It is critical that all team work is visible because if it is not visible then value cannot be managed. Very simple, if you are the product owner, then you are responsible for all work in the team’s backlog.

Bad smells that are symptoms of this problem include team members talking about their “day jobs” or a team with multiple backlogs. One backlog that is well managed by the PO and the other that is managed ad hoc by individuals of the team. This blindness lets us live for a while in fool’s paradise. For the product owner they are not forced to make hard choices between their pet project and all the other work the team needs to do to keep the lights on. And then we wonder why so many agile teams have lousy SAY/DO metrics.

Personally I think the term “product owner” is misleading because it implies responsibility for a specific product. In the modern agile context, a product may only be a subset of the teams overall work. The term product owner is a carryover from the early days of agile methodologies when the exemplar teams in the early textbooks were focused building a single product and the product owner represented the beginning and end of the value stream. Today most agile teams work in complex multi-system environments, and all of that work from multiple sources comes to the team and all of that work should be in one team backlog. The agile development team is one step in the value stream. So our model of the agile development team also needs to be updated.

I think we need to start thinking of the product owner as a “backlog owner” who is responsible for everything the team works on. Those who own a specific initiative or product are not “product” owners unless they can also take on the backlog owner role. The Scaled Agile Framework actually explicitly calls this out these different roles distinguishing between an “epic owner” role and a product owner role. While the same person may fulfill both roles, it is clearly states the interests of the EO may diverge from the PO role.

So be honest with yourself, are you really the product owner? Are you really interested and capable of working with the team to prioritize all the work in the backlog. You don’t have deep technical knowledge of everything in the backlog. The team can advise you and make recommendations. While the buck does stop with you, product ownership is still a negotiation, but it’s a negotiation about all the work, not just the bit you are interested in.