Thoughts from Agile 2019: Is it time for agilists to finally join the online revolution?

The Agile Alliance “Agile 20XX” conference is arguably the premier agile conference. Each year some 2500 people gather together to learn, to see and be seen, to share ideas, and even share a few ideas over a few drinks.  The “Agile 20XX” conferences are great places to catch the pulse of the industry. Of course, business agility is the hot topic these days, but one undercurrent that really caught my attention this year at Agile 2019 in Washington DC, was the rising interest in online or virtual consulting and training. For the first time that I was aware of, there was a real buzz about delivering agile training and consulting services online. Not as a cheap substitute for development teams in far off lands, or those who do not apparently warrant “the luxury of in-person delivery” but rather as a legitimate and perhaps superior way of delivering services.

I was at Agile 2019 to deliver an experience report which told the story of how we leveraged online assets to give everyone on a large globally distributed SAFe solution train an equal voice in the SAFe problem solving workshop. If you are interested in our experience report, you can find it at the agile alliance website “The Sun Never Sets on the Problem Solving Workshop”.  Our experience report was followed immediately by Joe Fecarotta of Accenture/SolutionsIQ who explained how he is successfully training and coaching Accenture staff online. Later Shane Hastie & Shannon Ewan of IC-Agile ( led a session on Adapting Training from the Back of the Room (TBR) for Remote Learning  online. For those not familiar with TBR, these are practices which move the instructor away from the classic “talking head, death by power point” model of course delivery to a more participative learning model. Take a look at Sharon Bowman’s website; if this interests you, and it should if you are doing any kind of training.  Shane & Shannon’s session was about developing practices for developing online participative models that are consistent with the TBR principles. Looking around the well-attended session I noticed many agile coaches whom I consider thought leaders in our industry. Simply put, it seems that virtual training is moving from the fringes to the mainstream.

I am convinced the time for virtual training and consulting has come for many reasons. First the collaboration technology has simply got way better. We have always believed that face to face conversations are best, and in absolute terms this is true, but today face to face doesn’t have to mean in the same physical location. When the manifesto was written in 2001, the Internet and online collaboration technology were in their infancy. In 2001 I had dial up for access. Now I have 300Mbits/sec coming into my house, bandwidth that allows for multi-channel high-definition video transmission.  It’s not just speed, it’s also the resolution of the HTML5 standards wars which means there are now amazing online collaborative tools. Second, product development and especially software development is a global industry, and regular in person face to face meetings are often impractical. Third the millennials – sorry folks, I know you hate us old boomers referring to you this way –  are moving into leadership roles and with them there is a major organizational change in attitude towards online collaboration. These are people who grew up in an online environment and are totally comfortable working there. I often like to joke if educated and experienced engineers could do what my daughter does in Minecraft I would be out of a job – seriously, if you have kids watch them playing Minecraft. Fourth is cost, easily 20 to 25% of the cost of a consulting engagement is expenses. Either my customer would like to save that 20-25% or I would like to have that 20 to 25% go into my pocket rather than Air Canada’s and Marriott’s. But it’s not just monetary cost. It is also the physical and social costs on the people who have to regularly travel to deliver those services. There are fewer and fewer people who are willing to travel.   There is no joy in spending many hours in an aluminum tube breathing recycled air for 16 hours at a stretch.

Given all this we have to ask, is it really necessary to have a coach or trainer onsite day after day, week after week? Recently a family issue prevented me from traveling to a coaching engagement, so I executed the engagement for the week using a variety of collaboration tools like Slack. In our opinions, I was just as effective working from home as if I had been onsite. Of course, I had already established a strong personal relationship with my client because I had previously been onsite, already knew the people well and they know me. However, this was a clear indicator that I did not need to be physically onsite every week.   It helps to build in-person relationships to enable remote collaboration.  Furthermore, I am currently working a completely online engagement with a client where we are successfully launching a number of Scrum teams. We have delivered all the training and coaching online. Of course, as per the advice in Shane & Shannon’s Agile 2019 session, we had to re-design our course delivery style to leverage the online capabilities, but we were still able to deliver an interactive and engaging experience. Trainers and coaches do not need to be onsite week after week after week to successfully deliver services.

Finally, and most importantly there is an environmental reason for moving more of our work online and that is the carbon footprint for air travel.  While air travel only contributes between 2 to 3 % of green house gasses, it is the small cadre of frequent flyers like myself who make up the bulk of airline passengers. For me it’s a bit of a personal paradox because I try to live a low carbon lifestyle.  I don’t own a car; I mostly use transit or ride a bicycle. Then every week I fly across the North American continent expelling the equivalent of a year’s worth of driving in green house gasses.  What if I could reduce that to say one week out of four? It is not hard to notice there is already a backlash starting against excessive air travel. Some academics in Scandinavian countries now have to justify their carbon budget if they choose to travel to a conference. It would not surprise me that if in 5 years frequent flyers like myself will be frowned upon much in the same way as smokers are today.

I want to leave something behind for my children, and if I can reduce my carbon footprint, reduce costs to my client, be in my own bed at night, delivering as good or better experience as on-site, then I am all for it. It’s time to follow our children’s leadership and join them online.

Apollo’s Legacy: A personal reflection of the 50th anniversary of the Lunar landing

I only saw it in black and white

July 21st marked the fiftieth anniversary of Neil Armstrong setting foot on the moon culminating one of the greatest human engineering achievements in history. I remember watching that first step on a friend’s black and white 25 inch console television set.  My first thought after recalling that moment at the 50th anniversary was “am I really that old?” But after that shock wore off my thoughts turned to value of the Apollo program. Was it worth it? After all the cost of that first step was the equivalent of hundreds of billions of dollars in today’s money.

Most people argue the benefits of the Apollo program in terms of the technological spin-offs. It is often suggested the modern semi-conductor industry would not exist today at least in its present form if not for the Apollo program. That may be, but I think the greatest legacy of the Apollo program was how it inspired whole generations to become excited about engineering as a career. How millions of young men and women were inspired, became engineers and worked to make people’s lives better.

Many people remember the late 60s as a time of social upheaval but many of today’s engineers remember it as the time that inspired them to become engineers. Not only the moon landing, but also TV programs and movies like Star Trek and of course Stanley Kubrick’s classic space ballet – 2001.  Both painted a picture of a future we not only wanted to live in, but to also create. I wanted to design and fly amazing space craft. I wanted to be one of the pilots of the Orion space shuttle or one of the engineers who designed and built the Clavius lunar base. Or the creator of HAL – the amazing and psychotic Artificial Intelligence controlling the Discovery.  I was going to create and live in the world of 2001.

The world I thought I would live in at 40

Of course it was not to be. The reality was that in 2001 I wasn’t designing cool space craft or lunar bases. Rather I was usually sitting in a rather ordinary conference room on Earth in yet another project status meeting during the development of a new cellular telephone system. Somehow my career placed me in a cube farm designing telephone switches, and railway signalling systems.  I can’t recall a cube farm in Star Trek, or 2001. Nor do I recall project status meetings. Worse, later in my career I became a candidate passenger for the Golgafrincham “B” ark, and became a management consultant. It’s really, really hard to be stuck on Earth when your imagination is in the stars. Or is it?

The Golganfrincham B Ark.
For those not with Douglas Adam’s “Hitchhiker’s Guide to the Galaxy, the “B” Ark was an attempt by the Golgafrinchams to remove all the “useless” people from their society such as account executives, public relations experts, and of course management consultants by telling them the planet was doomed. The Golgafrinchams gave them the “privilege” of being the first to leave Golgafrincham aboard a giant space ship called “B” Ark.

Even though I didn’t get to design moon bases and space craft, much of the work I did created value for a lot of people. The work I did with thousands of others designing digital exchanges and cellular telephone systems dramatically improved the communications services available to millions of people around the world. Railway signalling enabled cities to increase the capacity of their transit systems. Maybe not as cool as a lunar base, but we made a difference in peoples’ lives. More than that I got to work with many amazing people and I even had fun doing it – maybe with the exception of the project status meetings.

While the technological spin-offs of the Apollo program are always cited as its lasting economic legacy, I think it was how the program inspired so many of us to become engineers. We didn’t get to build space stations or lunar bases or psychotic AIs (not yet anyways), but we created wealth and comfort for many.  Every day people make phone calls, take a train to work, watch television, drive their car, turn on an electric light and get a drink of water without any thought of the extraordinary efforts made to create this world.  For me this is what it means to be an engineer, we make people’s lives better, and our work is trusted such that no one gives a second thought about it.

So, what will be the next great engineering achievement that will inspire the upcoming generations to become engineers? While I would like to think of an Apollo repeat, that is a Mars landing, I don’t see that having the same impact as the original moon landing. We have done our job well and space exploration is just yet another interesting domain, among many. What, I see in the so called “millennials” and generation “z” is an overpowering desire to make the world a better place. David Pink argues that people are purpose driven and the current generations of children and young engineers are inspired by making the world a better place more than ever. Coping with climate change, developing new energy sources and distribution systems, ensuring there is enough food, clean water, and proper shelter for a future with 8 billion people.  Building the systems and creating the wealth that allow all those 8 billion people to live in comfort and dignity while realizing their aspirations will be our greatest engineering achievement.

The Backlog Is at the Center of the Team!

Placing the Backlog at the Center of a Scrum Team

Most models of agile software development portray the product backlog outside of the team. This is because, with the original small team focus of agile methodologies, the world began and ended with some kind of customer representative. For example, in Scrum, there is a Product Owner who is responsible for ensuring the team has a “ready” supply of backlog items and accepts the increment of working software delivered. What happens beyond the Product Owner is apparently not the team’s concern, and the team does not care how the Product Owner got the concept, how the analysis was performed, or how the design was created, just as long as they are not starved for stories.

Traditional Representation of Scrum with the “Product” Backlog Drawn Outside of the Sprint Cycle

From my point of view, this traditional view encourages a coding-centric view of agility and is misleading because it implies the purpose of the entire organizational ecosystem is to supply a software team with near “ready” backlog items which the team codes and tests into “working software”.

Coding is but one step in the creation of a valuable product, and to view agile teams strictly as just “design-build-test” teams is to completely devalue the work done by everyone else in the value stream. Nearly 40 years ago, in an article entitled “No Silver Bullet: Essence and Accidents of Software Engineering” (IEEE Computer, Vol. 20, No. 4. April 1987. pp. 10-19. Accessed 7 July 2019., Frederick Brooks wrote:

The hardest single part of building a software system is deciding precisely what to build.”

While there is clearly no valuable software product without “working software”, there is certainly no valuable product without the analysis and design to determine precisely what to build. It’s not enough to build the thing right, to create value we also have to build the right thing. It is wrong to offload and sweep under the carpet the hardest part of systems development to some undefined upstream process just to create the illusion of coding agility. Analysis and design still matter.

What if, rather than representing the backlog as a means for just “feeding” a development team, we call out ALL the work required to create a product or service and show EVERYTHING the team is fully responsible for. They are responsible for all the work which must be done to deliver each increment of working software –  not just coding and testing, but also analysis, design, and whatever other upstream and downstream activities are necessary to go from concept to cash. We have seen far too many organizations where analysis and design work is “invisible” or managed using a traditional stage gate model.

“Product” Backlog Drawn Inside the Scrum Sprint Cycle

Placing the backlog at the center of the team is an attempt to create a new visual where the whole team is responsible for all the steps required to create product, from ideation to delivery. The whole team not only builds the system, but is also involved in “…deciding precisely what to build”. The whole team performs the analysis, design, implementation, and deployment steps.

This means that everything we pull into planning will not necessarily be written as a traditional user story that yields some idealized slice of feature functionality. There are going to be backlog items representing analysis, design, packaging, and other work. For example, during a Scrum-style sprint planning meeting, a Business Analyst may take on the work to model some of the backlog items, perhaps to create a use case brief for a feature.

Don’t get me wrong, I am not advocating for traditional long drawn out analysis. Agility is about fast learning cycles, and our analysis and design needs to be part of the fast learning cycle. While the output may not be production ready “working software” – the output still needs to add verifiable knowledge that creates valuable learning. Pulling a line from the Scrum guide, any work we pull into a sprint from the backlog should still “… include test descriptions that will prove its completeness when done”. Just showing a “textbook-perfect” use case diagram at a sprint review is not useful.  Rather, if we pull a backlog item to “analyze and design,” there still has to be value generated in the form of validated learning. There has to be some demonstration, some proof that the model works – perhaps a code fragment implementing a slice of the model, an executable simulation, or even just a walk through. While we are arguing against a coding-centric view of agile teams, the intent behind the Agile Manifesto value statement “working software over comprehensive documentation”, that is getting something valuable done still applies.

Some may legitimately raise a red flag here suggesting that I am encouraging the waterfalling of sprints – a sprint to analyze, a sprint to code, a sprint to test. This can indeed happen, but I would prefer this problem be explicit and an issue the team can work to resolve in a retrospective, rather than sweeping it under the carpet and having the team believe they are truly agile and righteous. My argument is for making ALL the work visible. Placing the backlog at the center brings all those other people, who are traditionally thought of as being on the outside, and makes them visible and part of the team. It encourages us to apply agile thinking to analysis and design and whatever other processes are required to transform an idea into a valuable product or service.

End the User Story Tyranny

Somehow an agile myth has arisen that user stories are the only way for an agile team to capture needs. As a result, I have seen the crazy and inappropriate application of user stories create not only a lot of angst and confusion, but also a lot of resentment in agile teams. In my coaching practice, many of the issues I see with agile teams can be traced back to a low quality expression and understanding of needs. A common prescription for this is the rigid enforcement the user story format and the demand for precisely written acceptance criteria. Coaches and trainers love this because they can create the illusion of making a useful intervention. The team can now write textbook-perfect user stories. The problem is the original issues have not disappeared, because form is not a substitute for function. The team is still “agile in name” only.

Part of why Agile works is its focus on getting something done quickly. User stories are a really good way to quickly capture and create understanding about what valuable working software needs to do. User stories enable the rapid delivery of working software by capturing the desired behaviour of a small slice of the system. By delivering small slices of working software to our stakeholders we can quickly learn what the really need. So, what’s not to like about user stories? How can I assert that these beautifully intentioned and simple artifacts are imposing a tyrannical rule on a team, or even an entire program? When used appropriately, user stories are a wonderful and amazing tool. However, when we choose to disregard context and force all needs to be expressed as a user story, then they can become a tyranny imposed on the team or the program. Often the result is a frustrated team that believes agile does not work for them.

User stories were originally an XP artifact and were popularized with the publication of Mike Cohn’s “User Stories Applied” in 2004. Back in 2004, the scope of agile methodologies – at least for textbook writers – was a team of seven plus or minus two who worked directly with a stakeholder (e.g. the customer representative, or product owner). The systems they tended to build were customer-facing and had direct users. The world began and ended with the product owner or customer representative who co-wrote the user stories with the team. However, over the last 15 years we have dramatically stretched agile thinking from those small customer-facing teams to much bigger teams working on larger and more complex systems. User stories are not necessarily the most effective or appropriate tool for capturing in expressing needs in all domains.

Many of us tightly associate user stories with agile because when we were learning Scrum, our instructors dutifully took us through user story writing and judged us on our ability to express needs in the <As a> …<I want> …< So that> …format with precisely scoped acceptance criteria. But if you search the Scrum Guide you will not find a single reference to user stories because user stories are not a Scrum artifact. All Scrum declares is:

The Product Backlog lists all features, functions, requirements, enhancements, and fixes that constitute the changes to be made to the product in future releases. Product Backlog items have the attributes of a description, order, estimate, and value. Product Backlog items often include test descriptions that will prove its completeness when Done” – Scrum Guide

User stories are certainly a good way to satisfy this definition of a backlog item, but they are not the only way – and while user stories can make good backlog items, good backlog items are not necessarily user stories. Unfortunately, some agile methodologies and tools have bolstered the idea that user stories are the only true backlog item by defining their backlog items as “stories”.

This post is not an indictment of user stories. Rather it is about how dogmatically focusing on the user story format without consideration of context or understanding creates the illusion of agility – what is sometimes called cargo cult agile – while dooming the team to frustration and even failure. Agile teams work because they focus on getting something done, and the function of a user story is to create understanding about some small sliver of value the team can get done within a short timebox. Our job is to create a flow of value for our stake holders. If a user story can do that for you, then a user story is the right tool for the job. But user stories are just one “form” for performing this function. Do not let the user story tyranny stop you from doing what makes sense in your context and capturing stakeholder needs in whatever format works. After all, Mike Cohn is not going to come to your site and give you a gold star for beautifully written user stories. But your stake holders might for rapidly delivering a flow of value to them.

Be Honest: Are You Really the Product Owner?

Product Owner?

The product owner role is a manifestation of the agile manifesto’s values, particularly – “Customer interaction over contract negotiations” and  “Individuals and Interactions over processes and tools”.  The product owner provides delay busting  leadership and clarity for the team. The conversations with the product owner facilitated by user stories align the product owner and the team on a shared vision.  Rather than slow the development cycle and suppress learning through comprehensive documentation and change control boards, the product owner directly interacts with the team to learn what is really needed. From that learning the team and product owner refine, and reprioritize  the backlog to maximize value. The product owner is a value manager, exploiting fast learning cycles to re-prioritize the and revamp the backlog. Without an engaged product owner there is only at best incremental development. There is no agility.

One of the most common bad smells in many agile environments is the disengaged or unavailable product owner. However, there is a more insidious variant of this problem, one that can creates the appearance of an engaged product owner but in reality the product owner is dis-interested in and disengaged from the majority of the work team is responsible for.

In this variant of the dis-engaged product owner, the supposed PO is responsible for a specific project – or even a set of projects – that the team works on.  The problem is the team has other work in their backlog besides the PO’s pet projects. This may be more than the usual on-going maintenance, support, defect fixes, and the inevitable shoulder taps a development team must cope with because it may also include work on other projects that are not in their PO’s portfolio. The team is effectively left to fend for themselves on prioritizing and managing this “dark” or “off balance sheet” work.

i uses the term dark work as a metaphor for matter that is invisible but has significant affect. The problem with this dark work is that it is invisible, and therefore not managed. But like dark matter its effect on the team is significant, even destructive. Or as  my other metaphor “off balance sheet” work implies, dangerous liabilities that is not being accounted for.   It is critical that all team work is visible because if it is not visible then value cannot be managed. Very simple, if you are the product owner, then you are responsible for all work in the team’s backlog.

Bad smells that are symptoms of this problem include team members talking about their “day jobs” or a team with multiple backlogs. One backlog that is well managed by the PO and the other that is managed ad hoc by individuals of the team. This blindness lets us live for a while in fool’s paradise. For the product owner they are not forced to make hard choices between their pet project and all the other work the team needs to do to keep the lights on. And then we wonder why so many agile teams have lousy SAY/DO metrics.

Personally I think the term “product owner” is misleading because it implies responsibility for a specific product. In the modern agile context, a product may only be a subset of the teams overall work. The term product owner is a carryover from the early days of agile methodologies when the exemplar teams in the early textbooks were focused building a single product and the product owner represented the beginning and end of the value stream. Today most agile teams work in complex multi-system environments, and all of that work from multiple sources comes to the team and all of that work should be in one team backlog. The agile development team is one step in the value stream. So our model of the agile development team also needs to be updated.

I think we need to start thinking of the product owner as a “backlog owner” who is responsible for everything the team works on. Those who own a specific initiative or product are not “product” owners unless they can also take on the backlog owner role. The Scaled Agile Framework actually explicitly calls this out these different roles distinguishing between an “epic owner” role and a product owner role. While the same person may fulfill both roles, it is clearly states the interests of the EO may diverge from the PO role.

So be honest with yourself, are you really the product owner? Are you really interested and capable of working with the team to prioritize all the work in the backlog. You don’t have deep technical knowledge of everything in the backlog. The team can advise you and make recommendations. While the buck does stop with you, product ownership is still a negotiation, but it’s a negotiation about all the work, not just the bit you are interested in.

Put the Story Back Into User Stories

User Story – A Promise for a Conversation

User stories are a lovely manifestation of “individuals and interactions over processes and tools”. The team and the stake holder get together and have a conversation where a story about a need is told. The ensuing conversation helps create understanding and clarity. User stories facilitate collaboration, and create a shared clear understanding of what is really needed between customer representatives and development teams in a fast and economical way. Originally an artefact from eXtreme Programming (XP) user stories are now a part of most agile methodologies.

Unfortunately somewhere along our journey towards agile maturity we seem to have forgotten about the “story” part of user stories. Agile coaches love to teach the so called “3Cs” of user stories, Card, Conversation, and Confirmation and yet a common problem I have observed multiple times with many teams is that we have forgotten about the conversation. It seems the product owner writes the user stories and just drops them onto the team. There is very little of the collaborative writing and revision that fosters understanding. The promise for the conversation seems to have been broken.

This processes and tools approach to user stories is common when the product owner/customer representative is geographically separated from the team (e.g off shore development team). The lack of a conversation – the lack of interaction between individuals – now means heavyweight processes and tools are needed to communicate what could have been understood in a simple conversation. I have seen user stories where the acceptance criteria were 6 pages of psuedo code. All this did was push the cost of development up stream from the off-shore development team to the business analyst writing the user story. Worse, it dramatically slowed the decision making process and the cycle time. These are the real economic cost of trying to save money.

As agile coaches we have to share some responsibility for this. Many development problems are traceable to “poor” user stories and our typical solution as coaches is to emphasize well formatted user stories….those that meet the INVEST criteria, those written in the “As-a, I want, Such that” format popularized by Mike Cohn. We chastise those who do not write “high quality” acceptance criteria. However, high quality user stories are not a proxy for understanding. Are we failing to ask the questions, do these stories create a shared clear understanding of what the need is? Are they still a manifestation of “individuals and interactions” or have they become “processes and tools”?

As product owners, as business analysts, as developers and as coaches we need to put the story back into user story. Not break the “promise for a conversation” but embrace the individuals and interactions that co-write the user stories that creates the shared clear understanding of what the need really is. The first time a team sees a user story must not be the iteration or sprint planning meeting. For without the story, there is no user story.

Why Methodology Matters

Software methodology is easily most practitioners least favourite topic and yet I have devoted a better part of my career to studying and deploying software methodologies. I often like to explain why I have followed this path by citing a wonderful case study that comes from the auto industry, specifically the New Unified Motor Manufacturing Incorporate (NUMMI). This facility in Fremont California is now where Tesla manufactures its cars but it once was a General Motors facility. It was also reputed to have the worst workforces in the auto industry and the plant was closed in 1982. By 1984 it was re-opened as a joint venture between Toyota and GM, and had one of the highest quality ratings and productivity in the industry – with the same workforce. What changed was how people worked together to get the job done – their methodology.

When I use the term “methodology” I’m not talking about bureaucratic prescription of practices, roles and artifacts and then forcing everyone adapt their work practices to the methodology. Neither am  I talking about the puritanical enforcement of practices and narrow focus  often seen in so called agile methodologies. Rather I am talking about creating the shared clear understanding necessary for people to effectively work together and create value. I am talking about people knowing what they need to work on, who they need to work with, and how to effectively get the job done. Most importantly I am talking about people taking ownership of their methodology and not being forced to abdicate that responsibility to some obscure “center of excellence”.

When I reflect on my career and compare the more successful and more enjoyable projects to the ones we slogged through, and the projects that were outright disasters, the differentiator was not technical engineering talent, rather it was how well we worked together and how quickly we could learn and adapt. It didn’t matter if we assembled a team of superstars, if they could not work together to get the job done, then the job didn’t get done.

An interesting case study from my own personal experience, helps illustrate this point. I was part of a team parachuted in to a major industrial manufacturer to help recover a failing project. They were at the tail end of a long waterfall project and for a lack of better words, flailing badly. Defect opening rates were way higher than defect closing rates. People would work long hours on a critical defect, only to be interrupted and dragged off to a new higher priority crisis. The project deadline was looming and this was no mere presumed date on the calendar, failure to deliver by the deadline date, would mean total failure.

The project was shut down for two weeks to introduce a different methodology. During those two weeks the teams were re-organized and most importantly, trained together. Then in a mass “release planning” event, work was prioritized and collaboratively re-planned.  With the restart of the project, people’s attitude changed from “we are all doomed” to “we may have a fighting chance”. The project was delivered.  I will not claim any magic methodological silver bullet or personal brilliance for this. A lot of people worked massively hard and long hours to deliver. However, what I will claim is just like NUMMI, changing how people worked together, and how they think about their work – their methodology – created the opportunity for people to go from failing to delivering.

This is why methodology matters, why we need to think about it, and this is why I write about it.