The One Where The Product Backlog Was Too Big

Overview

Andy, the ScrumMaster, was guiding the team through a session of planning poker to estimate the initial Product Backlog so the team could come up with a release plan for the new project. Murray, the Product Owner, had provided the team with the Product Backlog in time for the planning session although it did have to be rescheduled a couple of times as he couldn’t get hold of a couple of the account reps to get their requirements.

The team had timeboxed three hours for the estimation but they had only really just scratched the surface of the Product Backlog and time was nearly up. A quick look at the spreadsheet showed they were probably only about 10% of the way through the product Backlog and that was quite depressing; it was noticeable how the initial energy within the team had all but disappeared and there was still no end in sight.

Unfortunately, even though the team was co-located they had to use a tool to manage the Product Backlog because of the size; they would have preferred to use index cards but the number of index cards needed would have made that impractical. Andy could hardly complain because Murray had done a very thorough job in preparing the Product Backlog and the team now had a very good understanding of what was ahead of them on this project. In terms of fulfilling his responsibilities as a Product Owner, Murray was definitely on the right track.

 

He still felt something was not quite right though – it didn’t feel very agile – and wondered how to help his team keep create the release plan before they lost all motivation. Andy called a break, saying they would re-convene in an hour and asked Murray for a chat. “Are all of these items absolutely necessary?” he asked.

 

“Absolutely” came Murray’s reply, “I have even prioritised them as you asked. Priorities 1 through 5. I mean they wouldn’t be called requirements unless they were required, would they?”

 

Murray started to look worried. “Why? What’s the problem?” he asked.

 

“I don’t know.” said Andy “But something’s not right here. We have the team estimating a load of work and, at this rate, it’s going to take them the rest of the week just to estimate it all.”

 

“Could we try something quicker than planning poker?” Murray suggested

 

“Sure we could.” Andy said “We could try affinity estimation or some other quicker technique but I’m not sure that is really the point. Plus the team are getting a lot of benefit from the discussions that are coming out of the planning poker session”

 

“Could we cut the Product Backlog down to a more manageable size?” Andy asked

 

“Not really” Murray said “I don’t want to run the risk of forgetting something after the account reps have actually submitted their requirements”

 

When the team came back from the break Andy put his unease to the team.

 

“On the one hand, I think we have learned a huge amount about the product and the product backlog. The effort put in to estimating these items has been immense. Personally I am seeing a big drop in the energy levels and am wondering what the best thing to do next would be. What do you guys think?”

 

By playing back what he was seeing to the team he immediately did two things. First of all he acknowledged the elephant in the room that everyone else was concerned about but didn’t want to mention, and secondly reminded the team that if something isn’t working they have the power to change it.

 

“How about we see how much work we have actually estimated before carrying on with the rest?” came the suggestion from Greg, one of the developers. The rest of the team seemed keen on that and so Andy to guide them through commitment based planning to try and get a feeling for their velocity and then use that figure to see how many Sprints it would take them to complete the Product Backlog items they had already estimated.

 

It turned out to be eight four-week Sprints.

 

“So nearly eight months work.” Greg summarised “If we extrapolate that out, assuming an average representation of effort over the Product Backlog items, we are talking about nearly four years worth of work on the Product Backlog. This is with just what we know now, let alone what will probably come up over the next 3 or 4 years! Is it really worth the effort of estimating the rest of this stuff right now?”

 

Analysis

Andy’s situation is far from rare. I regularly see teams with Product Backlogs that are unmanageably large and have even heard of teams with Product Backlogs in the 10,000 items range

 

The typical justification for this is to avoid losing information about what is required; users are also considered to be happier if they know that there is a plan to get to their requirement at some point rather than be told “it’s not going to get done”. Ultimately the Product Backlog is the responsibility of the Product Owner as it is their job to ensure a positive return on the investment of the project. If things do need to get done then they are perfectly within their rights to hold the item on the Product Backlog. Most Product Owners, however, are aware of the benefits of regular “tidying” or “grooming” of the Product Backlog, keeping it in a manageable state.

 

The larger the Product Backlog the more overhead there is for a Scrum team, the more items need to be re-prioritised and re-estimated, the more churn there is on requirements and the more stale they become. In Scrum we ask the ScrumMaster to help the Product Owner in their grooming responsibility and Scrum also allows the development team to spend up to 10% of their Sprint preparing for the next Sprint so, effectively, they are involved in grooming the Product Backlog too.

 

There are a number of options for Andy and the team here. As Murray suggested they could try a quicker estimation technique – such as affinity estimation, where team members don't vote but instead collaboratively move the items along a scale – to speed up the estimation process. 

 

This is often a very useful technique to cover more ground in an estimation session. My personal preference is that this is only used when a team is used to working together and, ideally, after a good investment of time into establishing benchmarks and assumptions through something like planning poker. Hopefully the development team were estimating the Product Backlog in priority order and so the items that had been estimated were the most important ones. If this is true then the decreased discussion and debate in the affinity estimation approach is likely to be less of a risk. However, in this particular situation, I agree with Andy that this probably wasn’t going to address the underlying problem.

 

Another option would be for Murray to group the Product Backlog items into themes of related requirements/features/user needs so that the teams would have less items to size. They would probably need a different set of cards for their planning poker session as these items would most likely be much bigger than the items they were estimating before but it would fit nicely with Scrum. We expect a Product Backlog to be very pyramid/iceberg shaped in terms of size relative to priority. See image below:

 

Self-Organisation Survey Results

Thanks to everyone who contributed to my survey on self-organisation. I presented a summary of the results at NDC2010 in Oslo and have more detailed results available for download from the articles section.

The One Where The Product Owner Was Also The Designer

Overview

Analysis

Summary

The One Where The ScrumMaster Was Also A Developer

This blog post is the first in a series of stories from real Scrum teams (although the names etc have been changed to protect anonymity). 

Overview

Marcel had been at the company for 7 years and, in that time, had come to be known as the “go-to guy” for Java work. In fact he was known affectionately as the “Java Guru”. The last project was a really important one for the whole company and, naturally Marcel was a central part of the effort to build this product. On more than one occasion he had come up with a breakthrough to help get the team and project back on track and, with a monumental effort from everyone, the project just about met the deadline. 

In an attempt to get things under a little more control and avoid this type of situation in the future, management had decided that it was time to start using Scrum. One problem they had was they couldn’t afford a dedicated ScrumMaster and so, after the whole team were trained in Scrum, management asked Marcel if he would take the role on as he was the most experienced person on the team.

Marcel seemed apprehensive but had not got to where he was by saying no when asked if he could take on more responsibility so agreed. He found this challenging almost immediately when working with the team and the Product Owner to come up with the Sprint plan – how much time was he going to spend doing his ScrumMaster work? And therefore, how much of the Product Backlog could he and the team commit to? During the sprint, finding time to chase down the team impediments while also trying to complete his tasks was a constant headache and he often found himself stopping mid-task to shift his focus to a problem the team needed resolving. 

Also, the team didn’t seem to be very self-organising as they were supposed to be with Scrum and he couldn’t work out why.

Analysis

Can a person be a ScrumMaster and a member of the development team at the same time? This is a question that I think I have been asked every week for the last 5 years (or at least it feels like it). From Marcel’s point of view he was obviously very conflicted, between his duties as a member of the team and his responsibilities as a ScrumMaster in servant-leadership of the development team.

First of all this dual-role has made planning (both at a sprint level and release level) even harder than it would normally be as there is an additional unknown variable thrown into the mix, namely how much of Marcel’s time can he spend on development activities and how effective will the rest of the team be without a full-time ScrumMaster? Marcel and the rest of the development team could agree on a figure (say 30%) that they think Marcel would be able to spend on items on the Sprint Backlog as one way of handling it. This at least gives them something to go with but there is another issue here – which items should Marcel commit to?

A team member who is not totally committed (which is effectively what Marcel has become) poses a new problem for the team in that, if he takes on the high-risk or high-priority items, there is a very high chance that he will be interrupted during his work leading to either delay or, in some cases, mistakes. For this reason, the team may decide that it would be prudent for Marcel to focus more on the lower risk items, letting the “full-time” people tackle the higher risk items. This mitigates risk but also loses the value that the “Java Guru” could and should be adding to the team and the Sprint.

Marcel will also probably end up doing less of the job he likes and I see the loss of passion and enjoyment a lot in people taking on these dual roles. This will almost always lead to a loss of productivity, not just in the individual but in the rest of the team as well as they subconsciously get dragged down by it. Of course, this could be the beginning of a whole new career direction for Marcel but, in my experience, that is the exception rather than the norm.

One of the typical problems I see in teams that employ the developer/ScrumMaster dual role is that the rest of the team is slower to self-organise, and in many cases, just don’t get there. I put this down to two factors: firstly the lack of focus of the ScrumMaster (and the fact that quite often it is not the right person) holds a team back due to the lack of coaching and facilitation time available; and secondly, the implied hierarchy of one person with two roles. This person with the dual-role at least appears to be a little bit more important than the rest of the development team and this can stop the rest of the team “stepping up” and increase the feeling of responsibility of the person with the dual-role.

An option for Marcel and the team could be to rotate the role with each member of the development team taking on the dual-role for a Sprint at a time. This would remove the implied hierarchical challenges and mitigate the impacts on Marcel’s technical passion. It would also give every member of the team an opportunity to get some experience in removing impediments and serving the team, almost increasing the self-organisational capabilities of each team member in turn. However, you are almost certain to have someone in the team who is either unsuitable for this role, or not happy about taking it on and therefore are almost guaranteed to have at least one Sprint where the team struggles. Also, a big part of being an effective ScrumMaster is the networking within the organisation: knowing who to speak to and where to go when a certain problem comes up. By only giving each person one Sprint to play the role, nobody is going to get the opportunity to build up their network and “get their teeth into” the role.

Another option would be to duplicate the dual-role i.e. have two people (or possibly the whole team) who are both developer/ScrumMasters. This way whoever is least “in the zone” with their work could pick up the ScrumMaster responsibilities and the risk is spread a little. Obviously we are then bringing the sub-optimisation that comes with task switching to more people instead of one and making planning even harder and there is no focal point for impediment removal.

In this particular scenario, the team found that because Marcel couldn’t be relied upon to perform his development “heroics” others in the team had to step up technically which, although meant a short-term velocity drop, was a massive boost to long-term productivity and teamwork.

Summary

Scrum itself is not particularly prescriptive about this situation although its values of commitment and focus imply to me that it is not recommended. Also in practice it is usually sub-optimal in that it compromises both the development efforts of the individual and the effectiveness of the ScrumMaster role, usually leading to lower velocity of the team. Therefore it is probably best for this situation to be avoided wherever possible, or at least until the team is mature enough to be able to cope with a part-time ScrumMaster. The team would really have to work out how to balance things such that they have the greatest chance of success – would having Marcel fully focussing on his role as developer with a less effective ScrumMaster be better for the team than having a dual-role within the team?

A secondary consideration here is the message that the organisation is sending by not supporting the team with the ScrumMaster that Scrum requires. This is a fundamentally important role and a full-time role too. It isn’t really a question of whether the organisation can afford a ScrumMaster but rather can they afford NOT to have one? This is a question certainly worth asking. Although the options above are all valid ways of coping with the issue, a better solution would be to look at the issue itself – very much what a ScrumMaster is there to do. 

Getting RE-TRAINED to be a ScrumMaster

Does self-organisation work?

I am conducting some research on one of the fundamental underpinnings of agile that has been interesting me of late. How effective is self-organisation? And also, do teams actually want to be able to self-organise or is a turn off for them?

At the moment I am gathering data so if you know of any good papers/presentations on the subject or just wouldn't mind sharing your experiences then please get in touch - geoff@inspectandadapt.com - I would be grateful.

I will also be launching a small questionnaire shortly to try and capture some more specific information on the subject and hopefully I will be presenting my findings at some conferences this year. 

Article on InfoQ

A recent article of mine has been published on InfoQ if you are interested in reading it. It focusses on overcoming anxiety and fear to make Scrum stick in your organisation.

 http://www.infoq.com/articles/making-scrum-stick 

When The Going Gets Tough

When the going gets tough...

 

I hear the phrase "the current climate" a lot these days and, undoubtedly things are tougher for most, if not all, companies. I worry that some companies are using it as an excuse however. I get the feeling that that there are two types of company out there, the ones that were never really serious about being agile and the ones that were (and are); these are the times that it true colours become very visible. "The current climate" quite rightly makes everyone take stock and become even more careful with what we spend our money on, driving even more for value for money and I see two opposite reactions. The companies that cut whatever costs they can (training, travel, consultants, headcount, communication media, team meetings etc) and revert back to the more comfortable world of contracts, documents, process and predictive plans. 

 

I think this is a great time because it is exactly situations like "the current climate" that we need to be agile. We need to make sure we don't let failing projects carry on forever, we need to get greater return from our projects earlier through iterative, incremental delivery and we need to stay and work together. We should invest more in our people (training etc), make sure more than ever that people get together to work through the issues on their projects, ensure tighter relations with all aspects of the project (customers, 3rd parties etc) rather than trying to "look after number 1" through tougher conditions and contracts. Agile methods such as Scrum give you a massive competitive edge and in "times like these" we need all that we can get so, to me, it screams for an agile approach. The companies that are truly behind this need to stick to their principles, because it is easy to regress and lose heart, but those that do stick at it they will surely thrive in "the current climate"

ScrumMaster & Chief Architect

I was having a discussion the other day about the differences/similarities between the Chief Engineer in a Toyota-style team and the ScrumMaster in a Scrum team. This has been brought on recently by the thought in some places that the "leader" (note not manager) of the team should be technical in nature and, in the case of the chief engineer, should be even more technically competent than the team members themselves. I can see why this would have advantages, not least when applying the "Go See" principle, in software teams the place to "Go See" is the code itself. A non-technically minded person would be unable to apply this concept and thus would be unable to act as either a check on the team or a guide for the team.

 

When asked if the ScrumMaster has to be technical, my answer has always been "no". This might be because I was never technical and I seemed to cope quite well, coaxing a team of very good people to achieve and solve problems themselves - why does it need to be a manager who applies "Go See" at the code level? A lot of the most effective teams I see employ peer review far more effectively than management review. It might be because they had good people on the team.

 

An inherent danger in the ScrumMaster being the most technically capable person on the team is that it stifles the creative input of the rest of the team; they become scared of suggesting things in case they looked stupid compared to the genius that is inevitably going to come from the guru. When this person also has the hat of ScrumMaster, this can also infer some authority - of which there is none in the role of ScrumMaster. Conversely, as I understand it, Toyota's chief architect had, if not absolute power, then certainly the final say.

 

This is a similar issue to the ScrumMaster being the line manager of the team, something I also strive to avoid not because it is an inherently impossible combination but because it creates a more-likely conflict of interests and negative consequences. Achieving a high-performing, self-organising team is incredibly difficult and few teams get there. They are, in my opinion, even less likely to achieve this peak if they have a ScrumMaster with either of these types of "baggage".

 

I like the popular racing car analogy, where the Product Owner is seen as the driver, the team as the car and the ScrumMaster as the oil and instruments warning that the engine is being optimally or over-used (working at an unsustainable pace or taking risks). I believe by blurring the ScrumMaster role with the Chief Architect role this compromises the neutrality of the role which is a key balancing act in creating a high-performing team.

 

Another thing that makes me wonder if it could work is the demands on that person to remain the most technically astute person on the team - keeping their hand in - while also taking on the role of ScrumMaster, which is certainly a full time role in the beginning, if not for the first couple of years. The Chief architect role in the Toyota teams was obviously successful however, and is certainly not a model that I advocate ignoring or declaring incompatible with Scrum. I guess it just concerns me with its potential impacts on hyper-productivity.

LSUG - Team Ownership

At the last meeting of the London Scrum User Group, a collection of interested people had an interesting discussion about how to get Scrum team members engaged and taking ownership of the process, especially at things like the Sprint Retrospective. The point that came up was an interest in why that feeling might have been there in the first place and this could obviously be any number of things. There is often the fear of speaking up as this could make you prone to blame, possible inhibitions in front of "seniors" or "technical experts" and so some people would rather defer to others. There is also the factor of WIIFM (what's in it for me) i.e. what benefit do I get from taking responsibility? This could be because the change effect isn't running as deep as it could be and so they don't feel that this "self-organisation thing" is really happening and perhaps there are even conflicting messages coming from other parts of the organisation about whether it is OK to make suggestions take ownership. Most of the discussions came down to a feeling about a lack of safety or a comfortable environment in which people felt OK to speak up.

Then how to create a safe environment? This is arguably something that takes time and a lot of (repaid) trust but one quick way to move in that direction is to lead by example. Sophie, one of our group, mentioned how she had deliberatly put herself "out there" and made herself vulnerable in the hope that others would follow her lead. This is arguably a brave thing to do but undoubtedly a very effective way of establishing trust in a group.

If we are talking about an experienced (in a years under the belt kind of way) as a lot of us were, this is a slightly different problem but still in the same ball park as safety. Other team members don't want to say anything that might be negated or dismissed by "the expert" and likewise, he/she is very happy with this hero-like status. A quite common technique amongst the people present was to actually ban said person from writing any code for a sprint or two and, instead, encouraging them to mentor other members of the team to explicitly try and bring their confidence levels up.

This was an interesting discussion in a pleasant, well-attended evening. I look forward to attending again soon.

The LSUG wiki can be found here and the next meeting is at The Square Pig, Holborn on 14th May

Cragilist or Scrumbut(t)

There seems to be a lot of discussion at the moment about teams and companies who are "not really doing it". There are arguments in favour of those who take baby steps and constantly strive to improve themselves over time.

There are also arguments that most teams get stuck half way, once they have implemented the easy bits, and stagnate with a partial Scrum implementation.

People have called this state a number of things but the most amusing I have heard are Cragilist (crappy agilist) and Scrumbut (we're doing Scrum but...). Some people (usually the Americans) add an extra t to make it Scrumbutt.

The best teams I see use this information as their primary metric for improving themselves, running the Nokia test and aiming to improve on those scores sprint on sprint. 

Why do so many teams turn into cragilists? Probably a combination of factors:

 

• Change is hard and keeping up motivation and discipline for a long time takes its toll
• The easy bits of Scrum are easy to implement (daily scrums, sprint reviews etc) and that seems like enough change for the time being...
• Organisations de-motivate people. Companies that don't reward the good behaviours or encourage or provide incentives to those breaking the mould wear individuals down. Also a failure for the organisation to change and remove the barriers to successful Scrum teams is a contributor here.

There is even discussion of whether such measurements could be brought into the certification process which is an interesting concept. In my view, application of this measurement stops becoming useful when used by those outside of the team to judge that team. I think this would fall into that category because as soon as something is riding on the figures coming out of the Scrum test (pay, bonus, certification etc), the sooner the figures start being manipulated 

Fellowship Of The Scrum

The Fellowship of the Scrum

 

Somebody asked me the other day "What did you learn from the Lord of the Rings?". A strange question you might think. And when you ask strange people strange questions you undoubtedly get strange answers. I thought about it and, given the way my mind works and the prejudices it has, I thought that "the fellowship" quite strongly resembled a good Scrum team. This was a group of individuals plucked from various specialisms (bow, axe, sword etc) from diverse backgrounds, some with tensions (Gimli and Legolas) that come to the surface in a classic "storming phase" which is both inevitable and necessary in every Scrum team. They are also tasked by a pseudo-Product Owner in Gandalf with a highly challenging yet motivating goal to defeat their opponent.  Aragorn emerges as the ScrumMaster but, for a while, the team naturally flip-flops without any internal protection or impediment-removal.

 

There was no opportunity to come up with the perfect plan and the team had to make several significant adjustments to their plan as they came across more information and more problems. Even though there was a king in the group and one person who was tasked with carrying the ring, there was no one person in charge of the group. They self-organised and, at different times along the journey, different members of the group led, picking up the more weary at that particular point in time. Another thing that I noticed when looking back on this "team" was that they very rarely did anything on their own; they almost always tackled things as a team or, at least, in pairs, picking up skills from and playing off each other.

 

If we run with the example of Gandalf playing the role of Product Owner, we also see the benefits a Scrum team gets from the close links with their Product Owner. Gandalf was not always with the team and certainly didn't define how the problem would be solved but he did help them with domain knowledge, guidance and even getting his hands dirty. At other times he was away from the team but still looking forward and furthering their cause like a good Product Owner does within a Scrum team.

 

Obviously a normal Scrum team is not faced with such a life-or-death scenario like "the fellowship" were and should aim to work at a much more sustainable pace than this group but I think it's a fairly good, if somewhat whimsical, analogy of a team spirit coming from a cross-functional group of individuals galvanizing around a challenging goal.

Generalists or Specialists

Scrum has as one of its primary principles the concept of self-organising teams and the ideal here is that any member of the team should be able to take any item off the backlog and get it "done". As these product backlog items should be vertical slices of functionality, this is often difficult for teams primarily due to functional expertise - having experts in each area of the development lifecycle. Due to traditional career progression paths we often have an issue in that testers are encouraged to become developers in order to gain promotion and developers are then encouraged to become designers etc. This is, imho, very destructive as it effectively promotes people out of their areas of expertise as well as promoting waterfall, silo thinking. Also, by emphasising various functional experts, we are backing ourselves into a corner where we are restricted with what we can deliver by the skills we have available. As a business we would much rather be in a position where we can deliver what we and our users need; and having a group of cross-functional people helps us reduce that risk to our project.

This is not to say that everyone becomes a homogenous resource - certain people will always have more skills in some areas than others - but it almost inevitably calls for people to focus on pulling in for the team and picking up other areas than they might have done before hand. I would still expect people with the passion in certain areas to take responsibility for sharing those skills with the other team members in a mentoring and coaching capacity. I have had a few conversations with people who are concerned with this, that their desires to focus on their area of expertise would not be supported by an organisation wishing to "go agile" and that overall technical levels will be diluted by the lack of focus and dedication to their specialisms.

Some of those concerns are driven by "the hero syndrome" where people worry that they will no longer get the respect that they got when they were the only person who could deal with a particular problem. This is definitely going to be reduced in a good agile team as the "heroes" are bottlenecks to teams delivering and something that adds risk to a project. Some of the concerns can also be addressed by finding new ways of creating respect for experience such as mentoring and coaching, thus bringing up the skills of the other team members.

My view is that we should be aiming for a team of generalising specialists: people that pull in together to get what's needed done but still spend time furthering their areas of passion so long as this benefits the team by helping building up the skills of the other team members rather than holding them up waiting for the "hero".

Organisational Butterflies

Maybe it's because I'm an external coach for the organisation, maybe it's because I'm not assigned to any particular project, maybe it's because I've come across so many people in my capacity as trainer and coach, or maybe it's because I'm just nosy... I seem to have so many conversations with so many people from so many projects in one organisation that it seems that I am uniquely placed to bring parties together when needed.

Today was a case in point, I was in one building talking to someone about an experience report we have been pulling together when we got talking about, let's call it "Project X", which she was getting some stress about. I happened to mention that I was just on my way to the Scrum of Scrums meeting and that it would probably be a good idea for her to come along as well - get people who need to talk talking. It made me think of the term butterfly from openspace sessions - someone that just flits from one session to another, taking a bit of information, leaving a bit of information and helping the pollination process a little here and there.

This interested me even more because we have been talking a lot recently about becoming self-sufficient - one of the aims of all of my engagements is for the internal employees to own and drive the change - and I think a few organisational butterflies would be a good thing not just for the success of an agile transformation, but for projects and programmes to work in general.

Workstreams in a Scrum project

I was talking with a team today who were considering splitting their one Scrum team into 3, allowing groups to focus on their own "streams". There was very strong opinion from some members of the team that a split would be constructive, allowing smaller, cross-functional groups to get more focus without the "distractions" of the other team members and what they are working on. On the other hand, there was also a significant feeling that the team would risk becoming silos of domain knowledge and they might lose any advantage that being a team gives them over being a collection of individuals. There was a also a concern that if each team was only focussed on a stream of work, would they lose track of the bigger picture.

I've seen teams split into streams very successfully in the past. They have managed the extra communication overhead of co-ordinating themselves pretty well and can track overall progress just as easily as they can track their progress within their own sprint. However, that is usually only necessary when the team is so big (15 people plus) that the day to day working of the team is a little too much. In this situation, we were only talking about 4 developers, a couple of testers and some BA's and when they really talked through the issues it seemed that most of their problems that were leading them down this route were a lack of domain knowledge within the team and, more importantly, trying to do too much. These streams were really broad and they would have been much more effective focussing as one team on a thicker vertical slice within a sprint rather than trying to do 3 ultra thin vertical slices that weren't really very closed related.