Perspectives and limits

Criticism and the state of the evidence

Scrum is often presented as the most widespread agile framework, and some of the sharpest criticism comes from its own pioneers. They describe a Scrum that applies the rules without their spirit and serves mainly to put pressure on developers. This article examines these drifts, the certification market and the productivity promises, then takes stock of what research allows us to say and of the status of the patterns in A Scrum Book.

When Scrum is used to pressure the team

Ron Jeffries, one of the creators of Extreme Programming and a signatory of the Agile Manifesto (2001), gave this situation a name in an article published on his website in September 2016: "Dark Scrum". In it he writes that "Scrum can be very unsafe for programmers." Those who hold power turn up every two weeks to tell the team what to build. At the Daily Scrum, someone from outside extracts information from the Developers and then tells them what to do, instead of letting them organize their day around their shared goal. The Sprint Review begins by reminding everyone what the team "promised", when it had only answered that it would try.

Jeffries sees the cause in managers' anxiety. When they do not see working software, they worry and push harder. The remedy he proposes is for the team to deliver every Sprint an Increment that is genuinely finished and tested, because what is visible changes the conversation. To that end he recommends technical practices, including writing tests before the code and continuous integration. This is the opinion of an experienced practitioner, with no measurement to back it, and it places on the team part of the solution to a problem that comes from its management.

Other practitioners have described similar drifts. Johannes Schartau, Christiaan Verwijs and Barry Overeem, in the Zombie Scrum Survival Guide (2020), call "Zombie Scrum" what looks like Scrum from a distance and is nothing like it when you get close. Among the symptoms, they list the absence of a usable product at the end of the Sprint and stakeholders who never show up. In 2018, Martin Fowler, another signatory of the Manifesto, spoke of "faux-agile: agile that's just the name, but none of the practices and values in place", and called the imposition of methods on teams by what he named the "Agile Industrial Complex" "an absolute travesty". The first pattern in A Scrum Book, The Spirit of the Game, starts from the same observation: you can follow the rules and spoil the game (see The spirit of the game).

A certification market

In 2002, Ken Schwaber co-founded the Scrum Alliance, which created a series of certifications. The best known, Certified ScrumMaster, now requires sixteen hours of training, usually spread over two or three days, followed by a one-hour online test: you must answer 37 out of 50 multiple-choice questions correctly. The certification must be renewed every two years, with proof of continuing education hours. Schwaber left the organization at the end of 2009 and founded Scrum.org, which offers a parallel series of "Professional Scrum" certifications. Jeff Sutherland, the Guide's other author, founded the consultancy Scrum Inc.

The Scrum Guide (2020) states that Scrum is free. Yet around this text a trade in courses and certificates has grown up, whose main players are the very people who define the framework. Courses provide a shared vocabulary, but a certificate obtained in two days attests that someone knows the rules, not that they know how to protect a team from a sales director in a hurry. We know of no independent study showing that teams supported by certified Scrum Masters get better results than others.

Twice the work in half the time

In 2014, Jeff Sutherland published with his son J. J. Sutherland a book for the general public, Scrum: The Art of Doing Twice the Work in Half the Time. Taken literally, the subtitle amounts to multiplying productivity by four, since twice the work would be done in half the time.

Sutherland's earlier publications go further. In "Shock Therapy", a paper presented at the Agile 2009 conference with Scott Downey and Björn Granvik, he states that properly implemented Scrum brings teams to performance five to ten times higher than waterfall development. He reports there that six teams at MySpace and at the Swedish company Jayway, guided by a coach who imposes Scrum without deviation, improve their velocity (the sum of estimation points completed per Sprint) by more than 240% in a few weeks. The same year, an experience report co-signed by Arline Conan Sutherland, Jeff Sutherland and Christine Hegarty credits Scrum with increased velocity and quality in American churches between 2005 and 2009.

These texts are experience reports written by the framework's founder, about teams that he or his colleagues were coaching. They have no comparison group, and their unit of measurement is set by the teams themselves. No independent study has, to our knowledge, established a gain of this size. The 2,000-to-1 productivity differences between teams that A Scrum Book repeats (see The Developers) call for the same caution: even assuming they are accurate, they compare very different teams and projects and say nothing about what Scrum brings to a given team.

A velocity that doubles with no extra work. Let us go back to the vehicle rental team. Each Sprint it finishes about ten items of comparable size, which it originally estimated at 3 points on average: its velocity is 10 × 3 = 30 points. Management sets a velocity target. Without any deliberate cheating, the estimates drift, because everyone knows the numbers have to look good. Four Sprints later, the team still finishes ten items, now estimated at 7 points on average. Its velocity is 70 points, a rise of 40 / 30, about 133%, for exactly the same work.

A velocity figure therefore cannot be used to compare a team with itself before and after a change of method, still less with another team. This is Goodhart's law: a measure that becomes a target ceases to be a good measure (see Estimation and velocity and Goodhart's law, in French).

What research has examined

The general difficulty is set out in the article What is the evidence worth? in the Agility corpus (in French). Projects cannot be randomly assigned to use Scrum or not, and the systematic review by Tore Dybå and Torgeir Dingsøyr (2008) rated the strength of evidence in studies of agile methods as very low. Scrum raises two further problems of its own.

The first is that the "treatment" changes from one organization to another. In 2022, Hron and Obwegeser surveyed the studies that describe modifications of Scrum. The framework is adapted in very varied contexts, including outside software, and they classify these adaptations according to nine goals, such as remote work, and seven ways of modifying the method, such as adding procedures or artifacts. Two studies that speak of "Scrum" are therefore not necessarily speaking of the same thing.

The second is that the framework protects itself against refutation. The 2020 Guide declares the framework immutable and specifies that implementing only parts of it is possible, but that the result is then not Scrum. When a team fails, it becomes easy to say that it was not doing "real" Scrum, and a claim that explains failure as well as success can no longer be put to the test.

More recent work looks for what sets effective Scrum teams apart from the others. Christiaan Verwijs, co-author of the guide to Zombie Scrum, and Daniel Russo published in 2023 a study carried out over seven years. They built a model from thirteen field studies, then tested it with a questionnaire completed by about 5,000 developers from some 2,000 teams. Their model makes a Scrum team's effectiveness depend on five factors: its responsiveness, concern for stakeholders, continuous improvement, team autonomy and management support. This result agrees with Jeffries's diagnosis. It has the limits of a survey: the answers are self-reported, the analysis establishes associations and not causes, and the teams are compared with one another, not with teams that work differently.

Patterns as hypotheses to be put to the test

The introduction to A Scrum Book explains where its patterns come from. The authors draw them from their observations of Scrum teams on several continents and in several sectors, in their successes as in their failures. Each pattern was reviewed by at most a dozen practitioners, and dozens of candidates were dropped because they did not go beyond anecdote or lacked empirical validation. The word "empirical" here has the meaning it has in Scrum: the solution has been seen to work several times. It does not refer to a controlled comparison. The stars that accompany each pattern express the authors' conviction, and solutions that failed without anyone writing them up do not appear in the collection, which exposes the whole to survivorship bias (in French).

The authors draw the practical consequence themselves. They advise introducing one pattern at a time, so as to be able to judge afterwards without ambiguity whether it worked, and going back if it brought nothing. They ask that each pattern show, in the team's own situation, that it produces the expected effect. Read this way, a pattern is a reasonable hypothesis, grounded in other teams' experience, that you test in your own setting.

A housing grants office. In a French inter-municipal authority (a communauté de communes, which groups neighboring towns and villages), six officers process applications for home improvement grants. Their manager, back from a two-day course, puts all of Scrum in place at once, just as the office is changing its software. Three months later, nobody can say what helped or hindered.

The following year, the head of the office goes about it differently. Complex files sit for weeks on a single officer's pile, and she formulates a hypothesis drawn from the swarming pattern (see Swarming): if two or three officers handle each stuck file together until it is finished, the processing time will fall. She first measures, over eight weeks, the median time between the submission of a complete file and the decision, which is 24 days. During the eight weeks of the trial, it drops to 17 days, 7 days less, a fall of about 29%.

The team keeps the practice, but the head of the office notes that the trial took place at a time when fewer applications were coming in, and she plans to compare the same quarter from one year to the next. The result holds for this office and proves nothing for another.

What is established, what comes from experience, what is debated

Status	Claims
Established	The evidence on agility in general is weak. Scrum is often modified in practice. The productivity gains announced by Sutherland rest on his own experience reports and not on independent studies. A Scrum Master certificate attests a short course and a questionnaire.
Practitioners' experience	The patterns in A Scrum Book. The diagnosis of Dark Scrum and Zombie Scrum. Jeffries's remedy of genuinely finished Increments and sound technical practices.
Plausible, from surveys	The Scrum teams that report being effective are also those that say they have autonomy, the support of their management and involved stakeholders.
Debated	The superiority of Scrum over other ways of working. The usefulness of certifications. The transfer outside software. The "immutable" character of the framework and the argument that a failure proves one was not doing real Scrum.

Limits and pitfalls

The first pitfall is to conclude that Scrum does not work. A low level of evidence means that the effect is poorly measured, not that it is zero. The other ways of organizing work are no better established, and Scrum draws on reasonably well-supported ideas, such as working in small batches.

The second is the reverse: putting every failure down to poor implementation. The "fake Scrum" excuse is sometimes fair, but it does not spare you from asking whether the framework suits the work. A team that mainly handles urgent, unpredictable requests may have good reasons to prefer a continuous flow to Sprints.

The third is to aim the criticism at the team. According to Jeffries, Dark Scrum arises from decisions made above the team, and the study by Verwijs and Russo places management support among the factors associated with effectiveness. The Involve the Managers pattern, presented in Protecting the Sprint from interruptions, addresses this difficulty.