A grid built forward ends with everyone scoring three out of five. Build it from the decisions you owe, and know that adding rows changes which competitor wins.
A competitive benchmark built forward always ends the same way. Someone lists every criterion anyone can think of, twenty rows accumulate, each competitor gets scored, and the result is a table where every serious supplier lands between three and four out of five. Nothing separates. The grid gets presented, admired and never used again.
The reason is not laziness. It is that the criteria were chosen before anyone knew what the benchmark was for, and criteria chosen that way are mostly criteria on which everybody is adequate.
Build it backwards instead: start from the decisions you owe in the next two quarters, derive the criteria that would change those decisions, and fill in only those. Everything below explains why that ordering matters more than it sounds, and what public data can carry the parts you cannot guess.
The grid partly decides the answer
Two findings from preference measurement apply directly to a comparison table, and both are uncomfortable.
Adding rows changes who wins. Experimental work on attribute importance established a number-of-levels effect: holding the actual range of variation constant, increasing the number of levels on which an attribute is expressed increases its measured importance. Split “support” into four rows and support becomes more important than it was when it occupied one, without anything about the suppliers having changed.
A benchmark is exactly this operation performed casually. Every time you decompose one criterion into three, you increase its weight in the reader’s judgment. The person who builds the grid is making a strategic argument, whether or not they intend to.
And comparison itself changes the values. In the founding experiment on this, participants valued two used dictionaries, one with 10,000 entries in like-new condition and one with 20,000 entries and a torn cover. Seen side by side: $19 and $27. Seen one at a time: $24 and $20. The preference reverses, because the entry count means nothing alone and the torn cover means something to everyone.
The follow-up matters more for benchmarking. When the researchers simply supplied a reference range, telling participants what a good and a bad value looked like on the uninterpretable attribute, the reversal disappeared entirely.
That is the whole design principle for a benchmark that works. Its job is not to score. Its job is to supply the reference range on the criteria a reader cannot otherwise interpret. If you write “response time: 4 hours” the reader learns nothing. If you write “response time: 4 hours, against a category range of 1 hour to 3 business days”, you have made a criterion decidable.
There is a second, older objection to the standard benchmark, and it is about what is being compared at all.
The founding statement on competitive strategy draws a hard line between operational effectiveness, which is performing similar activities better than rivals, and strategic positioning, which is performing different activities or performing them differently. A twenty-row feature comparison is entirely a measure of the first. It tells you who is better at the same things.
And the article is explicit that the first is not durable on its own:
“the absence of trade-offs is a dangerous half-truth that managers must unlearn. Quality is not always free.”
The rows that would tell you something are the ones nobody puts in a benchmark: what has this competitor given up. Which segment do they not serve. Which channel have they abandoned. Which desirable attribute have they sacrificed to be good at another one.
Those are findable. A competitor with no self-service tier has given up the small end. One with a fourteen-day onboarding has given up the buyer in a hurry. One who publishes prices has given up the ability to discriminate, and one who does not has given up the buyer who wants to decide alone.
A benchmark of abandonments has one useful property that a feature grid never has: it shows you where the space is empty.
Differentiation is lower than everyone assumes, and it does not stop anyone buying
One finding should temper the whole exercise before you build it.
Research covering 17 categories across two countries found a very low level of perceived differentiation between competing brands, and that customers went on buying them anyway. Brands that were not perceived as different succeeded just as well as those that were.
That is worth sitting with. It does not mean positioning is pointless. It means the market is not waiting to be told how different you are, and a benchmark whose purpose is to prove differentiation is answering a question buyers are not asking.
The version that survives this finding is a benchmark aimed at a decision, not at a claim: given what these suppliers actually do and do not do, what should we stop doing, start doing, or price differently. Those decisions land on the positioning, the site and the campaigns at once, which is why they stall unless the three are held together rather than split between suppliers.
There is a second reason a differentiation-proving benchmark disappoints, and it is arithmetic rather than psychology. Most criteria anyone can name are ones on which every viable supplier already clears the bar. Support exists. Security exists. Onboarding exists. A criterion where all five competitors are adequate contributes nothing to a decision but a great deal to the length of the table, and, through the number-of-levels effect above, quietly adds weight to whichever theme happens to have been decomposed into the most rows.
A quick test before a row goes in: could this criterion, on its own, make you choose a different supplier? If not, it is context, not comparison. Move it to a footnote or drop it.
For a company in the United States, a surprising amount of the structural work is free and official.
How many competitors exist, and where. The Census Bureau’s Statistics of U.S. Businesses reports firm counts, establishment counts, employment and payroll by industry code, employment size band, state and county. For reference year 2022 it records 6,395,635 employer firms nationally, of which 103,741 employ between 100 and 999 people. County Business Patterns provides the same picture by establishment rather than by firm, which is what you want if you sell to a site rather than to a head office.
How stable the field is. The Bureau of Labor Statistics publishes establishment survival by opening year. Across cohorts opening in 2010, 2015 and 2020, the pattern barely moves: about 80 percent survive one year, about half survive five, and a little under 40 percent survive nine. If your benchmark lists nine competitors, that history says roughly four of them will not be there in five years, and it says nothing about which four.
What none of it gives you. Public data will not name your competitors, will not tell you their pricing, and will not explain why they win. Financial filings, where they exist, show size, growth and solvency. They do not show which objection a competitor answers better than you, which segment they have quietly stopped chasing, or what their buyers actually compared them against in the meeting.
That last item is only obtainable one way, and it is the one most benchmarks skip: asking the buyers who chose someone else what was in front of them at the time.
Write the decisions first. Three to five, in one line each, that you owe an answer to in the next two quarters. Should we build a self-service tier. Should we publish prices. Should we drop the smallest segment. Should we shorten onboarding. A decision you are not actually facing does not deserve a row.
Derive the criteria from the decisions. For each decision, name the one or two facts about competitors that would change your answer. If no fact about a competitor would change it, the decision is not a competitive question and the benchmark cannot help.
Cap the grid. The method that measures how people trade off attributes recommends six or fewer, on the stated grounds that beyond that respondents resort to simplifying tactics that distort their preferences. Your readers are not more robust than experimental subjects. A benchmark with twenty rows is not more thorough, it is unreadable, and the person who built it has already decided the answer through the choice of rows.
Put a range on every criterion. Not a score, a range. Best observed, worst observed, and where each supplier sits. That is what makes an uninterpretable attribute decidable, and it is the single change that most improves a grid.
Add the abandonment column. For each competitor, one line: what they have visibly given up. It is often the most useful cell in the table, because it is where a decision for your own company becomes visible.
Record who compiled it and when. A benchmark is a snapshot of public evidence on a date, assembled by a person with a point of view. Both facts belong on the page, because the next reader will otherwise treat the grid as a measurement.
And check it against buyers, not against websites. Marketing pages tell you what a competitor claims. Buyers who chose them tell you what was actually in the room, which is the only version that determined an outcome.
A benchmark built this way is shorter, less impressive in a deck, and considerably harder to ignore. It ends in a decision rather than in a table, which is the only test worth applying: if the grid could have come out either way and nothing would have changed, it was never a benchmark. It was a summary.
Frequently asked questions
Why does every competitor score about the same in my benchmark?
Because a grid built forward collects every criterion anyone can name, and on most of them every serious supplier is adequate. A benchmark filled with criteria that do not separate anyone produces a table where nobody separates.
Does adding more criteria make a benchmark more rigorous?
No, it makes it less interpretable. Experimental work shows that increasing the number of levels of an attribute increases its measured importance even when the actual range does not change. The grid you build partly determines the answer you get.
How many criteria should a benchmark have?
Few. The reference method for measuring what people trade off recommends six attributes or fewer, on the stated grounds that beyond that respondents resort to simplifying tactics that distort their preferences. A twenty-row grid is not more thorough, it is unreadable.
How do I count my competitors without buying a database?
The Census Bureau's Statistics of U.S. Businesses gives firm counts by industry code, employment size band, state and county, for free. County Business Patterns gives the same by establishment. Neither will name your competitors, but both will tell you how many exist and where.
What will financial statements never tell me?
Why a competitor wins deals. Filings and credit reports show size, growth and solvency. They do not show which objection the competitor answers better, which segment they have quietly abandoned, or what their buyers actually compared them against.