The four experiments everyone cites about buttons have, between them, no published per-variant sample size and no published confidence interval. One of them is not an experiment at all. Three of the ten results in the most reproduced collection are negative, which the summaries tend to drop.
Meanwhile the same button is governed by a specification with exact numbers in it, which you can verify in an afternoon without a single visitor.
This page separates the two, because the industry spends its attention almost entirely on the half it cannot verify.
The red button test, in full
It is the most cited result in conversion optimisation. Here is everything its source actually reports.
The finding. Red outperformed green by 21%, meaning 21% more people clicked the red button.
The sample. More than 2,000 visits in total, across both variants combined. No per-variant split is published. No conversion counts are published.
The duration. A few days of traffic. No dates, no window.
The statistics. No confidence interval. No p-value. The article makes a general remark that A/B testing often requires thousands of trials, but publishes no figure for this test.
What the author himself wrote, and this is the part that never travels with the statistic: that the results cannot be generalized to all situations, that the most that can be said is that they hold for the conditions in which they occurred, in that page design, on that site, with that audience, and therefore that readers should not go out and blindly switch their green buttons to red without testing first.
What to take from it. A real result, honestly reported, with an explicit warning attached that the industry removed before repeating it fifteen thousand times.
The $300 million button was not a button test
The second most cited item in this field is a case study, and its headline number is an extrapolation.
What changed. On an unnamed retail site, a “Register” button that forced account creation before purchase was replaced with a “Continue as Guest” option.
What was measured. Roughly $6 million in additional revenue in the first week, and password reset requests falling by about 80%.
Where the $300 million came from. The author states it plainly: some arithmetic let them calculate the revenue being abandoned in carts by everyone who could not authenticate, and that is where the annual figure came from. It was computed, not observed.
The diagnostic numbers behind it. About 40% of visitors to the authentication page were requesting a password reset. Fewer than 25% of those requests were completed. Of those who completed one, fewer than 20% went on to purchase.
Why it still matters. Because the intervention was not cosmetic. It removed a mandatory account creation step from a purchase flow. The lesson is about forced friction, not about buttons, and certainly not about colour.
The measurement caveat the author raised himself. The authentication pages had never been instrumented before the analysis, which is why the problem had gone unnoticed.
The collection of ten, including the three that went backwards
The most reproduced set of button experiments is worth reading in the original rather than in the listicles that quote it.
What it contains. Ten before-and-after cases from 2013: wording changes, a colour change, a font colour change, a size change, an arrow removal, a placement change.
The famous one. Changing “Start your free 30 day trial” to “Start my free 30 day trial” produced a reported 90% increase in trial signups, over three weeks.
The range. From +304% for moving the call to action below the fold, down to -18% for a font colour change.
Three of the ten are negative. The font colour change lost 18%. A bigger button lost 10.6%. Removing a green arrow lost 12.3%. Those three almost never appear in the summaries.
What is not published, for any of the ten. Sample size. Duration. Confidence level. Not one case carries them.
Why not, in the author’s own words. Challenged on exactly this point in his own comment thread, he explained that most clients are not happy about showing their data to competitors, and that he includes as much as they will allow. He then offered an aggregate: most cases based on samples above 2,000 visitors, a declared 98% confidence level.
The honest reading. A practitioner reporting real client work under a confidentiality constraint, which is a reasonable thing to do, and which is also not evidence anyone else can act on.
There is a serious quantitative treatment of this problem, and it changes how you read every number above.
The power requirement. A published analysis by a statistician working inside an A/B testing company puts the requirement near 6,000 conversion events per group to detect a 5% uplift at 80% power, and around 1,600 per group to detect a 10% uplift.
Note the unit. Conversion events, not visits. The famous button tests report visits in the low thousands, which is a different and much smaller quantity.
What under-powering does. The same analysis states that shortening a test that needs two months to two weeks makes it under-powered, and that almost two-thirds of winning tests will be completely bogus.
What peeking does, which is worse. Checking a test daily and stopping as soon as the p-value drops below 5% produced a positive result 41% of the time in a simulation where the two variants were identical. The conclusion drawn is that at least 80% of winning results obtained that way are worthless.
The base rate that frames all of it. The same paper cites data from a large search company suggesting a new variant is generally only 10% likely to cause a true uplift, with 90% having no effect or degrading performance.
And the multiple comparisons problem. Run twenty tests and you will on average get one winner even if none of the variants does anything.
The winner’s curse. Tests that do win tend to overstate the size of the effect, and the effect is stronger the smaller the sample. So the published 21% and the published 90% are, if anything, the inflated versions.
Switch to the accessibility specification and the vagueness disappears. These are numbers, with conformance levels, that you can check without any traffic at all.
Text contrast, level AA. The visual presentation of text and images of text has a contrast ratio of at least 4.5:1. Large text may go to 3:1, where large means at least 18 point, or 14 point bold.
Text contrast, level AAA.7:1, with large text at 4.5:1.
The button itself, level AA. Non-text contrast requires a ratio of at least 3:1 against adjacent colours for user interface components and graphical objects. This is the one people miss: it governs the button’s own border or fill, separately from the text sitting on it. Inactive components are exempt.
Target size, level AA. New in the current version: the target for pointer inputs is at least 24 by 24 CSS pixels, with five exceptions: spacing, an equivalent control elsewhere on the page, inline targets constrained by line height, targets whose size the browser controls, and cases where a particular presentation is essential or legally required.
Target size, level AAA.44 by 44 CSS pixels, with only three exceptions: inline, user agent control, and essential.
Colour alone, level A. Colour is not used as the only visual means of conveying information, indicating an action, prompting a response, or distinguishing a visual element. Note what this does and does not say: it forbids relying on colour alone. It expresses no preference between colours.
The accessible name, level A. For components with labels that include text, the name contains the text presented visually. A button reading “Send” must not carry a completely different accessible name.
Worth stating plainly, because the search volume on this question is enormous and the answer is short.
No standards body says it. The accessibility specification governs contrast ratios, never hues. There is no criterion anywhere that prefers one colour to another.
No ergonomics standard says it. The relevant international standards on visual display cover contrast and legibility, not commercial performance.
No platform says it. None of the advertising platforms publishes guidance on button colour as a conversion lever.
And the best-known practitioner in the field says it does not exist. The author of the ten-case collection wrote that there are no set rules for which colours work, and that rules like “never use red, it’s a stop colour” or “green is always best, it’s a positive colour” are, in his words, plain stupid.
The likeliest explanation for the famous result. Red won on a site whose brand palette was green. That is a contrast effect, which is a property of the relationship between the button and its surroundings, not of the colour red.
What that means for your page. The useful question is not which colour converts. It is whether your primary action is the highest-contrast element on the page. That question has an answer, and you can measure it.
What to do instead
Six steps, ordered by how much they cost and how reliably they pay.
Meet the specified thresholds first. 4.5:1 on the text, 3:1 on the button itself, 24 by 24 pixels minimum, no meaning carried by colour alone, accessible name containing the visible label. This costs an afternoon, needs no traffic, and is the only part of this subject with defensible numbers.
Make the primary action the highest-contrast element on the page. One primary action, visually dominant. If two things compete for that role, neither is winning.
Write the button as the completion of a sentence. The visitor is finishing the thought “I want to…”. Buttons that read as an action the visitor is taking generally beat buttons that read as an instruction from you. This is a copywriting principle, not a tested law, and I am labelling it as such.
Stop testing buttons and start testing offers. Your traffic almost certainly cannot detect a 5% button effect, since that needs thousands of conversion events per group. It can detect the difference between two genuinely different offers, because that difference is large.
If you do test, decide the sample size and the end date before you start. Then do not look until you get there. Peeking is what turns a null result into a 41% chance of a false winner.
And record what you did. Most organisations rerun the same inconclusive test every eighteen months because nobody wrote down that it was inconclusive the first time.
The specified requirements cost an afternoon. Button colour tests cost months and cannot resolve at your traffic. Source : W3C WCAG 2.2 and Goodson, Qubit (2023)
The red button test reports 2,000+ visits across both arms, no per-variant split, no conversion counts and no confidence interval.
Its own author warned against generalising, and told readers not to switch their buttons without testing first. That warning never travels with the statistic.
The $300 million button was a case study, not a test. The measured result was about $6 million in week one; the annual figure was calculated from abandoned carts.
The ten-case collection publishes no sample size for any case, by the author’s own explanation, and three of its ten results are negative.
A 5% uplift needs roughly 6,000 conversion events per group to detect at 80% power. Visits are not conversion events.
Stopping a test when it looks significant produced a winner 41% of the time on identical variants, and at least 80% of such winners are worthless.
A new variant is only about 10% likely to produce a true uplift in the first place.
What is specified is exact: 4.5:1 text contrast, 3:1 for the button itself, 24 by 24 pixels at AA and 44 by 44 at AAA, never colour alone, accessible name containing the visible label.
Fix the verifiable half this week, then test offers rather than buttons. Book a diagnostic, or see how we approach B2B websites.
Frequently asked questions
Does button colour affect conversion?
Nobody has published evidence that would let you answer that in general. The most cited colour test reports 2,000+ visits across both variants and no confidence interval, and its own author warned against generalising from it.
What did the red versus green test actually find?
That red outperformed green by 21% on one page, on one site, over a few days, with more than 2,000 visits in total. The author wrote that the result holds only for the conditions in which it occurred.
Was the $300 million button a real experiment?
No, it was a case study. The measured result was roughly $6 million in additional revenue in the first week and an 80% drop in password reset requests. The $300 million annual figure was calculated from abandoned cart revenue, not measured.
Why should I distrust the famous button test collections?
Because the sample sizes are not published. The author of the best-known collection explained in his own comments that clients would not let him disclose the data, and offered only an aggregate claim after the fact.
How many conversions does a valid test need?
One quantitative analysis of the problem puts it near 6,000 conversion events. Most published button tests report visit counts in the low thousands, which is a different and much smaller thing.
What is wrong with stopping a test when it reaches significance?
It manufactures winners. Checking daily and stopping at a p-value under 5% produced a positive result 41% of the time on tests where no real difference existed, and at least 80% of winners obtained that way were worthless.
What is actually specified for a button?
Text contrast of 4.5:1, non-text contrast of 3:1 for the button itself, a pointer target of at least 24 by 24 CSS pixels at AA and 44 by 44 at AAA, no reliance on colour alone, and an accessible name containing the visible label.
So what should I do instead of colour tests?
Meet the specified thresholds, which costs nothing and needs no traffic. Then test the offer and the wording of the promise rather than the button, because that is where the differences are large enough for your traffic to detect.