SEO test calculator

How the numbers are made

The calculator simulates a site the way penlid’s acceptance grid does, and judges every simulated test with the real estimator. Counts in each 28-day window are Poisson draws. Every page has an early window (days −56 to −29), a before window (days −28 to −1) and an after window; each page’s after window also drifts by a random factor whose log has a standard deviation of tau, which is the noise untouched pages carry. Your page’s after window additionally carries the change you are asking about. The count you type is taken as each page’s steady rate per window.

Each simulated test then goes through the code the product runs. eligibleControls picks the comparison pages from the early window alone, then estimateLift judges the before and after windows of the pages it chose. Both are imported from /assets/lib/stats.js, the one file the paid evaluator also uses, so the calculator and the product cannot disagree about a verdict. A run in which fewer than 3 pages are usable counts as a run with no verdict, because that is what you would be told.

Every figure comes from a stated number of simulated runs and a seed that is a hash of what you entered, so the same inputs give the same numbers every time and a colleague who enters them reads what you read. Sampling error is about ±1.4 points on each chance in the tables, near 50% at the standard 5,000 runs, and larger when the accuracy is set lower. The 80% changes carry more error than that, and the range simulation noise leaves each one in is printed beside it.

The “change for an 80% chance” is found by bisection on simulated power: the size is narrowed seven times, each step a fresh simulation, from the table you see. The figure printed is where a curve fitted to every point that search simulated crosses 80%, which is steadier than the last step alone, and the range beside it is that crossing’s 95% range: how far simulation noise alone could move it. The “Significance threshold” is the median, over simulated tests where nothing changed, of the estimator’s own mde_pct: the smallest estimated lift that would have been called significant. It is where a verdict starts to read “improved”, not the lift the test can see: a real lift of that size is caught only about half the time in the tests that have enough comparison pages (less often where many tests have too few), which is why the 80% change is bigger. It is a typical value, not a guarantee for your page.

Estimating tau from your own file

The file needed is a page-by-day table: Date, Page, Clicks and Impressions columns, one row per page per day, covering at least 84 days. Search Console’s own Export button gives separate Pages and Dates tables, which cannot be joined into one, so this tool cannot use them; the Search Analytics API, a Data Studio table with Date and Page dimensions, or a spreadsheet add-on produce the right shape. The content decay check and the did-it-work tool on this site read the same file.

Windows count back from the newest day in the file, 28 days each. For each pair of consecutive windows it takes every page that had at least 30 in the window before them (the early window, which is not part of the page’s own before-and-after ratio), computes the page’s log ratio and its counting variance, and finds the tau at which the pages’ spread matches counting noise: the estimator’s own Paule-Mandel estimate. The headline is the middle of the pairs’ values, so one odd month does not set it. A pair with fewer than 12 usable pages, or with fewer than 26 of a window’s 28 days present, is skipped and named.

Penlid itself accepts a comparison page from 5 in that window, but tau is read from the spread between pages, and below about 30 a month counting noise swamps that spread and the estimate comes out too low. On simulated pages of 5.6 to 28 clicks a month, with a true tau of 0.15, a floor of 5 read a tau² of 0.0097 against the true 0.0225 on average, and 18 of 60 files read exactly 0, which the page would have called “very alike”. So smaller pages are not used, and when too few pages are big enough the tool says so and you pick one of the presets. The estimate is selected for you only when the file pins tau down. It is shown but not selected when its 95% range runs into the next preset’s zone (below 0.10 is “very alike”, 0.10 to 0.25 “typical”, 0.25 to 0.35 “volatile”), when the top of its range is more than 1.5 times the estimate, or when the estimate is exactly 0: the method cuts it off at 0 when the pages spread no more than counting noise would make them spread, so 0.00 says the file cannot see the drift, not that there is none. Reading tau too low makes every chance look better than it is; the top of the range is offered as the cautious choice. An estimate that is not selected still never leaves you calculating with a tau below what your file reads: if the tau already selected in step 4 is lower than the estimate, the page selects the cautious end instead and says so, and you can choose the estimate itself or a preset if you know your pages better than the file does. A tau selected at or above the estimate is left as it was.

Opening a very large file can hold the page for about half a second: the parser shared with the other tools reads it in one go. Working out tau from the rows afterwards is done in short slices, the way the simulation is, and the page stays responsive.

Methods: the rules and thresholds

Each number below is a constant in public/assets/lib/power.js or public/assets/lib/stats.js. The project’s tests compare the numbers quoted here with those constants; that is a check on the numbers, and not on every sentence. They are judgement calls, and they are the owner’s to change.

The three answers

A page with 0 in the 28 days before the change is a new page: no lift can be measured. 1 to 9 earns a direction (“improved” or “declined” at p < 0.05) and no size. 10 or more earns a size. This is the rule estimateLift applies to the before window, from SIZED_MIN_BEFORE.

Comparison pages

At least 3 are needed and at most 8 are used. Candidates are ranked by closeness in size to the page and the closest 30 are kept, as at publication. On every simulated run a candidate is used only if it had at least 5 in its early window (days −56 to −29) and, for a sized answer, that count is within ten times the page’s before count; the best 8 that pass are used. A run with fewer than 3 usable pages has no verdict.

Tau presets

Very alike 0.05, typical 0.15, volatile 0.35. The acceptance grid’s tau went to 0.35, its counts to 11,200 in 28 days (400 a day) and its comparison pages to 8; outside those the page says so.

Chance of a verdict

The share of simulated tests that returned “improved” (for a lift) or “declined” (for a drop), counting refused tests as having no verdict. Lifts are +10%, +30%, +50%, +80% and +150%. Drops are −10%, −30%, −50%, −80% and −90%; a drop of 150% cannot happen.

The change that gets 80%

A chance of 80% (the usual target for a test’s power), found by bisection in 7 steps on simulated tests, each step half the standard run count. The figure printed is where a straight line fitted on the normal-quantile scale to every simulated point with a chance between 50% and 95% reaches 80%, counting each point’s chance among the simulated tests that had enough comparison pages to give a verdict at all and reading 80% of all tests the same way (a test with no verdict can never read “improved”, so the chance cannot climb past the share that had one); the range beside it is that crossing’s 95% range, 1.96 standard errors either side, widened when the line misses its points by more than their noise. If there is nothing sound to fit, the bisection’s own answer and the bracket the search started from are printed instead. If no lift up to +99,900%, or no drop up to −99.9%, gets there, it says so and why.

False alarms

The share of simulated tests, with no change at all, that returned “improved” or “declined”. At the 95% level it should be at or under about 5%. The acceptance grid held every recorded scenario to 7%, and its worst was 5.85%. A figure above 7% is flagged on the page as outside what was tested.

Accuracy

2,000, 5,000 (standard) or 20,000 simulated runs per scenario. A calculation runs 11 scenarios plus the 80% searches, so about 18 times the per-scenario count. The binomial standard error of a figure p is √(p(1−p)/runs).

When the answer arrives

The after window starts 1 day after the change and runs 28 days. Google restates the newest days for 7 days, and reports about 3 days late, so the answer is 28 + 7 + 3 = 38 days after the change goes live. These are the server’s own constants (EVAL_WINDOW_DAYS, REVISION_WINDOW_DAYS, REPORTING_LAG_DAYS), mirrored here, and a test runs the server’s functions and compares.

Estimating tau

At least 84 days in the file (three windows of 28), at least 12 usable pages in a pair, at least 26 of a window’s 28 days present, and the newest 6 pairs at most. A page is read only if it had at least 30 in the window before the pair. The interval printed is the Q-profile interval for tau². An estimate whose 95% range reaches into the next preset’s zone, whose range tops out at more than 1.5 times the estimate, or which is exactly 0, is not selected for you, and the top of the range is offered as the cautious end. If the tau already selected is below the estimate, the page selects the cautious end instead; otherwise it leaves the selection as it was.