How we measure, and what we can't

1. Sampling AI engines

Answers from language models are stochastic: ask the same question twice and you can get two different sets of recommended vendors. A single observation is therefore not a measurement, and any tool reporting one as though it were is misleading you.

We sample each tracked prompt repeatedly across each engine on a schedule, and we record the model version returned with every sample. Model drift is a real confound — an engine silently upgrading its model can move your citation rate without anything on your site changing — so the dashboard names the model version behind each engine's rate, and says so explicitly, with the date, when the version changed inside the window you are reading. A rate that spans a model change is two measurements, not one. There is no citation chart to annotate yet; the change is stated in words under the per-engine rates instead.

Prompts are deliberately unbranded. Asking "is Acme any good?" guarantees Acme is mentioned and measures nothing. We ask the comparison, alternatives and buyer's-guide questions a real customer types.

Which engines, exactly. Claude, with live web search is always sampled. Every other engine is sampled only where two things are true: its key and adapter are configured on this deployment, and your plan's engine count reaches it. Until both hold it is not sampled and not reported. We deliberately do not print the current number here — a sentence on a marketing page cannot know what the deployment is configured with, and one that says "one engine" the day a second is switched on is worse than no sentence. The free check names the engines this deployment can sample right now, from the running code; every result — the free check and the dashboard alike — names the engines that actually answered, taken from the run's own record. A tool that lists engines it did not ask is reporting a number nobody measured.

2. Confidence intervals

Citation rate is a proportion — k citations out of n samples — so we report it with a Wilson score interval at 95%. Wilson is used rather than the textbook normal approximation because the normal interval behaves badly at small n and near 0% or 100%, which is exactly where most brands start.

All three read "30%". They do not mean the same thing, and we don't display them as though they do.

Two figures are reported as significantly different only when their intervals do not overlap. When they overlap, we say "no significant change" — even when the point estimates differ, and even when the difference would flatter us.

This is the rule the dashboard applies when it compares one engine against another: every pair of engines is checked, and the page names the pairs whose intervals are actually apart. Two things about it are stated rather than hidden. It is conservative — non-overlapping intervals imply a difference, but two proportions can genuinely differ while their intervals still touch, so a pair we do not name is "not shown to differ", not "shown to be the same". And it carries no multiple-comparison correction: with five engines there are ten pairs, and testing ten pairs at 95% finds a spurious one more often than one pair does. The number of pairs compared is printed beside the verdict so the reader can weigh it. What the dashboard will never do is claim a comparison it did not perform — where there are fewer than two engines with samples, it prints nothing at all.

3. Detecting decay

A page's recent 28 days are compared against the preceding 84, using Search Console data. The critical adjustment: we subtract the site-wide move over the same window. Without that, every page on a site looks like it's decaying every January, and you'd spend your refresh budget on pages that are fine.

Pages below 3,000 impressions in the trailing 90 days are marked unknown rather than classified. Beneath that volume the day-to-day noise swamps the signal, and a confident label would be false precision.

The part nobody else does

4. Attributing a change to an effect

The question is not "did traffic go up?" — it is "did this page do better than the pages we left alone would have predicted?" Everything below is how that is calculated, in the order it happens. The estimator is called lr-re-2, and every rate and power figure in this section is a measured result of testing it on simulated sites where the true answer was known; the percentages used to explain it, such as "+30%", are examples.

Which pages are the comparison

When Penlid changes a page it records the change and, at that moment, freezes a pool of untouched pages from the same site: up to 30 of them, ranked by how close they were in traffic and in trend, with a modest bonus for sharing the first URL path segment (/blog/, /guides/, /pricing/). That segment is a site's own information architecture, already there for free; there is no content-based grouping, because that would take a model call per page. The pool is frozen because the one fact that cannot be rebuilt afterwards is which pages you left alone.

Which of the pool qualify is decided later, on days that are not part of the measurement. The change is measured over the 28 days before it and the 28 days after it. The pool is narrowed to the best eight, and which pages qualify is read from each page's own clicks over a separate four-week stretch only: the 28 days that end 29 days before the change (days −56 to −29). To be used, a page needs at least 5 clicks in that stretch and, when the page you changed had 10 or more clicks before the change, has to be within ten times its size. That size comparison sets the page's clicks in the stretch against the changed page's own clicks in the 28 days before the change (days −28 to −1), so the changed page's baseline is the one figure from the measured days that it reads. Because the stretch used to qualify pages is separate from the two the lift is measured over, qualifying them cannot tilt the result towards "it worked". In the worst simulated case, qualifying them on the "before" days instead added 1.08 percentage points of false "improved" results; qualifying them on the separate stretch changed it by −0.08. The test harness records one lean that reading the baseline does have, and it runs the other way: among runs at the size boundary that were given a verdict, false "declined" results outnumbered false "improved" ones.

One part of the ranking is not separate, and we say so: pages are also put in order by their trend, and the trend is read over the 56 days before the change, which includes the days the change is measured against. That only orders pages that already qualified, and matters only when more than eight qualify. None of the simulated scenarios on this page has more than eight qualifying pages, so none of its figures measures what that ordering does; matching on the earlier stretch's trend alone is the next change we want to make. Pages that were themselves changed at any time from 56 days before the change to 28 days after it are left out and counted, and the count is printed beside the result. That is a wider stretch than either window the lift is measured over, because a comparison page changed in its own qualifying stretch has a "before" that is that change's "after". If fewer than three untouched pages are left, the result is "not measurable" rather than a number.

What is calculated

Each page's clicks after the change are compared with its clicks before, as a log ratio: the natural log of (after + ½) over (before + ½). Ratios remove the difference in size — a 20% rise is the same number on a 100-click page and on a 10,000-click page — and the half-click keeps a page with no clicks in one window from dividing by zero. The headline figure is the changed page's ratio set against what the untouched pages' ratios predict, turned back into a percentage: "+30%" means 30% more clicks than they predicted, and it can never fall below −100%, because a ratio cannot.

Between-page drift. Pages drift apart on their own: a season hits one more than another, a ranking wobbles, a link lands on one and not its neighbour. The untouched pages show how much. If their ratios are spread further apart than counting noise alone would spread them, the surplus is estimated as between-page drift using the Paule–Mandel method — the amount of drift that makes the untouched pages disagree with each other exactly as much as they actually did. The changed page is assumed to be able to drift by the same amount, so the bar for a result is higher on a site whose pages wander than on one whose pages move together.

The interval. With three to eight comparison pages, the amount of drift is itself uncertain, and that uncertainty matters as much as the counting noise. So the interval is not worked out at one assumed level of drift. It is a calculation over every level of between-page drift consistent with the untouched pages, each weighted by how plausible it is given how much those pages actually disagreed, combined into one reference distribution. The 95% interval and the p-value are both read from that one distribution, so "the interval excludes zero" and "p < 0.05" are the same statement and cannot disagree. The p-value, the interval and the number of untouched pages used are stored with every estimate and printed under the verdict; the verdict never arrives without its working.

If fewer than three comparison pages remain, the result is inconclusive or not measurable and says why. We do not fall back to a bare before-and-after number and let it imply a cause.

How much starting traffic a result needs

This is counted on the page you changed, in the 28 days before the change:

• None. There is no baseline, so there is no lift to measure. The result says "New page" and gives the clicks and impressions it did get. It never says "no measurable effect", because nothing was measured.
• 1 to 9 clicks. You get a direction and no size: "Went from 3 to 58 clicks while the untouched pages moved +4.0%. The direction is clear (p = 0.004); the starting point is too small to put a size on it." Any percentage from a starting point that small is mostly the luck of its first few clicks. We tested lowering the threshold to 5 clicks: the intervals for 5 to 9 clicks covered the true value 97.78% of the time on average but only 89.65% in the worst scenario, so it stays at 10.
• 10 or more. A size, its 95% interval and p-value. When the answer is "we cannot tell", the result also says the smallest lift the measurement could have detected, which is the honest answer to "did it work?" when the evidence is not enough.

The pre-trend check

The same calculation is run one stretch earlier, treating days −56 to −29 as "before" and days −28 to −1 as "after". If the page was already moving differently from its comparison pages before the change (p < 0.05), the result carries a warning that says so. It is a warning and not a gate on purpose: refreshing a page that is already decaying is the main thing this product is for, and refusing to measure it would refuse the main use.

What it can find, measured

The test: simulated sites, with a known lift put on one page and nothing on the others, run through the real estimator. In the table, "clicks a day" is the changed page's traffic before the change (4 a day is about 112 clicks in a 28-day window), "drift" is the between-page drift - each page's after level is multiplied by a random factor with a standard deviation of 15% - and "pages" is the number of comparison pages. 2,000 simulated experiments per scenario, 200,000 for the first three rows; results of the run on 2026-10-01.

Two of those deserve a plain reading. A +80% lift on a 4-click-a-day page is found 92.01% of the time, which only just meets the 92% we set as the target: 184,022 of 200,000 simulated experiments against the 184,000 it needs, so read it as "about 92%" and not as headroom. An exact textbook t test, run on the same simulated data, finds 90.79%, so the estimator is not losing power that a standard test would have kept. And drift between pages costs a lot: with 15% drift and a +80% lift on six comparison pages, the lift is found 36.95%, 59.65%, 79.55% and 82.20% of the time at 1.4, 4, 40 and 400 clicks a day (the table's 78.90% is the 40-clicks-a-day case run again on another seed; at 2,000 experiments a scenario any one figure can be a point or so off); with 35% drift the same four scenarios give between 17.35% and 26.25%. A change that is real and smaller than the table's lifts will often come back "we cannot tell", and the result says how big a lift it could have seen.

The other direction, when nothing was done to the page: across 675 simulated scenarios (page sizes from 0.4 to 400 clicks a day, comparison pages from a fifth to ten times its size, drift up to 35%, site-wide moves of −30%, none and +50%), a change was called significant 3.26% of the time on average when the estimator runs alone and 2.87% when the pages are chosen the way the product chooses them, where a 95% test allows 5%. The worst single scenario was 5.85% either way. "Improved" specifically was called falsely at worst 3.10% and 3.05%, and at the stricter 1% level the worst was 1.20%. The 95% interval contained the true lift at least 94.70% of the time for pages with 1.4 clicks a day or more. Each scenario is 2,000 runs, so any single figure can be a point or so off.

The largest remaining error: pages picked because they were declining

Refreshing pages because they were declining is the largest remaining source of false "improved" results, and we cannot remove it. A page that has just had a bad four weeks is likely to have a better next four whatever you do to it — that is regression to the mean, and no comparison group can subtract it, because the comparison pages were not chosen for having had a bad four weeks. We measured it: pages whose last 28 days sat at least one standard deviation below their own average, and which were then changed in no way at all, came back "improved" 5.50% of the time at 4 clicks a day and 5.75% at 40 clicks a day with no drift between pages, and 9.15% and 9.20% with 15% drift. When the pages were not picked that way, "improved" was called falsely at most 3.10% of the time. The pre-trend warning above appeared on only 10.9% to 22.7% of those pages, so it is a useful flag and not a safeguard.

What reduces it: when a result matters, choose which pages to change without looking at how they were trending — every page in a set, or alternate pages — and the comparison is clean. Where you did pick on decline, read an "improved" as "at least as good as the pages left alone, and possibly some recovery that was coming anyway".

The free did-it-work tool

The free tool runs this same calculation on your export, and on identical counts produces an identical result. On a day-by-day export it chooses comparison pages the same way, from days −56 to −29, and shows which days those were on each card. A Compare export holds only the two periods being compared, so there is no separate stretch to choose from: every other page with at least 10 clicks across the two periods is used, the pre-trend check cannot run, and the result says so above the number. Because the choice cannot be made without leaning on the answer, the tool makes none.

5. Fact verification

Checkable claims — statistics, dates, prices, attributions, product facts — are extracted from every draft and verified individually against live sources. Each gets a stored record: verdict, confidence, source URL and the exact supporting quote.

"Supported" requires a source stating that specific fact. A source discussing an adjacent statistic is recorded as unverifiable, not supported. A draft with any contradicted claim is held for human review regardless of its overall score.

What this method cannot tell you

Published deliberately, because a measurement product that hides its limits is just a confident one.

• It is not a randomised experiment. Control pages are matched, not randomly assigned. Matching on pre-period behaviour removes a lot of confounding but cannot remove all of it. A change that happens to coincide with something else page-specific will still be misattributed.
• Picking pages because they were declining inflates "improved". Pages that had a bad month tend to have a better one next. The comparison pages cannot subtract that, and the pre-trend warning catches only a minority of cases. Section 4 has the measured rates.
• We only know about changes we were told about. A comparison page that someone edited outside Penlid looks untouched to us, and a few of them can move the comparison. Pages changed through Penlid, or listed in the free tool, are left out and counted.
• Few comparison pages means a wide interval. With three to eight untouched pages the amount of drift between pages is itself uncertain, and the interval says so by being wider. On a small site the honest answer is often "we cannot tell", with the smallest lift the measurement could have seen.
• We can't see inside the engines. We observe what an engine answered, not why. A citation-rate change may be your content, a model update, a change in the engine's retrieval, or a competitor's PR win — the ledger narrows this, it doesn't eliminate it.
• Consumer interfaces differ from APIs. Where we sample through an API, behaviour may differ from the consumer app: personalisation, location, memory and A/B tests all vary. We sample without personalisation and say so.
• Prompt selection shapes the result. Which questions you track changes your measured visibility. There is no neutral prompt set. We show you every prompt in full so you can judge the sample yourself.
• Google AI Overviews has no official API. Where sampled, it comes via a third-party SERP provider, which introduces their sampling and geography as an extra layer of uncertainty.
• Search Console has known gaps. Data is delayed by 2–3 days, queries below a privacy threshold are omitted entirely, and Google deletes everything older than 16 months. Our archive preserves your history from the day you connect — but we cannot recover what Google has already discarded.
• Small sites may never reach significance. If your pages don't clear 3,000 trailing-90-day impressions, this method will keep returning "insufficient data", and a page with fewer than 10 clicks in the 28 days before a change gets a direction at best. We would rather say that than invent a number to justify the subscription.