Check explainer · v1

Near-duplicate pages: when a whole group says the same thing

Automated

Last reviewed

Ten pages that say the same thing are not ten chances to rank — they are one page split ten ways. The crawl groups them and fails every member.

How this check works

Body text is fingerprinted in five-word shingles (pages under 50 words do not participate), and pages whose fingerprints are about 90% similar or more land in one near-duplicate group. Every member of the group fails, naming up to three peers and the group size — canonicalization signals are split across the set.

What does a failure look like?

Location pages cloned from one template with only the city swapped; product variants that differ by a colour name; a migrated site that kept both URL shapes live — groups that each look findable alone and dissolve into one indistinguishable cluster together.

How to fix it, step by step

  1. Decide the group’s keeper: the strongest URL for the topic.
  2. Merge the unique value into the keeper and consolidate the rest with canonicals or 301s — differentiation is the other option when each page genuinely serves a segment.
  3. Re-run the crawl and confirm the group breaks up or shrinks to one URL.

What does a pass or fail look like?

Good

No group above the similarity line; every near-neighbour pair differs in substance.

Fail

Six location pages share ~95% of their body text and form one group — six URLs, one page’s worth of signal.

Frequently asked questions

How is similarity measured?

By fingerprinting body text as overlapping five-word shingles and comparing the fingerprints. The group forms when a page is about 90% similar to the group representative; the reported percentage is the least-similar member.

Why does every member fail, not just the duplicates?

Because the defect belongs to the set: none of the pages can prove it is the canonical version. Fixing one member leaves the rest splitting the same signals, so the finding names them all.

Is this the same as duplicate titles or descriptions?

No. Those rules compare single fields; this one compares body content, which catches cloned templates even when titles and descriptions were varied. Both can fire on the same page.

This rule is one of the checks the audit runs. Run the full site audit

Free preview; no account needed.

Visdenda check explainers · one page per published ruleNear-duplicate pages · v1