What a Link Index Actually Contains

A backlink index is the record of one crawler’s journey through the web, kept according to one company’s retention rules. It is not a copy of the web, not a sample in any statistical sense, and not a subset of what Google knows. Understanding how it’s assembled explains nearly every surprising thing a link tool ever shows you.

Three properties do most of the explanatory work: it has a frontier, it has a budget, and it forgets on a schedule.

The frontier: what gets discovered at all

A crawler starts from known pages and follows links outward. Which means the only way into the index is to be linked from somewhere the crawler already visits.

Consequences that are easy to miss:

A page with no inbound links from crawled pages is invisible. So is any link on it. If someone writes about you on a site nothing else links to, that link may never enter any commercial index.

Discovery is biased toward the well-connected web. Neighbourhoods that are densely interlinked get crawled thoroughly; isolated corners don’t. This bias is systematic, not random, which is why an index can’t be treated as a sample you could correct for.

Platform structure matters. Sites that expose archives, tags, and sitemaps get crawled deeply. Sites that hide old content behind search boxes, infinite scroll, or JavaScript-only navigation get crawled shallowly, and links in their older content go unseen.

Blocked means absent. A site that disallows a vendor’s user agent in robots.txt, or challenges non-browser traffic, contributes nothing to that vendor’s index no matter how many links it hosts. Different vendors get blocked by different sites, which alone would guarantee they disagree.

The budget: what gets crawled, how often

Crawling is expensive, so every vendor allocates. Pages are prioritised — by the vendor’s own authority estimate, by change frequency, by how many links point at them.

This produces two effects that show up constantly in reports.

Discovery lag varies enormously by source. A link on a busy homepage is seen quickly; a link in a five-year-old post on a small site may take months. The timestamp implications are in what first seen and last seen actually mean.

Recrawl intervals vary the same way. Low-priority pages are revisited rarely, so links on them stay in the index long after removal, and removals on them are noticed late. That’s the source of most of the noise in lost-link reports and index churn.

Neither the priority function nor the interval distribution is published by anyone.

Retention: what stays after it’s gone

Every index has a policy for links it can no longer observe: mark as lost immediately, wait for N consecutive failures, keep in a historical table, purge after some interval. These policies differ between vendors and change over time.

The direct consequence: a “live links” count is a claim about what the crawler last saw, not about what exists. Two vendors with identical crawls and different retention policies would report different live totals.

It also means a larger number can indicate a slower forgetter rather than a better finder, which is why “found 40% more links” is not a quality claim. Measuring which is which on your own site is the point of benchmarking two link indexes.

What the index stores per link, and how it can differ

Even for a link both vendors saw, the stored record differs.

• URL normalisation — protocol, www, trailing slash, case, tracking parameters, fragments. Whether two URL strings are one page is a decision.
• Dedup rules — a link repeated on one page, or the same link across a paginated archive, may be one row or many.
• Subdomain grouping — whether blog.example.com is a distinct referring domain.
• Redirect attribution — whether a link through a redirector is attributed to the redirector or the destination, and how many hops are followed.
• Attribute parsing — nofollow, ugc, sponsored, page-level meta directives, JavaScript-inserted attributes. See how tools count nofollow, UGC, and sponsored links.
• Rendering — whether the crawler executes JavaScript. A link inserted client-side exists for some crawlers and not others.

Six decisions, each defensible, each different between vendors, and each multiplying the divergence in the totals.

What follows for interpretation

An index is a lower bound on links, and an upper bound on live links. It misses links it never crawled, and it holds links that have died. Both errors, simultaneously, in unknown proportions.

Absence is weak evidence. “This tool doesn’t show that link” is compatible with the link not existing, the page being uncrawlable, the source being blocked, and the link being JavaScript-inserted.

Presence is stronger evidence, with a caveat. The crawler saw it at some point. Whether it’s still there requires opening the page.

Cross-vendor arithmetic is invalid. Subtracting one index’s count from another’s yields a difference between two crawl-and-retention regimes.

Index growth is not your growth. When a vendor expands its crawl, your count rises. So does everyone’s, which is exactly what the cohort test in when a metric moves but nothing changed detects.

The one thing an index is unambiguously good at

Describing the shape of a link profile. Which kinds of sites cite you, how concentrated the citations are, what anchors they use, whether the pattern looks accumulated or arranged. Shape is robust to coverage gaps in a way counts aren’t: if you’re missing 30% of links roughly evenly across categories, the shape survives and the total doesn’t.

That’s a good argument for building reports around distributions and compositions rather than headline totals, which is most of what makes a report durable — see building a link report you can defend.

What nobody outside the vendors knows

Index size in any comparable unit, crawl budget allocation, the priority function, recrawl interval distributions, retention thresholds, rendering coverage, and how any of these have changed. Vendors publish some figures; the figures are self-reported, defined differently by each vendor, and not independently verifiable.

Which is the honest bottom line and not a complaint. These are large, expensive, genuinely impressive systems doing an intrinsically incomplete job on an intrinsically uncrawlable web. The error isn’t using them. It’s forgetting that the number in the interface is a property of a crawler.