Figuring out how many broken links the internet has

When a link on your website breaks, usually nobody tells you. Users just leave. To understand how big this problem actually is, I wanted a number: what share of the links on the internet no longer work? Getting that number turned out to be harder than expected. Here is the journey of how I got there.

Idea 1: Scrape the entire internet

My first idea was to scrape the entire internet and cross-reference every link I found. However, this requires a lot of storage. Because of the nature of how I would later interact with the data, HDD storage is not sufficient. The 8TB SSD in my laptop is already relatively full, and I had a feeling that this wasn't going to be enough anyways, so Gemini-3.1-Pro recommended that I rent one of Hetzner's SX servers for this idea. But before I was going to do that, I wanted to ensure that my idea would 1. be practical and 2. give me results that I can trust.

Getting a list of every website

The problem already starts with the domain names. How do you get a list of every registered domain? Well, there are websites that offer just that, with some lists only costing 10€. I had acquired such a list a few months prior and got around 310 million domains. But some websites might not have a domain and just run on an IP address. Since there are only 2³² IPv4 addresses, that address space becomes searchable if you're patient enough. IPv6 addresses would not be considered, because the 2¹²⁸ address space is much larger.

The storage problem

The next step would be to set up a Python script and try to fetch the sitemap of every one of these websites. The sitemap files would be saved to disk and there would be some kind of deduplication logic, as multiple domains might point to the same website. Let's assume we have 150 million websites in total (not every domain points to a website), with the sitemap files being 50KB in size on average. That step alone would already cost me 7.5TB of storage.

But sitemap files feature a lot of repetitive patterns, which means that they can be compressed very well. Using standard .zip compression, I would be able to decrease the storage cost to about 1TB.

Using the sitemaps, I would fetch every page found on each specific website. Since sitemaps are usually automatically generated, we won't be finding that many dead links at this step. Saving the entire page content of every page from every website on the internet, with images etc., would not be feasible. Therefore we would only save the list of hyperlinks we found on each page and put them into a database.

Cross-referencing the links

Most dead links are external links (i.e. links pointing to a different domain), like an old Reddit post that shows how to solve a niche problem, or a news article from 2018 reporting about a startup that no longer exists. But I have also personally experienced dead links pointing to the same website, like one FAQ article pointing to another one which no longer exists. In my opinion that is especially embarrassing, as it could be detected automatically with very basic tools.

Let's assume we have successfully scraped every page. Now I would go through every scraped page and see if each hyperlink it points to also exists in the database. If yes, the link is valid. If not, we would fetch that specific link and check for 4XX status codes or a soft 404.

Why it fails in practice

This is how it could work in theory, but in practice it's not that easy. Along my route of scraping the entire internet, I would likely get blocked by an increasing amount of firewalls and Captchas. That would lead to a lot of false positives.

With a 1 Gbit/s connection on the rented server, I would be able to download a maximum of 10.8TB (~9.82TiB) per day. Because there is a practical limit to how many websites I can scrape at once from the amount of CPU cores and internet speed, the entire project could likely take 30 days or more. Within that timeframe some previously working links might no longer function, leading to additional false positives. After all, I wanted to capture how many broken links the web has at a specific point in time.

Idea 2: The Common Crawl

Then I had a better idea. Instead of scraping the entire internet myself, why not use the Common Crawl?

The Common Crawl is a non-profit organization that has been crawling the web since 2007 and publishes everything it collects for free. Roughly every month a new crawl snapshot is released, containing billions of web pages, and anyone can download and analyze it.

WARC, WAT and WET

The Common Crawl offers multiple variants available for download:

The downloads are offered in the Apache Parquet format (.parquet), which I've never worked with before. It's like a .zip file but for databases.

The compressed WARC is an insane 85 TB in size. So I thought I could just use the WET, which is 6.06 TB compressed, but that only contains the text displayed on hyperlinks, like "Click here", not the actual hyperlink itself. Gemini-3.1-Pro pointed out to me that the WAT contains a list of all hyperlinks from each page. The metadata it contains might also come in handy later. So the file I would have to download is 14.66 TB in size.

The remaining problems

Whilst relying on Common Crawl data reduces the complexity of the logic I'd have to implement myself, it does not resolve the false positive problem caused by firewalls and Captchas. In fact, that problem would only get a lot worse, for the following reasons:

1. A Common Crawl does not contain every web page on the internet.
2. Some websites explicitly block Common Crawl bots.

I could apply the scraping logic from my first idea to reduce the number of false positives; however, the sync issue would still remain. There would likely be a significant time gap between when the Common Crawl was captured, when it was published, and when I would query a page to double-check if a link is actually dead.

Idea 3: The URL index

After spending some more time on the Common Crawl website, I looked into the other data formats that they offer:

• Robots.txt files
• Non-200 responses
• URL index files
• Columnar URL index files

The "URL index files" and "Columnar URL index files" contain the same data, just in different formats. I didn't initially look into the URL index, as I thought it would just be a list of all queried links during a Common Crawl run, but then I found out that they also contain the status code of every scraped link, which is exactly what I needed. This means that I no longer have to rely on the cross-referencing logic; I can basically do my study for a subset of the internet (e.g. whatever the current Common Crawl contains). Because the Common Crawl is very large, I would still get more than accurate enough data. The URL index is also significantly smaller in size, which means I don't have to rent a special storage server. So I decided to try it out.

However, there isn't a big "Download" button on the Common Crawl website that you can just click on. I decided to use cc-downloader to download the data.

At 200GiB, the download for the CC-MAIN-2026-17 completed overnight and I resumed the next day. Now I had Gemini-3.1-Pro write me some Python scripts to analyze the data. Each line of the index is a URL key, a timestamp and a JSON record. That JSON record holds the HTTP status code:

Results: how many links are broken?

First I wanted to figure out the percentage of dead links on the internet.

200 responses: 84.26%
200+301..303+307,308 responses: 92.99%
404 errors: 5.16%
410 errors: 0.13%
500 errors: 0.12%
other status codes: 1.60%

The full file can be downloaded at: https://faray.tech/fragments/broken-link-study/counts.txt

This means at least 5.41% of links on the internet are no longer functional. If you think about links you have bookmarked in the past, that percentage would likely double. In fact, a study from 2013 showed that 49% of links referenced in US supreme court documents no longer work.

The most valuable companies

Next I wanted to find out which websites from the world's most valuable companies had the most broken links. It was definitely a surprise to see Qualcomm, Danaher and Waymo at the top of the list. What wasn't surprising to me was that the websites at the top of the list are from B2B companies. This is likely because B2B companies tend to run large legacy sites with thousands of product pages, PDFs, event pages and campaign microsites that are frequently restructured or quietly taken offline. Far fewer people notice and report the breakage than they would on a consumer product.

The full file can be downloaded at: https://faray.tech/fragments/broken-link-study/domains-by-broken-links.txt

Keep in mind that the Common Crawl bot is blocked by some websites, so the list is not fully in the correct order.

The most visited websites

However, what matters more than knowing which websites have the most broken links is knowing which of the world's most visited domains have broken links. I looked at what the most visited domains on the internet are from a few different sources and decided to use the following 100 domains. Here are the first rows of the result:

The full file can be downloaded at: https://faray.tech/fragments/broken-link-study/domains-by-popularity.txt

Some domains that block the Common Crawl bot were intentionally replaced with alternatives.

Conclusion

In the current race of AI startups, the average lifespan of a website is only decreasing, and most companies do not have any automated tools to deal with broken links. In fact, in most cases the companies aren't even aware of their problem until users complain.

The first step in the right direction would be to serve "410 Gone" instead of "404 Not Found" responses for deleted pages. If a page or product no longer exists, we should let the user know. If a page was renamed, or the link a user clicked on had a typo in it, we can redirect them to the correct page. LinkAutocorrect does both of these things for you automatically and yes, there is a free version.

I hope my research was useful to you. What might also be interesting is to investigate how the percentage of broken links evolved over time.

Discuss this blog post on our Discord server.