Creating Sitemaps for Very Large Websites is Tricky

Over the weekend, I started working on creating a sitemap for my business research project. I made one permanent url per company page and one permanent url per sic category.

The result is a theoretical sitemap that is about 605,000 items long. That’s a lot of XML, which means a LOT of data, which means the sitemap.xml is like… 20-50 megabytes. What could possibly go wrong.

Continue reading