Creating Sitemaps for Very Large Websites is Tricky

Over the weekend, I started working on creating a sitemap for my business research project. I made one permanent url per company page and one permanent url per sic category.

The result is a theoretical sitemap that is about 605,000 items long. That’s a lot of XML, which means a LOT of data, which means the sitemap.xml is like… 20-50 megabytes. What could possibly go wrong.

Continue reading

How I Built a Data Analytics Platform

Recently, I was tasked with building a method for performing data analysis as a service, as well as building a data science team that could perform those services for various industries. So, I quickly kicked off a deep exploration of the field and the technologies available in the space. I found what I learned to be very engaging and it got me thinking not only about technical challenges in performing analysis, but also managing the process of doing data science in a business context.

Continue reading