> dozen of huge blogs vs tens of thousands of onepagers Try crawling ONE wordpre...

		anc84 on July 6, 2018 \| parent \| context \| favorite \| on: How to crawl a quarter billion webpages in 40 hour... > dozen of huge blogs vs tens of thousands of onepagers Try crawling ONE wordpress blog with less than 10 posts and you will be surprised just how many pages there are due to pagination on different filters and sorting options, feeds, media pages, etc.