Basically the scrappers do not bother to cache your website or if they do, with an insanely low ttl. Also they do not specialize the content. So the worst hit sites are something like git hosting due the bfs style scrape (every link). The worst part is alot of this is done via tunneling so ip can be different each time or from residential ops. Which makes it annoying.