It's becoming more of an issue now that we are growing.
We have had conversations with the YouTube guys about this and they have asked us to use API's instead of directly scrapping pages.
Quora is currently rate limiting us a ton as well. We again have to directly reach out to them.
Most oEmbed endpoints aren't rate limited because they are generally not resource intensive. They are also meant to be used client side and therefore expect more traffic.
Embedly also does a fair amount of caching that reduces the number of calls that we have to make. This limits us, because we aren't a straight pass through, but helps us handle spikes of traffic.
We have had conversations with the YouTube guys about this and they have asked us to use API's instead of directly scrapping pages.
Quora is currently rate limiting us a ton as well. We again have to directly reach out to them.
Most oEmbed endpoints aren't rate limited because they are generally not resource intensive. They are also meant to be used client side and therefore expect more traffic.
Embedly also does a fair amount of caching that reduces the number of calls that we have to make. This limits us, because we aren't a straight pass through, but helps us handle spikes of traffic.