My experience is that many people want to buy scraped data that is against a large website's TOS such as Google, Amazon or Craigslist and the act of selling that data directly in a marketplace is likely to draw legal action from these companies as it's someone else's copyrighted data. Most companies that want you to scrape their data provide an api or would at least be willing to sell you the data or sell you access to an api.
That's why people simply hire people, rent out proxies and store the data in their own internal databases, because then the companies have to figure out who the source of the scraping is and be able to prove it, which isn't a lucrative target for the scraping companies.
That being said, there are clear examples of companies that sell processed data that's been analyzed by data analysts and isn't the raw scraped data and they have successful and lucrative business models.
> My experience is that many people want to buy scraped data that is against a large website's TOS such as Google, Amazon or Craigslist and the act of selling that data directly in a marketplace is likely to draw legal action from these companies as it's someone else's copyrighted data.
I came here to make substantively the same point. Scraping for your own personal use and re-use as a derived product is reasonably cool but can border into IPR concerns. Scraping to re-sell as a product, thats really stepping over a line.
This has been a problem since the days of the yellow pages phonebook. There are rules around when a catalog is in the public domain, page contents with data I don't think cross into the "its ok to just copy and resell" space.
I scrape several sources. World population and GDP stats are highly useful but finding good canonical sources was hard for a while there. I now use a Python API provided by the world bank, for most of them. Now I just have the problem that by economy, the inputs span 2019-2022 and I can't get a single authority for population estimates right "now" worldwide. But, scraping to resell that? I think it would be wrong.
Scraping internet data is perfectly fine and legal. It may be subject to copyright or ToS (which are not the same) but as long as you don't redistribute copyrighted content, or affirmatively accept any ToS, you're doing nothing wrong. So the potential risk lies entirely with the seller, not the buyer, and could potentially be reduced or circumvented in various ways. Just saying
Yes. but the article was almost entirely about being that seller. The premise was, "why don't more people scrape the web and sell it" -And if I was an IPR owner and detected a scraper, and then tracked back from their origin IP robot to their web service, I'd be sending my legals round.
Offering a brokering service to "share" access to the investment in scraping, has to be done carefully. If for example its brokering access to the page specific structure in some meta notation and you as a consumer scrape direct? thats actually quite cool. If its running a cache of the data re-presented in JSON form for you to consume, or monetising an Elastic Stack of the data and you read it from them? I think its getting iffy to be that seller.
It all comes down to what data: PII (personally Identifiable Info) or IP (Intellectual Property) are strongly protected. There are a lot of useful info not PII or IP that can be collected, used and sold, like prices (Ryanair case against Expedia being a crucial case)
I'm somewhat ignorant on this matter, but I was under the impression that in some jurisdictions (e.g. UK) scraping as a whole sale concept is pretty grey area at best not just on PII & IP
Regarding the legality of scraping factual data...
In the EU there's a 'database right' that covers specific collections of facts. The US has categorically refused to issue any kind of "IP"[0] on facts, and doesn't have 'database rights', but in practice you can sprinkle a few fictitious bits of information in your dataset, retain copyright on the lies, and sue people who don't independently verify the factuality of every bit of information in the dataset. So the US has database rights.
No clue if people are sticking fictional data on their websites to sue scrapers with, but I wouldn't be surprised if they were.
[0] Intellectual property is the right to dictate to competitors the conditions upon which they are allowed to compete, if any.
Doesn't section 230 say that most internet orgs are just aggregators and not publishers? The copyright in this case lies with the content creators and not the orgs? I can see individuals might have a car to sue for copyright, but do these internet orgs have any legal leg?
Maybe copyright is not the correct law to cite, but Craigslist sued companies that scraped it's data and provided it to a wider audience and won like hotpads, etc.
Not a copyright lawyer and I don't know what the actual law is against scraping, but just pointing out that previous companies offering raw scraped data from a large platform were sued out of business.
Not all content is covered byIp (intellectual property). Long textx (like an article), images, or videos are. But factual information (such as prices, a hot one in web scraping) are not.
On the legal side, legislation in the US, UK and EU is clear on PII (personally identifiable Informtion) and IP (Intellectual Property). Web scraping can be perfectly legal when these elements are considered.
Here my article https://blog.databoutique.com/p/web-scraping-legal-context
That's why people simply hire people, rent out proxies and store the data in their own internal databases, because then the companies have to figure out who the source of the scraping is and be able to prove it, which isn't a lucrative target for the scraping companies.
That being said, there are clear examples of companies that sell processed data that's been analyzed by data analysts and isn't the raw scraped data and they have successful and lucrative business models.