3) Provide APIs so people don't need to scrape your site.
Trying to essentially DRM your web site so that it's human-readable and not machine-readable is not only inherently impossible to do effectively (like any DRM), but is also solving the wrong problem.
1 and 3 are the most effective. Option 2 is useful until there is any real demand for your data, and then you're back to trying to prevent automated scraping by your paying customers.
If you are charging someone for access, then presumably you are requiring them to log in to access the data. In that case, it should be pretty straightforward to throttle or completely block them if they are doing something (e.g., scraping) that violates your terms of service. Then you charge them again to use your API.
#3 is only an option if you are trying to prevent scraping just to reduce your bandwidth caused by rapid, repeated page requests. If you are trying to prevent someone one from just coming in and scooping up all your data, then providing an API is worse than just allowing the scraper to scrape.
There is no bullet proof way to stop it. So you make it as painful for the scraper as possible. I like the randomized classes/ids and the extraneous random invisible table cells and divs.
My point is that trying to provide data that can be used in one way but not another way is a pretty ridiculous concept. If you have a lot of data, you should have an API.
The idea that you can have a ton of data accessible on the web but somehow only let humans access it, and not computers, is quite simply untenable. My point with #3 (and to a lesser extent #1) is that it makes little sense to solve this problem at all.
That is an opinion. One that I don't agree with. Companies spend a lot of time and money collecting/refining/cleansing/etc the data so they can present it to their users. It is not ridiculous for them to want to keep some script from capturing it all in one swoop. A lot of sites have clauses in the TOS that prohibit mechanized data harvesting. But there are a lot people that place zero value on the work others do to create something so they wait for someone else to do the work and then just come in and share it. If you sit at your school desk and wait for the girl next to you to finish her exam and then you lean over and share her answers... is that not cheating? Sure she doesn't own the data she just wrote down. And most likely it was all in the course book anyway. And of course you aren't stealing her answers since you don't effect her ability to turn in her exam. But there is the total lack of respect for the time and effort that other people put into doing the work. This everything is free and share ALL the things attitude is not sustainable.
Um, wow. I was actually just discussing technical feasibility. Nowhere did I state, or imply, or do I believe, that we shouldn't respect the time people put into work.
I'd explain what I meant further, but by jumping into a full-blown rant on me on piracy after I expressed a technically minded opinion about the futility of DRM on a web page, I think you've clearly demonstrated that you're not worth engaging further.
sorry. I misunderstood your comment. my bad. a bit punchy today. we cool?
Edit: Yes. Technically it does seem futile. It does become a game of cat and mouse (not unlike the spam/anti-spam battle over the years). Slow them down. Make them work for it. I used to work with a crazy old engineer that always told me: "I don't lock my car. If a thief wants my radio... he'll get it regardless. I'd rather deal with just a missing radio instead of a missing radio AND a broken window." Did I mention he was crazy? lol But I thought there is probably a large subset of people that would steal a radio out of an unlocked car but would not break a window to get in. Who knows.
I can understand the "slow it down" idea, but it just doesn't seem worthwhile. If your data is that valuable, charge for access. Then it doesn't matter if they download 100 pages a second, since they're then paying for 100 pages a second.
But if you put data on the internet for people to view in their browsers, you should not be surprised when people consume the data and use it for their own purposes. Site scrapers fall in this category.
As mikeash said, trying to obfuscate your site (because it can be consumed by more than just a person's eyes) is solving the wrong problem. You can't have (own) your cake (data) and eat it (render it publicly) too.
Your analogy is flawed. When you sit for an exam you agree not to cheat (whether explicitly, as a student who has signed acceptance of an offer to study subject to the institution's regulations, or owing to the law of the land).
I don't think anybody is arguing everything should be free and uncopyrightable. People are simply observing that preventing information flow is difficult, and - whether you like it or not - there are plenty of people out there who don't share your sense of ethics. Furthermore, much data is uncopyrightable (facts) and it's only the presentation and organisation of the data that is copyrightable. If your business can be destroyed because somebody copied your data I'm sorry to say but you didn't actually have a business to begin with.
That is true. I'm sure there are sites that don't provide an API for that reason. Ad revenue on the site might be what allows them to provide the data for free. But that wasn't what I was talking about.
1) Don't have any data worth scraping.
2) Charge for access.
3) Provide APIs so people don't need to scrape your site.
Trying to essentially DRM your web site so that it's human-readable and not machine-readable is not only inherently impossible to do effectively (like any DRM), but is also solving the wrong problem.