Now that we've learned how to do web scraping with requests and Beautiful Soup,
it's time to step back for a moment and have a think about what we're allowed to
do and what might not be a good idea when we're scraping data from other
websites. Because after all, we don't own that data, right?
When you think about services like Google or Bing or any other search engine,
essentially what they're doing is they're constantly scraping data from all of the
websites that are listed on the internet.
And that's how they manage to get the information about what's on each page
and for it to show up for users who use their search service.
Now we have to step back for a moment and think about what is the law on web
scraping? What is legal and what is illegal?
Even as we were looking at Hacker News
just now I noticed that one of the articles in fact talks about the Genius law
suit with Google. And in terms of recent history,
there's two really famous cases, which is Genius suing Google
because they're saying that Google is scraping the song lyrics from their
website and they're actually displaying it without taking people to Genius.
So for example, if we're looking at the lyrics for Code m Monkey,
you can see that Google automatically shows the lyrics straight inside of
Google.
That means that a user can potentially just get all the information they need,
say all of the song
lyrics to this song, without ever needing to visit the website where this lyric
might come from.
And that lyric might've been compiled by somebody on Genius.
Genius has a lyric annotation website. And of course,
as with all websites,
they rely on users actually visiting their website to make money or to show ads.
And if Google simply just shows it in their search results,
then this can be a problem for websites like Genius.
So they sued them over this and actually ended up losing the lawsuit.
Another really famous example of a lawsuit over scraping is hiQ versus
LinkedIn. So hiQ was scraping data from LinkedIn to use commercially.
So LinkedIn sued them and ended up losing in the lawsuit.
Based on these lawsuits,
we have a little bit of a better idea of what is legal when it comes to web
scraping and what is not legal.
The law actually seems to favor web scraping in the sense that you're allowed to
scrape a website data
as long as you think about a couple of things.
A lot of people have been writing about web scraping being legal based on the
LinkedIn versus hiQ case.
But the important thing to remember is that this is not a blanket sort of,
you can do whatever you want, scraping any website's data.
It only means that data that is publicly available and not copyrighted is
probably legal for companies to scrape. Now,
if you are using this data privately
like we are creating some sort of service for ourselves, then it doesn't really
matter. You're just a user.
The difficulty comes when you're trying to commercialize that data,
when you set up a business and your business kind of involves somebody else's
data. That is a bit of a gray area. Now,
the things that we definitely know are
that you can't commercialize copyrighted content.
So if you scrape data from YouTube and you scraped the video data,
you can't just use that video on your own website. That is still not allowed
because that video is copyrighted and it's created by a YouTube user and the
copyright belongs to that user, not to you. So this is still illegal.
This might also apply to other things like a Medium blog post that somebody else
wrote or a piece of music that's being hosted on Spotify.
So copyrighted content you can't commercialize.
The second thing is that you can't scrape data that's behind authentication.
So if you have to log into Facebook in order to scrape the data,
then that's pretty much illegal.
And the reason for this is when you sign up as a user to any of these services
like Facebook or Twitter or Instagram,
there's a policy in there that you are agreeing to when you sign up that says
I agree to not use this data that I obtained on this website commercially.
But the data that is not behind authentication,
so any website that you can access as it is,
they can't bind you to a policy because you haven't agreed to anything.
So if the website has data that just out there in the open that you can access
without logging in and the content is not something that can be copyrighted,
then it is fair game legally. Now,
just because it's legal doesn't mean that you can actually do it.
A lot of websites will use captcha or recaptcha in order to prevent bots like
our Python code to get data from their websites. Every single time,
you're agreeing to one of these captchas, it's testing
whether to see if your actually a real human
or if you just a bit of code that is trying to access their data. Captcha was the
old version where you had the type in some squiggle letters and recaptcha is the
new version where you just have to tick a checkbox.
And it's actually really interesting how it works.
It looks at things like how your mouse approaches the checkbox,
how you maybe quiver a little bit before you actually check it
and other things like your cookies and the store data that they have on you.
Essentially, this service is used by websites to prevent people
to scrape their data using a bot. The other thing to remember is that,
you know, if you get sued by somebody like LinkedIn
because you're using their data and you're building a business on it
like hiQ is, then you can
at any moment be hit with a really expensive lawsuit
and you are going to have to pay a lot of money to lawyer up in order to contest
this and actually to fight them in court.
Unless you have the money to lawyer up and fight a company like LinkedIn,
it's really important to know what are the implications of web scraping,
especially when you're selling that data as a part of your business.
But in addition to the sort of legal side of things,
the other part that you should really think about is the ethics of web scraping.
This is basically putting aside what is legal and what is illegal,
but more thinking about what is right
and what's wrong because let's say that you've built a website and you've got
some sort of bot that's constantly scraping it for data, data that you know,
has been generated by your own users
that's really precious and that you might even charge for it, then,
is it really right for somebody to do that?
So I often follow the rule where if I don't want something to happen to me,
I try to not do that to others. In terms of the ethics, a couple of things
I would recommend abiding by is
if you come across a website and they have a public API
which we've already learned about and we know how to use, then always
always go for the API. If it requires an application, then apply for it.
Don't just go ahead and try to take their data when there's already a route for
you to use and access their data.
The second thing is to respect the web owner, because you know,
you don't want somebody to access your website a million times a second,
potentially making your website go down
or it could count as a DDoS
attack where it affects other users using the website.
When you are on a website,
they actually provide a way for you to tell what it is that you can scrape and
what it is
you can't. At the very end of the URLs after the.com or.co.uk,
if you put a forward slash and put robots.txt, you can see
this is the advice that they give to any bots that are potentially scraping
their website.
User agent is the person who is scraping, the person or the bot that's scraping,
and it tells you what are the things that it disallows.
So it doesn't want you to access the /vote?, /reply?,
/submitted?, /threads?.
So basically any of these end points are ones that they don't really want you to
use. For example,
here I've access the /reply?
which is a way to log in and reply to a particular comment.
Now that really shouldn't be a bot kind of action
because then it means the data that's generated
or the replies on here will be automated, right?
You actually want humans to comment and reply on the articles rather than some
sort of robot.
So these are the paths that they don't want you to access as a bot.
And finally, it even tells you a crawl-delay.
So this is the number of seconds that you should leave between each time you hit
up the website.
If we're writing Python code and we're using Beautiful Soup and response to
scrape data from YCombinator,
we could potentially get that code to run every fraction of a second right?
I could just write a for loop and just get this to keep scraping again and again
and again.
But that means that you're adding a lot of extra traffic and a lot of extra
demand on their servers which could potentially mean that real users,
real humans who want to access their website might not be able to do it at a fast
speed. So this is the reason why
when a lot of people accessing the same website,
say when a new ticket has been released for Glastonbury or some sort of big
concert, that the website can go down.
Its because a lot of servers can't cope with so much demand.
And when that demand is coming from a for loop,
then you can imagine that you're just adding a lot of extra work onto the web
server. So always respect their crawl-delay
if you see one in the robots.txt, and even if you don't see one,
just try to limit your rate so that you don't max out their server.
I recommend not scraping more than once a minute.
The YCombinator's of robots.txt is actually quite permissive.
It allows you to do pretty much anything you want,
but that's not true for all websites.
If you look at the robots.txt for LinkedIn,
you can see that they really don't want anyone to scrape it.
There is a bit of legal jargon,
there's a lot more disallows that you can see, right?
This is probably not a website where I would scrape their data and try to build a
company around.
So remember that this is a piece of text that the website owners have written
for you to look at to see what you can do and you can't do with their website.
So before you scrape a website,
always go to the root of their URL and check out their robots.txt and follow
the ethical codes of conduct when you're trying to commercialize a project.
So this is just the quick tip on the law and ethics of web scraping
just so that you don't get into trouble in the future.