4 Things No One Tells You About Web Scraping
Web scraping can change your life as a data scientist. Here's why.
If you’ve been following me for some time, you might know that I publish a lot of content about web scraping.
I like the topic; unfortunately, it’s not welcomed by everyone out there. That said, in this new newsletter “Let’s be Frank” I will help you become a better data scientist by giving weekly advice without filters, which I wouldn't be able to share anywhere else.
This time I’d like to share with you 4 things that (almost) no one tells you about web scraping.
It can make you a better data scientist
Can you start a data science project without data?
I think the words “DATA scientist” speak by themselves. Data is essential in any project. Your company can provide you with data, but whenever you want to go beyond your company’s needs, you won’t have data available.
This is why you need to learn web scraping. This skill will help you collect real-world data (forget about those Kaggle datasets that almost everyone has worked with) and build your own dataset in order to start your personal projects and become a better data scientist.
It’s legal (at least in some circumstances)
Some people believe that doing web scraping is always illegal.
That doesn’t surprise me after the Cambridge Analytica scandal, but that’s not always the case.
There’s a firm named HiQ that won a CFAA lawsuit against Microsoft after scraping data from LinkedIn. What’s the difference between the Cambridge Analytica and HiQ case?
Cambridge Analytica scraped data illegally thanks to Facebook’s leaky platform, while HiQ got the public data legally.
I’m not going to delve into details here, but I want you to know that there’s a legal side of web scraping and encourage you to do your own research (as a rule of thumb, always check the Terms of Service (ToS) of a website).
It can help you make money
Yeah, you heard that right. Web scraping can help you make money in different ways.
The most obvious way is to sell the data you scrape. That said, I don’t recommend you to make money this way. Why? The data might be copyrighted and get you into trouble.
But you don’t need to sell data to make money! According to The Economist, the world’s most valuable resource is no longer oil, but data.
There are alternatives to make money with web scraping. You can do arbitrage trading, reselling, sell research, create a website/app, and more. The applications increase based on the type of work you do. Believe it or not, web scraping helps me come up with hundreds of viral article ideas that increase my earnings as an online writer.
You can scrape some websites with 0 coding knowledge
Anybody can scrape a website!
You don’t even need to write code and build your own scraper from scratch. There are tools that have pre-built templates that will help you scrape popular websites such as Twitter, Amazon, and Instagram.
That said, when it comes to scraping any website you want, writing code is the best option.