When, Why, and How To Use Web Scraping In A Nutshell

Summary The internet is a rich source of information, but a majority of it isn’t accessible programmatically through APIs or databases. To address that shortcoming there are a variety of web scraping frameworks that aid in extracting structured data from web pages. In this episode Attila Tóth shares the challenges of web data extraction, the ways that you can use it, and how Scrapy and ScrapingHub can help you with your projects. Announcements Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great. When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $60 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show! This portion of Python Podcast is brought to you by Datadog. Do you have an app in production that is slower than you like? Is its performance all over the place (sometimes fast, sometimes slow)? Do you know why? With Datadog, you will. You can troubleshoot your app’s performance with Datadog’s end-to-end tracing and in one click correlate those Python traces with related logs and metrics. Use their detailed flame graphs to identify bottlenecks and latency in that app of yours. Start tracking the performance of your apps with a free trial at datadog.com/pythonpodcast. If you sign up for a trial and install the agent, Datadog will send you a free t-shirt. You listen to this show to learn and stay up to date with the ways that Python is being used, including the latest in machine learning and data analysis. For more opportunities to stay up to date, gain new skills, and learn from your peers there are a growing number of virtual events that you can attend from the comfort and safety of your home. Go to pythonpodcast.com/conferences to check out the upcoming events being offered by our partners and get registered today! Your host as usual is Tobias Macey and today I’m interviewing Attila Tóth about doing data extraction with web scraping. Interview Introductions How did you get introduced to Python? Can you start by explaining what web scraping is and when you might want to use it? How did you first get started with web scraping? There are a number of options for web scraping tools in Python, as well as other languages. What are the characteristics of the Scrapy project and community that have made it stand out and retain such widespread popularity? One of the perpetual questions with web scraping is that of copyright and content ownership. What should we all be aware of when scraping a given website? What are some of the most challenging aspects of crawling and scraping the web? What are some of the features of Scrapy that aid in those challenges? Once you have retrieved the content from a site, what are some of the considerations for storing and processing the data that we should be thinking about? How can we guard against a scraper breaking due to changes in the layout of a site, or simple updates that weren’t accounted for in the initial implementation? What are some of the most complicated aspects of scaling web scrapers? For someone who is interested in using Scrapy, what are some of the common pitfalls that they should be aware of? What are some of the most interesting, innovative, or unexpected projects that are built with Scrapy and ScrapingHub? What are the most interesting, unexpected, or challenging lessons that you have learned while working with web scrapers and ScrapingHub? What resources would you recommend to anyone who is looking to learn more about web scraping? Keep In Touch LinkedIn Picks Tobias Gov’t Mule Attila Awesome Web Scraping Awesome Scrapy Closing Announcements Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story. To help other people find the show please leave a review on iTunes and tell your friends and co-workers Join the community in the new Zulip chat workspace at pythonpodcast.com/chat Links Web Scraping ScrapingHub Java Android Scrapy JSoup HTMLUnit Selenium Pandas robots.txt Puppeteer Splash The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA