Fastest Python Web Scraper - Exploring Sessions, Multiprocessing, Multithreading, and Scrapy
In this video, we will make a fast web scraper. We will begin with BeautifulSoup.
🚀 The first script takes 128 seconds and after optimization, takes as little as 2.5 seconds.
Finally, we will create a scrapy spider without optimization and see what kind of results we get.
We will use BeautifulSoup, Requests, Sessions, Multithreading, Multiprocessing, and Scrapy.
👩💻 Source Code: https://github.com/eupendra/fast_scra...
You can jump to the sections you like:
00:31 Scraper Objective
00:44 Creating Scraper with Requests+BS4
9:20 First Run
10:07 Sessions
13:58 Multiprocessing
17:22 Multithreading
22:36 Scrapy Without Optimization
Related videos
-------
👩💻 Watch the Playlist to Learn the Basics of Scrapy: • Scrapy for Beginners
👨💻Join all courses on my site: https://coderecode.com/membership
----------------------------------------------
What is Web Scraping?
In a nutshell: Web Scraping = Getting Data from Websites with Code
What is Scrapy?
Scrapy is a Python library to makes web scraping very powerful, fast, and efficient.
There are other libraries too like BeautifulSoup, for web scraping. However, when it comes to true power and flexibility, Scrapy is the most powerful.
Why Learn Scrapy?
Most powerful library for scraping
Easy to master
Cross-platform: doesn't matter which OS you are using
Cloud-ready: Can be run on the cloud with a free account
Most Important: You would be able to earn by taking up some of the web scraping gigs as a freelancer
#scrapy #fast #beautifulsoup #multiprocessing #multithreading
~-~~-~~~-~~-~
Please watch: "Making Scrapy Playwright fast and reliable"
• How to make Scrapy Playwright fast an...
~-~~-~~~-~~-~
On this page of the site you can watch the video online Fastest Python Web Scraper - Exploring Sessions, Multiprocessing, Multithreading, and Scrapy with a duration of hours minute second in good quality, which was uploaded by the user Code [RE] Code 12 July 2021, share the link with friends and acquaintances, this video has already been watched 11,846 times on youtube and it was liked by 316 viewers. Enjoy your viewing!
