Web Scraping Difficult to Scrape Sites | Alternative approach for Captcha + Python Code

Published: 01 January 1970
on channel: Python 360

9,170

114

Scraping data from a Real Estate site using 3rd party software to try and get property listings. Starting with a review of the problem I show an example of so software designed to 'auto detect' fields and whilst it may be great with less restrictive sites, I demonstrate some issues with trying to scrape the details pages.

You also see how the auto detect struggles with the next page navigation, and how I make a quick and dirty workaround using Python code to format a list of URLs.

I discuss the philosophy of how you might approach web scraping, making judgements about whether you have the time and money/resources to scrape the site, or if you should avoid it and recommend a large specialist such as scrapinghub or equivalent.

Do you pick the 'low hanging fruit' or do you subscribe to a proxy, or subscribe to a cloud based scraping platform?

By the end of the video you will have learned some of the limitations of paid software, and some of the advantages. The site I demonstrate with was not 'scrapeable' just using requests, or Selenium, or Scrapy. Even the scraping platform has the same issues that I exposed in an earlier video where I rotated the ip address every 5 requests.

Note : This is from late 2020 (December) so earlier methods of scraping the site don't/wont apply.

~ Chapters ~
0:00 Intro - Talking about response 200 / Captcha
6:13 Looking at a scraping service
16:36 Adding fields
21:00 Ich Bin Kein Roboter
24:42 An overview of web scraping
32:44 Running a task - getting some useful data
37:52 I write some Python code to generate valid URLs
47:09 Getting good data from 2583 properties
1:00:00 Exporting the good data
1:11:49 Testing URLs with Python and Requests
1:18:31 Adding the correct headers

If you are looking for alternatives to bs4, Scrapy or Selenium, also see Code Monkey King's demo using JQuery :    • Real World WEB SCRAPING project that ...

-- Useful Links --

Code to generate URLs with page numbers for scraper
========================================
🌏 https://github.com/RGGH/BeautifulSoup...

Visit redandgreen blog for more Tutorials
=========================================
🌏 http://redandgreen.co.uk/about/blog/

Subscribe to the YouTube Channel
=================================
🌏    / drpicode

Follow on Twitter - to get notified of new videos
=================================================
🌏   / rngweb

Buy Dr Pi a coffee (or Tea)
☕ https://www.buymeacoffee.com/DrPi

Thumbs up yeah? (cos Algos..)

#webscraping #tutorial #python

On this page of the site you can watch the video online Web Scraping Difficult to Scrape Sites | Alternative approach for Captcha + Python Code with a duration of hours minute second in good quality, which was uploaded by the user Python 360 01 January 1970, share the link with friends and acquaintances, this video has already been watched 9,170 times on youtube and it was liked by 114 viewers. Enjoy your viewing!

2,280