Web Scraping Yellow Pages with Python | Subscriber request using Scrapy and Xpath

Опубликовано: 01 Январь 1970
на канале: jobstr

724

Scraping Yellow Pages.
A demonstration of finding selectors in Chrome, testing them in the browser developer tools, and then making Scrapy iterate through using accurate and not so accurate selectors.
The main focus of this video is on refining the xpath in the browser developer tools before pasting into your Scrapy spider code.

The full code is here on my GitHub page : https://github.com/RGGH/Scrapy9

: Chapter timings : -

0:00 Introduction - what are we scraping? Why?
0:32 The spider code so far
0:55 The parse method
1:43 Ctrl + F in Chrome
2:57 xpaths : using predicates
4:15 Using "*" wildcard versus "p" or "div" or "span" element
8:22 Finding website addresses
9:55 xpath using 2 predicates
11:28 Updating the Scrapy spider code...adding the selector for 'website'
13:40 Test
14:15 Troubleshooting
14:46 Quick change of selector for website! - No need for 2 predicates!
15:07 Running the spider
15:25 The "."
16:55 Success
17:55 Summary
20:25 Closing comments

If you want more info relating to xpaths, here are some examples plus a great site to check out:

⦿ response.xpath("//div[@class='article']//a/img/@src")
⦿ response.xpath('//a[contains(text(), "Next »")]/@href')
⦿ response.xpath('//div[@class='pibox']/descendant::*/text()')

https://devhints.io/xpath

contains is a really useful one, and it can future proof your spider, as even if the css changes, "Next" should still be spelled the same!

All the best,
Dr Pi.

На этой странице сайта вы можете посмотреть видео онлайн Web Scraping Yellow Pages with Python | Subscriber request using Scrapy and Xpath длительностью часов минут секунд в хорошем качестве, которое загрузил пользователь jobstr 01 Январь 1970, поделитесь ссылкой с друзьями и знакомыми, на youtube это видео уже посмотрели 724 раз и оно понравилось 12 зрителям. Приятного просмотра!

13,624

228