UK Bin Collection - Selenium Tutorial
I'm excited to share a complete end-to-end video tutorial now available! This walkthrough shows you exactly how to set up and use a DevContainer to build a custom Home Assistant bin collection integration using Selenium.
Repo : https://github.com/robbrad/UKBinCollection...
Peterborough Council Integration : https://github.com/robbrad/UKBinCollection...
🔧 What’s Covered:
In this in-depth tutorial, we walk through the full process of adding a new Selenium-based UK council (Peterborough) to the UK Bin Collection custom Home Assistant component. Starting from a fresh environment, we set up the dev container, clone the repo, debug the scraper live, and handle everything from cookies and dropdowns to edge cases like flaky elements and page timing. We also explore testing, GitHub Actions, and how to contribute your own council integration.
Setting up a development environment with DevContainers (VS Code + Docker)
Structuring and scaffolding a new council scraper integration
Writing a custom scraper using Selenium and BeautifulSoup
Debugging, testing, and improving the integration flow
How to hook everything up into a custom component for Home Assistant
🧠 Learning by Doing (and Failing!)
This tutorial intentionally includes mistakes and troubleshooting moments—because let’s be honest, real-world development isn’t perfect! Watching how issues are identified and resolved is often just as valuable as a clean demo. I kept everything in to help you learn how to debug, adapt, and build confidence when writing your own integrations.
Drop a comment, like, and subscribe if it helps—or if you have questions! 😊
00:00 – Intro & Goals
Overview of the tutorial: setting up from scratch, adding a new Selenium-based council (Peterborough), using the dev container.
00:40 – Setting Up Your Environment
Cloning the repo, Docker Desktop setup, creating folders, launching VS Code.
02:50 – Dev Container Initialization
Rebuilding and reopening in the VS Code dev container, pulling required images.
04:45 – Understanding the Containers
Explanation of all Docker containers involved – dev, Selenium hub, Home Assistant, video recording sidecars.
06:45 – Running & Viewing Tests
Intro to Pytest integration in the repo, using the built-in test runner, viewing Selenium test sessions.
08:55 – Selenium Test Debugging & Videos
Running a Selenium council test, using localhost sessions UI, and reviewing test video recordings.
11:00 – Creating a New Council – Peterborough
Copying from Barnet as a base, starting Peterborough integration.
13:00 – Investigating the Council Website
Testing user flow manually, inspecting web elements, confirming the need for Selenium.
17:00 – Creating the Python Integration File
Setting up the council Python file and registering it in input.json.
19:00 – First Debug Run & Breakpoints
Running the test in debug mode, setting breakpoints, inspecting variables live.
23:00 – Handling Cookie Popups (Selenium)
Identifying and accepting cookie banners, fixing XPath for custom button elements.
27:00 – Debugging Click Issues
Working around scrolling and visibility issues in Selenium – refining scrollIntoView.
31:30 – Postcode & Form Interaction
Typing into input fields, sending keys via Selenium, and handling form buttons.
35:00 – Dropdown Selection Automation
Selecting address from dynamically rendered dropdowns using XPath and waits.
40:00 – Navigating to Results Page
Clicking continue, waiting for elements, verifying the presence of the results content.
45:00 – Extracting HTML with BeautifulSoup
Targeting the specific div containing the bin data, creating a focused soup object.
50:00 – Using ChatGPT to Adjust Parsing Logic
Feeding updated HTML into GPT to refactor and correct the data extraction logic.
55:00 – Troubleshooting Date Format & Data
Validating JSON schema, fixing date parsing issues, checking HTML structure.
01:00:00 – Final Debug & Validation
Adding delays, adjusting wait conditions, confirming the scraper is reliable.
01:05:00 – Formatting & Committing Your Code
Using Black formatter, staging changes, writing conventional commit messages.
01:08:00 – Testing & Pull Request Process
Running tests locally and in CI, creating pull requests, and validating passing councils.
01:14:00 – Fixing Other Failing Councils
Applying consistent try/except patterns and driver.quit() to prevent test hangs.
01:22:00 – Retesting & Cleanup
Re-running integration tests, inspecting logs, validating successful PR merges.
01:26:00 – Home Assistant Integration Testing
Launching HA in the dev container, testing the custom component live.
01:30:00 – Wrap Up & Contribution Guidance
Encouragement to contribute, tracking broken councils, and improving test reliability.
Nesta página do site você pode assistir ao vídeo on-line UK Bin Collection - Selenium Tutorial duração hora minuto segundo em boa qualidade , que foi baixado pelo usuário Robert Bradley 13 Abril 2025, compartilhe o link com seus amigos e conhecidos, no youtube este vídeo já foi visto 143 vezes e gostou 4 espectadores. Boa visualização!
