AI-Assisted Data Scraper
A weekly digest that collects, filters and ranks itself in the cloud
Personal IP, built independently.
The problem
Checking the same sources every week for the handful of items worth reading is the kind of job that quietly eats an hour and never gets done properly. The sources are mostly noise, the good items are easy to miss, and a laptop left running a script is not a system. This build was for job boards, but the shape is the same for tender alerts, competitor price watching, new-listing lead lists and supplier stock checks.
What I built
A single cloud function that a scheduler wakes at eight on Sunday morning. It fetches from four sources, normalises them into one shape so an outage at one never stops the run, and checks every listing against a rules profile written in plain JSON: hard excludes first, then a required match against target roles, then a location and time-zone check. Every decision comes back with the rule that made it, so a filter that’s too strict can be seen and fixed rather than silently dropping the one that mattered. Anything it’s unsure about is kept and flagged, not thrown away. A small memory in cloud storage stops the same listing being sent twice, and the result is emailed as a short ranked digest.
Nothing in it is open to the internet. The function only answers signed calls from the scheduler, and its permissions are granted on the one secret and the one storage bucket it uses.
The stack
Python on Google Cloud Functions, Cloud Scheduler for the Sunday trigger, Cloud Storage for the seen-listing memory and results, and Secret Manager for the mail credentials. The local test run and the live cloud run share the same code, switched by one environment variable.
What changed
A ranked digest arrives every Sunday morning, with no server to maintain and nothing sent twice. Swap the sources and the rules file and the same fetch, filter and digest loop watches whatever a business needs watched.
Proof
