Cape Town City Centre: Web Scraping Engineer posted by PST.AG

About the Role

We are seeking a Data Extraction Engineer to design and maintain our next-generation data acquisition platform, leveraging declarative extraction specifications to treat web scraping as a specification-driven discipline.

Key Responsibilities

• Design and maintain declarative extraction specifications—using Pydantic models, JSON schemas, or domain-specific languages—that describe exactly which fields to capture, their types, and validation rules.
• Implement pipelines that translate these specifications into executable extraction plans, leveraging both classical (Scrapy, Playwright) and AI-augmented (LLM-based semantic parsing) backends.
• Build reusable specification libraries for recurring data types (product prices, tariff codes, regulatory texts) to accelerate onboarding of new sources.
• Deploy self-healing spiders that automatically detect website layout changes and repair themselves using Model Context Protocol (MCP) servers (e.g., Scrapy MCP Server, Playwright MCP).
• Integrate semantic extraction (Scrapy-LLM, custom LLM pipelines) to eliminate selector brittleness—spiders rely on field descriptions, not fragile XPaths.
• Orchestrate complex, multi-step browsing workflows with agentic frameworks (BMAD/TEA, AutoGPT-like agents) that reason about page state, adapt to anti-bot measures, and correct their own behaviour in real time.
• Move beyond one-off scrapers: build a component-based extraction platform where selectors, login handlers, and pagination logic are shared, versioned, and tested.
• Implement monitoring, alerting, and automatic rollback for failed extraction runs.

Requirements

• Bachelor’s degree in Computer Science
• 3 years of experience in web scraping or data extraction

Qualifications

• Specification-Driven Extraction – Experience defining extraction requirements via schemas (Pydantic, JSON Schema) and executing them through both traditional crawlers and LLM-based semantic parsers.
• Self‑Healing & Semantic Extraction – Hands‑on use of Scrapy‑LLM, Scrapy MCP Server, or similar systems that decouple field definitions from page structure.
• Agentic Workflows – Familiarity with frameworks that give LLMs browser control (Playwright MCP, BMAD/TEA) to handle complex, non‑deterministic crawling tasks.

Salary & Benefits

Salary details will be discussed during the interview process.