Automating market intelligence in recruitment patterns

Executive Summary

Idwala, a consultancy specialising in market research, partnered with gecco to automate its market intelligence gathering. Previously, Idwala spent 24 hours each week manually tracking job postings across eight major companies to identify industry trends. gecco implemented a fully automated workflow capable of processing around 200 new roles weekly, categorising them by department and strategic importance. This solution not only ensures complete and consistent data capture but also delivers annual savings of £48,000 in analyst time. Within the first year, Idwala achieved a remarkable return on investment of 788%.

Objective

To automate the collection and categorisation of job posting data from leading companies, enabling Idwala to track industry transitions towards reduced-risk products and identify emerging strategic priorities.

Scope

The project included eight major players and the automation encompassed web data extraction, AI-driven categorisation, and weekly reporting.

Audience

This case study is designed for consultancies and research organisations exploring how to automate market intelligence processes, particularly those reliant on publicly available data sources.

Challenges and Drivers for Change

An Idwala analyst was dedicating one full working day each week to manually reviewing careers portals across eight company websites. This involved examining approximately 200 job postings weekly, extracting details into spreadsheets, and applying categorisation rules.

The manual approach introduced several issues:

• Human error – occasional missed postings or inconsistent categorisation.
• Resource drain – 20% of an analyst’s working week lost to administrative tasks, limiting capacity for deeper strategic analysis.
• Inconsistent formats – varying job board structures made data extraction slow and prone to mistakes.
• Workflow disruption – the weekly reporting cycle created peaks in workload that affected other client-facing projects.

As the Idwala team noted: “We knew the data was valuable for understanding industry transformation, but the collection process was unsustainable.” A scalable solution was required to ensure accuracy, efficiency, and continuity.

Strategic Goals and Vision

The project’s vision was to remove manual processes whilst improving data accuracy and consistency. The strategic goals were clear:

• Reduce time spent on data collection by at least 90%.
• Guarantee 100% capture of all new job postings.
• Apply consistent categorisation logic across every position.

Idwala wanted a system that would run automatically each week, delivering structured, ready-to-analyse data. The consultancy sought to shift analyst effort from spreadsheet management to higher-value strategic insights. Reliability, scalability, and adaptability to website changes were non-negotiable requirements.

Solution and Approach

gecco implemented a comprehensive automation workflow using a secure data automation platform, scheduled to run every Monday. The system:

1. Extracted job postings from each company’s careers portal via web scraping APIs.
2. Identified new roles by comparing URLs with previously processed data.
3. Extracted details including title, location, department, and full description.

OpenAI language models were deployed to perform intelligent categorisation:

• In depth classification – distinguishing roles linked to department and region.
• Risk assessment – evaluating strategic importance based on role responsibilities and seniority.

This dual-layer categorisation delivered nuanced intelligence into industry hiring priorities.

Implementation took two weeks, covering development, testing across all job boards, and refining AI prompts to ensure categorisation accuracy. gecco worked directly with Idwala’s analysts to translate existing categorisation methods into AI-ready logic. The final workflow integrated seamlessly with Idwala’s digital infrastructure, automatically updating data and issuing weekly email summaries.

Results and Outcomes

The solution now processes approximately 200 new job postings per week with a 100% capture rate. Analyst time investment dropped from 24 hours to zero, representing annual savings of £48,000. This freed Idwala’s team to focus entirely on strategic interpretation and client delivery.

Additional benefits include:

• Improved accuracy – consistent categorisation eliminates human subjectivity.
• High reliability – the system maintained 99.8% uptime over six months, with no missed reporting cycles.
• Faster insights – email summaries provide instant visibility of hiring patterns by company and category.

Get in touch

Ready to transform your business? gecco specialises in automating complex data collection and analysis workflows.

Book a demo