A job seeker, a recruiter or a niche job newsletter usually ends up running the same search three times: once on LinkedIn, once on the employers' own career sites, and once on the remote boards. The results overlap. The same role shows up on LinkedIn and on the company's Greenhouse or Lever page, often with the pay range on only one of them, and counting or posting both copies inflates everything downstream.
This post shows a daily search that covers all of them in one run and returns one row per job.
Disclosure: I'm Aurelix, and I built the Multi Job Board Scraper. It's a paid tool (prices below).
What one run covers
- LinkedIn's public job search, read logged-out.
-
About 50 bundled company career sites on Greenhouse, Lever, Ashby and SmartRecruiters (25 searched per run by default, home-country employers first), plus any boards you add, such as
greenhouse:stripe. - Remote OK and We Work Remotely, kept when the job is open to your country or worldwide.
Every row has the same fields whatever its source: title, company, location, city, ISO country, workplace_type, salary_min / salary_max / salary_currency where published, posted_date, job_url, apply_url and the description. A job found on two sources is one row: sources lists both, also_listed_on links the other copy, and the gaps (pay, apply link, description) are filled from it. That row is charged once.
A daily feed of new jobs, deduplicated
pip install apify-client
import csv
import os
from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("aurelix/multi-job-board-scraper").call(
run_input={
"keywords": ["data engineer", "analytics engineer"],
"country": "CA",
"location": "Toronto, ON",
"datePosted": "past24h",
"maxResultsPerSource": 50,
"incremental": True,
"stateKey": "toronto-data-daily",
}
)
jobs = list(client.dataset(run["defaultDatasetId"]).iterate_items())
fields = ["title", "company", "city", "workplace_type", "salary_min", "salary_max",
"salary_currency", "posted_date", "sources", "apply_url", "job_url"]
with open("new_jobs.csv", "w", newline="", encoding="utf-8") as f:
w = csv.DictWriter(f, fieldnames=fields, extrasaction="ignore")
w.writeheader()
for j in jobs:
w.writerow({**j, "sources": "+".join(j.get("sources") or []),
"apply_url": j.get("apply_url") or j.get("job_url")})
both = [j for j in jobs if len(j.get("sources") or []) > 1]
paid = [j for j in jobs if j.get("salary_min")]
print(f"{len(jobs)} new jobs, {len(both)} found on more than one source, {len(paid)} with a pay range")
for j in both[:5]:
print(f"- {j['title']} at {j['company']}: {', '.join(x['job_url'] for x in j['also_listed_on'])}")
With incremental on, the first run returns everything that matches; each later run with the same stateKey returns only jobs no earlier run returned, so a daily schedule pays only for new jobs. To run it without your own script, save the input as a task in the Apify Console and schedule it daily; a Slack webhook URL in slackWebhookUrl sends a short digest of the new jobs and nothing when there are none.
A few input choices that matter:
-
sources:["linkedin", "careersites", "remoteboards"]by default. Set it to["linkedin", "careersites"]to compare pay and posting date on LinkedIn and the employer's own site for the same job. -
remoteOnly: for a remote-jobs newsletter. Pick the country your readers live in. -
starterList:"canada"uses the bundled Canadian employers;"none"searches only the boards you list incompanies. -
includeDescription: false: faster runs when you only count or list jobs.
Why this tool
- $1.50 per 1,000 jobs on the free plan and $0.90 on paid plans, with no platform usage on top, cheaper than the leading tools.
- LinkedIn, employers' own career sites (Greenhouse, Lever, Ashby and SmartRecruiters: about 50 bundled, plus your own) and Remote OK and We Work Remotely in one run.
- The same job found on several sources is merged into one row, with the pay and apply link filled in, and charged once.
- An only-new-jobs mode for scheduled runs.
- What it doesn't cover: Indeed, Glassdoor and ZipRecruiter.
Limits
- No Indeed, Glassdoor or ZipRecruiter. Sources are LinkedIn, the career-site platforms above and two remote boards.
-
Career sites are a bundled list of about 50 employers, 25 per run by default. Add the boards of the companies you care about in
companies. - LinkedIn caps a search at about 1,000 results. For more, run one search per city.
- Pay is only what the source publishes. Many LinkedIn rows have none unless the employer's own site lists it.
- 30 countries are supported; LinkedIn rows that name another country are dropped.
Cost
At the time of writing the Store page lists $1.50 per 1,000 jobs on Apify's free plan and $0.90 per 1,000 on paid plans, plus $0.001 per run. One daily search returning about 25 new jobs costs about $1.16 a month on the free plan. Apify's free plan includes $5 of credit a month, about 3,300 jobs.
If you only need one source, see How to pick a LinkedIn or career-site job scraper, which covers the LinkedIn Jobs Scraper and the Greenhouse Jobs Scraper.
Top comments (0)