Curious what everyone’s approach is for handling cron jobs that depend on external APIs.
Had a data sync job that pulled from a vendor API every hour. Worked great for months until the vendor started rate-limiting us during peak hours. The job would fail, retry on the next cron tick, hit the rate limit again, and basically spin for 3-4 hours until traffic died down.
What I ended up doing was adding exponential backoff inside the job itself — first retry after 5 min, then 15, then 60. But it felt wrong because now the job’s execution time is unpredictable and can overlap with the next scheduled run.
The other approach I considered was just letting it fail and having a separate “catch-up” job that runs less frequently and handles any gaps. Cleaner separation but more moving parts.
For those of you running scheduled jobs that depend on third-party services: do you build the retry logic into the job, handle it at the scheduler level, or just accept that some runs will fail and deal with it downstream?
Also wondering if anyone’s hit issues with jobs that silently succeed but return partial data — like the API responds 200 but only gives you half the records because of an undocumented pagination change. That one took me a while to catch.
systemd timers are definitely nicer than cron for this (and basically all cron jobs). there’s both a restart on failure with backoff if you want it to handle the retry, or a way to schedule it based on the last exit time so the script can manage the backoff like you are doing now
on the last part, page numbers are hard to trust. I assume you don’t have a choice, but that’s why I prefer next page tokens when designing an api instead of numbered pages. then you know the data you’re getting is exactly after the previous data