Rate-limit-aware unattended batch automation CLI
A packaged command-line tool that runs a daily batch unattended, never acts twice on the same item, respects a service's rate limits with its own conservative request budget and backs off when throttled, and verifies afterwards that its work registered.
Problem
A recurring batch of actions against a web service had to run every day without supervision, without ever acting twice on the same item and while staying well inside the service's rate limits.
Goal
A cross-platform command-line tool that runs the daily batch unattended, never acts twice on the same item, and stays well inside the service's documented limits.
My role
Sole developer: packaging, the core engine, reliability hardening and verification.
Solution
A pip-installable command-line tool with commands for running, status and verification, a dry-run mode and platform-appropriate data directories, designed to be a polite client of the service it talks to.
How it works
Every request passes through a sliding-window request budget set below the service's documented short-window limit, so the tool slows itself down before it would reach the limit.
Only when a request is actually throttled does the tool read the server's retry hint, given either in seconds or as a date, and after consecutive throttles it falls back to capped exponential backoff.
Actions are deduplicated across runs in a local state file, so a rerun never repeats an item, and a daily counter resets each day.
A problem item is skipped and recorded rather than aborting the whole run, and a status file reports live progress.
After a run, a verifier checks that the items processed actually registered.
The tool is designed to run unattended under a system timer.
Technical challenges
A conservative self-imposed budget: the tool counts its own requests in a sliding window set below the documented limit and waits before it would exceed it.
Handling the retry hint in both of its forms, and capping backoff after repeated throttles.
Confirming after a run that every processed item registered.
Skip-not-abort error handling that still leaves a clear record.
Migrating loose scripts into a packaged, installable tool.
Capabilities
- Batch actions with dry run
- Cross-run deduplication
- Self-imposed request budget
- Retry-hint handling in both forms
- Capped exponential backoff
- Daily caps
- Post-run verification
- Live progress status
- Unattended scheduling
Technology
Impact
Unattended daily runs designed so that an item is never acted on twice and the request budget is never exceeded. No metrics are claimed.
Skills demonstrated
- HTTP client engineering
- Rate-limit and backoff design
- Duplicate-safe batch design
- Command-line tool packaging
- Unattended scheduling
Demo
PlannedAn in-page simulation against a fictional work-queue service that holds generic work items: the request-budget gauge fills and the tool pauses before the limit, the fake service throttles in three ways (a retry hint in seconds, a retry hint as a date, and repeated throttles that trigger capped backoff), one malformed item is skipped and recorded, the verifier confirms the batch registered, and a rerun redoes nothing. It imitates no real platform and no social action.
Not built yet. A demo here uses only invented names and data.