feat: add rate limiting, retry with backoff, and max count cap

- Add configurable request delay between paginated API calls (default 1.5s)
- Add retry with exponential backoff on HTTP 429 and Twitter error code 88
- Add hard max count cap (default 200, absolute ceiling 500)
- Add rateLimit config section with requestDelay, maxRetries, retryBaseDelay, maxCount
- Add normalization tests for rateLimit config
This commit is contained in:
jackwener
2026-03-07 19:02:49 +08:00
parent 0f26e20abb
commit 55c48b077b
6 changed files with 125 additions and 31 deletions

View File

@@ -13,3 +13,9 @@ filter:
replies: 2.0
bookmarks: 5.0
views_log: 0.5
rateLimit:
requestDelay: 1.5 # seconds between paginated requests
maxRetries: 3 # retry count on 429 / rate-limit errors
retryBaseDelay: 5.0 # base delay for exponential backoff (seconds)
maxCount: 200 # hard cap for single fetch