A successful HTTP response is only one part of a useful eBay data pipeline. Your application also needs to know when the underlying listing evidence was collected, whether another worker already requested it, and what to do when a refresh fails.
Those decisions affect the product people see. Repeated cached observations can make a chart look busier than the market really is. A burst of retries can delay everyone in a workspace. An unknown shipping amount converted to zero can change which listing looks cheapest.
This guide builds an operating model for SoldFetch search and item requests: a shared queue, a durable request ledger, an observation store, and explicit retry decisions. The policy example runs offline and makes no API requests.
Keep fetch attempts separate from listing observations
Store one record for each request attempt and another for each distinct observation. An attempt describes your infrastructure: operation, normalized parameters, start time, status, error code, latency, and request ID. An observation describes listing evidence: marketplace, item ID, displayed amount, currency, condition, warnings, and source retrieval time.
Keep X-SoldFetch-Cache, X-SoldFetch-Data-Fetched-At, and X-SoldFetch-Usage-Units with the attempt. Search responses also include fetchedAt, and listing rows include scrapedAt. Record your own ingestion timestamp separately. A response received at noon may contain evidence collected earlier.
An observation key can combine marketplace, item ID, retrieval time, and a digest of the relevant raw fields. Keep the unmodified response so you can explain later normalization changes. A failed refresh creates a failed attempt; it should not create a zero-price observation or erase a valid earlier record.
Schedule around the source data's cache age
SoldFetch uses different cache durations for different results:
| Result | Current cache duration |
|---|---|
| Sold search and category results | 6 hours |
| Active search results and active-item details | 15 minutes |
| Ended-item details | 24 hours |
| Empty search or category results | 5 minutes |
Identical normalized queries can share cached results across workspaces. A cache hit keeps the original retrieval timestamp and does not extend expiry. An ordinary successful cache hit still consumes one request unit; a new response is not proof of a new source observation.
Decide your application's acceptable observation age before setting a schedule. A sold-comps report refreshed every minute can repeatedly pay for the same six-hour observation. An inventory screen may tolerate a different age for active listings. Show the evidence timestamp beside the result so users can judge freshness themselves.
Cache expiry also does not promise a source update at that moment. A later request may fail, find unchanged evidence, or encounter unavailable data. Build the display around the last successful observation and its age.
Share one request budget across workers
The public plan limits apply across all API keys in a workspace:
| Plan | Data requests per minute | Concurrent data requests |
|---|---|---|
| Basic | 10 | 1 |
| Starter, Growth, Scale | 60 | 5 |
Use a workspace queue with a shared rate budget and a concurrency semaphore. Multiple keys or several queue consumers do not multiply these allowances. Leave headroom for interactive requests if scheduled jobs use the same workspace.
Dispatch only as quickly as both limits permit. On Basic, spacing starts alone is insufficient if one request is still running when the next starts. Hold the concurrency slot until the attempt settles, then release it even when an error occurs.
SoldFetch also shares provider capacity across requests. Your queue can remain within its workspace allowance and still receive a temporary provider-capacity response. Keep that condition separate from a local programming error or exhausted monthly quota.
Classify errors before deciding to retry
Read the HTTP status and structured code together. A 429 with rate_limited means the workspace rate or concurrency allowance was hit. A 429 with quota_exceeded means the allowance is exhausted; a short sleep does not replenish it. Shared upstream pressure can return 503 with upstream_rate_limited.
Honor Retry-After when returned; SoldFetch expresses it in seconds. Add jitter after the minimum wait, use a finite attempt budget, and stop when the overall job deadline expires. Authentication failures, unsupported filters, and invalid parameters need a correction before another request.
This pure classifier makes those decisions visible. Its action strings are application policy, not SoldFetch response fields:
def retry_action(status, code, error=""):
if status is None:
return "retry_same_key_after_backoff"
if 200 <= status < 300:
return "store_success"
if status == 429 and code == "quota_exceeded":
return "pause_for_allowance"
if status == 409 and code == "idempotency_conflict":
if error == "Previous request failed; use a new Idempotency-Key":
return "retry_new_key_after_backoff"
if error == "Equivalent request is still running":
return "retry_same_key_after_backoff"
return "stop_and_review_key"
if status == 429 and code == "rate_limited":
return "retry_after_confirmed_failure"
if status in (502, 503, 504):
return "review_transient_failure_before_retry"
return "stop_and_review_request"
Here status=None means the client has no conclusive response, such as a connection timeout. It does not mean the server failed. The caller still needs a maximum attempt count, deadline, delay calculation, and error log. A persistent 503 caused by disabled data access will not be repaired by an automatic retry loop.
Persist idempotency keys across ambiguous outcomes
Create a unique Idempotency-Key for each logical REST data request and save it before sending. Retry the same normalized operation and parameters with that key when the outcome is ambiguous. Changing the key immediately after a network timeout can turn one intended request into two charged successes.
A completed successful replay within the 24-hour retention window returns the saved result with zero additional usage units. It still passes workspace rate checks. Reusing the key with different parameters returns 409. A fresh observation requires a fresh logical request and key.
There is one important failure case: a request recorded as failed cannot be replayed into a new attempt with that key. The API returns Previous request failed; use a new Idempotency-Key. After confirming failure and deciding the condition is retryable, use a new key. An in-progress conflict means wait and keep the existing key; a parameter conflict means correct the client.
This distinction belongs in durable job state. Save the key, attempt outcome, and parameters together so a restarted worker can resume safely. Do not apply the REST replay rule to background-job creation without checking that endpoint's own contract.
Use status checks as observations
Check capabilities before a substantial run. It describes the current deployment's data access, enabled marketplaces, supported operations, and constraints. A disabled data gate is different from an empty search result.
Use the status board to inspect recent marketplace observations and incident context when errors rise. The board reflects monitoring observations; it is not a service-level guarantee or a promise that a particular query will succeed. Keep its observation time in view, especially if a marketplace reading is stale or unknown.
A practical pause rule combines your own recent failure rate with this context. Stop dispatching a large sweep after several consecutive provider failures, retain unfinished jobs, and retry later within a defined window. Continue serving previous evidence with a visible age and refresh state.
Make the pipeline explainable before scaling it
For a small first run, inspect a cache hit, an ambiguous timeout, an exhausted-quota response, and a confirmed failed-key conflict using local fixtures. Confirm that retries preserve keys when appropriate and never turn missing values into prices.
Report distinct observations separately from request attempts. Track charged units, queue delay, cache hits, source age, and failed refreshes. Keep X-Request-Id available for support while excluding API credentials from logs.
Start with the search reference, rate-limit guide, and data-quality guidance. Then create an API key and run one bounded search before expanding the queue. A useful pipeline produces evidence you can trace back to its source, including when the most recent refresh did not succeed.
Download the offline example
Download the complete Python example. The companion includes synthetic fixtures and offline checks. It makes no network requests.
