Skip to content

fix: Move RQ metadata counter updates after successful API call #799

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
vdusek wants to merge 1 commit into from
+6 −3
Closed

fix: Move RQ metadata counter updates after successful API call #799

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions src/apify/storage_clients/_apify/_request_queue_single_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -204,10 +204,9 @@ async def mark_request_as_handled(self, request: Request) -> ProcessedRequest |
if cached_request := self._requests_cache.get(request_id):
cached_request.handled_at = request.handled_at

if request.handled_at is None:
should_update_counters = request.handled_at is None
if should_update_counters:
request.handled_at = datetime.now(tz=timezone.utc)
self.metadata.handled_request_count += 1
self.metadata.pending_request_count -= 1

try:
# Remember that we handled this request, to optimize local deduplication.
Expand All @@ -224,6 +223,10 @@ async def mark_request_as_handled(self, request: Request) -> ProcessedRequest |
logger.exception(f'Error marking request {request.unique_key} as handled.')
return None
else:
# Update counters only after a successful API call to avoid inconsistency on failure.
if should_update_counters:
self.metadata.handled_request_count += 1
self.metadata.pending_request_count -= 1
return processed_request

async def reclaim_request(
Expand Down