Setting Time Limit on specific task with celery

CeleryCeleryd

Celery Problem Overview


I have a task in Celery that could potentially run for 10,000 seconds while operating normally. However all the rest of my tasks should be done in less than one second. How can I set a time limit for the intentionally long running task without changing the time limit on the short running tasks?

Celery Solutions


Solution 1 - Celery

You can set task time limits (hard and/or soft) either while defining a task or while calling.

from celery.exceptions import SoftTimeLimitExceeded

@celery.task(time_limit=20)
def mytask():
    try:
        return do_work()
    except SoftTimeLimitExceeded:
        cleanup_in_a_hurry()

or

mytask.apply_async(args=[], kwargs={}, time_limit=30, soft_time_limit=10)

Solution 2 - Celery

This is an example with decorator for an specific Task and Celery 3.1.23 using soft_time_limit=10000

@task(bind=True, default_retry_delay=30, max_retries=3, soft_time_limit=10000)
def process_task(self, task_instance):
   """Task processing."""
        pass

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionjimstandardView Question on Stackoverflow
Solution 1 - CelerymherView Answer on Stackoverflow
Solution 2 - CelerygogascaView Answer on Stackoverflow