A good explanation is in this post Celery Execution Pools: What is it all about?.

Suppose there is a large tasks.py file, like this:

@celery.task()
def task1():
    ...

@celery.task()
def task2():
    ...
    task1.delay()
    ...

...

A good idea is to split it on the smaller files, but Celery auto_discover by default search tasks in package.tasks module, so one way to do this is to create a package tasks and import tasks from other files in __init__.py.

__init__.py

from .task1 import task1
from .task2 import task2

__all__ = ['task1', 'task2']

task1.py

@celery.task()
def task1():
    ...

task2.py

from .task1 import task1

@celery.task()
def task2():
    ...
    task1.delay()
    ...

With docker-compose it can be done the next way

flask_app:
    ...
    healthcheck:
      test: wget --spider --quiet http://localhost:8080/-/health
celery_worker:
    ...
    command: celery worker --app app.celeryapp -n worker@%h --loglevel INFO
    healthcheck:
      test: celery inspect ping --app app.celeryapp -d worker@$$HOSTNAME

Where /-/health is just a simple route

@app.route("/-/health")
def health():
    return 'ok'

If there is a need to monitor several applications, there are two ways:

• run celery flower instance for each application

celery flower --app app1.celeryapp --port=5555
celery flower --app app2.celeryapp --port=5556

• run celery flower with broker option, there will be less options to control tasks

celery flower --broker=redis://redis/0

There is a good checklist to build great celery async tasks.

I want only to add how to autoreload celery worker in development mode. Before there was an --autoreload option, but now it’s removed. For this task I use watchdog.

pip install watchdog
watchmedo auto-restart --recursive -d app -p '*.py' -i '*.pyc' -- celery worker --app app.celeryapp --queues my_query -n my_queue@%h --loglevel INFO

If you have some long background tasks, sometimes it’s useful to notify a user about the progress. In this article i put an example for flask, celery and socket.io. In short form it’s described in flask-socketio documentation.