How to Build Flask Background Jobs with RQ, Celery, and Redis

How to Build Background Job Processing in Flask: Simple RQ Worker and Medium Celery with Redis Using Docker Compose?

In modern web applications, certain operations are too slow or resource-intensive to run synchronously within an HTTP request. This is where task queues come to the rescue. They allow you to offload time-consuming work to background processes while keeping your web application responsive and scalable. This guide provides a brief yet comprehensive introduction to three essential Python task queue technologies: RQ Worker, and Celery. The app refers to a Flask Application or a Python automation script as well

Prerequisite:

This tutorial is part of the Docker Series.

📚 View the Complete Docker Series

⬅ Previous Part

Preliminary:

In this tutorial, I will walk you through how to build a simple one (RQ_worker) and a medium one (Celery) using Docker Compose.

The file and folder for both containers are as follows:

1) Simple_redis

2) Medium_redis

What is Redis?

Redis (which stands for REmote DIctionary Server) is an open-source, in-memory data structure store. It is primarily used as a database, cache, and message broker.

Because it holds all its data in RAM rather than on a hard drive, it is incredibly fast, often handling read and write operations in less than a millisecond.

Docker and Docker command line

Before beginning, let me launch Docker Desktop, and to run Docker, simply go to the app directory and then type it in PowerShell.

docker compose up -d --build

To check if the Docker container is running

docker ps

To display the real-time logs of a specific service

docker compose logs

To stop Docker from running,

docker compose down

To remove a particular container

docker rm -f {container ID}

1) Small project, simple tasks: Redis with Redis Commander, RQ Scheduler, RQ Worker, RQ Dashboard, and app.

(i) docker-compose.yaml

name: simple_redis

services:
  #1 Redis: The message broker for RQ
  redis:
    image: redis:7-alpine
    container_name: redis
    restart: unless-stopped
    command: redis-server --appendonly yes
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5
    networks:
      - app-network

  #2 Redis Commander: Web-based GUI for Redis
  redis-commander:
    image: rediscommander/redis-commander:latest
    container_name: redis-commander
    restart: unless-stopped
    ports:
      - "8081:8081"
    environment:
      REDIS_HOSTS: local:redis:6379
      HTTP_USER: admin
      HTTP_PASSWORD: admin123
    depends_on:
      redis:
        condition: service_healthy
    networks:
      - app-network

  #3 Your Main Application (Flask Web server)
  app:
    build:
      context: .
      dockerfile: Dockerfile
    image: flask-rq-app
    container_name: flask-app
    restart: unless-stopped
    working_dir: /app
    command: python app.py
    ports:
      - "8000:8000"
    environment:
      PYTHONUNBUFFERED: "1"
      REDIS_URL: redis://redis:6379
    volumes:
      - .:/app
    depends_on:
      redis:
        condition: service_healthy
    networks:
      - app-network

  #4 RQ Worker: Executes jobs from the queue
  rq-worker:
    build:
      context: .
      dockerfile: Dockerfile
    image: flask-rq-app
    container_name: rq-worker
    restart: unless-stopped
    working_dir: /app
    command: rq worker high default low --url redis://redis:6379
    environment:
      PYTHONUNBUFFERED: "1"
      REDIS_URL: redis://redis:6379
    volumes:
      - .:/app
    depends_on:
      redis:
        condition: service_healthy
    networks:
      - app-network

  #5 RQ Scheduler: Manages scheduled tasks
  rq-scheduler:
    build:
      context: .
      dockerfile: Dockerfile
    image: flask-rq-app
    container_name: rq-scheduler
    restart: unless-stopped
    working_dir: /app
    command: rqscheduler --url redis://redis:6379
    environment:
      PYTHONUNBUFFERED: "1"
      REDIS_URL: redis://redis:6379
    volumes:
      - .:/app
    depends_on:
      redis:
        condition: service_healthy
    networks:
      - app-network

  #6 RQ Dashboard: Monitor workers, queues, scheduled jobs, and failed tasks
  rq-dashboard:
    image: eoranged/rq-dashboard
    container_name: rq-dashboard
    restart: unless-stopped
    ports:
      - "9181:9181"
    environment:
      RQ_DASHBOARD_REDIS_URL: redis://redis:6379
    depends_on:
      redis:
        condition: service_healthy
    networks:
      - app-network

volumes:
  redis_data:

networks:
  app-network:
    driver: bridge

What are the containers and their functions?

This file defines six core services:

Redis: The message broker, storing jobs and results.
Redis Commander: A web GUI for viewing and managing Redis data
app: The main web application (e.g., Flask, Django) that enqueues tasks.
rq-worker: Executes jobs pushed onto the queue.
rq-scheduler: Manages and enqueues scheduled jobs.
rq dashboard: web-based monitoring interface for Redis Queue (RQ)

What are those images and ports?

Container Images Ports

Redis redis:7-alpine - localhost:6379
Redis-commander (GUI for Redis) rediscommander/ - localhost:8081

redis-commander:latest

RQ dashboard (GUI for RQ) eoranged/rq-dashboard - localhost:9181
Main application flask-rq-app - localhost:8000
RQ Worker flask-rq-app
RQ Scheduler flask-rq-app

Every container will communicate through Redis instead of localhost.

A particularly good aspect of your configuration is that the Flask app, worker, and scheduler all use the same image (flask-rq-app). This avoids building three separate images and keeps dependencies consistent across all services.

What are the differences between Redis-commander and RQ Dashboard?

(i) Redis Commander

Function: It is a web-based user interface used to view and manage raw data inside your Redis database.

This allows me to inspect the following:

queues
keys
cached data
scheduled jobs

through the browser.

Use Case: Debugging data, manually deleting or editing keys, and verifying whether your application is caching or storing raw data correctly.

Redis Commander required a login in the above example.

ID: admin, and

Password: admin123

(ii) RQ Dashboard

Function: It is a web-based monitoring interface specifically built for Redis Queue (RQ).

This allows me to inspect the following:

Monitor workers,
queues,
scheduled jobs,
failed tasks

Use Case: Monitoring the health of your asynchronous background workers, inspecting stack traces of failed jobs, and retrying failed tasks.

Will they be replaceable?

No, neither Redis Commander nor RQ Dashboard can replace the other. Because they operate at completely different layers of your stack

Will RQ Scheduler do the work?

No, it doesn't. It only places them into Redis. The worker actually runs them.

Why is a health check important?

It prevents the Flask app from starting before Redis is ready.

How to save Redis data?

Your Redis data will be saved:

container recreation
docker-compose down
reboot

Unless the volume is explicitly removed.

(ii) Dockerfile

# Use an official lightweight Python runtime as the base image
FROM python:3.13-slim

# Set the working directory inside the container to /app
WORKDIR /app

# Copy only the requirements file first to leverage Docker's cache layering
COPY requirements.txt .

# Install the Python dependencies specified in requirements.txt
# --no-cache-dir keeps the image size smaller by not saving the download cache
RUN pip install --no-cache-dir -r requirements.txt

# Copy the rest of the local application code into the container
COPY . .

# Inform Docker that the container listens on port 8000 at runtime
EXPOSE 8000

# Define the default command to run your application when the container starts
CMD ["python", "app.py"]

(iii) Requirements.txt

Flask
redis
rq
rq-scheduler
python-dotenv
gunicorn

Preview the architecture for a simple project as follows:

Please check out how Redis and RQ_worker apply to the sales invoice automation script.

---------------------------------------------------------------------------------------------------------------------------------------------------
🎁 Get Your FREE Flask Cheat Sheet
Get more Flask, Python automation, Docker, and HTMX tutorials delivered to your inbox.
✓ Practical coding tutorials
✓ Automation tips for SMEs
✓ New project ideas and templates
Download my FREE Flask Cheat Sheet (PDF)
Get My Free PDF
-----------------------------------------------------------------------------------------------------------------------------

2) Medium project, some scheduling: Redis with Celery, Redis Commander, Celery Beat, Flower, Celery Worker, and app

name: medium_redis

services:
  # Redis: Message broker for Celery
  redis:
    image: redis:7-alpine
    container_name: celery-redis
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5
    restart: unless-stopped
    networks:
      - celery-network

  # Redis Commander: Web-based GUI for Redis
  redis-commander:
    image: rediscommander/redis-commander:latest
    container_name: redis-commander
    ports:
      - "8081:8081"
    environment:
      - REDIS_HOSTS=local:redis:6379
      - HTTP_USER=admin
      - HTTP_PASSWORD=admin123
    depends_on:
      redis:
        condition: service_healthy
    restart: unless-stopped
    networks:
      - celery-network

  # Flower: Web-based monitoring for Celery
  flower:
    image: mher/flower:latest
    container_name: celery-flower
    ports:
      - "5555:5555"
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/0
      - FLOWER_PORT=5555
      - FLOWER_BASIC_AUTH=admin:admin123
    depends_on:
      redis:
        condition: service_healthy
    restart: unless-stopped
    networks:
      - celery-network

  # # Celery Beat: Pushes scheduled tasks into Redis at defined intervals
  celery-beat:
    build: .
    container_name: celery-beat
    command: 
      celery -A app.celery beat --loglevel=info
    # For non-Django projects, use:
    # command: celery -A app.celery beat --loglevel=info
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/0
    volumes:
      - .:/app
    depends_on:
      redis:
        condition: service_healthy
    restart: unless-stopped
    networks:
      - celery-network

  # Celery Worker: Executes tasks
  celery-worker:
    build: .
    container_name: celery-worker
    command: > 
       celery -A app.celery worker
       --loglevel=info
       --concurrency=4
  -Q celery,high_priority,low_priority
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/0
    volumes:
      - .:/app
    depends_on:
      redis:
        condition: service_healthy
    restart: unless-stopped
    networks:
      - celery-network

  # Flask App: Enqueues tasks to Celery and serves HTTP endpoints
  app:
    build: .
    container_name: celery-app
    ports:
      - "8000:8000"
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/0
      - FLASK_ENV=development
      - FLASK_APP=app.py
    volumes:
      - .:/app
    depends_on:
      redis:
        condition: service_healthy
      celery-worker:
        condition: service_started
    restart: unless-stopped
    networks:
      - celery-network

volumes:
  redis_data:

networks:
  celery-network:
    driver: bridge

What are the containers and their functions?
This file defines four core services:

Redis: The message broker, storing jobs and results.
Redis Commander: A web GUI for viewing and managing Redis data
Flower: A web-based monitor for Celery
Celery-beat: Scheduler for periodic tasks
Celery Worker: Executes tasks
app: Your main web application (e.g., Flask, Django) that enqueues tasks.
What are those images and ports?
Container Images Ports
Redis redis:7-alpine - localhost:6379
Redis-commander (GUI for Redis) rediscommander/ - localhost:8081
redis-commander:latest
Flower (GUI for Celery) mher/flower:latest - localhost:5555
Main application build: . - localhost:8000
Celery Worker build: .
Celery Beats build: .
What is Flower?
Function: It is a real-time, web-based monitoring and administration tool for Celery, a popular distributed task queue used in Python.

This allows me to inspect the following:
task monitoring
worker status
task history
failure tracking
Use case:
Monitoring Workers: Tracking worker status, start/stop events, and system statistics.
Task Tracking: Viewing real-time task progress, arguments, execution times, and return values or error stack traces.
Remote Control: Allowing you to dynamically adjust worker pools, rate limits, and cancel or revoke tasks directly from the UI.
Broker Insights: Inspecting queue lengths and message details from your backend message broker (like Redis or RabbitMQ).

Both the Redis Commander and Flower require a login in the above example.
ID: admin, and
Password: admin123

What is the role of celery?
Celery uses Redis for 3 roles:

message broker
result backend
caching (optional)

What Redis health check? How does it work?

The Redis health check is one of the most useful features in your Docker Compose file because it helps ensure that dependent services (Flask, Celery Worker, Celery Beat, Flower, etc.) don't start before Redis is actually ready to accept connections.

Option	Description
`test`	Runs the command `redis-cli ping` inside the Redis container. If Redis is running correctly, it responds with `PONG`.
`interval: 5s`	Docker performs the health check every 5 seconds.
`timeout: 3s`	If Redis doesn't respond within 3 seconds, that check is considered a failure.
`retries: 5`	Docker marks the container as unhealthy after 5 consecutive failed checks.

Why is a Redis health check important?

If Redis is ready, the output is "Pong," and Docker then marks the container as healthy. Without a health check, Docker only knows that the Redis process has started—it doesn't know whether Redis is actually ready to serve requests.

Otherwise, it serves an error:

Connection refused or
Cannot connect to Redis

(ii) Dockerfile

# Use an official lightweight Python runtime as the base image
FROM python:3.13-slim

# Set the working directory inside the container to /app
WORKDIR /app

# Copy only the requirements file first to leverage Docker's cache layering
COPY requirements.txt .

# Install the Python dependencies specified in requirements.txt
# --no-cache-dir keeps the image size smaller by not saving the download cache
RUN pip install --no-cache-dir -r requirements.txt

# Copy the rest of the local application code into the container
COPY . .

# Inform Docker that the container listens on port 8000 at runtime
EXPOSE 8000

# Define the default command to run your application when the container starts
CMD ["python", "app.py"]

(iii) Requirements.txt

Flask
Flask==3.0.3
celery==5.4.0
redis==5.0.8
python-dotenv==1.0.1
gunicorn==22.0.0

Preview the architecture for a simple project as follows:

The comparison

I have discussed both RQ and Celery above; let me dive in further to make a comparison between them.

1) RQ: The "Lightweight" Choice

RQ is built on the philosophy of simplicity. It is essentially a wrapper around Redis specifically for job queueing.
Best for: Small to medium-sized applications, projects where you are already using Redis and don't want to add complexity, or situations where you simply need a background worker with as little friction as possible.
Pros: Very "Pythonic" and easy to read. If you can write a standard Python function, you can turn it into an RQ task. It integrates beautifully with Flask and other web frameworks.
Cons: Limited flexibility. Because it only supports Redis, you are locked into that infrastructure. It lacks the complex workflow engines found in Celery and requires additional packages for features like periodic scheduling.

2) Celery: The "Mediumweight" Choice

Celery is the industry standard for Python background tasks. It is designed to handle extremely complex distributed systems.
Best for: Large-scale applications, complex task workflows (e.g., "if Task A succeeds, run Task B and C in parallel, then run Task D"), and projects that might need to switch message brokers in the future (e.g., moving from Redis to RabbitMQ).
Pros: Highly configurable, supports advanced patterns like retries with exponential backoff, rate limiting, and sophisticated task routing.
Cons: Overkill for simple projects. It has a significantly steeper learning curve, a larger memory footprint, and requires more "plumbing" to set up correctly.

Which one should you choose?

Choose RQ if:
You are building a small-to-medium project.
You want to get up and running in minutes, not hours.
You are already using Redis and don't need the complexity of RabbitMQ.
You prefer a simple, clean, and predictable API.
Choose Celery if:
You are building a large, distributed enterprise application.
You need complex orchestration (task chaining, grouping, parallel execution).
You anticipate needing to scale across many different types of infrastructure or need to support high-availability brokers like RabbitMQ.
You require robust monitoring tools (like Flower) to visualise a massive number of concurrent workers.
Final Wrap-up

In this tutorial, we learned how to set up background job processing in Flask using two different approaches: a simple solution with RQ worker and Redis for lightweight task handling and a more powerful medium-level setup using Celery with Redis for scalable task queues, scheduling, and distributed workers.

Published: July 2026
Last Updated: July 2026

---------------------------------------------------------------------------------------------------------------------------------------------------

Thanks for reading!

If you haven't subscribed yet, join my newsletter to receive future Python and Flask tutorials.

Subscribe

---------------------------------------------------------------------------------------------------------------------------------------------------

Need a similar system for your business?

I build custom Flask web applications and Python automation solutions for SMEs and solopreneurs.

Hire me

---------------------------------------------------------------------------------------------------------------------------------------------------

About the Author

Kelvin Loh is a Python developer focused on Flask, desktop applications, and business automation solutions. He shares practical tutorials and real-world coding projects to help developers and small businesses build useful applications.

Kelvin Loh Dot Click | Python Desktop, Web & Data Automation

Search This Blog