Skip to main content

How to Build Background Job Processing in Flask: Simple RQ Worker and Medium Celery with Redis Using Docker Compose?



In modern web applications, certain operations are too slow or resource-intensive to run synchronously within an HTTP request. This is where task queues come to the rescue. They allow you to offload time-consuming work to background processes while keeping your web application responsive and scalable. This guide provides a brief yet comprehensive introduction to three essential Python task queue technologies: RQ Worker, and Celery. The app refers to a Flask Application or a Python automation script as well

Prerequisite:

This tutorial is part of the Docker Series.

📚 View the Complete Docker Series

⬅ Previous Part

 Preliminary: 
In this tutorial, I will walk you through how to build a simple one (RQ_worker) and a medium one (Celery) using Docker Compose.

The file and folder for both containers are as follows: 
1) Simple_redis
2) Medium_redis

What is Redis?

Redis (which stands for REmote DIctionary Server) is an open-source, in-memory data structure store. It is primarily used as a database, cache, and message broker.

Because it holds all its data in RAM rather than on a hard drive, it is incredibly fast, often handling read and write operations in less than a millisecond.

Docker and Docker command line

Before beginning, let me launch Docker Desktop, and to run Docker, simply go to the app directory and then type it in PowerShell.

docker compose up -d --build
To check if the Docker container is running 
docker ps
To display the real-time logs of a specific service 
docker compose logs
To stop Docker from running, 
docker compose down
To remove a particular container 
docker rm -f {container ID}

1) Small project, simple tasks: Redis with Redis Commander, RQ Scheduler, RQ Worker, RQ Dashboard, and app. 

(i) docker-compose.yaml
name: simple_redis

services:
  #1 Redis: The message broker for RQ
  redis:
    image: redis:7-alpine
    container_name: redis
    restart: unless-stopped
    command: redis-server --appendonly yes
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5
    networks:
      - app-network

  #2 Redis Commander: Web-based GUI for Redis
  redis-commander:
    image: rediscommander/redis-commander:latest
    container_name: redis-commander
    restart: unless-stopped
    ports:
      - "8081:8081"
    environment:
      REDIS_HOSTS: local:redis:6379
      HTTP_USER: admin
      HTTP_PASSWORD: admin123
    depends_on:
      redis:
        condition: service_healthy
    networks:
      - app-network

  #3 Your Main Application (Flask Web server)
  app:
    build:
      context: .
      dockerfile: Dockerfile
    image: flask-rq-app
    container_name: flask-app
    restart: unless-stopped
    working_dir: /app
    command: python app.py
    ports:
      - "8000:8000"
    environment:
      PYTHONUNBUFFERED: "1"
      REDIS_URL: redis://redis:6379
    volumes:
      - .:/app
    depends_on:
      redis:
        condition: service_healthy
    networks:
      - app-network

  #4 RQ Worker: Executes jobs from the queue
  rq-worker:
    build:
      context: .
      dockerfile: Dockerfile
    image: flask-rq-app
    container_name: rq-worker
    restart: unless-stopped
    working_dir: /app
    command: rq worker high default low --url redis://redis:6379
    environment:
      PYTHONUNBUFFERED: "1"
      REDIS_URL: redis://redis:6379
    volumes:
      - .:/app
    depends_on:
      redis:
        condition: service_healthy
    networks:
      - app-network

  #5 RQ Scheduler: Manages scheduled tasks
  rq-scheduler:
    build:
      context: .
      dockerfile: Dockerfile
    image: flask-rq-app
    container_name: rq-scheduler
    restart: unless-stopped
    working_dir: /app
    command: rqscheduler --url redis://redis:6379
    environment:
      PYTHONUNBUFFERED: "1"
      REDIS_URL: redis://redis:6379
    volumes:
      - .:/app
    depends_on:
      redis:
        condition: service_healthy
    networks:
      - app-network

  #6 RQ Dashboard: see what your RQ workers and queues are doing
  rq-dashboard:
    image: eoranged/rq-dashboard
    container_name: rq-dashboard
    restart: unless-stopped
    ports:
      - "9181:9181"
    environment:
      RQ_DASHBOARD_REDIS_URL: redis://redis:6379
    depends_on:
      redis:
        condition: service_healthy
    networks:
      - app-network

volumes:
  redis_data:

networks:
  app-network:
    driver: bridge

What are the containers and their functions? 
This file defines six core services: 
  1. Redis: The message broker, storing jobs and results. 
  2. Redis Commander: A web GUI for viewing and managing Redis data
  3. app: The main web application (e.g., Flask, Django) that enqueues tasks. 
  4. rq-worker: Executes jobs pushed onto the queue. 
  5. rq-scheduler: Manages and enqueues scheduled jobs. 
  6. rq dashboard: web-based monitoring interface for Redis Queue (RQ)
What are those images and ports?
               Container                                                   Images                              Ports
  • Redis                                                   redis:7-alpine                     - localhost:6379
  • Redis-commander (GUI for Redis) rediscommander/             - localhost:8081
    •                                                              redis-commander:latest 
  • RQ dashboard (GUI for RQ)             eoranged/rq-dashboard   - localhost:9181
  • Main application                               flask-rq-app                        - localhost:8000
  • RQ Worker                                         flask-rq-app
  • RQ Scheduler                                    flask-rq-app
What are the differences between Redis-commander and RQ Dashboard?
(i) Redis Commander 
Function: It is a web-based user interface used to view and manage raw data inside your Redis database.

Use Case: Debugging data, manually deleting or editing keys, and verifying whether your application is caching or storing raw data correctly.

Redis Commander required a login in the above example.
ID: admin, and
Password: admin123

(ii) RQ Dashboard
Function: It is a web-based monitoring interface specifically built for Redis Queue (RQ).

Use Case: Monitoring the health of your asynchronous background workers, inspecting stack traces of failed jobs, and retrying failed tasks.

Will they be replaceable?
No, neither Redis Commander nor RQ Dashboard can replace the other. Because they operate at completely different layers of your stack 


(ii) Dockerfile 
# Use an official lightweight Python runtime as the base image
FROM python:3.13-slim

# Set the working directory inside the container to /app
WORKDIR /app

# Copy only the requirements file first to leverage Docker's cache layering
COPY requirements.txt .

# Install the Python dependencies specified in requirements.txt
# --no-cache-dir keeps the image size smaller by not saving the download cache
RUN pip install --no-cache-dir -r requirements.txt

# Copy the rest of the local application code into the container
COPY . .

# Inform Docker that the container listens on port 8000 at runtime
EXPOSE 8000

# Define the default command to run your application when the container starts
CMD ["python", "app.py"]
(iii) Requirements.txt
Flask
redis
rq
rq-scheduler
python-dotenv
gunicorn


2) Medium project, some scheduling: Redis with Celery, Redis Commander, Celery Beat, Flower, Celery Worker, and app

name: medium_redis

services:
  # Redis: Message broker for Celery
  redis:
    image: redis:7-alpine
    container_name: celery-redis
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5
    restart: unless-stopped
    networks:
      - celery-network

  # Redis Commander: Web-based GUI for Redis
  redis-commander:
    image: rediscommander/redis-commander:latest
    container_name: redis-commander
    ports:
      - "8081:8081"
    environment:
      - REDIS_HOSTS=local:redis:6379
      - HTTP_USER=admin
      - HTTP_PASSWORD=admin123
    depends_on:
      redis:
        condition: service_healthy
    restart: unless-stopped
    networks:
      - celery-network

  # Flower: Web-based monitoring for Celery
  flower:
    image: mher/flower:latest
    container_name: celery-flower
    ports:
      - "5555:5555"
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/0
      - FLOWER_PORT=5555
      - FLOWER_BASIC_AUTH=admin:admin123
    depends_on:
      redis:
        condition: service_healthy
    restart: unless-stopped
    networks:
      - celery-network

  # Celery Beat: Scheduler for periodic tasks
  celery-beat:
    build: .
    container_name: celery-beat
    command: 
      celery -A app.celery beat --loglevel=info
    # For non-Django projects, use:
    # command: celery -A app.celery beat --loglevel=info
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/0
    volumes:
      - .:/app
    depends_on:
      redis:
        condition: service_healthy
    restart: unless-stopped
    networks:
      - celery-network

  # Celery Worker: Executes tasks
  celery-worker:
    build: .
    container_name: celery-worker
    command: celery -A app.celery worker --loglevel=info 
         --concurrency=4 -Q celery,high_priority,low_priority
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/0
    volumes:
      - .:/app
    depends_on:
      redis:
        condition: service_healthy
    restart: unless-stopped
    networks:
      - celery-network

  # Application (Flask example)
  app:
    build: .
    container_name: celery-app
    ports:
      - "8000:8000"
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/0
      - FLASK_ENV=development
      - FLASK_APP=app.py
    volumes:
      - .:/app
    depends_on:
      redis:
        condition: service_healthy
      celery-worker:
        condition: service_started
    restart: unless-stopped
    networks:
      - celery-network

volumes:
  redis_data:

networks:
  celery-network:
    driver: bridge

What are the containers and their functions?
This file defines four core services:

  1. Redis: The message broker, storing jobs and results. 
  2. Redis Commander: A web GUI for viewing and managing Redis data
  3. Flower: A web-based monitor for Celery
  4. Celery-beat: Scheduler for periodic tasks
  5. Celery Worker: Executes tasks
  6. app: Your main web application (e.g., Flask, Django) that enqueues tasks. 
What are those images and ports?
               Container                                                   Images                              Ports
  • Redis                                                            redis:7-alpine             - localhost:6379
  • Redis-commander (GUI for Redis)          rediscommander/     - localhost:8081
    •                                                                       redis-commander:latest
  • Flower (GUI for Celery)                             mher/flower:latest     - localhost:5555
  • Main application                                        build: .                          - localhost:8000
  • Celery Worker                                             build: .
  • Celery Beats                                               build: .
What is Flower?
Function: It is a real-time, web-based monitoring and administration tool for Celery, a popular distributed task queue used in Python.

Use case: 
  • Monitoring Workers: Tracking worker status, start/stop events, and system statistics.

  • Task Tracking: Viewing real-time task progress, arguments, execution times, and return values or error stack traces.

  • Remote Control: Allowing you to dynamically adjust worker pools, rate limits, and cancel or revoke tasks directly from the UI.

  • Broker Insights: Inspecting queue lengths and message details from your backend message broker (like Redis or RabbitMQ).

Both the Redis Commander and Flower require a login in the above example.
ID: admin, and
Password: admin123

(ii) Dockerfile

# Use an official lightweight Python runtime as the base image
FROM python:3.13-slim

# Set the working directory inside the container to /app
WORKDIR /app

# Copy only the requirements file first to leverage Docker's cache layering
COPY requirements.txt .

# Install the Python dependencies specified in requirements.txt
# --no-cache-dir keeps the image size smaller by not saving the download cache
RUN pip install --no-cache-dir -r requirements.txt

# Copy the rest of the local application code into the container
COPY . .

# Inform Docker that the container listens on port 8000 at runtime
EXPOSE 8000

# Define the default command to run your application when the container starts
CMD ["python", "app.py"]
(iii) Requirements.txt

Flask
Flask==3.0.3
celery==5.4.0
redis==5.0.8
python-dotenv==1.0.1
gunicorn==22.0.0

Final Wrap-up

In this tutorial, we learned how to set up background job processing in Flask using two different approaches: a simple solution with RQ worker and Redis for lightweight task handling and a more powerful medium-level setup using Celery with Redis for scalable task queues, scheduling, and distributed workers.

Published: July 2026
Last Updated: July 2026

About the Author

Kelvin Loh is a Python developer focused on Flask, desktop applications, and business automation solutions. He shares practical tutorials and real-world coding projects to help developers and small businesses build useful applications.
   


Labels:

Comments

Post a Comment

Popular Posts

How to Design a Location Tracking Module for Desktop Business Systems Using Python?

This tutorial will show you how to create an interactive map application that allows users to input geographic coordinates and visualize locations on an interactive map using Python's Tkinter GUI framework enhanced with TTKBootstrap styling and the TkinterMapView widget.  Prerequisite: This tutorial is part of the standalone tutorial. 📚 View the  standalone tutorial Preliminary   Before I begin, it is recommended to activate the virtual environment before installing the relevant dependencies. python -m venv venv venv\Scripts\activate pip install ttkbootstrap tkintermapview As usual, we need to import the relevant module and set the root values from tkinter import * import ttkbootstrap as tb import tkintermapview from ttkbootstrap.dialogs import Messagebox root = tb.Window(themename='darkly') root.title('Find My Map') root.geometry("1200x1200") I would like to divide it into three sections.  a) Input Panel Layout,  The headers are mainly a label...
Post a Comment
Read more

How to Create Flask Forms with CKEditor and Flask-WTF?

In this tutorial, we will be integrating Flask-WTF and Flask-CKEditor into your Flask application! You'll learn how to set up the editor, securely handle formatted HTML content, and create a seamless user experience that enhances any project that requires user-generated content. Let's get started! Prerequisite: This tutorial is part of the Flask CKEditor Project Series. 📚 View the Complete Flask CKEditor Series                                                                                                                                                  ➡ Next Part Preliminary Before I begin, it is recommended to activate the v...
Post a Comment
Read more

How to Store Application Data with SQLite in Python?

  In the previous tutorial, I used TinyDB as our storage solution to keep application data in a simple JSON-based format. While TinyDB is lightweight and easy to use, many web applications require a more structured and scalable database system. In this tutorial, we will switch to SQLite , a powerful relational database that integrates well with Python and web frameworks. By using SQLite, you will learn how to store, manage, and query data in a more structured way for real-world applications. Prerequisite: This tutorial is part of the Flask CKEditor Project Series. 📚 View the Complete Flask CKEditor Series ⬅ Previous Part                                                                                                         ...
Post a Comment
Read more