Temporal vs Celery for AI Workflows: Durable Python Jobs, Retries and Human Approval (2026)
Updated November 21, 2026 · 16-minute read
A customer uploads a 200-page contract. Your application extracts the text, calls an LLM, checks the answer, waits for a human reviewer, writes the result to a database, and sends a webhook. The first model call takes 90 seconds. A worker restarts after the model responds but before the database write. A retry starts the model call again. Now you have duplicate cost, uncertain job state, and a support ticket asking whether the document is finished.
This is where choosing a background-job system becomes an architecture decision. Celery is an excellent distributed task queue. Temporal is a durable workflow engine. They can both run Python code asynchronously, retry failures, and scale workers, but they solve different levels of the problem. This guide compares them for AI workloads and builds a small Temporal workflow you can adapt to document processing, RAG ingestion, model evaluation, or agent jobs.
Who should read this?
- Python developers running LLM calls, document pipelines, RAG ingestion, or agent tasks outside an HTTP request.
- Teams deciding whether to keep Celery or introduce Temporal.
- Backend engineers who need retries, human approval, progress tracking, and recovery after worker restarts.
- Architects designing multi-step AI jobs that may run for minutes, hours, or days.
The short answer
Choose Celery when the unit of work is primarily a task: send an email, resize an image, generate one report, or process a queue of independent jobs. Its task model, broker ecosystem, retries, and Canvas primitives for chains, groups, and chords are often enough.
Choose Temporal when the unit of work is a long-lived business process: run several model and database steps, wait for an approval, resume after a day, compensate for a partial failure, or preserve the exact progress of a workflow when workers or infrastructure restart.
Do not migrate to Temporal just because an LLM call is slow. Migrate when coordinating the steps around that call has become harder than executing the call itself.
1. Understand the architectural difference
Celery: dispatch tasks to workers
A Celery application sends task messages through a broker such as RabbitMQ or Redis. Workers consume those messages, execute registered task functions, and optionally store results in a result backend. You can compose tasks with Celery Canvas: a chain runs tasks in sequence, a group runs them in parallel, and a chord runs a callback after a group completes.
This model is straightforward and works very well for independent background jobs. You still need to design the application-level state machine when a business process spans multiple tasks, long waits, user decisions, and external side effects.
Temporal: persist the progress of a workflow
Temporal separates a Workflow from its Activities. Workflow code describes the sequence and decisions. Activities perform side effects such as calling an LLM, querying a database, or sending a webhook. The Temporal Service records workflow events and activity outcomes. If a Workflow Worker disappears, another Worker can replay the event history and reconstruct workflow state without rerunning Activities that have already completed.
Temporal does not host your Python business code. You deploy and operate Workers yourself; Temporal provides the durable orchestration service that coordinates them. For local development, the Temporal CLI includes a development server and Web UI.
| Concern | Celery | Temporal |
|---|---|---|
| Primary abstraction | Task message and worker | Durable Workflow made of Activities, timers, and messages |
| Simple background jobs | Excellent fit | Supported, but may be more infrastructure than needed |
| Retries | Task retry and automatic retry options | Activity Retry Policies recorded as part of workflow execution |
| Multi-step orchestration | Canvas chains, groups, chords; application owns broader process state | Workflow code coordinates steps and durable state |
| Long human wait | Usually model it with database state and scheduled tasks | Workflow can wait for a Signal or Update without holding a worker thread |
| Recovery after worker loss | Broker acknowledgement and redelivery rules; tasks must tolerate duplicates | Workflow history is replayed; in-flight Activities may retry |
| Operational footprint | Broker, workers, and optional result backend | Temporal Service plus your Workers and application infrastructure |
| Workflow code constraints | Normal Python task code | Workflow code must be deterministic; external I/O belongs in Activities |
This is not a feature-count contest. Celery is not merely a primitive queue, and Temporal is not a replacement for every task queue. The useful distinction is whether you need to manage tasks or preserve the lifecycle of a process.
2. Model an AI job before choosing the tool
Consider a document summarization pipeline. A production version might look like this:
- Accept a document and create a durable job identifier.
- Extract or retrieve the document text.
- Call a model to produce a structured summary.
- Validate the output and possibly run a second evaluation step.
- Wait for a reviewer if the document is high-risk.
- Write the approved summary and notify downstream systems.
With Celery, you could represent each step as a task and connect them with a chain or chord. Your database would usually hold the authoritative process state, and your code would decide what to enqueue next after a failure, approval, or timeout.
With Temporal, the Workflow itself is the process definition. It calls Activities, records their results, and can wait for a message or timer. The application database still stores business data, but you do not have to reconstruct the orchestration state from a collection of task rows and ad hoc callbacks.
3. Set up a local Temporal development environment
Prerequisites:
- Python 3.10 or later.
- The Temporal CLI installed and available as
temporal. - An OpenAI API key and a model identifier available to your API account for the example below.
- An application repository with document lookup and summary persistence functions. The two repository functions in the sample are integration points, not a database implementation.
Start the local Temporal development server in a terminal:
temporal server start-dev --db-filename temporal.db
The service listens on localhost:7233 and the Web UI is normally available at http://localhost:8233. The local server is for development, not a production deployment.
Create a virtual environment and install the Python dependencies:
python -m venv .venv
source .venv/bin/activate
# Windows PowerShell:
# .venv\\Scripts\\Activate.ps1
python -m pip install --upgrade pip
python -m pip install temporalio openai
Set the model configuration in the Worker environment. Keep secrets out of source control:
export OPENAI_API_KEY="..."
export OPENAI_MODEL="your-enabled-model"
export TEMPORAL_ADDRESS="localhost:7233"
4. Put external work in Activities
An Activity is the right place for network calls, database operations, file access, and model requests. The following example uses the official OpenAI Python SDK Responses API. Replace the repository imports with your own database or object-storage implementation.
Create activities.py:
import os
from openai import AsyncOpenAI
from temporalio import activity
from temporalio.exceptions import ApplicationError
from app.documents import get_document_text, upsert_summary
model_client = AsyncOpenAI(
timeout=120.0,
max_retries=0, # Let the Temporal Activity Retry Policy own retries.
)
@activity.defn
async def generate_summary(document_id: str, revision: int) -> str:
text = await get_document_text(document_id, revision)
if not text or not text.strip():
raise ApplicationError(
"The document contains no extractable text",
non_retryable=True,
)
response = await model_client.responses.create(
model=os.environ["OPENAI_MODEL"],
instructions=(
"Summarize the document accurately. Separate facts from "
"uncertainty. Do not invent missing details."
),
input=text,
)
return response.output_text
@activity.defn
async def persist_summary(
document_id: str,
revision: int,
summary: str,
) -> None:
# Implement this as an idempotent upsert in your application.
await upsert_summary(
document_id=document_id,
revision=revision,
summary=summary,
idempotency_key=f"{document_id}:{revision}",
)
Two design choices matter here:
- One retry owner: the OpenAI client is configured with
max_retries=0so SDK retries do not multiply the attempts configured by Temporal. If your provider SDK retries internally, account for those attempts and timeouts explicitly. - Idempotent persistence: the database write should use a unique key such as document ID plus revision. If an Activity runs again after an ambiguous failure, an upsert should not create a second summary row or repeat a downstream side effect.
The example returns the summary through Temporal so the next Activity can persist it. For large or sensitive documents, prefer storing the generated artifact in object storage and passing a short reference between Activities. Temporal records Workflow and Activity inputs and results in Event History unless you configure payload encryption or external storage.
5. Write the deterministic Workflow
Create workflows.py. The Workflow coordinates the Activities; it does not call the model or database directly.
from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy
with workflow.unsafe.imports_passed_through():
from activities import generate_summary, persist_summary
@workflow.defn
class SummarizeDocument:
@workflow.run
async def run(self, document_id: str, revision: int) -> str:
summary = await workflow.execute_activity(
generate_summary,
document_id,
revision,
start_to_close_timeout=timedelta(minutes=3),
schedule_to_close_timeout=timedelta(minutes=12),
retry_policy=RetryPolicy(
initial_interval=timedelta(seconds=5),
backoff_coefficient=2.0,
maximum_interval=timedelta(seconds=60),
maximum_attempts=4,
),
)
await workflow.execute_activity(
persist_summary,
document_id,
revision,
summary,
start_to_close_timeout=timedelta(seconds=30),
retry_policy=RetryPolicy(
initial_interval=timedelta(seconds=2),
maximum_attempts=5,
),
)
return summary
The first Activity gets a three-minute attempt timeout and a twelve-minute total schedule-to-close window. Its retry delay grows exponentially, up to one minute, with four attempts total. Tune these values to your model's real latency distribution and provider rate limits; they are example settings, not universal production defaults.
The import is marked as passed through the Workflow sandbox because the Activity definitions live in another module. Temporal's Python SDK runs Workflow code in a sandbox by default and checks for common sources of nondeterminism.
6. Run a Worker and start a Workflow
Create worker.py:
import asyncio
import os
from temporalio.client import Client
from temporalio.worker import Worker
from activities import generate_summary, persist_summary
from workflows import SummarizeDocument
TASK_QUEUE = "ai-document-jobs"
async def main():
client = await Client.connect(
os.getenv("TEMPORAL_ADDRESS", "localhost:7233")
)
worker = Worker(
client,
task_queue=TASK_QUEUE,
workflows=[SummarizeDocument],
activities=[generate_summary, persist_summary],
)
await worker.run()
if __name__ == "__main__":
asyncio.run(main())
Start the Worker in a second terminal:
python worker.py
Now create start_job.py to submit a document job:
import asyncio
import os
from temporalio.client import Client
from workflows import SummarizeDocument
TASK_QUEUE = "ai-document-jobs"
async def main():
client = await Client.connect(
os.getenv("TEMPORAL_ADDRESS", "localhost:7233")
)
document_id = "contract-842"
revision = 3
handle = await client.start_workflow(
SummarizeDocument.run,
document_id,
revision,
id=f"summarize-{document_id}-r{revision}",
task_queue=TASK_QUEUE,
)
print({"workflow_id": handle.id, "run_id": handle.result_run_id})
if __name__ == "__main__":
asyncio.run(main())
Run it with python start_job.py, then open the Temporal Web UI. You should see the Workflow Execution and its Activity events. In a FastAPI endpoint, the same pattern can return HTTP 202 with the Workflow ID instead of waiting for the model to finish. A separate status endpoint can query the Workflow or read your application's job table.
7. Add approval and status without keeping a request open
Suppose summaries for regulated or high-value documents need human approval. Do not keep a FastAPI request open while a reviewer takes several hours. Let the Workflow wait.
Temporal supports three message types:
- Query: read the Workflow's current state without changing it.
- Signal: send an asynchronous message, such as “review completed”. The sender does not wait for a result.
- Update: send a tracked request when the caller needs validation or a result from the Workflow.
A review step can set an approval state through a Signal or Update, and the Workflow can wait on that state with a Workflow condition. Add a Workflow timer or deadline so a review that never arrives does not remain open forever. Keep the actual authorization decision in your application: a Signal is a message, not proof that the sender is allowed to approve the document.
This is a different layer from an agent framework's internal state machine. If you already use LangGraph for agent reasoning and interrupts, Temporal can sit outside it to manage the longer business process. See our LangGraph human-in-the-loop and strict-resume deep-dive for the agent-level side of that design.
8. What the Celery version looks like
If the process is a single summarization task, Celery may be the simpler answer. A task can call the same repository and model adapter:
from celery import Celery
from openai import (
OpenAI,
APIConnectionError,
APITimeoutError,
RateLimitError,
)
app = Celery(
"ai_jobs",
broker="redis://localhost:6379/0",
backend="redis://localhost:6379/1",
)
model_client = OpenAI(timeout=120.0, max_retries=0)
@app.task(
bind=True,
acks_late=True,
autoretry_for=(
APIConnectionError,
APITimeoutError,
RateLimitError,
),
retry_backoff=True,
retry_backoff_max=300,
retry_jitter=True,
max_retries=5,
)
def summarize_document(self, document_id: str, revision: int):
text = get_document_text_sync(document_id, revision)
response = model_client.responses.create(
model=os.environ["OPENAI_MODEL"],
instructions="Summarize accurately. Do not invent missing details.",
input=text,
)
upsert_summary_sync(
document_id,
revision,
response.output_text,
idempotency_key=f"{document_id}:{revision}",
)
return {"document_id": document_id, "revision": revision}
This is a valid task-oriented design if your repository functions are synchronous and your task is independently retryable. The task is configured for late acknowledgement, so idempotency is essential: a task can execute more than once. Celery also documents cases where a child process termination is acknowledged even with late acknowledgement enabled; understand broker and worker-loss settings before treating redelivery as a guarantee.
For multi-step Celery pipelines, use Canvas rather than building an improvised chain of callbacks. But once you need a process to survive days of waiting, receive reviewer decisions, manage timers, and resume from a specific point, compare the amount of custom state and recovery code you are maintaining with the cost of operating Temporal.
9. Production trade-offs and failure handling
Retries are not exactly-once side effects
Both systems can execute a task or Activity more than once. A worker can complete an external request and lose its connection before recording completion. The orchestrator cannot safely infer whether the external side effect happened. Use idempotency keys, unique database constraints, upserts, and provider-supported request keys where available. For payments, emails, webhooks, or customer notifications, write an outbox record and dispatch it separately.
Do not put nondeterministic work in a Temporal Workflow
Workflow code must be replay-safe. Do not make direct HTTP calls, read the system clock, generate random values, mutate global state, or use normal thread-based concurrency inside a Workflow. Put those operations in Activities and use Temporal's Workflow APIs for timers, time, and orchestration. If replay fails after a deployment, inspect the Workflow history and versioning guidance before disabling the sandbox.
Control retry storms
A model provider returning HTTP 429 is telling you to slow down, not to launch more concurrent retries. Set bounded attempts, exponential backoff, jitter, and a sensible overall timeout. Apply per-tenant quotas and concurrency limits outside the Workflow as well. Our FastAPI and Redis multi-tenant LLM quota guide covers request and token controls.
Keep payloads small and protect history
Workflow inputs, Activity arguments, and results may be persisted in Event History. Do not pass full PDFs, entire database rows, API keys, or unnecessary personal data through Workflow arguments. Pass document IDs and object-storage references instead. Use a Payload Codec or Temporal Proxy to encrypt payloads before they reach the Temporal Service, restrict access to namespaces and the Web UI, and protect the keys used to decode data.
Separate queues by workload
Model inference, OCR, embedding, and short database writes have different CPU, memory, GPU, and latency profiles. Use separate Task Queues and Worker deployments for heavy inference versus lightweight I/O. In Celery, route long-running and short-running tasks to dedicated workers and set explicit network timeouts. In Temporal, Activity Task Queues let you scale the workers responsible for particular Activity types independently.
10. Troubleshooting checklist
- Workflow stays queued: confirm a Worker is running, connected to the expected Namespace, and polling the exact Task Queue name used by the client.
- Workflow fails on replay: look for network calls, system time, random values, mutable global state, or unsupported imports in Workflow code. Move external work into Activities.
- Model calls happen twice: inspect Activity attempts and timeouts. Make the model request and downstream persistence idempotent where possible; do not assume a timeout proves the provider did nothing.
- 429s repeat rapidly: reduce concurrency, honor provider retry guidance, add backoff and jitter, and enforce tenant-level quotas.
- Worker never notices a long Activity is stuck: use realistic Start-to-Close timeouts. For genuinely long-running Activities, send heartbeats and configure a Heartbeat Timeout.
- History or payloads become too large: move large artifacts to external storage, pass references, and consider Continue-As-New for long-running workflows with growing histories.
- Celery chord does not finish: verify the result backend is configured and that participating tasks do not ignore their results; Celery documents this as a chord requirement.
11. Which should you choose?
| Your workload | Starting recommendation |
|---|---|
| Send emails, resize files, run one independent report | Celery |
| Parallel embedding jobs with a final aggregation step | Celery Canvas may be sufficient; use Temporal if the whole pipeline needs durable lifecycle state |
| Multi-stage RAG ingestion with retries, reprocessing, and operator intervention | Temporal when recovery and process visibility justify the extra service |
| Agent job that waits for approval or an external event for hours or days | Temporal |
| One short LLM call behind an API endpoint | Neither may be necessary; use a normal request if latency and reliability permit |
| Existing stable Celery platform with manageable workflows | Keep Celery; do not migrate for novelty |
For a RAG system, Temporal is useful for orchestrating ingestion and reprocessing, while your vector store and evaluation framework remain separate components. Our guides on incremental local RAG and RAG evaluation with Ragas, DeepEval, and Phoenix cover those other layers.
Limitations and caveats
- The Temporal example is an integration pattern, not a complete application:
get_document_textandupsert_summarymust be implemented in your repository. - The sample uses the OpenAI Responses API. A local model or another provider can be substituted inside the Activity without changing the Workflow orchestration layer.
- Retry counts, timeouts, model names, and quotas must be chosen from your provider account and workload measurements. No benchmark or throughput claim is made here.
- Temporal adds a durable service, Worker deployments, versioning discipline, and operational learning. It is not automatically cheaper or faster than Celery.
- Celery's delivery and redelivery behavior depends on broker, acknowledgement, and worker-loss configuration. Do not interpret late acknowledgement as an exactly-once guarantee.
FAQ
Does Temporal replace Celery?
Not universally. Temporal is a stronger fit for durable multi-step processes; Celery remains a practical task queue for many applications. Keep Celery when it already solves the problem reliably.
Can I use Temporal with FastAPI?
Yes. FastAPI can authenticate a request, validate it, start a Workflow, and return HTTP 202 with the Workflow ID. A status endpoint can query the Workflow or read a separate application record. Do not hold the HTTP request open for a long-running model job.
Does Temporal run my Python code?
No. Your Workers run the Workflow and Activity code. The Temporal Service stores execution history and coordinates tasks, retries, timers, and messages.
Can I use Celery for human approval?
Yes, but you normally need to persist the approval state yourself and schedule or enqueue the next task when the user responds. Temporal provides Signals and Updates as first-class Workflow messages, which can make long-lived waits easier to model.
Is Temporal exactly once?
No external side-effect system should be treated as exactly once just because an orchestrator retries work. Activities may execute more than once. Design writes and external calls to be idempotent and reconcile ambiguous outcomes.
Can I use a local LLM instead of a hosted API?
Yes. Keep the provider call inside the Activity and replace the OpenAI adapter with an Ollama, vLLM, or other local inference client. The Workflow does not need to know which model serves the request. Add GPU-aware concurrency limits to avoid overwhelming the inference server.
Official documentation and references
- Temporal: What is Durable Execution?
- Temporal Python SDK developer guide
- Temporal Activities, retries, and idempotency
- Temporal Signals, Queries, and Updates
- Temporal Python Workflow sandbox
- Temporal payload codecs and encryption
- Temporal production deployment options
- Celery 5.7 task, acknowledgement, and retry documentation
- Celery Canvas: chains, groups, and chords
- OpenAI Python SDK reference
- Temporal Python SDK repository
Bottom line: Celery helps you run background tasks. Temporal helps you make a multi-step process survive failures, long waits, and worker restarts. Start with Celery for simple jobs; choose Temporal when durable orchestration is the actual requirement.
