smallage

Background tasks, cron and a dead letter queue on Redis Streams, with a Litestar integration. Smallage is the old name for wild celery.

Delivery is at-least-once. Handlers must be idempotent; a dedup key is provided for the cases where that is not naturally true. Treating delivery as exactly-once is a design error in the application, not a bug here.

Install

uv add smallage

The core — transport, worker, scheduler — never imports Litestar, and an import-linter contract keeps it that way, so smallage.core is usable behind another framework or none. That is a rule about imports rather than about installation: Litestar is an ordinary dependency, because the Litestar layer is half of what this package is for and an extra would stand in the way of nearly everyone installing it. Nothing obliges you to import that half.

Quickstart

from dataclasses import dataclass
from uuid import UUID, uuid4

from litestar import Litestar, post
from litestar.di import Provide

from smallage.litestar import QueueConfig, QueuePlugin, TaskRegistry

tasks = TaskRegistry()


@dataclass
class Settings:
    index_name: str = "documents"


def settings() -> Settings:
    return Settings()


@tasks.task
async def reindex(doc_id: UUID, settings: Settings) -> None:
    """`doc_id` is serialised; `settings` comes from the application."""


@post("/documents")
async def create() -> str:
    await reindex.enqueue(doc_id=uuid4())
    return "queued"


app = Litestar(
    route_handlers=[create],
    dependencies={"settings": Provide(settings, sync_to_thread=False)},
    plugins=[QueuePlugin(QueueConfig(registry=tasks, redis_url="redis://localhost"))],
)

Workers are the same application, started differently:

litestar workers run --queue high --concurrency 20

Two layers

• smallage.core — transport, worker, scheduler, retries, results. Depends on redis, msgspec, anyio. Importing Litestar from here is forbidden, and an import-linter contract enforces it.
• smallage.litestar — dependency injection, CLI, serialization, health, tracing.

Guides

• Design and invariants — the rules, and why each is a rule
• How this differs — against Celery, taskiq-redis and dramatiq
• Tasks — arguments, dependencies, what travels and what is injected
• Scheduling — delayed jobs, cron, missed occurrences
• Retries and the DLQ — the two counters, backoff, reading a buried job
• Priorities — queues, fairness, shards
• Broker mode — consuming streams somebody else writes
• Results and deduplication — waiting for an outcome, running once
• Testing — eager mode, assertions, a real worker in a fixture
• Operations — deployment, health, shutdown, depth
• API reference

Redis

Redis 7 or newer: depth comes from the consumer group's lag, which earlier versions do not report. Standalone, Sentinel and Cluster deployments are all covered by the test suite, against both Redis 7 and 8.