Kyu — durable job queue for Go

postgres + redis

Two backends. One job queue. Zero coordination code.

Kyu runs five concurrent subsystems after Start(). They don't talk to each other — each one talks to Postgres or Redis. That shared state is the coordination layer.

architecture.svg960×360 · 12×6 grid

Postgres owns the schedule and the history. Redis only holds a sorted set of IDs scored by priority — it's a disposable index, not a source of truth.

Worker pool

engine 01 · continuous

Pops the highest-score ID from Redis, fetches the full record from Postgres, runs your handler, and writes the outcome back. Claims are guarded by optimistic locking: a stale worker can never clobber a newer one.

Scheduler

engine 02 · every 5s

Asks Postgres which scheduled and backoff-delayed jobs are now due, then pushes their IDs into Redis. Postgres holds the schedule; Redis just gets told when to run things.

Stale reaper

engine 03 · every 60s

Resets jobs stuck in running past StaleJobTimeout — the workers that claimed them crashed. The job comes back as pending and runs again.

Orphan reaper

engine 04 · every 60s

Re-queues pending jobs that went missing from Redis — a worker died between popping and claiming, or the sorted set was cleared. Nothing waits forever.

Metrics server

engine 05 · on MetricsPort

Exposes a Prometheus /metrics endpoint the moment you set a port. Each queue instance owns a private registry, so multiple instances in one process never collide.

Ledger + dispatcher

the core split

Every state change is a Postgres write you can SELECT. Redis only answers "what runs next" at microseconds. You get relational durability and sorted-set scheduling without either one doing the other's job.

ledger · postgres

Every transition is a row you can query.

pending → running → completed. Or failed → re-queued with backoff, then dead. Each change is an update in a normal Postgres table — no vendor API, no hidden state, just SQL.

lifecycle.svg960×360 · 7 states

status.tsv7 rows · postgres

status	means	who changes it
pending	waiting to run, or its scheduled time hasn't arrived	enqueue, scheduler, reapers
running	claimed by a worker via optimistic lock	worker pool
completed	handler returned nil	worker pool
failed	handler returned an error, retries remain	worker pool → scheduler (backoff)
dead	handler failed, retries exhausted — kept forever	worker pool
cancelled	cancelled before it ran	CancelJob()

postgres + redis

Operable in production, not just in a demo.

Kyu ships the tooling real queues need — dead letters, a dashboard, a CLI — as part of the library rather than as add-ons you discover you need later.

Durable by default

f01

Redis is a cache; Postgres is a database, and Kyu treats them accordingly. Clear Redis entirely — your jobs are still there, still scheduled, still run.

Priority queues

f02

Priority maps to the Redis sorted-set score. Workers always pop the highest score first — process_payment at 10 beats send_email at 1.

Scheduled jobs

f03

Pass a ScheduledAt time and the scheduler promotes it when the clock catches up. No cron process, no second service.

Retries + exponential backoff

f04

Fail once → wait 1s. Twice → 2s. Three → 4s. Your downstream service gets a chance to recover instead of absorbing a retry storm.

Dead-letter queue

f05

Jobs that exhaust retries stay dead forever, queryable by SQL. Inspect them, Retry() one, RetryAllDead() the lot onto a dedicated queue, or purge.

Optimistic locking

f06

Workers stamp locked_by when they claim a job. Every transition is guarded by that stamp, so a worker with a lost claim can't clobber the current owner.

Middleware + panic recovery

f07

Wrap every job in logging, timing, or auth middleware. Panics in handlers are caught and recorded, not propagated into the worker.

Batch enqueue

f08

EnqueueMany writes a batch as one COPY into Postgres and one ZADD into Redis — atomic, fast, and IDs come back in input order.

Prometheus + Grafana

f09

Set MetricsPort and /metrics is live. The included docker-compose provisions Grafana with a prebuilt dashboard — queue depth, throughput, failures by type.

RunOnce cron mode

f10

Drains the queue and exits with code 0. Wire it to a Kubernetes CronJob or crontab — no long-running process needed for batch work.

Inspect · cancel · reset

f11

Inspect(id), CancelJob(id), Reset(id), Purge(status), Pause()/Resume(), Stats(), filtered + paginated ListJobs(). All of it, in the library.

Completion webhooks

f12

Set CallbackURL and Kyu POSTs {job_id, status, payload, error} on completion. Fire-and-forget with a 10s timeout — callbacks never slow down processing.

Scope — deliberate boundaries

f13

Handlers are plain Go functions compiled into your binary — no DSL, no sidecar, no sandbox; they share your toolchain and deploy pipeline. Kyu is a job queue, not a workflow orchestrator: no DAG engine, jobs are independent units of work, and multi-step sequences are expressed in application code (each handler enqueues the next job).

postgres + redis

From zero to processing in minutes.

The jobs table and its indexes are created by embedded goose migrations on the first Connect(). No code generation step, no extra tooling.

quickstart.gogo 1.22+

go get github.com/codetesla51/kyu

# the CLI ships as its own command
go install github.com/codetesla51/kyu/cmd/kyu@latest

q := kyu.New(kyu.Config{
    DSN:         "postgres://user:pass@localhost:5432/mydb?sslmode=disable",
    RedisAddr:   "localhost:6380",
    Workers:     5,
    MetricsPort: 9090,
})

q.Register("send_email", func(ctx context.Context, payload string) error {
    log.Printf("sending email: %s", payload)
    return nil
})

ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()

if err := q.Connect(ctx); err != nil { log.Fatal(err) }
if err := q.Start(ctx); err != nil { log.Fatal(err) }

// run immediately, processed by priority
jobID, err := q.Enqueue(ctx, "send_email", `{"to":"user@example.com"}`, kyu.EnqueueOptions{
    MaxRetries: 3,
    Priority:   1,
})

// scheduled for later
at := time.Now().Add(1 * time.Minute)
jobID, err := q.Enqueue(ctx, "process_payment", `{"order_id":"123"}`, kyu.EnqueueOptions{
    MaxRetries:  5,
    Priority:   10,
    ScheduledAt: &at,
    TimeOut:     10 * time.Second,
})

// batch, atomically
ids, err := q.EnqueueMany(ctx, []kyu.EnqueueRequest{
    {JobType: "a", Payload: `{}`, Options: kyu.EnqueueOptions{Priority: 1}},
    {JobType: "b", Payload: `{}`, Options: kyu.EnqueueOptions{Priority: 2}},
})

CLI included

one binary

kyu serve · workers + dashboard
kyu enqueue send_email '{"to":"u@ex.com"}'
kyu inspect <job_id> · kyu version

One command stack

docker compose

docker compose up --build
→ dashboard :8080 · metrics :9090
→ grafana :3000 (admin/admin)

Queue isolation

QueueName

Give each app a unique QueueName and its Redis sorted set is separate. Apps sharing a name compete for the same jobs — that's how multi-worker deployments scale.

ops

Watch the queue without leaving your browser.

kyu serve starts a web dashboard by default. Live stats stream over SSE, and every management action — retry dead jobs, purge a status, pause workers, enqueue from the UI — is a click away. It's embedded in the binary; nothing extra to install.

kyu dashboardkyu serve · :8080

Kyu dashboard: queue depth, per-status job totals, worker state, and a live jobs stream

live overview · jobs stream (SSE) · dead-letter actions · pause/resume

metrics.json/metrics · prometheus

metric	type	what it tracks
kyu_jobs_total	counter	total jobs ever submitted
kyu_jobs_processed_total	counter_vec	completed jobs, labelled by status
kyu_job_failures_total	counter_vec	failures, labelled by job_type
kyu_jobs_dead_total	counter	jobs that exhausted all retries
kyu_queue_depth	gauge	jobs currently waiting in Redis

measured

Fast where it should be, honest about the rest.

Dispatch overhead is under a microsecond — but that's not the ceiling. Cross-layer load testing with Barrage found the real bound is Postgres write latency, which is exactly where durability comes from.

dispatch.benchgo test -bench

dispatch benchmark	cost	allocs
register	~52 ns/op	0
execute	~950 ns/op	5
execute + middleware	~1.2 µs/op	7
execute (parallel)	~600 ns/op	5

barrage.json60s · 300 enq/s

barrage load test	rate	p99	success
HTTP enqueue	251.5/s	275ms	100%
Postgres	749.9/s	966ms	100%
Redis	1250/s	41.9ms	100%

Barrage drives HTTP, Postgres, and Redis on one clock and correlates them by time bucket — it shows cross-layer behavior under simultaneous load, not a per-job causal trace. 60s run, 20s ramp, 300 enqueues/s, 900 DB ops/s, 1500 Redis cmds/s. Full investigation log: github.com/codetesla51/kyu → benchmarks/README.md

postgres + redis

Neither Redis-only nor Postgres-only. Both.

Redis-only queues are fast until they lose the queue. Postgres-only queues are durable but scan for what to run next. Kyu deliberately splits the difference: Postgres is the ledger, Redis is the dispatcher.

comparison.svg960×360 · 2 queues → 1

comparison.csvkyu · asynq · river · machinery · bullmq

feature	kyu	asynq	river	machinery	bullmq node
storage	Postgres + Redis	Redis only	Postgres only	Redis / AMQP / Mongo	Redis only
job durability	survives Redis wipe	lost on flush	full	depends on backend	lost on flush
priority scheduling	sorted set		numeric (1–4)	—	sorted set
transactional enqueue		—		—	—
full job history	SQL	TTL-limited	SQL	limited	TTL-limited
stale reaper		limited		—	limited
orphan reaper		—	—	—	—
prometheus native		separate	—	—	separate
scheduled jobs
middleware			—
retries + backoff	exponential	exponential	exponential	basic	exponential
license	MIT	MIT	MPL	MIT	MIT