Temporal Learning Notes

1. The big picture (3 roles)
2. Workflow vs Activity
3. Triggering: who calls run()?
4. Why name-based decoupling is powerful
5. Durability & replay
6. Anatomy of an event history
7. Task queues under the hood
8. Ordering
9. Workflow ID reuse
10. Testing tiers
11. Temporal ≈ AWS Step Functions

1. The big picture — three roles

Temporal is a client–server system. Your code never runs on the server; the server is a coordinator with a durable memory. Three roles talk only through the server.

Client triggers → server schedules & persists → worker pulls, runs, reports.

One sentence: Your code runs on your workers, always. The server stores state and hands out work; it never executes your code.

2. Workflow vs Activity

	Workflow	Activity
Role	Orchestrator — decides what & in what order	The actual side effects (I/O, API calls)
Code shape	a class, `@workflow.defn` / `@workflow.run`	a function, `@activity.defn`
Rule	Must be deterministic — no I/O, no clocks, no randomness	May be non-deterministic; retried on failure
Why	the server replays it to recover, so it must be reproducible	I/O is non-deterministic, so it is quarantined here

The workflow never does the work itself — it calls workflow.execute_activity(book_flight, ...), a durable, replay-safe RPC.

Workflow = the brain. Activities = the hands. Cert-rotation analogue: CertRotationWorkflow orchestrates; copy_cert / restart_redis do the SSH.

3. Triggering — who calls run()?

Never you, directly. Calling run() yourself is just plain Python — no durability. The worker calls it, after a client triggers it.

client = await Client.connect("localhost:7233")
handle = await client.start_workflow(
    "TripBookingWorkflow",        # which workflow (by NAME)
    args=[{...}],                  # argument to run()
    id="trip-Ada",                # unique workflow id
    task_queue="trip-booking",    # must match the worker's queue!
)

Chain: client.start_workflow → server records "Started" + schedules a task → worker polling that queue picks it up, instantiates the class, calls run(), reports back.

task_queue is the matchmaker. If the client's queue doesn't match a worker's polled queue, the workflow sits scheduled but never runs (classic gotcha).

4. Why name-based decoupling is powerful

There are two coupling boundaries, and they differ:

Pair	Coupled by	Shared code?
Worker ↔ workflow definition	direct import (the class)	Yes — ship together
Client/trigger ↔ worker	name string + task queue	No — just agree on a name

Our booking-api proves it: it triggers "TripBookingWorkflow" by string and imports none of trip-booking's code.

The aha: A control-plane API can trigger heavy workflows without bundling their logic. The REST API calls start_workflow("CertRotationWorkflow", task_queue="cert-ops") carrying no SSH/cert code; a separate worker fleet (placed near the Redis hosts) owns the real work. They share only a name + queue — so the API scales for request load, workers scale for job load, and you redeploy worker logic without touching the API.

5. Durability & replay

Why a database? It is the feature. Every step appends an event to the workflow's durable history. On crash, a new worker replays that history: it re-runs the workflow code, but substitutes recorded results for already-completed activities, then resumes at the first un-recorded step.

The activity is atomic; the workflow is checkpointed. Every activity boundary is a durable save-point. There is no mid-activity resumption — a crash inside an activity re-runs the whole activity. So: make activities idempotent, and choose activity granularity by what you're willing to repeat.

This is why workflow code must be deterministic: replay must produce the identical sequence of decisions, or it won't line up with history (→ a non-determinism error).

6. Anatomy of an event history

Our first run (trip-Ada → FLIGHT-SEA-JFK) produced 11 events — more than expected. The reason: a Workflow Task (run the workflow code to decide) is different from an Activity Task (run the activity to do). Workflow code runs in short bursts, one per decision point.

One Workflow Task per "the workflow needs to think again" — here, twice.

Generalizes: flight→hotel→car (3 activities) ≈ 4 Workflow Tasks + 3 Activities ≈ ~4 events per activity. This is why histories have size limits. (Also seen: sticky queues route later Workflow Tasks back to the same worker for cached replay.)

7. Task queues under the hood

A task queue is not a message broker (no Kafka/Rabbit/SQS). It's a logical routing label, served by the horizontally-scalable Matching Service, durably backed by the database.

How it scales (no Kafka in dispatch)

Partitioned queues: one logical queue = N partitions across Matching nodes. Scale a hot queue by adding partitions.
Sharded History: by workflow ID (e.g. 512/4096 shards). Frontend/Matching/History scale horizontally.
The database is the real ceiling — high scale → Cassandra over single Postgres.
Many use cases = many task queues on one cluster.

Kafka asterisk: only the optional Visibility / advanced-search path (Elasticsearch, sometimes a Kafka buffer). Never in task dispatch or correctness.
Caveat: exact partition/shard defaults are version-specific.

8. Ordering

Task queues are NOT ordered (not guaranteed FIFO). r-101 enqueued before r-102 can still finish after it — due to partitioning, multiple workers, retries, or variable durations. Queues trade ordering for scale.

Ordering that matters lives in workflow identity, not the queue:

Within one workflow ID: strictly, deterministically ordered (one History shard, one append-only log).
Across workflow IDs: no guarantee, by design.

When you need ordering

Need	Pattern
Strict order across N requests	signals to a single long-running workflow (a serializer)
Per-entity serialization	workflow_id = entity key (e.g. `rotate-cluster-A`)
Order irrelevant	default fire-and-forget (fastest)

9. Workflow ID reuse

Workflow ID = logical identity (reusable). Run ID = the specific execution (always unique). We verified live: re-POSTing "Ada" after trip-Ada completed was accepted, giving a new run ID — two executions under one workflow ID.

Default (AllowDuplicate): an ID is rejected only while one is still running; once closed, it can be reused.

Policy	Behavior
AllowDuplicate (default)	reuse freely once previous is closed
AllowDuplicateFailedOnly	reuse only if previous failed
RejectDuplicate	never reuse, even after close
TerminateIfRunning	kill a running one, start the new

10. Testing tiers

	Workflow test	API test
How	`WorkflowEnvironment` — ephemeral in-process server	mock the client via `dependency_overrides` (≈ Spring @MockBean)
Needs a server?	spins up its own, throwaway	none — fully isolated
Tests	the workflow really produces the result	the API translates request → trigger, returns 202

Both are build-time / unit tests: run once per build, carry their own (or no) infra. The Hydra-style post-deploy gating tests (point at a real deployed stage, gate promotion) are a separate tier we haven't built.

Lesson: the API test first "passed" only because the Docker server happened to be up — a false pass. Inject the dependency and override it: a test must pass for the right reason.

11. Temporal ≈ AWS Step Functions

Step Functions is "workflow-as-configuration, fully managed"; Temporal is "workflow-as-code, you run the workers."

	Temporal	Step Functions
Workflow	real code (Python/Go/Java)	state machine (declarative ASL JSON/YAML)
Activity	your code on your workers	Lambda-backed task states
Compute	you run the worker fleet	fully managed by AWS

Historical note: AWS SWF was co-created by the people who later built Cadence → Temporal — so Temporal is a lineage descendant of AWS's original workflow service. (Step Functions also has an "Activities" mode with a polling worker, but default usage is Lambda tasks with no worker to manage.)

Contents