A Job Queue Is a Command With a Tracking ID

Every API eventually grows a “do this later” requirement, and the .NET answers span a spectrum: Task.Run (a prayer), IHostedService plus a table you invented (a weekend), Hangfire/Quartz (a dependency and its dashboard), or a message broker (an operations team). FastEndpoints' job queues sit in an interesting spot on that spectrum: the queueing semantics are the library's, the storage is entirely yours, and the unit of work is — once again — the ICommand from Part 11. Same command, third execution mode: in-process now, remote now (Part 12), or persisted for later.

The developer contract

Turning a command into a queueable job is a marker interface and a different verb:

public class GenerateStatement : ICommand, ITrackableJob<JobResult<string>>
{
    public Guid CustomerId { get; init; }
    public Guid TrackingID { get; set; }        // set by the queue
}

// enqueue instead of execute:
var trackingId = await new GenerateStatement { CustomerId = id }
    .QueueJobAsync(executeAfter: DateTime.UtcNow.AddMinutes(5),
                   expireOn: DateTime.UtcNow.AddHours(2), ct: ct);

The handler is the same ICommandHandler you would write anyway. What you must supply is storage — an implementation of IJobStorageProvider<TStorageRecord> with methods shaped like “store this record”, “give me the next batch of pending jobs for this queue”, “mark complete”, “purge stale”. The record type is yours too (EF entity, Mongo document, Dapper row) as long as it carries the IJobStorageRecord fields: queue ID, tracking ID, execute-after, expire-on, completion flag, and the serialized command. The library ships no storage — which I count as a feature: your jobs live in your database, in a table you can query, index, and back up with everything else. (Community packages cover MongoDB and EF Core if you want them.)

The part worth reading: how the worker sleeps

Job systems die by polling — a fleet of workers hammering SELECT ... WHERE pending every second, forever, mostly finding nothing. The state machine in JobQueue.cs is the most careful anti-polling design I have seen in a small library, and the enum doc comments narrate it:

// Src/JobQueues/JobQueue.cs (FastEndpoints, MIT)
enum QueueState
{
    /// Non-distributed queue that hasn't seen any jobs yet. Blocks indefinitely
    /// on the semaphore (no DB polling). Initial probe fetch skips ExecuteAfter
    /// filtering to discover any existing jobs.
    Dormant,

    /// Queue has discovered or received jobs. Polls the DB periodically
    /// with _semWaitLimit timeout.
    Active
}

One queue exists per command type (queue ID = a hash of the type name), each with its own executor task and its own SemaphoreSlim. A queue that has never seen a job blocks on the semaphore indefinitely — zero database traffic. Local QueueJobAsync releases the semaphore (storage write first, then wake); the worker drains batches up to a concurrency limit (default: processor count, configurable per queue along with execution time limits). Once Active, the semaphore wait gains a timeout — the periodic re-check that catches scheduled executeAfter jobs coming due, and, in distributed mode (DistributedJobProcessingEnabled on your provider — multiple instances sharing one store), jobs enqueued by other instances, which is why distributed queues start Active instead of Dormant. The initial probe deliberately ignores executeAfter so a restarted app immediately discovers work persisted before the crash.

It is a design you can hold in your head: wake on local signal, poll only when there is reason to believe work exists, own the trade-off knobs. And because each command type is its own queue, a flood of cheap SendEmail jobs cannot starve GenerateStatement — per-type isolation you usually have to configure explicitly elsewhere.

Failure, retry, expiry — the sharp edges

Here the library makes you read the fine print, and I respect that it has fine print rather than vague promises:

• Retry is infinite by default. A failing job is re-attempted after a delay (default 5s, configurable) until it succeeds or expires. There is no built-in max-attempt counter — attempt counting, poison-job quarantine, and backoff policy belong to you (your storage record can carry an attempt count; your provider's fetch can filter on it). Miss this and one poisoned job retries for four hours.
• Everything expires. Default expiry is 4 hours after enqueue; expired jobs stop being picked up and are handed to your provider's stale-job purge. The CreateJob guard clauses insist on UTC dates and on expireOn > executeAfter — small validations that kill whole categories of scheduling bugs at enqueue time.
• At-least-once, not exactly-once. A job completed but not yet marked complete (crash between the two) will run again. Handlers must be idempotent — the same rule every queue in the world imposes, but worth saying because the API's friendliness can lull you.

Cancellation got real engineering attention. JobTracker<TCommand>.CancelJobAsync(trackingId) must handle a job that is pending (mark it in storage), executing right now (signal its CancellationTokenSource), or racing between states — and the source walks a persist-before-signal sequence with explicit rollback commentary so the executor can't re-pick a job that storage still considers pending while its token fires. Concurrency dances like this are exactly where homegrown job tables go wrong; reading this method is a free education in why.

Progress and results ride the same tracker: a command declaring ITrackableJob<JobResult<T>> can report percent-complete from inside the handler and expose a final result, which an endpoint can poll by tracking ID — the classic “202 Accepted + status URL” API pattern with the plumbing already built (your provider implements IJobResultProvider to persist results; the dispatch code type-checks the result type and throws a named error on mismatch — crash-early again).

Where it sits on the spectrum

Compared to Hangfire: no dashboard, no cron scheduling (one-shot delays only — pair a queue with a trivial IHostedService timer if you need recurrence), no storage included. In exchange: your schema, your queries, no dependency lock-in, per-command-type isolation, typed commands instead of serialized method-call expressions (Hangfire's expression serialization is a versioning trap I have been bitten by — the VersionKit series exists partly because of it). Compared to a broker: nothing to operate, but also no cross-language consumers and throughput bounded by your database.

The fit, in one sentence: APIs that need durable, retried, trackable background work measured in thousands-per-hour, not millions-per-minute — statement generation, exports, webhook fan-out, provisioning steps. For that band, a job queue whose entire storage story is one interface over your existing database is the right amount of machinery.

We have now covered everything FastEndpoints does at runtime. The next two parts are about confidence and constraints: first the testing story — route-less typed integration tests, fixture lifecycles, and unit-testing endpoints without HTTP (Part 14, Route-less Tests) — and then the build-time story, where source generators replace reflection so all of this survives Native AOT (Part 15).