Cancellation and shutdown

Make operations cancellation-aware

Cancellation starts cooperatively. A resident task must observe TaskContext:

use taskvisor::{TaskContext, TaskError};

async fn do_work() -> Result<(), TaskError> {
    // Application work goes here.
    Ok(())
}

async fn run_one_operation(ctx: &TaskContext) -> Result<(), TaskError> {
    ctx.run_until_cancelled(do_work()).await?
}

async fn run_with_more_branches(ctx: &TaskContext) -> Result<(), TaskError> {
    tokio::select! {
        _ = ctx.cancelled() => Err(TaskError::Canceled),
        result = do_work() => result,
    }
}

run_until_cancelled drops the wrapped future when cancellation wins. Cancellation wins a tie. An already-cancelled context does not poll the wrapped future. Use it only when dropping that future is a safe way to cancel the exact operation. Check the operation's cancellation-safety contract. An external commit, acknowledgement, or partly consumed input may need an explicit protocol. The Tokio sleep in graceful_worker.rs is a simple drop-safe example. See the context implementation for cancellation priority and child scopes.

Know which deadline you are setting

Taskvisor deadlines cover different waits:

API or setting	What it bounds	What expiry means
TaskSpec::with_timeout or TaskDefaults::with_timeout	One attempt after Task::spawn returns its future.	Cancel the attempt context and destroy the future. The timeout is retryable under policy if cleanup succeeds.
add(spec).ownership_timeout(d).execute()	Ownership admission before registry command commit.	Return RuntimeError::OwnershipAdmissionTimeout. Start no task and publish no lifecycle event for the request.
submit(request).ownership_timeout(d).execute()	Ownership admission before controller command intake.	Return ControllerError::OwnershipAdmissionTimeout. Later queues, admission, and execution are outside this deadline.
cancel(target).termination_timeout(d).execute()	This caller's terminal wait after the registry claim.	Return RuntimeError::TaskTerminationTimeout. The cancellation request remains active.
with_grace for remove or cancel	One registered task's cooperative stop after a new claim.	Request abort and commit ForceAborted. cancel(target).execute().await still returns its claim boolean.
with_grace for shutdown	One shared drain of tasks and pending removals.	Return RuntimeError::GraceExceeded if work did not drain in time. Commit force-abort where needed.
with_subscriber_shutdown_timeout	The separate shared drain of subscriber queues.	Drop queued events after the deadline. A callback already running cannot be interrupted.

Both ownership-admission timeouts happen before command intake. Neither starts work or publishes a lifecycle event for the request. See management intake for the ownership timeout modifiers. These deadlines are not interchangeable. None rolls back external side effects. Timers cannot interrupt synchronous Rust code in the middle of a poll.

Understand attempt timeouts

The attempt runner starts the timeout after Task::spawn returns its future. On expiry, it cancels the attempt context and drops the future. It does not poll the future again to run async cleanup or undo side effects. A blocking future destructor can delay attempt release beyond the configured timeout.

Stop one registered task

remove(target).execute() requests a stop and returns its claim boolean without waiting for terminal cleanup. cancel(target).execute() waits until registry membership is removed and the final outcome is committed. Both use the configured grace period when they create a new stop claim for registered work.

If the task does not stop within grace, Taskvisor requests abort and commits ForceAborted. A plain cancel(target).execute() can then return Ok(true). It does not return GraceExceeded for this task. A cancellation operation that joins an existing removal waits for the same cleanup and returns Ok(false). Use a TaskWaiter to learn the task's final outcome; the boolean reports only who made the stop claim.

The registry stop commands create or join the claim. The join reporter applies the task's grace period.

Cancel work with a caller deadline

The termination_timeout(duration) modifier limits how long the caller waits for registered task cleanup. Controller ordering, command-queue admission, and the registry claim happen outside that timer. A timeout stops this caller's wait. It does not undo cancellation or change the task grace period. If task completion is observed at the timeout boundary, completion wins. Queued controller work is removed directly, and the termination timeout does not apply to that path. A watched queued submission then resolves to Rejected with RejectionKind::RemovedFromQueue, not to Canceled. The independent fail_fast() modifier makes command-queue admission fail fast. The remaining behavior is unchanged. If the caller already holds a TaskWaiter, a TaskTerminationTimeout neither consumes nor cancels it. The waiter can still deliver the eventual terminal outcome. See the caller-wait implementation.

Join shutdown

handle.shutdown().await joins one shared shutdown workflow. It affects the whole runtime and every handle clone. Concurrent and later callers receive the same cached result.

The workflow has concurrent parts:

• It closes admission and signals shutdown.
• It waits for earlier committed registry commands before starting the task grace window.
• The registry requests cancellation and spends one shared grace period on tasks and pending removals.
• The controller rejects pending submissions as its loop exits. This can overlap the registry grace period.
• Taskvisor joins runtime and controller cleanup, then drains subscriber queues up to their separate deadline.

If task cleanup exceeds grace, shutdown reports RuntimeError::GraceExceeded and force-aborts work as needed. The shutdown coordinator owns the shared result. The cleanup workflow owns the drain order and deadlines.

After a shutdown future is first polled, dropping that caller's future does not stop the detached workflow. Taskvisor schedules that workflow on the Tokio runtime that successfully started the supervisor. Polling shutdown from another runtime does not transfer cleanup ownership to it. The startup runtime must remain alive and driven until the shared operation completes. Its return does not prove that force-aborted synchronous code, a running subscriber callback, or a user destructor has finished. Dropping the final public owner can request cancellation, but a destructor cannot await cleanup or report errors.

Separate logical completion from physical release

This diagram follows one watched, registered task after cancellation reaches the registry. It shows a cooperative stop and one possible force-abort path. Controller routing and queued work are not shown.

sequenceDiagram
accTitle: Cancel one watched task
accDescr: Cancellation separates the logical outcome, physical actor exit, and deferred destruction of retained values.
participant Caller
participant Registry
participant Task as Task actor
participant Waiter as TaskWaiter
participant Cleanup as Deferred cleanup

Caller->>Registry: cancel(id).execute()
Registry-)Task: Request cancellation
alt Task returns Canceled within grace
    Task-->>Registry: Physical actor exit
    Registry->>Registry: Remove membership
    Registry-)Waiter: Canceled
    Registry-)Cleanup: Retained user values
else Grace expires while physical code still runs
    Registry-)Task: Request abort
    Registry->>Registry: Remove membership
    Registry-)Waiter: ForceAborted
end
par Caller result
    Registry-->>Caller: Ok(claimed)
and Remaining cleanup
    opt Force-abort path shown above
        Task-->>Registry: Physical exit observed
        Registry-)Cleanup: Retained user values
    end
    Cleanup->>Cleanup: Destroy values
end

The force-abort branch shows logical completion before physical exit. Physical exit can also happen earlier. Final destruction and the caller receiving its result have no fixed order. The terminal commit removes membership and sends the outcome. The attempt reaper retains force-aborted actors until physical release.

Taskvisor exposes separate views for these stages:

Question	Interface
What final logical result did watched work reach?	TaskWaiter and TaskOutcome
Is the task still registered?	list
Is a physical attempt still active?	alive_snapshot and is_alive
Are user values retained or awaiting destruction?	ownership_snapshot

After ForceAborted, a task can be absent from list while its name remains in alive_snapshot. After physical exit, the name can leave alive_snapshot while ownership is still in use. Its ownership unit stays charged until the retained user values finish isolated destruction. A destructor panic permanently retires that unit from finite capacity instead of releasing it.

These snapshots can become stale immediately. They are not one atomic per-task record. See Configure Taskvisor for the ownership fields and Final outcomes and lifecycle events for outcome semantics.