Final outcomes and lifecycle events

Choose the result path

Taskvisor has two result paths with different contracts:

Path	Contract	Use it for
TaskWaiter and TaskOutcome	One direct final result, outside the event bus.	Application decisions.
Subscribe and Event	Best-effort bounded delivery.	Logs, metrics, traces, and live status.

A watched outcome is independent of event loss while the process and runtime remain alive. It is not durable storage. TaskWaiter::wait can return OutcomeUnavailable if its completion channel closes unexpectedly. Dropping the waiter does not cancel the task.

flowchart TB
accTitle: Final outcomes and event delivery
accDescr: Direct final outcomes bypass the event bus and the bounded subscriber queues.
Runtime["Runtime and controller"]
Waiter["TaskWaiter"]
Bus["Event bus: drop oldest when full"]
Relay["Event relay"]
Queues["Subscriber queues: drop incoming when full"]
Callbacks["Subscriber callbacks"]
Runtime -->|"direct final outcome"| Waiter
Runtime -->|"lifecycle events"| Bus
Bus --> Relay
Relay --> Queues
Queues -->|"serial per subscriber"| Callbacks

The event path has two separate loss points. Neither controls the direct outcome channel.

Separate API errors from task outcomes

An API error means the call did not return its documented success value. It does not prove that no command or state change was committed. Check the contract of that method. A TaskOutcome reports how watched work finally ended.

flowchart TB
accTitle: API errors and final task outcomes
accDescr: A watched add and its waiter return separate results at separate boundaries.
Add["add(spec).watch().execute().await"]
NoWaiter["No watched registration returned"]
Wait["waiter.wait().await"]
Unavailable["Result channel failed"]
Outcome["Work reached a final state"]
Add -->|"Err(RuntimeError)"| NoWaiter
Add -->|"Ok(waiter)"| Wait
Wait -->|"Err(OutcomeUnavailable)"| Unavailable
Wait -->|"Ok(TaskOutcome)"| Outcome

Boundary	Type
Checked configuration constructors and setters	ConfigError
BackoffPolicy::new	BackoffError
SupervisorBuilder::try_build	BuildError
Runtime lifecycle, management, wait, and shutdown	RuntimeError
Controller preparation and command intake	ControllerError
One task attempt	TaskError
Final result of watched work	TaskOutcome
Application code that combines runtime and controller calls	Error

SupervisorBuilder::build and Supervisor::new panic when checked construction would fail. Use try_build to handle typed construction errors.

Ok(TaskOutcome::Failed { .. }) means outcome delivery succeeded and the task ended in failure. It is not an API error. For controller work, submit(request).watch().execute().await? confirms command intake. The waiter can later deliver TaskOutcome::Rejected if slot or registry admission fails.

Handle final outcomes

Final outcomes distinguish:

Outcome	Meaning
Completed	The final attempt succeeded and the restart policy stopped the task.
Failed	Retryable failure stopped under policy or retry limit.
Fatal	The task reported a permanent failure.
Canceled	Cancellation was requested or reported.
ForceAborted	Taskvisor stopped waiting before the task stopped cooperatively.
Panicked	The actor or protected attempt-owned cleanup panicked before final outcome delivery.
Rejected	Admission rejected the work, or queued controller work was removed.

Use TaskOutcomeKind and RejectionKind for branching, metrics, and alerts. Treat reason strings as diagnostic text. A panic while polling task code becomes a retryable task failure instead of Panicked. Removing watched controller work while it is still queued or waiting for registry-command capacity produces Rejected with RejectionKind::RemovedFromQueue, not Canceled. Once work enters the registry, cancellation uses the normal task lifecycle even if the task is still waiting for a concurrency permit.

Taskvisor delivers the final outcome before deferred cleanup destroys the retained task object and actor result. That later destruction can block or panic. It cannot change an outcome already delivered through TaskWaiter. Failures on that later path are reported as best-effort runtime diagnostics, not a new TaskOutcome::Panicked.

Branch on a final outcome

Match typed variants for application decisions and use stable labels for telemetry. Keep a fallback arm because TaskOutcome is non-exhaustive.

use taskvisor::{TaskOutcome, TaskWaiter};

async fn report(waiter: TaskWaiter) -> Result<(), Box<dyn std::error::Error>> {
    let outcome = waiter.wait().await?;

    match &outcome {
        TaskOutcome::Completed => println!("completed"),
        TaskOutcome::Failed { reason, .. } => println!("failed: {reason}"),
        TaskOutcome::Fatal { reason, .. } => println!("fatal: {reason}"),
        TaskOutcome::Rejected { kind, .. } => println!("rejected: {kind:?}"),
        other => println!("ended: {}", other.as_label()),
    }

    Ok(())
}

Use TaskOutcome::kind or typed variants for classification, not reason strings.

Understand ForceAborted

ForceAborted normally follows the configured grace period. Last-owner fallback and signal-setup failure cleanup cannot wait for that period. The physical attempt can remain active until synchronous task code returns control to Tokio.

Treat events as observability

Choose the interface for the question:

Need	Interface
Application decision	TaskWaiter and TaskOutcome
Readable demo or small-tool logs	LogWriter with logging.
Structured service telemetry	TracingBridge with tracing.
Application-owned metrics	A custom Subscribe implementation
Registry membership	list
Physical attempt activity	alive_snapshot
Retained values and cleanup pressure	ownership_snapshot
Per-key admission state	controller_snapshot

The shared event bus and every subscriber queue are bounded. Events can be lost at the shared bus or in an individual subscriber queue. When the shared bus is full, it drops the oldest event and retains the newest one. When one subscriber queue is full, Taskvisor drops the incoming event for that subscriber. Callbacks run one at a time for each subscriber.

Choose shared or dedicated callback execution

Subscriber callbacks are synchronous and run outside Tokio worker threads. Choose the execution mode from the work done inside the callback:

• Use Shared, the default, for short, bounded, non-blocking work. Examples are an atomic counter as in custom_subscriber.rs, in-process metrics as in metrics.rs, or a non-blocking handoff.
• Use SubscriberExecution::Dedicated when a callback can block synchronously and must not delay other subscriber lanes. Override Subscribe::execution. Examples are standard-output writes in LogWriter and synchronous dispatch in TracingBridge.
• Do not perform async I/O, long blocking work, or substantial processing in either callback mode. Copy the required event fields and hand them to an application-owned bounded queue and worker.
• Do not use a subscriber for a result that application logic cannot lose. Use TaskWaiter instead.

All Shared lanes use one fixed library-owned OS worker. A blocked callback on that worker delays the other shared lanes. Each Dedicated subscriber adds one native thread and can run concurrently with the shared worker and other dedicated subscribers. Its own bounded queue can still overflow.

Dedicated is a blocking-isolation mode, not a faster callback mode. In one Taskvisor 0.9.0 reference run on Linux/aarch64 with 14 logical CPUs, the matched short-callback fan-out benchmark measured this complete lifecycle-and-delivery boundary:

• current-thread: one Dedicated subscriber at 55.407 K tasks/s, eight Dedicated at 6.434 K tasks/s, and eight Shared at 48.200 K tasks/s;
• multi-thread with four workers: one Dedicated at 48.960 K tasks/s, eight Dedicated at 9.126 K tasks/s, and eight Shared at 53.909 K tasks/s.

The run establishes that eight Dedicated subscribers were slower for this measured workload. It does not establish a portable ratio or that every dedicated callback workload is slower on every host. The benchmark measures 256 watched task completions plus matching TaskFinished delivery to every short-callback subscriber; it is not a callback-only microbenchmark. The fan-out benchmark contract defines the included and excluded work.

Taskvisor does not elastically add callback workers or retire them merely because they are idle. A callback still running when the shared shutdown deadline expires may continue on its detached worker after shutdown returns. Taskvisor does not wait for callback-worker thread-local destructors; they may also continue after shutdown returns. Shutdown deadlines can also drop events. SubscriberOverflow diagnostics report loss when possible.

Correlate events

Match Event.kind before reading its optional fields. The Event API and EventKind API list the fields for each event:

Field	Meaning
id	The task or submission ID, when present. Use it to correlate one submission across controller and runtime events.
task	The event's subject: usually a task name, but a slot name for controller events. Diagnostics may use a subscriber or runtime component name.
outcome_kind, rejection_kind	Typed final-outcome or rejection categories. Use their stable labels for telemetry.
seq	Process-local construction order, not the order of concurrent effects. It does not survive a process restart.

Do not treat task as a task ID or parse reason as a stable category. The metrics example uses a subject label for task and explains how to keep labels bounded.

See outcomes.rs, custom_subscriber.rs, logging.rs (requires logging), tracing.rs (requires tracing), and metrics.rs.

Source: final outcomes, event fields, event bus, event relay, and subscriber delivery.