Python TaskGroups: make sibling failure and cleanup part of the design

First principles

Concurrent work is easier to reason about when its lifetime has a visible owner. An asyncio.TaskGroup makes the lexical block that creates child tasks responsible for waiting for them. When the block exits normally, it waits for its children. If a child fails with an exception other than cancellation, the group requests cancellation of unfinished siblings, waits for their cleanup, and raises the failure or failures as an exception group. The parent does not silently forget a still-running child.

This is different from launching tasks and keeping their handles in a list. A list does not define what to do when the first task fails, when the parent is cancelled, or when the function returns early. Those policies must be implemented explicitly. Structured ownership gives a default relationship between parent and children, but it does not decide whether partial results are useful, whether an operation should be retried, or whether side effects can be undone.

Cancellation in asyncio is cooperative. The runtime raises CancelledError at an await point, giving code a chance to release resources. A coroutine that catches cancellation and continues can outlive the intended scope. Cleanup belongs in finally blocks or context managers, and CancelledError should generally be re-raised after cleanup. Swallowing it can break timeout and shutdown behavior higher in the task tree. A TaskGroup waits for children to finish their cancellation cleanup; a child that never cooperates can delay that wait.

When multiple children fail close together, the group can surface more than one failure. Python's except* syntax handles matching leaves of an ExceptionGroup while preserving unmatched failures. This matters because handling one expected error class should not accidentally erase an unrelated programming error from another task. Keep exception classification close to the operation that can interpret it; a broad catch at the task-group boundary often loses useful context.

Runnable example

Run with Python 3.11 or newer. The first task fails, the sibling observes cancellation, releases its resource in finally, and the parent handles the expected exception type. This demonstrates ownership and cleanup, not a general retry policy.

import asyncio


async def load_profile():
    await asyncio.sleep(0)
    raise LookupError("profile is absent")


async def watch_updates():
    try:
        await asyncio.Event().wait()
    finally:
        print("update watcher released")


async def main():
    try:
        async with asyncio.TaskGroup() as group:
            group.create_task(load_profile())
            group.create_task(watch_updates())
    except* LookupError as failures:
        print(f"profile failures: {len(failures.exceptions)}")


asyncio.run(main())

The output includes the cleanup message before the failure summary because the group waits for the cancelled sibling to finish. If the child owned a response body, file, or lock, its finally block or async context manager is where that resource should be released. A TaskGroup does not make resource ownership implicit; it makes task lifetime easier to align with resource scope.

Contextual failure case

A service starts three concurrent calls to build a response. One call fails quickly, but the other two continue consuming sockets and database capacity after the response is already doomed. The handler returns the first exception, losing the task references. Later, these tasks log errors with no request context and may write results that the caller never uses. Under load, the “minor” orphan work compounds into pool exhaustion.

Wrapping related tasks in a group fixes the lifetime mismatch, but only if the children cooperate. If a child catches BaseException, suppresses CancelledError, and loops, the group cannot complete promptly. If the child delegates to a blocking thread or remote service, cancelling the coroutine does not necessarily stop the external effect. A deadline can stop waiting, but it does not retract a committed write. Use idempotency or reconciliation for ambiguous effects and a bounded executor for blocking calls.

Partial success also requires an explicit contract. If two independent lookups may fail while a third result remains useful, one group-wide exception may be too coarse. Catch a narrowly understood exception inside the individual task and return a typed result, or run independent groups with separate policies. Do not convert every exception into None: that makes a missing record indistinguishable from a timeout, permission failure, or programming defect.

Verification

Write tests around observable lifetime, not just return values. Use asyncio.Event objects to signal child startup and cleanup; avoid arbitrary sleeps as synchronization. Make one child fail, assert that a sibling enters its finally block, and assert the parent does not finish before cleanup completes. Add a separate test where the parent task is cancelled and verify the same resource release. Finally, make two children fail with different exception types and prove an except* clause handles only the intended type while the other remains visible.

For service incidents, log task name, request correlation ID, operation, and cancellation reason, but avoid logging full payloads or credentials. Compare task counts and active dependency operations before and after the handler exits. Python queue ownership covers the related distinction between a queue's completion counter and durable business success.

Exercise

Add a third task that raises ValueError while the profile lookup raises LookupError. Catch only the lookup failure with except*, and allow the validation error to remain visible. Then change the watcher so it delays cleanup using an event. Demonstrate that the task group waits for that event, and decide what operational deadline should bound cleanup. Explain why cancelling the parent cannot guarantee that a remote write was rolled back.

Safety and caveats

Task groups require Python 3.11 or later. Exception-group behavior is part of the language and runtime contract; frameworks may add their own cancellation scopes and shutdown rules. Do not shield work casually: shielding can intentionally let a task outlive its caller and therefore needs a new owner. A task group is not a transaction. It coordinates lifetimes and exceptions, while data consistency, duplicate prevention, and compensation remain application responsibilities.