mirror of
https://github.com/anthropics/skills.git
synced 2026-08-02 13:05:28 +08:00
Lance Martin
1f630fdf92
Syncs the claude-api skill with the current upstream source.
## Managed Agents — five new features
- `effort` on the agent's `model` object (a level string or `{"type": "<level>"}`).
It is agent-configuration only: setting it in a per-session `model` override is
silently ignored.
- Optional `version` on agent update, for optimistic concurrency. Omit it for
last-write-wins.
- `initial_events` on session create, collapsing create plus first send into one
call. Validation is all-or-nothing and only `user.message` and
`user.define_outcome` are accepted.
- Environment and memory-store webhooks: four `environment.*` events and three
`memory_store.*` events.
- Event deltas on per-thread streams. Previews are thread-scoped, so a child
thread's previews never reach the session-level stream.
## Corrections to behavior the skill already documented
- Tool output offload triggers at 100,000 characters (~25k tokens), not 100K
tokens, and covers built-in tools rather than MCP alone.
- `system.message` works on four models, checks only the primary model, appends
system context instead of replacing the prompt, and is accepted during a
`requires_action` idle when it trails a tool result in the same request.
- Vault-to-MCP credential matching is normalized (scheme and host lowercased,
default ports and trailing slashes stripped), not byte-exact.
- Multiagent depth greater than 1 is a validation error, not silently ignored.
- Outcome `interrupted` fires even when evaluation never started, and then
carries an empty-string `outcome_evaluation_start_id`.
- Deployment jitter is 15% of the run interval, floor 5s, cap 9 minutes.
- Webhooks retry three times with jittered 5-120s backoff, then drop silently.
Auto-disable is duration-based with three named triggers.
- `session.status_terminated` means completion or error.
- `agent.thinking` is a progress signal and carries no thinking content.
- `processed_at` is already populated on first sighting for
`user.define_outcome`, `user.custom_tool_result`, and `user.tool_result`.
- Session creation does not provision the sandbox.
- Skill `version` applies to Anthropic-authored skills too.
- Console-created vault credentials are header-injection only, and vault
environment-variable substitution skips secrets in URL paths.
- Console trace URLs need the real workspace ID when the API key is not in the
Default workspace.
## Elsewhere in the skill
- Note partner pricing under the first-party price table: Microsoft Foundry
bills at standard API rates, while Amazon Bedrock and Vertex AI are
partner-operated with separate pricing.
- Correct the tool-runner human-in-the-loop guidance, which previously pointed
readers at the manual loop for approval gates the runner's per-turn hooks
already cover.
- Repoint links away from the retired documentation hosts.
- Dedupe eagerly loaded SKILL.md content that the per-section files already
cover.
109 lines
6.5 KiB
Markdown
109 lines
6.5 KiB
Markdown
# Managed Agents — Outcomes
|
|
|
|
An **outcome** elevates a session from *conversation* to *work*: you state what "done" looks like, and the harness runs an iterate → grade → revise loop until the artifact meets the rubric, hits `max_iterations`, or is interrupted. A separate **grader** (independent context window) scores each iteration against your rubric and feeds per-criterion gaps back to the agent.
|
|
|
|
The SDK sets the `managed-agents-2026-04-01` beta header automatically on all `client.beta.sessions.*` calls; no additional header is required for outcomes.
|
|
|
|
---
|
|
|
|
## The `user.define_outcome` event
|
|
|
|
Outcomes are not a field on `sessions.create()`. You create a normal session, then send a `user.define_outcome` event. The agent starts working on receipt — **do not also send a `user.message`** to kick it off.
|
|
|
|
You can collapse both calls into one by passing a single `user.define_outcome` in the session's `initial_events` array — same event, same rules, one round trip (see `shared/managed-agents-core.md` → Seeding a session with `initial_events`). More than one `user.define_outcome` in that array, or one without a `rubric`, rejects the whole create with a 400.
|
|
|
|
```python
|
|
session = client.beta.sessions.create(
|
|
agent=AGENT_ID,
|
|
environment_id=ENVIRONMENT_ID,
|
|
title="Financial analysis on Costco",
|
|
)
|
|
|
|
client.beta.sessions.events.send(
|
|
session_id=session.id,
|
|
events=[
|
|
{
|
|
"type": "user.define_outcome",
|
|
"description": "Build a DCF model for Costco in .xlsx",
|
|
"rubric": {"type": "text", "content": RUBRIC_MD},
|
|
# or: "rubric": {"type": "file", "file_id": rubric.id}
|
|
"max_iterations": 5, # optional; default 3, max 20
|
|
}
|
|
],
|
|
)
|
|
```
|
|
|
|
| Field | Type | Notes |
|
|
|---|---|---|
|
|
| `type` | `"user.define_outcome"` | |
|
|
| `description` | string | The task. This is what the agent works toward — no separate `user.message` needed. |
|
|
| `rubric` | `{type: "text", content}` \| `{type: "file", file_id}` | **Required.** Markdown with explicit, independently gradeable criteria. Upload once via `client.beta.files.upload(...)` (beta `files-api-2025-04-14`) to reuse across sessions. |
|
|
| `max_iterations` | int | Optional. Default **3**, max **20**. |
|
|
|
|
The event is echoed back on the stream with a server-assigned `outcome_id` and `processed_at`.
|
|
|
|
> **Writing rubrics.** Use explicit, gradeable criteria ("CSV has a numeric `price` column"), not vibes ("data looks good") — the grader scores each criterion independently, so vague criteria produce noisy loops. If you don't have a rubric, have Claude analyze a known-good artifact and turn that analysis into one.
|
|
|
|
---
|
|
|
|
## Outcome-specific events
|
|
|
|
These appear on the standard event stream (`sessions.events.stream` / `.list`) alongside the usual `agent.*` / `session.*` events.
|
|
|
|
| Event | Payload highlights | Meaning |
|
|
|---|---|---|
|
|
| `span.outcome_evaluation_start` | `outcome_id`, `iteration` (0-indexed) | Grader began scoring iteration *N*. |
|
|
| `span.outcome_evaluation_ongoing` | `outcome_id` | Heartbeat while the grader runs. Grader reasoning is opaque — you see *that* it's working, not *what* it's thinking. |
|
|
| `span.outcome_evaluation_end` | `outcome_evaluation_start_id`, `outcome_id`, `iteration`, `result`, `explanation`, `usage` | Grader finished one iteration. `result` drives what happens next (table below). |
|
|
|
|
### `span.outcome_evaluation_end.result`
|
|
|
|
| `result` | Next |
|
|
|---|---|
|
|
| `satisfied` | Session → `idle`. Terminal for this outcome. |
|
|
| `needs_revision` | Agent starts another iteration. |
|
|
| `max_iterations_reached` | No further grader cycles. Agent may run one final revision, then session → `idle`. |
|
|
| `failed` | Session → `idle`. Rubric fundamentally doesn't match the task (e.g. description and rubric contradict). |
|
|
| `interrupted` | Emitted whenever a `user.interrupt` arrives while an outcome is active — **even if evaluation hadn't started**. In that case `outcome_evaluation_start_id` is an empty string rather than an event ID, so don't use it as a lookup key without checking. |
|
|
|
|
```json
|
|
{
|
|
"type": "span.outcome_evaluation_end",
|
|
"id": "sevt_01jkl...",
|
|
"outcome_evaluation_start_id": "sevt_01def...",
|
|
"outcome_id": "outc_01a...",
|
|
"result": "satisfied",
|
|
"explanation": "All 12 criteria met: revenue projections use 5 years of historical data, ...",
|
|
"iteration": 0,
|
|
"usage": { "input_tokens": 2400, "output_tokens": 350, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 1800 },
|
|
"processed_at": "2026-03-25T14:03:00Z"
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Checking status & retrieving deliverables
|
|
|
|
**Status** — either watch the stream for `span.outcome_evaluation_end`, or poll the session and read `outcome_evaluations`:
|
|
|
|
```python
|
|
session = client.beta.sessions.retrieve(session.id)
|
|
for ev in session.outcome_evaluations:
|
|
print(f"{ev.outcome_id}: {ev.result}") # outc_01a...: satisfied
|
|
```
|
|
|
|
**Deliverables** — the agent writes to `/mnt/session/outputs/`. Once idle, fetch via the Files API with `scope_id=session.id`. This is the same session-outputs mechanism documented in `shared/managed-agents-environments.md` → Session outputs (including the dual-beta-header requirement on `files.list`).
|
|
|
|
---
|
|
|
|
## Interaction rules & pitfalls
|
|
|
|
- **One outcome at a time.** Chain by sending the next `user.define_outcome` only after the previous one's terminal `span.outcome_evaluation_end` (`satisfied` / `max_iterations_reached` / `failed` / `interrupted`). The session retains history across chained outcomes.
|
|
- **Steering is allowed but optional.** You *may* send `user.message` events mid-outcome to nudge direction, but the agent already knows to keep working until terminal — don't send "keep going" prompts.
|
|
- **`user.interrupt` pauses the current outcome** — it marks `result: "interrupted"` and leaves the session `idle`, ready for a new outcome or conversational turn.
|
|
- **After terminal, the session is reusable** — continue conversationally or define a new outcome.
|
|
- **Outcome ≠ session-create field.** Don't put `outcome`, `rubric`, or `description` on `sessions.create()` — outcomes are always sent as a `user.define_outcome` event.
|
|
- **Idle-break gate is unchanged.** In your drain loop, keep using `event.type === 'session.status_idle' && event.stop_reason?.type !== 'requires_action'` — do **not** gate on `span.outcome_evaluation_end` alone (on `needs_revision` the session keeps running). See `shared/managed-agents-client-patterns.md` Pattern 5.
|
|
|
|
For the raw HTTP shapes and per-language SDK bindings beyond Python, WebFetch `https://platform.claude.com/docs/en/managed-agents/define-outcomes.md` (see `shared/live-sources.md`).
|