> ## Documentation Index
> Fetch the complete documentation index at: https://docs.novita.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# OpenTelemetry

> Reference for Sandbox telemetry exported over OTLP/HTTP, including CPU, memory, disk, organization quotas, usage, and lifecycle events.

Sandbox telemetry uses the OpenTelemetry Protocol (OTLP) to export resource metrics, organization quotas and usage, and lifecycle events to a compatible receiver.

## Configure your OTLP endpoint

> [!WARNING]
>
> **Permissions and team-wide scope**
>
> * Only team members with the **Owner** or **Admin** role can manage the OpenTelemetry configuration.
> * The configuration applies to the **entire team**, not just the member who saves it.
> * The `user_uuid` attribute identifies the **team owner's account UUID**, not the current member's account UUID.
> * Authorized administrators can read the **full Header values**, including any credentials they contain. Treat these values as secrets accessible to those administrators.

Open [Sandbox Console Settings](https://novita.ai/sandbox-console/settings) and find the **OpenTelemetry** section.

1. Enter the base URL provided by your OpenTelemetry Collector or observability platform in **OTLP Endpoint**.
2. If your receiver requires authentication or other headers, add them under **Headers**. You can configure up to 32 headers.
3. Turn on **Enable export**, then click **Save**.

![OpenTelemetry settings with an OTLP endpoint field, optional headers, an export toggle, and a Save button](https://mintcdn.com/novitaai/SeQhkhcNE0WmcPcX/images/sandbox/opentelemetry-settings.png?fit=max\&auto=format\&n=SeQhkhcNE0WmcPcX\&q=85\&s=9b81d8fedbf089d17c7fd73bb5e28841)

## Available telemetry

| Data | What you can monitor | OTLP signal |
| - | - | - |
| Sandbox resources | Allocated CPU cores, CPU utilization, and allocated and used memory and disk | Metrics |
| Organization quotas and usage | Concurrent sandbox, CPU, and memory usage, plus organization and per-sandbox limits | Metrics |
| Lifecycle events | Operations such as create, pause, resume, and delete, including status, latency, and errors | Logs |

This page describes the available metrics and event information. To query data directly through the SDK or CLI instead, see [Sandbox Metrics](https://docs.novita.ai/guides/sandbox-metrics.md) and [Sandbox Events](https://docs.novita.ai/guides/sandbox-events.md).

## Resource attributes

Each resource contains these string attributes:

| Attribute | Description |
| - | - |
| `service.name` | Identifies the telemetry category: `novita.sandbox.events`, `novita.sandbox.metrics`, or `novita.sandbox.organization`. |
| `novita.sandbox.user_uuid` | UUID of the team owner's account. |

All metric names and business attributes below use the default `novita.sandbox.` prefix. The prefix is determined by the service configuration.

## Sandbox resource metrics

Resource metrics use `service.name = novita.sandbox.metrics`. All six metrics are **Gauges**, representing values at the time of sampling. Each data point includes a sample timestamp.

| Metric name | Unit | Description |
| - | - | - |
| `novita.sandbox.cpu_core_total` | `{core}` | Number of allocated CPU cores. |
| `novita.sandbox.cpu_utilization` | `1` | CPU utilization, preserving the source value without scaling. |
| `novita.sandbox.mem_total_mb` | `MB` | Allocated memory. |
| `novita.sandbox.mem_use_mb` | `MB` | Used memory. |
| `novita.sandbox.disk_total_mb` | `MB` | Allocated disk space. |
| `novita.sandbox.disk_use_mb` | `MB` | Used disk space. |

> [!NOTE]
>
> CPU utilization is exported without multiplying or dividing by 100. Do not infer a 0–1 range from the unit `1` alone. Memory and disk metrics use the unit string `MB`, but their values are calculated from bytes divided by `1024²` (MiB).

### Resource metric attributes

Each data point includes the following string attributes:

| Attribute | Description |
| - | - |
| `novita.sandbox.event_id` | Event identifier associated with the sample. |
| `novita.sandbox.region_id` | Region identifier. |
| `novita.sandbox.template_id` | Template identifier. |
| `novita.sandbox.sandbox_id` | Sandbox identifier. |
| `novita.sandbox.user_uuid` | UUID of the team owner's account. |
| `novita.sandbox.member_uuid` | Member identifier from the source record. |
| `novita.sandbox.cluster_id` | Cluster identifier. |

Use these attributes to filter by sandbox or compare resource usage across templates and regions. Memory utilization can be calculated as `mem_use_mb / mem_total_mb`, and disk utilization as `disk_use_mb / disk_total_mb`, when the corresponding total is greater than zero.

## Organization quotas and usage

Organization metrics use `service.name = novita.sandbox.organization`. All metrics are **Gauges** with integer values. Each data point includes a collection timestamp.

### Quotas

| Metric name | Unit | Description |
| - | - | - |
| `novita.sandbox.limit.concurrent_instances` | `{sandbox}` | Maximum number of concurrent sandboxes. |
| `novita.sandbox.limit.concurrent_cpu_core` | `{core}` | Maximum total concurrent CPU cores. |
| `novita.sandbox.limit.concurrent_mem_mb` | `MB` | Maximum total concurrent memory. |
| `novita.sandbox.limit.disk_mb` | `MB` | Disk space available to a single sandbox. |
| `novita.sandbox.limit.max_length_hours` | `h` | Maximum lifetime of a single sandbox. |
| `novita.sandbox.limit.max_ram_mb` | `MB` | Maximum memory for a single sandbox. |
| `novita.sandbox.limit.max_vcpu` | `{core}` | Maximum CPU cores for a single sandbox. |

### Current usage

| Metric name | Unit | Description |
| - | - | - |
| `novita.sandbox.usage.concurrent_instances` | `{sandbox}` | Current number of concurrent sandboxes. |
| `novita.sandbox.usage.concurrent_cpu_core` | `{core}` | Current total concurrent CPU cores. |
| `novita.sandbox.usage.concurrent_mem_mb` | `MB` | Current total concurrent memory. |

Compare each concurrent usage metric with its corresponding limit to monitor remaining capacity. For details on sandbox limits, see [Quota and Limits](https://docs.novita.ai/guides/sandbox-quota-limit.md).

### Organization metric attributes

Each data point includes the following string attributes:

| Attribute | Description |
| - | - |
| `novita.sandbox.region_id` | Region identifier. |
| `novita.sandbox.cluster_id` | Currently populated with the region identifier for organization metrics. |
| `novita.sandbox.user_uuid` | UUID of the team owner's account. |
| `novita.sandbox.member_uuid` | Member identifier from the configuration; may be empty. |

> [!NOTE]
>
> When an optional source value is missing, the sender omits the corresponding `limit.concurrent_cpu_core`, `limit.concurrent_mem_mb`, or `usage.concurrent_mem_mb` data point. Do not treat an absent point as zero. Other metrics are still sent when their value is zero.

## Lifecycle events

Lifecycle events use `service.name = novita.sandbox.events` and are exported as **logs**, not metrics. They describe sandbox lifecycle operations rather than command stdout or stderr. For command output, see [Log Streaming](https://docs.novita.ai/guides/sandbox-log-streaming.md).

### Event information

Each event includes a timestamp, severity, message, and the attributes listed below.

* **Event time**: When the event occurred. If the source timestamp is missing, the current time is used.
* **Observed time**: When the event batch was prepared for export.
* **Severity**: `ERROR` when the state is `fail` or the HTTP status code is at least `500`; otherwise `INFO`. The severity text contains the event state, such as `success` or `fail`.
* **Message**: `sandbox lifecycle event: <event_name>`.

### Event attributes

| Attribute | Type | Description |
| - | - | - |
| `novita.sandbox.event_id` | String | Event identifier. |
| `novita.sandbox.region_id` | String | Region identifier. |
| `novita.sandbox.template_id` | String | Template identifier. |
| `novita.sandbox.template_name` | String | Template name. |
| `novita.sandbox.sandbox_id` | String | Sandbox identifier. |
| `novita.sandbox.user_uuid` | String | UUID of the team owner's account. |
| `novita.sandbox.member_uuid` | String | Member identifier from the source record. |
| `novita.sandbox.event_name` | String | Event name, such as `create`, `pause`, `resume`, or `delete`. |
| `novita.sandbox.state` | String | Event state. |
| `novita.sandbox.error_msg` | String | Error message. |
| `novita.sandbox.latency_ms` | Integer | Request latency in milliseconds. |
| `novita.sandbox.status_code` | Integer | HTTP status code. |

Filter events by `sandbox_id` to investigate a particular sandbox. Use `event_name`, `state`, `status_code`, and `error_msg` to identify failed operations, and `latency_ms` to inspect request latency.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.