Identify capacity adjustments and next steps from GPU monitoring data.
Combine GPU metrics and workload status with contractual requirements and operating policies. Run repeatable checks to identify capacity adjustments and provide the evidence for review.
Illustrative AI data center scenarioBuilt with fictional operational data. The Always-on Decision Agent and approval and execution interfaces are planned for open-source release.
GPUaaS teams need to meet capacity and performance commitments while using their fleet efficiently. Reviewing allocations alongside actual workloads and service requirements helps distinguish GPUs that could be reassigned from services that need a capacity review.
Look at allocations alongside SM and Tensor activity, memory use, power, and training progress. Distinguish an abandoned development session from a job waiting on I/O or communication, or capacity reserved for a legitimate need.
A total GPU count does not tell you whether a job can be placed. Check GPU type and memory, node and network topology, multi-GPU placement requirements, and tenant allocation policies.
Review traffic, queue time, P95 and P99 latency, throughput, and memory together. Peak demand, model loading time, and recovery headroom all affect how much capacity could be removed.
Technical background: NVIDIA DCGM metrics · Kueue placement requirements · Triton performance and resource analysis
Reports for operational decisions
Training needs capacity to start queued work. Inference needs capacity that meets service targets. These two examples show the evidence, possible actions, and metrics for assessing a change.
Illustrative reports built with fictional data.
Fictional GPU pool · 4 nodes
An illustrative snapshot of operating conditions
Eight unallocated GPUs are spread across the pool. A job requiring eight GPUs on the same node remains queued because no node meets that placement requirement.
Fictional figures · 32 GPUs = 24 allocated + 8 unallocated
8 GPUs · One node · Specified GPU type and memory
Not every workload requires a single node; that is a requirement of this example. CPU, memory, quota, node health, and other constraints also need checking.
Confirm the session state and restart plans with its owner, then obtain approval. If release is feasible, check whether the eight GPUs on Node D would meet the queued job's placement requirements.
Low SM activity alone does not justify reclaiming a GPU. Check training progress, I/O and communication waits, checkpoints, reservations, and operating conditions together.
Confirm ownership and stop conditions → Approve → Request release and reassignment through the scheduler → Verify GPU release and job start.Measure the effect Queue time · Reassigned GPU-hours
Align DCGM and Prometheus metrics with Kubernetes and scheduler allocations, Pending reasons, tenant quotas, training logs, and session ownership by time and workload. Check operating policies, reservations, and change history before evaluating a different allocation.
Fictional inference service · Same model
An operating scenario built with fictional figures
Traffic has fallen, but four GPU-backed service replicas remain running. Current latency meets the example operating target. Review whether to keep capacity, reduce replicas, or scale with demand.
All figures below belong to this fictional scenario.
The count in this example refers to service replicas. It is not a GPU count or a cost measure; those require the actual resource allocation.
Validate candidates using comparable operating periods and load tests. Rule out changes that breach latency targets or leave inadequate recovery headroom.
Validate the change → Approve → Request execution through the orchestrator → Verify replica count and service performance.Measure the effect Allocated GPU-hours · Cost per request or token · Latency and error rate
Review SM, Tensor, and DRAM activity and memory use alongside inference requests, queue and processing times, error rate, and throughput. Check peak traffic, concurrency, batching, model loading, scaling and recovery times, minimum capacity, reservations, and isolation requirements.
Distinguish contractual SLAs from internal operating SLOs. Align aggregation windows and measurement points for P95 and P99; for LLM services, also check time to first token and inter-token latency. Distinguish physical GPU measurements from model memory requirements, and validate memory figures only with a defined observation window, aggregation method, and attribution scope.
Read the Triton metric definitionsAlways-on Decision Agent · Planned open-source release
Check capacity on an agreed schedule using the team's operating criteria, then deliver review targets and evidence in an analysis report. Help operators focus on possible responses rather than gathering the facts first.
Look for persistently low activity on allocated GPUs, growing queues, and capacity that appears excessive for the workload's demand.
Narrow the investigation to nodes, jobs, and tenants. Review metrics and job status alongside incident, maintenance, and change records and operating documents.
Evaluate release, reassignment, and capacity changes against SLAs, reservations, priorities, and stop/resume conditions.
Separate observations from possible causes, proposed actions, and open questions. Consolidate repeat findings so the team can focus on what needs a decision.
Expected value · Operations Reduce manual data gathering and repeated checks. Assess the effect through analysis and review time per case, and verification time after a change.
Monitoring tools show system conditions, and schedulers place workloads. Decision Layer connects these signals with operating policies and analysis procedures to identify what to adjust and why.
Within Decision Layer, the Semantic Layer holds metric definitions and calculation rules; the Decision Graph holds the check sequence and comparison methods. Runbooks, contracts, and change records are accessed through separately configured agents and data connections.
Approval and execution · Planned open-source interfaces
After the team approves a change, request execution through existing schedulers and orchestrators. Record request acceptance, change completion, and performance impact as separate states.
The responsible person reviews the workloads and tenants in scope, the permitted change, and SLA, reservation, and recovery conditions.
A configured execution module checks permissions and sends the request through scheduler or orchestrator APIs.
Distinguish accepted, in progress, completed, and failed requests. Check actual GPU allocations and service replica states.
Compare queue time, throughput, latency, errors, and allocated or reassigned GPU-hours. Freed capacity is not the same as cost or power savings; verify those separately with billing and metering data.
Existing operations systems retain responsibility for resource placement and service control. Approval and execution interfaces are planned for open-source release. Connections to schedulers and orchestrators are configured during implementation.
The public project lets teams reuse registered analytical code and procedures and inspect execution records. Applying it to GPU operations requires monitoring data connections and procedures configured around the team's operating criteria.
Configure data connections and procedures for your environment. The current public project does not include dedicated AI data center connectors.
The Always-on Decision Agent and approval and execution interfaces are planned for open-source release and are not part of the current public framework. System connectors and approval conditions are configured during implementation.
Different industries. A shared approach to evidence and judgment. In manufacturing, examine quality and process conditions together. In GPU operations, connect workloads and capacity with service requirements. Decision Layer makes the team's data and analytical criteria available for investigating a problem and reviewing the next action.
These sources inform the discussion of GPU observability, placement, and inference performance. The scenario figures are fictional and do not describe a completed HEARTCOUNT integration.
Define the GPU pool, the workloads, and the operating metric you want to improve.
Build the analysis around your team's criteria, then identify which actions a future integration should support.