Set KPI and ROI
The Composabl benchmarking feature allows you to compare the performance of different multi-agent systems against key performance indicators (KPIs). This tool helps you evaluate your agent system's effectiveness, track their improvements, and calculate potential return on investment (ROI).
Benchmarks are generated only after training is complete. You won't see benchmark data while training progresses, even for long-running jobs. If a training job fails, no benchmark data will be generated.
Note: Benchmarking does not influence agent system training, goals, or rewards. It is purely an analysis tool that helps you answer the question: "Based on how this agent system was trained and how we believe it should perform in real life, how much money is this agent system generating?"
Getting Started
Define the KPI for Your Project
The key performance indicator (KPI) for your project is the top-level objective that you will use to compare and evaluate your multi-agent systems' performance.
To define the KPI for your project, choose the sensor variable that represents the metric that best indicates performance success. For most processes, this is a metric with a clear business impact, such as product yield or energy use.
Setting up the KPI
To set up the KPI:
Navigate to the Benchmark page from the main dashboard
Click the "Settings" button in the top right of the KPI Performance Metrics section
In the "Set up KPI" section, configure the following:
Sensor Name: Select a sensor or preceptor variable to track (e.g., "Eps_Yield")
Target Value: Set the desired target for this KPI (e.g., 0.33)
Benchmark Unit: Select the unit of measurement (e.g., "% of Max Theoretical Yield"). Note that this is for UI readability only and doesn't affect calculations.
Setting up ROI Calculations
Return on Investment (ROI) calculations help quantify the financial impact of your agent systems' performance. The ROI is calculated based on the monetary value of the difference between your target value and the actual value of the KPI generated by that agent system, minus training costs.
In the Settings modal, navigate to the "Enter ROI criteria" section
Define the conditions for calculating ROI:
When: Select the metric to track (e.g., "Eps_Yield")
Condition: Choose how the metric changes (e.g., "increases by")
Percentage: Enter the percentage change that matters (e.g., 2%)
Value: Set the financial value of this change (e.g., $1,000,000)
Period: Select the time period (e.g., "per year")
For example: "When Eps_Yield increases by 1% of max theoretical yield, it is worth $1,000,000 per year."
You can change KPI and ROI settings at any time after training completes, and the benchmark results will be automatically recalculated. This allows you to explore different business scenarios without retraining your agent systems.
Reading the Benchmarking Dashboard
Understanding the Overview Panel
At the top of the benchmarking page, you'll find an Overview panel containing three key metrics:
Highest performing Agent System: Displays the outcome of the highest performing agent system
Lowest performing Agent System: Displays the outcome of the lowest performing agent system
ROI: Shows the calculated financial return based on your KPI settings
Each metric displays both the value and the relevant unit (e.g., "% of Max Theoretical Yield").
Bar Graph Visualization
The bar graph in the KPI Performance Metrics section provides a visual comparison of your agent systems:
Each bar represents an agent system's performance on your selected KPI (all bars correspond to the same KPI)
The height of the bar indicates the specific value of the KPI for that agent system (Y-axis)
Horizontal dashed lines show benchmark averages
The actual value of the KPI for each agent system is shown above the respective bar
Performance Ranking Table
Below the graph is a detailed table showing:
Agent System Number: Sequential ID for each agent system
Agent System Name: Name or description of the agent system
ROI: Calculated financial return based on the agent system's performance
Green arrows (↑) indicate positive ROI with the percentage and absolute value
Red arrows (↓) indicate negative ROI with the percentage and absolute value
Eps_Yield (variable) per % of Max Theoretical Yield: Performance metric with indicators for highest agent system performers
Last updated
