Controller: A skill agent that makes decisions based on programmed algorithms. Also called a programmed skill agent.

Decisions: The outputs of a Composabl system - its instructions for controlling the system

Design patterns: Common structures that can be used to quickly create multi-agent systems

Episode: An entire run through the task

Functional Pattern: A machine teaching design pattern used to orchestrate skills in sequences

Iteration: One decision during agent training or performance

Learned skill agent: A skill agent that uses DRL to make decisions and learn by practicing in simulation

Orchestration: Arranging agents as modular building blocks to work together to complete tasks

Orchestrator: A special type of skill agent that assigns decision-making control to the right decision-making skill agent based on current conditions

Perceptor: An ML model or other algorithm that interprets sensor data

Plan-Execute Pattern: A machine teaching design pattern used to orchestrate skill agents in pairs that work together to make decisions

Programmed skill agent: A skill agents that makes decisions based on programmed algorithms. Also called controllers.

Project: A collection of agents for the same use case that share the same simulator and top-level goal

Scenarios: Conditions that are associated with specific skills

Sensors: The part of the system that takes in information from the simulator or real environment - the eyes, ears, and other senses of the system

Simulator: The virtual environment where agents practice and improve performance

Skill agents: Modules within a multi-agent system that make decisions to complete all or part of a task

Strategy Pattern: A machine teaching design pattern used to orchestrate skill agents in hierarchies

Teacher: An algorithm that creates a skill agent that uses DRL to learn to make decisions

Composabl is the platform for creating industrial-strength multi-agent AI systems that make high-impact decisions in the real world.

With Composabl, you can convert expert knowledge about how a process works into a team of agents with specialized skills that work together to allow the system to make the right decision in every situation. These skill agents can be either programmed or learned through advanced AI techniques and orchestrated so that the multi-agent system performs effectively in every part of the process and under any conditions. For skill agents that learn by practicing, Composabl trains the agent systems in realistic scenarios until the agent system can succeed at the task and outperform the alternatives.

Quick Links

Find articles on key topics with these links.

Try Out a Tutorial

Integrate with Composabl

You can use any model or Python algorithm with Composabl for training agent systems, adding perception, analysis, and communication, and making decisions. See how to configure different types of modules in the UI and by publishing them via the data science workflow for agent system design, training, and deployment.

Create Modular Skill Agents

Composabl multi-agent systems are built on modular skills that break down a task into separate parts. Learn how to create skill agents to train with deep reinforcement learning.

Deploy Multi-Agent Systems

Once Composabl agentic systems are designed and trained, you can export them to the Composabl runtime to connect with your system. Learn how to deploy an agent within the runtime container and how to use Composabl's tools to analyze agent behavior during both training and deployment.

Login

Agent

Sim (composabl sim)

Perceptor (composabl perceptor)

Skill (composabl skill)

Selector (composabl selector)

Job (composabl job)

This tutorial will take you through the process of building agents for a realistic chemical manufacturing use case.

1. Learn about the use case (this page)

7. Experiment with your own designs

About the Use Case

Why an industrial mixer?

Mixing and blending tasks are great use cases for intelligent agent systems because they are both complex and high-value. In the example in this tutorial, process improvements could lead to millions in ROI.

Use Case Overview

In the industrial mixer use case, raw materials are stirred together inside a tank, undergoing a reaction that produces the desired end product.

The goal of the process is to convert as much of the raw material as possible. But as the chemicals mix and the conversion occurs, the tank heats up. If the temperature gets too high, a condition called “thermal runaway” occurs, potentially causing explosions and fires.

To produce as much chemical as possible, the operator must constantly adjust the temperature in the tank, keeping it high enough to allow productivity but low enough to avoid any thermal runaway risk.

As in all machine teaching use cases, this process can be summarized in the form of a goal (maximize yield) and a constraint (avoid thermal runaway) that must be balanced against each other:

The process is controlled by adjusting the mixture's temperature in the tank using a "jacket" filled with coolant. Lowering the coolant temperature in the jacket lowers the temperature in the tank, decreasing the risk of thermal runaway.

However, cooling the tank can also reduce yield. By how much? The answer varies unpredictably – temperature changes affect chemical concentration differently at different parts of the reaction. That nonlinear relationship between temperature and yield is why this is a nuanced process that benefits so much from intelligent automation.

Simulating a Complex Reaction

The industrial mixer simulator uses principles of chemistry to model the behavior of the system in real life.

Benchmark

The current automation solution is a linear MPC controller. As the current solution, this agent system's performance is the benchmark for the other designs.

The image below shows an MPC controller represented in the visual system of agent designs. The agent takes in sensor information about the temperature in the tank and the concentrations of the chemicals.

It passes that information to the skills layer of the agent system. The skills layer contains a single programmed skill agent: control reactor. This skill uses a mathematical model to determine the desired temperature set point for the tank. It also determines the control actions to take to achieve that temperature using the cooling jacket, and outputs those actions as decisions.

In simulation, this agent's conversion rate was 82%. That means that 82% of the reagents were turned into product, with 18% waste, but the risk of thermal runaway is high.

Like all technologies, model predictive control has a “personality,” a unique set of strengths and weaknesses that can be seen reflected in this performance. MPC is a rule follower. It is a way of controlling a process using mathematical relationships that can be fully mapped and programmed by human engineers. It works well in situations that are straightforward and linear.

The agent does a good job in the first steady state at the start. However, as the transition phase begins, the agent’s performance starts to fail. The wide shadow shows that this agent can’t adapt to the unpredictable conditions in the transition. Its performance becomes dangerously inconsistent, potentially allowing the temperature to exceed the thermal runaway checkpoint at nearly every point in the reaction.

When an MPC controller is used to control this process in the real world, a human operator needs to step in and take over control before the automated system lets the temperature cross the thermal runaway threshold.

Get Started

The following tutorials will walk you through the anatomy of some multi-agent systems that take different approaches to learning how to control the temperature of a mixer more effectively to maximize yield and avoid thermal runaway.

The DRL agent system is a simple design with only one skill agent. This agent system does not use machine teaching to decompose the task into skills that can be trained separately. Instead, the entire reaction is controlled by a skill agent trained with deep reinforcement learning.

Let's get started!

1. Create your first skill agent

This agent system has a single-skill agent called Control Full Reaction. To create this skill in the UI go to the skill agent page and click Create new skill agent

2. Set skill agent goals and constraints

Configure your agent to set the instructions for its training sessions. This agent has one goal, to maximize yield, and one constraint, to keep the temperature from going above 400 degrees Kelvin.

1. Click Add goal In the left drop-down menu, select Maximize, and in the right one, select Eps_Yield . This means the agent will train with the goal of maximizing the total product produced by the end of each episode.

2. Click Add constraint. In the left drop-down menu, select Avoid , and in the right one, select T . After you select T you're going to see a slider appear for you to set boundaries you want to train the system to avoid. In this case, we want to set the boundaries from 400 to 500.

3. Save your skill agent configuration and return to the Agent Orchestration Studio.

3. Create a Scenario

Set scenarios to tell each skill agent what specific conditions or phases of the process to practice in. This skill agent controls the full reaction, so it needs to practice with the reaction as a whole.

Go to the Scenarios page and select Add scenario, then name it Control full reaction and click Save . We're going to add two criteria to this scenario, and they are a reference temperature and concentration.

Control Full reaction: Cref Is 8.57, Tref Is 311 | Why these numbers?

4. Add the Skill Agent to Your Agent Configuration

Drag the skill control_reaction that you can now see on the left-hand side of your project onto the skills layer. Click on the skill agent once it's in the skill layer and assign the scenario.

5. Run Your Training Session

Now, we are ready to train your agent and see the results. First, select our built-in training cluster or one you own and have connected to the platform. Then set the number of cycles. For this tutorial, we suggest running 50. You can run multiple simulations in parallel to speed up training time. Under advanced, you can use GPUs instead of CPUs, set a rollout fragment length, and set the number of benchmark runs.

Once you have everything configured, click Allocate training cycles . This agent system design has only one agent, so all training cycles will be allocated to our DRL agent. In a multi-agent system, you can assign a different number of training cycles to different agents depending on the complexity of the skill.

6. View Results

When the training has been completed, you can view your results in the training sessions tab in the UI. This will show you information on how well the agent is learning.

You will likely see a steep learning curve as the agent experiments with different control strategies and learns from the results. When the learning curve plateaus, that usually means that the skill is trained.

Analyze the DRL Agent's Performance

Conversion rate: 90% Thermal runaway risk: Low

We tested this fully trained agent and plotted the results.

The DRL agent system performs well. Its relatively thin shadow means that it performs consistently over different conditions and stays within the safety threshold almost every time.

This agent controls the initial steady state well, staying on the benchmark line. But during the transition, the DRL agent goes off the benchmark line quite a bit. It doesn't notice right away when the transition phase begins, staying too long in the lower region of the graph and then overcorrecting. That's because DRL works by experimentation, teaching itself how to get results by exploring every possible way to tackle a problem. It has no prior knowledge or understanding of a situation and relies entirely on trial and error. That means it is potentially well-suited to complex processes, like the transition phase, that can’t be easily represented mathematically.

However, its behavior is erratic because it can’t distinguish between the phases. The DRL agent’s skills do better than the traditional automation benchmark, but still leave room for improvement.

This getting started guide takes you through the steps you will need to go through before you can start building your teams of agents.

Prerequisites

• A Composabl account

Create Your Use Case

Select the simulator you will use for the project and the assigned team. If you haven't created a team yet, you will need to do that before you create your first project.

Then, you can use the AI prompt to explain what you are attempting to achieve with your project and have it bootstrap based on your description. I will describe the use case skills, goals, and constraints and call it Chemical Process Control.

Click New use case and Create use case

Set Use Case Goals

To finish setting up your use case, set your top-level project goal.

Now, we will set the goal for this example. In this project, we want to:

• Maximize yield (make as much product as we can)

Goal Title and Description

Fill out the goal title and description for the top-level goal.

Goal Title: Chemical Process Control Goals

Description: The goal for this project is to maximize yield

Logical Structure

We are going to one condition to represent our goal.

1. To maximize yield, the objective is Maximize , and the variable is Eps_Yield

Now you can save your changes.

Now that you have created your new use case and set the goal, you can start creating agent systems!

Optional: Set Up the Data Science Workflow

Some of the steps in this tutorial use Composabl's data science workflow, which allows data scientists and controls engineers to publish any Python module to the Composabl platform to use in multi-agent systems.

To use this workflow, complete these steps:

Now that you have your project created and your goals set you're ready to start creating agents!

What is special about this agent is that it combines DRL and MPC, the technologies from the two single-skilled agent systems — the worst performers — to create the best performing agent.

In this example, the DRL skill agent first uses its powers of learning and experimentation to determine the goal temperature for the cooling jacket — the set point. It then passes this information on to the MPC skill agent, which uses its powers of control and execution to direct the agent on what action to take to achieve the desired temperature.

Remember how the strategy pattern is like a math class where each student solves the problems they are best at, as assigned by the teacher? In the plan-execute pattern, the students work in groups to solve problems together. Let’s say Student A is good at translating word problems into equations, while Student B is good at solving equations. Student A works on each problem first, and then passes it over to Student B, who produces the solution. No teacher is needed here, because the students divide each problem the same way.

Let's get started configuring this agent!

1. Publish the MPC Skill Agent to Your Project

Return to the agent orchestration studio and refresh the page. The skill agent will appear in the skills menu on the left of your page.

Explore the Code Files

All skill agents, perceptors, and orchestrators have at least two files in them. A Python file contains the code the skill agent will use, and a config file.

1. pyproject.toml, a config file with the following information.

File Structure

See the Code

MPC Skill Group Controller Skill Agent

pyproject.toml

controller.py

2. Build the Plan Execute Pattern Agent System in the Agent Orchestrator UI

First, drag the skill agent control_full_reaction from the hand side of the page to the skill layer. Once it's there, drag over the mpc-skill-group and make sure that it is dropped below the control_full_reaction skill agent and not beside it.

3. Run Your Training Session

We are ready to train your agent system and see the results. Select the cluster you want to use and the number of training cycles. We suggest you run 50 training cycles. You will see the skill agents training one at a time, and you assign the number of cycles you want each skill agent to use. It will automatically assign an equal number of training sessions for each skill agent, but in some agent system designs, some skill agents might require more training than others.

4. View Results

When the training has been completed, you can view your results in the training sessions tab in the UI. This will show you information on how well the agent is learning.

Analyzing the Plan-Execute Pattern Agent’s Performance

Conversion rate: 95% Thermal runaway risk: Very low

We tested this fully trained agent and plotted the results.

This agent is the best performer of the group. Combining two imperfect technologies together with Machine Teaching produces much better results than either technology achieves alone.

While skill agents can be built within Composabl's no-code studio or created externally and then imported, perceptors are always built outside of Composabl and then published to the platform to use in agents.

In this case, the perceptor is a pre-built ML model saved as a pickle file. This ML model is trained to interpret the sensor data and check for conditions that might indicate an elevated risk of thermal runaway, and then pass that information to the selector along with the rest of the sensor data.

Think of the perception layer as an additional set of senses that helps the agent system predict when something might go wrong, like a teacher monitoring the class for early signs of trouble.

Let's get started configuring this agent system!

1. Add the Perceptor Skill Agent to Your Use Case

Return to the agent orchestration studio and refresh the page. The skill agent will appear in the skill agents menu on the left side of your page.

Explore the Code Files

All skill agents, perceptors, and selectors have a minimum of two files in them. A Python file contains the code that the agent system will use, and a config file. Perceptors have some more files to load in ML models and other python packages.

File Structure

Thermal Runaway Perceptor

pyproject.toml

thermal_runaway_predictor.py

2. Copy the Strategy Pattern Agent System, name it Strategy Pattern with Perceptor, and add the Perceptor Skill Agent to your Strategy Pattern Agent System

Drag the Perceptor thermal_runaway_predictor that you can now see on the left-hand side of your use case onto the perception layer.

3. Run Your Training Session

We are ready to train your agent system and see the results. Select the cluster you want to use and the number of training cycles. We suggest you run 150 training cycles. You will see the skill agents training one at a time, and you assign the number of cycles you want each skill agent to use. It will automatically assign an equal number of training sessions for each skill agent, but in some agent system designs, some skill agents might require more training than others.

4. View Results

When the training has been completed, you can view your results in the training sessions tab in the UI. This will show you information on how well the agent system is learning.

The agent system training results will be a little bit different from the strategy pattern alone. That's because the thermal runaway predictor is making a difference in how the agent system performs.

Analyzing the Strategy Pattern Agent System’s Performance with Perception

Conversion rate: 92% Thermal runaway risk: Very low

We tested this fully trained agent system and plotted the results.

Adding perception improves agent system temperature control performance.

The red lines on the graph show where the perceptors helped the agent system make adjustments to avoid thermal runaway. This agent system gets the same yield as the strategy pattern agent, but the improved temperature control has reduced the thermal runaway incidents from low to 0.

You can integrate your simulator with the Composabl SDK by using the ServerComposabl class. This class provides the necessary methods for the simulator to interact with the Composabl SDK.

The methods of the ServerComposabl class allow the Composabl SDK to automatically take care of serializing and deserializing the different requests and responses.

To conform your simulator to the Composabl SDK, you must define a server implementation class that defines methods of how to talk with the Composabl SDK.

Set Up the Simulation Environment Instance

Make

Make is a request to create a new instance of the environment with the specifications requested.

• string env_Id; Identifier for the type of environment to create.

• dictionary env_Init; Initial configuration for the environment, as defined within the runtime configuration (link to section about how to define runtime configuration parameters)

Sensor_space_info

Sensor_space_info provides details about the environment’s sensor space.

Action_space_info

Action_space_info defines the agent system's action space.

Action_space_sample

The action_space_sample function returns an element of the simulator’s action space.

Run the Simulation Environment Instance

Reset

Reset is a request to reset the environment, and returns the first observation of the newly reset environment.

• observation Initial observation of the environment.

• Dictionary info Additional information about the reset environment.

Step

Step provides the agent system action to be applied to the environment. The return structure is as follows:

• observation; The observation following the action.

• float reward The reward received after taking the action.

• bool terminated Whether the episode has ended.

• bool truncated Whether the episode was truncated before a natural conclusion.

• Dictionary info Additional information about the step.

Close

Close denotes the simulator is done being used and may perform any necessary cleanups required.

Set_Scenario

Get_Scenario

Get_scenario returns the scenario that the simulation is currently running.

Create Visualizations

Get_Render

Get_render provides the current rendered image of the environment, either as a numpy array or a string.

The examples and code samples in this documentation refer to our sample use cases. These examples are real-world use cases with complex goals and constraints. In each case, the Composabl team has built agent systems that exceed the benchmark control technology by orders of magnitude.

Industrial Mixer

About the Use Case

The industrial mixer use case is a realistic case study of a chemical process control agent system controlling a continuous stirred tank chemical reaction. The agent system controls the temperature in a tank where a chemical reaction occurs to create a product.

As the chemicals are stirred together in the tank, the reaction produces heat at a nonlinear, unpredictable rate. If the tank isn’t cooled enough, it can reach dangerous temperatures, a condition called thermal runaway. If it’s cooled too much, not enough product will be produced. The agent system needs to balance these two goals, keeping the tank at the right temperature at every moment to optimize production while ensuring safety.

Explore Agent System Components

Production Scheduling

About the Use Case

The production scheduling use case is an complex production planning problem set in an industrial bakery. The agent system must determine the right amount of cookies, cakes, and cupcakes to make each day, directing teams of workers and equipment and responding to fluctuations in costs, pricing, and demand.

Multi-agent systems have structure, just like our brains. Different parts perform different functions.

This modularity is a key to building high-performing agentic systems quickly and effectively. Most complex systems, from factories to rockets to software programs, are broken into modular pieces to separate concerns, reuse design patterns, and integrate pre-built components. Like these systems, decision-making AI also works best when decisions are separated into modular concerns. Modular structure makes intelligent agent systems easier to build, test, and maintain.

How Agentic Systems Make Decisions

Multi-agent systems work through a feedback loop. The system gets information from the simulation or real environment and then makes a decision to adjust one or more of its control variables in some way. This creates a change that is reflected in new sensor information that the agent system can then process and use to decide on the next adjustment.

While agent systems are being built and trained, they are connected to simulators that give them feedback about how their actions affect the system they are learning to control. Once an agent system is deployed, the simulator is replaced by an equivalent connection to the real system.

The simulator and the multi-agent system function in a continuous feedback loop, with the system outputting actions that affect the simulation environment, and the simulator returning new information to the agent system based on every decision.

Sensors are the part of the multi-agent system that take in information from the simulator or the real environment about conditions and about the results of the agent system's actions. They are the equivalent of human senses – the eyes and ears. Sensor readings come from specific variables in the simulation or control system that measure various aspects of the state space.

Decisions are the system's outputs, adjusting one or more of the control variables to control the process dynamically throughout the performance of the task. Each decision during training is known as an iteration, while a series of iterations to complete the task is called an episode.

Inside the Multi-Agent System

The team of agents is organized in layers, with information and decisions passing through each layer in sequence.

Skill Agents

The skills layer, or decision-making layer, is the heart of a multi-agent system. This is where the system makes its control decisions. When a specific skill agent is activated, it will determine the control action the system should take.

Multi-agent systems contain multiple agents orchestrated together according to design patterns. Skill agents can work together in groups, sequences, hierarchies, or coordinated teams. This modularity allows the agent to learn more efficiently and perform better.

You can imagine skill agents being like students on a math team who are working together to complete a set of problems. Each student performs best solving a particular kind of problem: one is good at fractions and one at decimals. Depending on the type of problem, the appropriate student will use their expertise to solve the problem and produce the answer for the team. Sometimes one student might handle the entire problem, and for other problems more than one student might need to work together.

Just as different students’ capabilities make them able to solve particular problems, different skill agents may make use of different technologies. Some types of decisions are best approached through skill agents that can be programmed with mathematical calculations, rules, or optimization algorithms. Others that are more complex and nonlinear can be trained using deep reinforcement learning.

Examples of Skills

For an HVAC system regulating temperature in an office building:

• Control the system during the day

• Control the system at night

For a factory where responses are needed to different types of alarms:

• Handle safety critical alarms (programmed with rules)

• Handle simple alarms (programmed with calculations)

• Handle complex alarms (learned with reinforcement learning)

For a drone autopilot:

• Stabilize

• Get to landing zone

• Land

• Avoid obstacles

For a robotic arm used to grab and stack objects:

• Reach (extend the robot arm from the "elbow" and "wrist")

• Move (move the arm laterally using the "shoulder)

• Orient (turn the "wrist" to position the "hand")

• Grasp (Manipulate the "fingers" to clamp down)

• Stack (Move laterally while grasping)

Orchestrators

Orchestrator skills are the supervisors for your agent system. In the math class analogy, the orchestrator would be like the teacher. The teacher assesses the type of problem and assigns the right student.

In a multi-agent system, an orchestrator uses information from the sensors and perceptors to understand the scenario and then determine which skill agent is needed. Once the skill agent is called into service, it makes the decision.

For example, for HVAC control, an orchestrator would determine whether day or night control is needed, and then pass control to the appropriate skill agent. In the safety alarm example, the orchestrator determines the type of alarm and then passes the decision to the right skill agent. In the drone and robotic arm examples, the skills need to be performed in sequence. In these cases, the orchestrator assigns the appropriate skill agent as needed for each step in the process.

Perceptors

A perception layer is an optional enhancement layer. Perceptors process and translate sensor data into a format that can be used to make decisions. In other words, the perception layer inputs the sensor variables and outputs new variables deduced by calculation, machine learning, or other programming.

For example, if we design a team of agents as an autopilot for a drone, we might have sensor variables that measure pitch, yaw, roll (position of the drone), velocity in each of those three directions, and acceleration in each of those three directions. But what about stability? Stability is an important characteristic to understand while flying a drone, but there is no sensor variable that describes stability. It is too complex to be captured by a single sensor.

The perception layer allows us to create a variable for stability. It can be calculated using dynamics equations or trained with supervised machine learning. The new variable then becomes accessible to the rest of the multi-agent system along with the other sensor variables.

Examples of Perceptors

• Computer Vision: A camera sensor passes image or video feeds into a perceptor module that identifies object types and locations

• Auditory Perception: A microphone sensor passes machine sounds to a perceptor module that identifies which state the machine is in based on the sounds that it is making

• Prediction: A perceptor module inputs quality measurements and past agent system actions and predicts whether current actions will lead to acceptable quality measurements

• Anomaly Detection: A perceptor modules inputs market variables and detects when the market is changing regimes.

• Classification and Clustering: A perceptor module inputs machine and process data and classifies which of several conditions a manufacturing line is currently in.

The simulator is the part of the agent system ecosystem that tells you what will happen when you take a certain action. Whether based on historical data, physical principles, and math, or other methods, the simulation environment models the real system and allows your agent systems to train.

Explore Composabl's Simulators

To look under the hood and see how a simulator works, explore Composabl's public Python simulators hosted on Docker Hub.

You can build and train agent systems using these simulations to explore Composabl, educate yourself and your team, and create demonstrations and POCs. These are relatively simple simulations that can train agent systems quickly.

The best way to access these simulations is through the CLI.

To see the options available:

To connect to one of the simulators:

Simulation Help

If you have a simulator, this section of the documentation will explain how to connect it to Composabl so that you can use it to train agent systems.

If you don’t have a simulator, you may want to work with a Composabl partner to help you develop and connect one.

A use case is a collection of agent systems for the same use case. The best way to use Composabl is to build multiple agent systems within the same use case using different design patterns and variations. This allows you to iterate and improve your agent systems for the best possible performance.

All agent systems within a use case share the same goals and the same simulator.

Create a Use Case in the UI

To create a use case, click on New New Use Case in the upper right-hand corner of your dashboard.

You'll be prompted to enter the simulator associated with your use case and then choose your team.

Use the AI Use Case Creation Assistant

You also have the option to use Composabl's AI assistant to help set up the use case. This specialized AI assistant is designed to help solve the "cold start" problem and get you started designing your agent system more quickly.

Based on your written use case description, the AI will automatically create customized goals and agent skills within your use case. You can then access those goals and agent skills in the Agent Orchestration studio and edit and adjust them as needed.

When prompting the assistant, use as much detail as you can. You can tell the AI:

• What the problem or use case is

• What equipment or process the agent system will control

• What the overall goal is

• Anything you know about phases of the process or different scenarios

The performance goal is the most important KPI or metric you will use to evaluate your agent system's success. This goal directs your agent as it trains. The AI learning technology within the agent will reward the agent system when it gets closer to the goal, helping it to improve.

For many business use cases, the top-level goal will be expressed in terms of profit or ROI. There are many factors to consider in ​the production scheduling use case, but profit is the ultimate goal, so it is the most important KPI.

Setting Goals, Constraints, and Success Criteria in the UI

To edit a goal created by the copilot or to create a new goal, follow these steps:

1. Navigate to the use case page

2. Click Set up goal and enter a name and description for your goal.

3. Click Add condition to define the goal.

Use the dropdown menus to select the variables and then define the parameters for each goal, constraint, or success criterion you want to include.

Configure Goals

Goals define what a skill agent system should do. Goals apply to one of the sensor variables and are defined using one of three possible directives:

• Maximize: Maximize the value of the variable

• Minimize: Minimize the value of the variable

• Maintain: Keep the variable at a specified value

Configure Constraints

Constraints set the boundaries for the skill agent. They define rules that the agent system must follow during operation. Constraints are defined using one of two possible directives:

• Avoid: The skill agent learns to keep the variable from reaching a specified value or range through withholding rewards

• Terminate: When the actions of the skill agent lead to certain conditions within a variable, the skill agent has failed and must stop and start a new episode

Configure Success Criteria

Success criteria tell the skill agent when it's doing something right. They are defined using one of two possible directives:

• Approach: The skill agent learns to get close to a specified value by getting an increased reward

• Succeed: When the success criteria are achieved, the session ends, and a new one begins so that the skill agent can keep practicing and learn to win every time

Using Goals, Constraints, and Success Criteria for Multi-Agent AI System Design

Goals, constraints, and success criteria transform complex AI agent building into an intuitive process that eliminates the need for programming expertise.

You directly apply your domain knowledge without coding by simply defining what you want the agent to achieve and what conditions to respect. This approach reduces development time while ensuring AI agents embody real-world operational wisdom, bridging the gap between technical AI capabilities and practical industrial knowledge in complex physical environments.

Skills are the foundational building blocks for your intelligent agent system. They take action to achieve goals in key scenarios where your agent system needs to succeed. To build an agent with Machine Teaching, you will create multiple skill agents and then orchestrate them together.

You can use three different types of skill agents within Composabl.

For learned skills you can use either the UI or the SDK successfully to create teachers, but the SDK includes some fine-tuning options that are not available in the UI.

Controllers for programmed skills can be created only through the SDK. They can then be published to the UI for use in agent systems.

You can use either the UI or the SDK to create selectors.

The Composabl SDK offers a suite of advanced tools to train skills using deep reinforcement learning. Using the Python teacher class, you can fine-tune the rewards for your skills. Once you have configured a skill with the SDK, you can publish it to the UI to use in agent system designs.

Create a New Skill

To create a skill in the Python SDK, begin by logging in to the SDK by typing Composabl login from the CLI.

Then type composabl skill new.

Give the skill a name and a description in response to the prompts that follow. Choose whether your skill should be a teacher (learned with AI) or a controller (a programmed module like an optimization algorithm or MPC controller).

Specify the folder where you’d like to create the skill.

The Composal SDK will create a folder and Python teacher file from the template.

The Python Teacher Class

The Python teacher class offers several functions that you can use to fine-tune the training of your skills.

Functions for Training

Train with Rewards: the compute_reward Function

python
def compute_reward(self, transformed_sensors, action, sim_reward):
        self.counter += 1
        if self.past_ sensors is None:
            self.past_ sensors = transformed_ sensors
            return 0
        else:
            if self.past_ sensors ["state1"] < transformed_ sensors ["state1"]:
                return 1
            else:
                return -1

The compute_termination function tells the Composabl platform when to terminate a practice episode and start over with a new practice scenario (episode). From a teaching perspective, it makes most senses to terminate an episode when the agent system succeeds, fails, or is pursuing a course of action that you do not find likely to succeed. This function returns a Boolean flag (True or False) whether to terminate the episode. You can calculate this criteria however seems best.

python
def compute_termination(self, transformed_ sensors, action):
        return False

The success_criteria function provides a definition of skill success and a proxy for how completely the agent system has learned the skill. The platform uses the output of this function (True or False) to calculate when to stop training one skill and move on to training the next skill. It is also used to determine when to move to the next skill in a fixed order sequence. The agent system cannot move from one skill in a fixed order sequence to the next, until the success criteria for one skill is reached.

python
def compute_success_criteria(self, transformed_ sensors, action):
        return self.counter > 100

Here are some examples of success criteria definition:

• A simple but naive success criteria might return True if the average reward for an episode or scenario crosses a threshold, but False if it does not.

• A more complex success criteria might calculate root mean squared error (RMSE) for key variables across the episode and return True if the error is less than a customer specified benchmark, but False otherwise.

• A complex success criteria might compare a benchmark controller or another agent system to the agent system across many key variables and trials. It returns True if the agent system beats the benchmark on this criteria, but False otherwise.

Train with Goals

Training with goals lets you use a predefined reward structure rather than configuring the rewards individually. When you use a goal, your agent system will inherit the compute reward, compute termination, and compute success functions from the goal. (You will still have the option to further customize those functions as needed.)

The five goal types you can use are:

• AvoidGoal

• MaximizeGoal

• MinimizeGoal

• ApproachGoal

• MaintainGoal

Goals are added using specialized teacher classes rather than the general teacher class that you would otherwise use to teach skills. For example, for a skill named Balance that you wanted to train with a goal to maintain a specific orientation, you would use the MaintainGoal teacher class.

python
class BalanceTeacher(MaintainGoal):
	def __init__(self, *args, **kwargs):
super(),__init__(“pole_theta”, “Maintain pole to upright”, target=0, stop_distance=0.418)

The parameters you can use for goals are:

You can also use more than one goal for a single skill using the CoordinatedGoal teacher class. This is useful when your agent system needs to behave in a way that creates a balance between two goals that are both important.

Functions to Guide Agent System Behavior with Rules

Just like rules guide training and behavior for humans, providing rules for the agent system to follow can guide the agent system's decision-making more quickly to success. Rules guide the behavior of an agent system based on expertise and constraints.

Add Rules: the compute_action_mask Function

The compute_action_mask teaching function expresses rules that trainable agent systems must follow.

python
 # The action mask provides rules at each step about which actions the agent system is allowed to take.
    def compute_action_mask(self, transformed_ sensors, action):
        return [0, 1, 1]

The compute_action_mask teaching function works only for discrete action spaces (where the actions are integers or categories), not for continuous action spaces (where decision actions are decimal numbers). If you specify a mask for a skill whose actions are continuous, the platform will ignore the action mask.

The function returns a list of 0 and 1 values. Zero means that the action is forbidden by the rule. One means that the action is allowed by the rule. The function may change the returned value after each decision. This allows complex logic to express nuanced rules.

In the example above, the first action is forbidden for the next decision, but the second and third actions are allowed. The logic in the skill itself (whether learned or programmed) will choose between the allowed second and third actions.

All selectors have a discrete action space (they choose which child skill to activate), so you can always apply the compute_action_mask function to teach them.

Functions to Manage Information Inside Agent Systems

As information passes through perceptors, skills, and selectors in the agent system, sometimes it needs to change format along the way. You can use three teaching functions to transform sensor and action variables inside agent systems: transform_ sensors, transform_action, and filtered_ sensor _space.

To transform sensor variables, use the transform_sensor function to calculate changes to specific sensors, then return the complete set of sensor variables (the observation space).

python
def transform_sensor(self, sensor, action):
        return sensor

Two of the most common reasons for transforming sensor variables are conversion and normalization. For example, if a simulator reports temperature values in Fahrenheit, but the agent system expects temperature values in Celsius, use the transform_sensor function to convert between the two.

Normalization is when you transform variables into different ranges. For example, one sensor variable in your agent system might have very large values (in the thousands), but another variable might have small values (in the tenths), so you might use the transform_sensor function to transform these disparate sensor values to a range from 0 to 1 so that they can be better compared and used in the agent system.

You may want to transform action variables for the same reasons as sensor variables.

python
def transform_action(self, transformed_sensor, action):
    return action

Use the filtered_sensor_space function to pare down the list of sensor variables you need for a particular skill. Pass only the information that a skill or module needs in order to learn or perform well.

python
def filtered_sensor_space(self):
        return ["state1"]
Return a list of all the sensor variables that you want passed to the skill by this teacher.

Controllers are programmed skill agents used in Composabl agent systems. These may be optimizers, PID or MPC controllers, heuristics, or any other Python algorithms that are useful to control parts of a process. Configure controllers in the SDK and then publish them to the UI to use in agent systems.

Create a New Skill Agent

To create a skill agent in the Python SDK, begin by logging in to the SDK by typing Composabl login from the CLI.

Then type composabl skill new.

Give the skill agent a name and a description in response to the prompts that follow. Choose your skill agent type as a controller (a programmed module like an optimization algorithm or MPC controller).

Specify the folder where you’d like to create the skill agent.

The Composal SDK will create a folder and Python controller.py() file from the template.

The Python Controller Class

The Python controller class offers several functions that you can use to build your algorithms or rule-based controllers in Composabl.

Functions for Training

Controllers don't need training, since they are based on programming rather than learning, but they include functions that connect them to the rest of the agent system during training.

Initialize your algorithm: the __init__ Function

The __init__ function can be used to initialize your Algorithm and define initial configuration variables, this is called once when the Runtime starts. Let's supose that you want to use a MPC algorithm:

Process Observation to Compute Action: the compute_action Function

The compute_action function will process the observation and returns an action. This function returns a number that represents the action that will be sent to the simulation.

The compute_termination function tells the Composabl platform when to terminate a practice episode and start over with a new practice scenario (episode). From a controller perspective, it makes most senses to terminate an episode when the agent system succeeds, fails, or is pursuing a course of action that you do not find likely to succeed. This function returns a Boolean flag (True or False) whether to terminate the episode. You can calculate this criteria however seems best.

The success_criteria function provides a definition of the skill agent's success and a proxy for how completely the agent system has learned the skill. The platform uses the output of this function (True or False) to calculate when to stop training one skill agent and move on to training the next skill agent. It is also used to determine when to move to the next skill agent in a fixed order sequence. The agent system cannot move from one skill agent in a fixed order sequence to the next, until the success criteria for one skill agent is reached.

Here are some examples of success criteria definition:

• A simple but naive success criteria might return True if a certain sensors or KPI value for an episode or scenario crosses a threshold, but False if it does not.

• A complex success criteria might compare a benchmark controller or another agent system to the agent system across many key variables and trials. It returns True if the agent system beats the benchmark on this criteria, but False otherwise.

Functions to Manage Information Inside Agent Systems

As information passes through perceptors, skill agents, and selectors in the agent system, sometimes it needs to change format along the way. You can use three teaching functions to transform sensor and action variables inside agent systems: transform_ sensors, transform_action, and filtered_ sensor _space.

To transform sensor variables, use the transform_sensor function to calculate changes to specific sensors, then return the complete set of sensor variables (the observation space).

Two of the most common reasons for transforming sensor variables are conversion and normalization. For example, if a simulator reports temperature values in Fahrenheit, but the agent system expects temperature values in Celsius, use the transform_sensor function to convert between the two.

Normalization is when you transform variables into different ranges. For example, one sensor variable in your agent system might have very large values (in the thousands), but another variable might have small values (in the tenths), so you might use the transform_sensor function to transform these disparate sensor values to a range from 0 to 1 so that they can better be compared and used in the agent system.

You may want to transform action variables for the same reasons as sensor variables.

Use the filtered_sensor_space function to pare down the list of sensor variables you need for a particular skill agent. Pass only the information that a skill agent or module needs in order to learn or perform well.

Perceptors use the SDK and CLI workflow.

To access a template for a perceptor, type composabl perceptor new into the CLI. Composabl will then generate a perceptor template that you can populate with your information.

In this simple perceptor example we calculate the perceptor outputs that will be added as new sensor variables and we create a list of perceptors that comprise the perception layer.

In this tutorial, we will walk through how to integrate a trained machine learning (ML) model into your Composabl agent system as a Perceptor. A perceptor allows your agent system to interpret data from sensors, process it using a machine learning model, and output new variables that will help the agent system make better decisions.

The goal is to publish a pre-trained ML model as a perceptor that adds a new layer of perception to your agent system, enabling it to process sensor data in a more advanced way. This could be useful in a variety of scenarios, such as predictive maintenance, anomaly detection, or autonomous decision-making.

Step 1: Understanding the Perceptor

A Perceptor in Composabl is a module in the perception layer that inputs sensor data, processes it (potentially using an ML model), and outputs new variables that are automatically added to the list of available sensors.

For this example, let’s assume we are building a perceptor that uses a trained machine learning model to predict thermal runaway in a system.

Step 2: Setting Up the Trained Model

We will use a pre-trained ML model stored as a pickle file to predict thermal runaway based on certain temperature and chemical sensor readings. Here’s how to set up the trained ML model for use as a perceptor.

1. Store the ML Model: Assume the ML model has been trained and saved as a .pkl file. For this example, the model is stored in the path: ml_models/ml_predict_temperature.pkl.

2. Load the ML Model in the Perceptor: In the perceptor class, we will load the model and define how it processes the sensor data.

Step 3: Creating the Perceptor

Now, we’ll create the perceptor using the trained ML model to process the sensor data and predict thermal runaway events. The perceptor will be responsible for calling the model and returning the prediction as a new sensor variable.

We can start by creating the preceptor by using the Composable CLI with the following command:

composable preceptor new

The new preceptor will have the following file structure:

.└── perceptor_name/    
    ├── perceptor_name/    
        │   ├── __init__.py    
        │   └── perceptor.py    
    ├── pyproject.toml    
    └── README.md

3.1. Configuring your pyproject.toml file

[project]

name = "perc" # Keep this as the same name as the directory
version = "0.1.0"
description = "perc" # You can change this to a description of your project
authors = [    
    { name = "John Doe", email = "john.doe@composabl.com" },
] # Change this to your name and email
dependencies = [    
    "composabl-core",    
    "<third-party-lib>", # Add any third-party dependencies here
]

[composabl]# This metadata should not be changed with exption of the entrypoint
# after the : marker if you change the name of the class. Those names
# should match.
type = "perceptor"entrypoint = "perc.perceptor:DemoPerceptor"

3.2. Implementing the Perceptor in the perceptor.py file

Here’s the Python code to create the perceptor:

import pickle
from composabl import Perceptor, PerceptorImpl
from sensors import sensors  # Assuming sensors are already defined

class ThermalRunawayPredict(PerceptorImpl):
    def __init__(self, *args, **kwargs):
        self.last_Tc = 0  # Store the last observed temperature
        self.ml_model = pickle.load(open("ml_models/ml_predict_temperature.pkl", 'rb'))  # Load the trained ML model

    async def compute(self, obs_spec, obs):
        # Convert the observation to a dictionary using sensor names
        if type(obs) != dict:
            obs_keys = [s.name for s in sensors]
            obs = dict(zip(obs_keys, obs))

        # Calculate delta temperature (change in Tc)
        if self.last_Tc == 0:
            delta_Tc = 5  # Initial change
        else:
            delta_Tc = float(obs['Tc']) - self.last_Tc

        # Prepare the input for the ML model
        X = [[float(obs['Ca']), float(obs['T']), float(obs['Tc']), delta_Tc]]
        
        # Get the prediction from the ML model
        prediction = self.ml_model.predict(X)[0]

        # Update last_Tc for the next iteration
        self.last_Tc = float(obs['Tc'])

        # Return the prediction as a new sensor variable
        return {"thermal_runaway_predict": prediction}

    def filtered_sensor_space(self, obs):
        # Define which sensor values the ML model needs
        return ['T', 'Tc', 'Ca']

In this perceptor:

• We load the trained machine learning model from a pickle file.

• The compute() method takes in sensor data (e.g., temperature, chemical concentrations), processes it, and uses the ML model to predict whether a thermal runaway event will occur.

• The perceptor outputs the prediction as a new sensor variable, thermal_runaway_predict.

3.2. Adding the Perceptor to Your Agent System

Once the perceptor is defined, you can login to the Composabl editor and add it to your agent system.

Conclusion

In this tutorial, we covered how to publish a trained ML model as a perceptor in Composabl. This allows the agent system to integrate more advanced decision-making by processing raw sensor data through a machine learning model and outputting predictions as new sensor variables. This method can be applied in various domains, such as predictive maintenance, anomaly detection, and control systems.

Adding perception modules to your agent can provide richer, complex, condensed, and nuanced information to the decision-making parts of the agent system. For example, you might include a computer vision model in your perception layer that inputs images or video from a camera and outputs classifications of objects that it identifies. You can also add large language models as perceptors to take in and interpret information in natural language.

Each module in the perception layer for a Composabl agent system inputs the sensor variables, processes those variables in some way, and outputs one or more new variables that the platform will automatically add to the list of sensors.

Perceptors can use any supported Python function or library to calculate outputs. They can even call machine learning and large language models or their APIs.

The next three pages explain how to use the SDK and CLI workflow to create new perceptors or configure existing models as perceptors to use in Composabl agent systems.

Add Perceptors to Agent Systems

Just like skill agents, perceptors can be dragged and dropped into agent systems using the UI. Perceptors will always be situated in the Perception layer that comes before orchestrators and skill agents. That’s because perception needs to be applied to the sensor inputs to create new variables that are then passed to the skills layer for the agent system to use in decision-making.

Scenarios are situations where your agent system needs to behave differently to succeed.

Scenarios are created by carving out sections of the simulation space, as defined by specific configurations of variables and variable ranges. This allows you to train skill agents only in the scenarios where they will need to perform, leading to more efficient training and better performance. Orchestrators learn to recognize scenarios and pass control to the appropriate specialist skill agent to handle the scenario’s demands.

Types of Scenarios

How you define scenarios depends on the type of sensor variables you are working with.

Discrete variables are categories that describe a particular scenario. For each scenario, there is likely a perceptor in the agent system, such as a machine learning model, that inputs the sensors, processes the sensor values, and outputs the discrete variable category.

Continuous variables are numbers. When they are used to define a scenario, one number value determines the section of the space that your agent system will explore for decisions.

Sometimes a scenario is better defined by a range of continuous values than by a single continuous value. In that case, the scenario would be defined by a continuous variable range.

Here are some examples of how scenarios can be defined in different ways:

Scenarios Example

Let’s say that we are creating an agent system to plan the operations of a restaurant with varying demand for three different recipes. We could create three different scenarios:

• Low demand: 30 recipe A, 20 recipe B, and 10 recipe C

• Normal demand: 60 recipe A, 45 recipe B, and 10 recipe C

• High demand: 100 recipe A, 50 recipe B, and 25 recipe C

Once you have defined these scenarios, you could set the agent system to train in the low-demand scenario until the success criteria are reached. Then it would move to normal demand, and then to high demand. We would expect the agent to learn faster as it progressed through the scenarios, with cumulative knowledge building over time.

Set Up Scenarios in the UI

Add Scenarios to a Use Case

Add scenarios to your use case by clicking on Scenarios from the left-hand menu to open the Scenarios page.

Click Add scenario to create a new scenario. Give your scenario a name and then click Add condition to configure it. You will then have the option to choose from any of the available sensor variables and apply conditions to them.

• For continuous variables, select Is and enter the exact value.

• For continuous variable ranges, select Is between and enter the range.

• For discrete variables, select Is element of and enter the possible values.

Create Scenario Flows

In addition to identifying scenarios, Composabl also lets you teach your agent systems about how scenarios relate to each other. To do this, you can build scenario flows to establish sequences of scenarios for the agent system to practice.

Without scenario flows, the training platform will connect scenarios at random. But sometimes it is important for agent systems to practice scenarios in a specific order. For example, it might be important to practice flying a drone in high winds and then landing in the same conditions. Or an agent managing a production line might need to practice going from a scenario in which raw materials are scarce to one in which material costs rapidly rise. By creating a scenario flow, you can ensure that a given sequence of scenarios will be included in training.

Once you have scenarios built, create sequential sections of the task by clicking Add section.

After creating all the sections you need, drag and drop the scenarios into the sections.

Create as many flows as you need to capture the different scenario sequences your agent system will need to navigate.

Add Scenarios to Skill Agents

Scenarios are added to agent systems as part of configuring skill agents. Scenarios are added to skill agents so that they know what specific conditions they need to master. Not all scenarios are relevant to all skill agents. For example, a drone landing skill doesn’t need to practice taking off in windy weather.

The configurations you set up when building the scenario flows will appear in the skill agent configuration modal. For each section of the process, as defined by the flows, tell the orchestrator which scenarios it should apply by checking the boxes next to each scenario.

Skill agents can be arranged in sequences or hierarchies, in skill groups, or as coordinated skills that output multiple decisions together. The orchestration structures reflect common design patterns that can be used to accelerate the design and creation of agent systems.

Orchestrate Skills in Hierarchies and Sequences

For some agent system designs, the task will be broken down into different skill agents that each control the system under certain conditions. For these agent systems, a special skill called an orchestrator chooses the right skill agent at the right time to accomplish the task. Orchestrators are the specialized supervisor skill agents that orchestrate the skill agents together, determining which skill agent to activate based on the conditions the system needs to respond to.

To add an orchestrator to an agent system, drag the orchestrator into your agent system above the skills layer.

You will then be prompted to configure the orchestrator.

You will also be prompted to choose between an additional set of options that correspond to two separate Machine Teaching design patterns.

• Fixed-order sequence: perform the skills in a set order. This is used in the Functional Pattern, a design pattern that is useful for tasks that involve fixed sequences of actions.

• Variable order sequence: perform the skills in any order based on the orchestrator’s determination. This is used in the Strategy Pattern, a design pattern that is useful for tasks that require different control strategies to be used in different situations or conditions.

Orchestrate Skill Agents in Groups

Unlike agent system designs that use an orchestrator to assign control to skills one at a time, agent systems with skill groups use skills working together to make decisions.

Skill groups always consist of two or more skill agents. To create a skill group, simply drag the second skill under the first, and a skill group will automatically be created.

Skill groups are used for the Plan-Execute Pattern, where one skill determines what the action should be and a second skill then “turns the knobs” to implement the decision.

In the industrial mixer example, the DRL skill agent is able to train effectively because the actions of the MPC controller are predictable. That means that it can practice and learn knowing that variations in performance are due to its own actions.

In agent systems with multiple DRL skill agents arranged in plan-execute patterns, Composabl will always train the skills from the bottom to the top. In other words, the execute skill will have to achieve competence before the plan skill will start training. That allows each skill to effectively interpret the feedback from the system without confusion from each other.

Orchestrate Coordinated Skills

Some tasks require multiple skill agents to work together on a single decision, but in parallel rather than in sequence. Skill agents for these tasks use coordinated skills that learn to take action together toward a goal. Also known as Multi-Agent Training, coordinated skills are trained using a coach, rather than a teacher.

Examples of Coordinated Skill Agents

Traffic Optimization: Enhancing traffic flow and safety by teaching individual vehicles to navigate optimally and cooperate with each other.

Collaborative Robotics: Enabling robots to work together on tasks such as assembly in manufacturing or coordination in logistics.

Smart Grids: Optimizing energy distribution by having agents represent power plants, storage, and consumers to improve efficiency and stability.

Multiplayer Games: Creating adaptive and intelligent NPCs that can offer dynamic challenges to players in competitive or cooperative game settings.

Communication Networks: Improving network performance by optimizing resource allocation and traffic routing through agents representing network components.

Environmental Management: Balancing economic, ecological, and social goals in land use and resource management by simulating stakeholders as agents.

Healthcare Logistics: Strategizing resource allocation and treatment plans in scenarios like pandemics by considering the actions of hospitals, pharmacies, and patients as agents.

Supply Chain Optimization: Minimizing costs and delivery times in supply chains by coordinating agents representing various stages of the supply chain process.

Orchestrate Coordinated Skills with the SDK

Coordinated skill agents are not yet available in the UI. In the SDK, we have expanded the API to integrate Coordinated Skills through the add_coordinated_skill method on your agent. This method accepts a new class that gets configured, named CoordinatedSkill, just as with the Teacher or Controller classes we implement this class by inheriting from the Coach class.

The coordinated skill agent will now take the incoming observation and action spaces and pass it to the sub-skills as a shared environment observation and action taking. The sub-skills will then return their observations and actions, which will be passed back to the coordinated skill agent. The coordinated skill agent will then return the combined observations and actions to the agent system.

python
# ####################################################################################################
# Define the Coordinated Coach
# ####################################################################################################
class CoordinatedCoach(Coach):
    def __init__(self):
        self.counter = 0

    def compute_reward(self, transformed_ sensors, action, sim_reward):
        """
        Computes the reward for the given transformed observation and action
        :param transformed_ sensors: The transformed observation
        :param action: The actions dict
        :param sim_reward: The reward from the simulation
        :return: The reward, as a dictionary, with each key the sub-skill name and the value the reward
        """
        self.counter += 1
        return 1

    def compute_success_criteria(self, transformed_ sensors, action):
        # keep the episodes short to make testing quicker
        return self.counter > 100

    def compute_termination(self, transformed_ sensors, action):
        # keep the episodes short to make testing quicker
        return self.counter > 150

    def transform_action(self, composabl_ sensors, action):
        return action


# ####################################################################################################
# Construct your Agent
# ####################################################################################################
s1 = Skill("skill1", IncrementTeacher)
s2 = Skill("skill2", IncrementTeacher)

a = Agent()
a.add_coordinated_skill(CoordinatedSkill(
  "my-coordinated-skill",
  CoordinatedCoach,
  [s1, s2]
))

A Two-Part Platform

Composabl is a two-part platform with a no-code UI and a Python SDK and CLI. The interplay of these parts gives Composabl its combination of usability and power.

The two parts enable teams to work together effectively. People and roles who primarily use code, such as data scientists and controls engineers, use the SDK to create modules like ML models and deep reinforcement learning skills. Then, subject matter experts, process engineers, and others can use the no-code interface to easily create teams of agents from these modular building blocks and train them to succeed.

We designed the platform this way because, for complex, high-value processes, there will be some tasks that can only be done through coding, and some team members who prefer to work in code, and other tasks that are better performed through a visual interface. Both parts of the platform work together.

Most users use both parts of the platform to some extent but spend more time in the no-code UI or the SDK, depending on their expertise and role. How you will use the platform depends on your role and what part of the process you are working on.

Set Up Your Environment

Access Composabl

You can access Composabl Via a no-code UI and an SDK. They work together to enable you to build, train, and deploy autonomous agent systems.

No-Code UI

Python CLI and SDK Instalation

Installing the SDK

pip install -U composabl

Development

pip install -U composabl-dev

Note: The Composabl CLI and SDK require Python version 3.10.x

Workflow Steps

• Step 3: UI and SDK | Create agent systems

• Step 7: Notebook | Export Historian training data and perform detailed analysis

You can use an LLM as a perceptor to add language and communication capabilities to your agent system.

This allows you to create human-like assistants or copilots who can contribute natural language capabilities to your agent system. Composabl has several personas for LLM assistants to help structure your agent design.

• The analyst interprets sensor data and passes it to an interface that the user can access, allowing real-time monitoring of conditions and the agent system's responses.

• The executive reads external data sources in text and reports information to the agent system, such as trends in the business press that would help to anticipate demand for a product

• The plant manager allows operators to communicate directly with the agent system and gives it instructions based on information that would not be otherwise available in its sensor space

LLM perceptors can either:

1. Output language to the operator about what the agent system is doing (ex. the analyst)

2. Take in inputs in natural language and then transform them into information that the decision-making layer of the agent can use (ex. the executive and plant manager)

Create an LLM Perceptor

Step 1: Create the perceptor shell

From the CLI, when logged into Composabl, type composabl perceptor new. You will be prompted for a location to save your new perceptor, and then a new directory with your perceptor will be created.

This will include the pyproject.toml file that will allow you to publish the perceptor to the UI once it is created.

Step 2: Define the perceptor class

Within the perceptor.py file, create the API call and prompt for the LLM.

Analyst Perceptor Code Sample

The analyst displays information to the human user, but doesn't send information to the decision-making layer of the agent, so the perceptor returns 0.

Executive Code Sample

The executive in this sample related to the industrial mixer use case automatically queries a chemical engineering LLM for advice about control actions to take.

The perceptor returns an action that it recommends the decision-making layer of the skill agent to take. This becomes a new sensor variable that the skill agent teacher(s) will take into account when training the agent system in simulation.

Examples and Reference

Step 3: Filter the Sensor Space

Composabl agent systems can include text fields in perceptors, but they must be transformed or filtered out in the teacher.py file before training with DRL. For any text variables that are not transformed into a different data type, use the filtered_sensor_space method of the teacher to remove them.

Step 4: Publish the Perceptor

Publish the perceptor to the UI.

Naviage to the folder above your perceptor. Then publish your perceptor.

Select the organization and project that your perceptor to add your perceptor. The refresh your Agent Orchestration Studio to see the perceptor and add it to agents.

The strategy pattern is one of the key design patterns of machine teaching. When you use the strategy pattern, you break down the task into specific skill agents that each handle one aspect of the process to be controlled. This allows you to "teach" the agent system using subject matter expertise.

In the strategy pattern, each skill agent is either trained using deep reinforcement learning or controlled with a programmed algorithm. Then, a special skill agent called an orchestrator decides which skill agent should make the decision based on the current conditions.

In the industrial mixer problem, the process is divided into three skill agents based on the phase of the process. All three action skill agents and the orchestrator are trained with DRL: each skill agent practices in the conditions it will face and learns to control its part of the process by experimenting over time.

Think of the strategy pattern as like a math class with three students. Student A loves fractions, Student B is great at decimal problems, and Student C thinks in percentages. The orchestrator is their teacher. She reads each question, sees what kind of problem it is, and then assigns it to the student who can solve it best, because of their own special math talent.

Let's get started configuring this agent system!

1. Create your skill agents

This agent system has three skill agents called Start Process ,Control Transition , and Produce Product . To create these skill agents in the UI, go to the skill agent page and click Add skill agent . Create all three skill agents and then set the goals and constraints.

2. Set Skill Agent Goals and Constraints

The goal for these skill agents is to maximize yield, and the constraint is to keep the temperature from going above 400 degrees Kelvin.

The goals and constraints are exactly the same in all three skill agents. The agents become specialized during training, as each skill agent trains in a different scenario, corresponding with the three phases of the process. We will create these scenarios later in the tutorial.

1. Click Add goal In the left drop-down menu, select Maximize, and in the right one, select Eps_Yield .

2. Click Add constraint. In the left drop-down menu, select Avoid , and in the right one, select T . After you select T you're going to see a slider appear for you to set boundaries you want to train the system to avoid. In this case, we want to set the boundaries from 400 to 500.

3. Save your skill agent and return to the Agent Orchestration Studio.

3. Add the Skill Agents to Your Strategy Pattern Agent System

Drag the skill agents Start Process, Control Transition, and Produce Product that you can now see on the left-hand side of your use case onto the skills layer. Drag the skill agents from the side in the order you want them to be used.

3. Configure the Orchestrator

The green diamond that appears when you place multiple skill agents alongside each other is the orchestrator. This is the "math teacher" agent that decides which specialized skill agent should be chosen to make each decision.

The goals of the top-level orchestrator in an agent will typically be the same as the goals of the agent system as a whole. So, we can set it to Maximize Eps_Yield.

A fixed-order sequence is appropriate for a phased process like the industrial mixer reaction. That means the orchestrator has the skill agent apply the skill agents one at a time, rather than switching back and forth between skill agents.

4. Configure Scenarios

This is what allows the skill agents to differentiate from each other. The three specialized skill agents practice only in their designated phase of the process and learn to succeed in their own specific conditions. The orchestrator practices with the whole process so that it knows which skill agent to choose at any point.

4.1 Add Scenarios

Go to the Scenarios page using the left-hand navigation menu. Click Add Scenario to create a new scenario for your agent to use in training.

When building an agent system for your use case, you will define the scenarios based on your knowledge of the task and process. In this case, we provide the values that define the phases of the chemical manufacturing process. Create these scenarios for your agent:

• Full reaction: Cref Is 8.57, Tref Is 311 | Why these numbers?

• Startup: Cref Is 8.5698, Tref Is 311.2612 | Why these numbers?

• Transition: Cref Is 8.56, Tref Is 311, count Is 22 | Why these numbers?

• Production: Cref Is 2, Tref Is 373.1311 | Why these numbers?

4.2 Create Scenario Flows

Scenario flows allow you to connect scenarios that have a sequential relationship to ensure that your agent gets practice in navigating the different conditions in the order in which they will occur.

For this problem, you do not need to create sequential connections between the scenarios. Drag all the scenarios to the first column to make them available to your skill agents and orchestrators.

4.3 Add Scenarios to Skill Agents and Orchestrators

Once you have your scenarios set up and connected with scenario flows, you can add them to skill agents and orchestrators to tell the skill agents and orchestrators what conditions they need to practice in. This helps them to develop their specialized expertise.

In the Agent Builder Studio, click on each skill agent and the orchestrator in turn. Check the box for each scenario to apply to the skill agent.

• Start process: Startup

• Control Transition: Transition

• Produce product: Production

• Orchestrator: Full reaction

5. Run Your Training Session

We are ready to train your agent system and see the results. Select the cluster you want to use and the number of training cycles. We suggest you run 150 training cycles.

You will see the skill agents training one at a time, and you assign the number of cycles you want each skill agent to use. It will automatically assign an equal number of training sessions for each skill agent, but in some agent system designs, some skill agents might require more training than others. For example, in this use case, the transition phase is more difficult to control than the two steady states, so the Control Transition skill agent may need more training time than the others to become effective.

6. View Results

When the training has been completed, you can view your results in the training sessions tab in the UI. This will show you information on how well the agent is learning.

You will likely see a steep learning curve as the agent experiments with different control strategies and learns from the results. When the learning curve plateaus, that usually means that the skill agent is trained.

Analyzing the Strategy Pattern Agent's Performance

Conversion rate: 92% Thermal runaway risk: Low

We tested this fully trained agent and plotted the results.

In this tutorial, we will explore how to use the historian to validate the trained AI agent system in Composabl and training logs. The historian stores historical time-series data in an optimized format (parquet) - https://www.databricks.com/glossary/what-is-parquet, which helps in evaluating how the agent is performing during training.

Step 1: Accessing the Historian Data

The historian file stores time-series data essential for validating agent system training. There are several ways to access and store the historian data, but the recommended format is as a delta file (parquet).

1. Understanding the Format:

   • The historian data is typically large, around 500 megabytes for standard operations. It is stored in a Delta Lake file format, optimized for time-series data and supporting efficient queries.

2. Downloading the Historian File:

   • From the Composabl UI, download the historian file. This file will come in a compressed format (e.g., .gz).

   • After extracting it, you should see the delta file containing time-series data.

Step 2: Setting Up for Validation

1. Unpacking the Historian File:

   • If the historian file is compressed (e.g., .gz), unpack the file using a tool like gzip:

     Copy

     
     gunzip -k historian_file.gz 
     

   • Once unzipped, you’ll see a 10 MB+ delta file with historical time-series data.

2. Understanding the Delta File:

   • The delta file is optimized for fast reads and writes of time-series data.

   • It supports an append-only structure, which ensures that each new piece of data can be added efficiently without modifying the existing data.

Step 3: Querying the Historian Data

1. Setting Up a Query Environment:

   • To validate your agent system’s training, you’ll need to set up an environment that allows you to query the delta file. Delta Lake integrates well with systems like Apache Spark, but for simple querying, you can use tools like pandas in Python.

2. Querying for Agent Training Logs:

   • Extract and analyze relevant historical data from the delta file. Here's a simple Python example for querying the delta file using pandas:

   Copy

   
   import pandas as pd 
   
   
   
   # Load the historian delta file 
   
   df = pd.read_parquet('historian_delta_file.parquet') 
   
   df = df.sort_values(by=['timestamp'])
   
   df_data = df[df['category_sub'].isin(['step', 'skill-training','skill-training-cycle'])]
   #filter df with composabl_obs on "data" col only
   df_data = df_data[(df_data['data'].str.contains('composabl_obs')) | (df_data['category_sub'].str.contains('skill-training')) | (df_data['category_sub'].str.contains('skill-training-cycle'))]
   
   #df_data['data'] = df_data['data'].apply(lambda x: x if 'composabl_obs' in x else None)
   def convert_to_dict(x):
      try:
         return json.loads(x)
      except:
         try:
               return ast.literal_eval(x)
         except:
               return None
   
   df_data['data'] = df_data['data'].apply(lambda x: convert_to_dict(x))
   
   df_data['skill_name'] = df_data['data'].apply(lambda x: x['name'] if 'is_done' in x else None)
   df_data['skill_name'] = df_data['skill_name'].fillna(method='bfill')
   
   df_data['reward'] = df_data['data'].apply(lambda x: x['teacher_reward'] if 'composabl_obs' in x else None)
   
   df_data['obs'] = df_data['data'].apply(lambda x: x['composabl_obs'] if 'composabl_obs' in x else None)
   
   #df_data['done'] = df_data['data'].apply(lambda x: x["teacher_terminated"] if "teacher_terminated" in x else None)
   df_data['cycle'] = df_data['data'].apply(lambda x: x['cycle'] if 'cycle' in x else None)
   df_data['cycle'] = df_data['cycle'].fillna(method='bfill')
   
   df_data = df_data[df_data['category_sub'] == 'step']
   
   print(df_data)
   
   # group by runs
   df_group = df_data.groupby(['run_id','skill_name','cycle'])['reward'].mean()
   
   # Process observation data
   df_obs = pd.DataFrame(data=[[v[0] for v in list(x.values())] for x in df_data['obs'].values], columns=[list(df_data['obs'][0].keys())])
   
   df_obs['cycle'] = df_data['cycle']
   df_obs['run_id'] = df_data['run_id']
   df_obs['skill_name'] = df_data['skill_name']
   df_obs.columns = [x[0] for x in list(df_obs.columns)]
   
   # Episode Reward by Run Id
   for run_id in list(set([x[0] for x in df_group.index])):
      for skill in list(set([x[1] for x in df_group.index])):
         #df_group[run_id].plot(subplots=True, title=run_id)
         plt.plot(df_group[run_id][skill])
         plt.ylabel(f'Mean Episode Reward')
         plt.xlabel(f'Cycle')
         plt.title(f'{run_id} - {skill}')
   
         plt.show()
   

Key Benefits of Using the Historian for Validation:

• Optimized Data Handling: The Delta Lake format is designed for fast querying, making it ideal for time-series data.

• Efficient Storage: The append-only nature ensures that new data can be added without overwriting or modifying existing data, making it easy to track data over time.

• Continuous Monitoring: By continuously adding data to the historian, you can validate your agent system's long-term impact on machine performance, uptime, and safety.

Analyzing agent system behavior is a key part of using Composabl. This capability is key for:

• Building agent systems that beat performance benchmarks

• Monitoring the performance of your agent systems and simulators in real-time

• Analyzing your agent systems and simulators to understand what went wrong in case of problems

• Creating dashboards and reports to share with your team or customers

Composabl offers multiple options for analyzing agent system behavior. For high-level summaries, you can use the benchmark reporting features in the UI. For more in-depth information, you can use the Historian and raw benchmark data in the SDK, which creates a database of all the information gathered during training and deployment. You can also design an LLM module in your agent to provide customizable communications capacity using natural language.

Once you have completed training an agent and are ready to deploy it you can download it from the UI.

1. Go to the Training tab in your project.

2. Select the training session for the agent you want to deploy.

3. Click on the Artifacts drop down in the upper right of the screen.

4. Click Inference Model and you will down load a file called agent.json.

You can use Composabl to create skill agents that learn with deep reinforcement learning. Configure the Composabl teacher by setting goals, constraints, and success criteria for your skill agent. Composabl then turns these into reward functions and trains the skill agent.

Create a New Skill Agent

To create or edit a skill agent, follow these steps:

1. Navigate to the Skill Agents page

2. Click on a skill agent, or click + to create a new skill agent

3. Configure the skill agent

Use the dropdown menus to select the variables and then define the parameters for each goal, constraint, or success criterion you want to include.

Choose Implementation Method

To create a new skill agent that learns with deep reinforcement learning, select Teacher under the Implementation Method dropdown. The other option, Controller, is used for a programmed skill agent that is configured with the Composabl SDK and published to the Agent Orchestration Studio.

Configure Goals

Goals define what a skill agent should do. Goals apply to one of the sensor variables and are defined using one of three possible directives:

• Maximize: Maximize the value of the variable

• Minimize: Minimize the value of the variable

• Maintain: Keep the variable at a specified value or track a specified variable (such as a set point)

Configure Constraints

Constraints set the boundaries for the skill agent. They define rules that the skill agent must follow during operation. Constraints are defined using one of two possible directives:

• Avoid: The skill agent learns to keep the variable from reaching a specified value or range through withholding rewards

• Terminate: When the actions of the skill agent lead to certain conditions within a variable, the skill agent has failed and must stop and start a new episode

Configure Success Criteria

Success criteria tell the skill agent when it's doing something right. They are defined using one of two possible directives:

• Approach: The skill agent learns to get close to a specified value by getting increased reward

• Succeed: When the success criteria are achieved, the session ends, and a new one begins so that the skill agent can keep practicing and learn to win every time

In this tutorial, we will learn how to upload simulators via the Composabl CLI as well as your custom, adapted simulators to the Composabl UI.

Upload simulators via Composabl CLI

To upload simulators that already follows the Composabl simulation specification all you need to do is the fallowing command from the simulation folder:

composabl sim publish

• After that, you can go to the Composable editor and connect that sim to any project.

Upload Third-Party Simulators via Docker

Prerequisites

If your simulator isn't already compatible with the Composabl platofrm you will need to create gRPC bindings and upload it as a Docker image in the Composabl editor. You can follow along with these Composabl API patterns and the following structure:

.
└── your-simulator-folder/
    ├── docker/
    │   └── entrypoint.sh
    ├── src/
    │   ├── exceptions/
    │   │   └── invalid_usage.py
    │   ├── __init__.py
    │   ├── main.py
    │   ├── server_impl.py
    │   └── sim.py
    ├── Dockerfile
    └── requirements.txt

Going through the files:

• docker/entrypoint.sh: This file is the entrypoint of your Docker container.

• src/exceptions/invalid_usage.py: This file contains the exception class InvalidUsage that is used to raise exceptions in the simulator.

• src/__init__.py: This file is the initialization file of the module. No code is needed here, but for Python to recognize the folder as a module, this file is necessary.

• src/main.py: This file is the main file of the simulator. It uses the composabl_core.networking module to expose the simulator to the Composabl platform. This file is also available zipped along with this tutorial.

• src/server_impl.py: This file contains the implementation of the server that will be used to run the simulator.

• src/sim.py: This file contains your implementation of the simulator itself. Usually, a Env (inheriting from gym.Env) class is implemented here, and it is used to run the simulator.

• Dockerfile: This file is the Dockerfile that will be used to build the Docker image of your simulator.

• requirements.txt: This file contains the Python packages that are necessary to run your simulator. It is used to install the necessary packages in the Docker image.

gRPC Spec for Implementation

Under the hood, the Composabl SDK uses gRPC to communicate with the Composabl platform.

To create a simulator that works with the Composabl API, you have to implement the following gRPC methods:

service Composabl {
  // Creates the environment with specified configurations.
  // - MakeRequest: Contains parameters to configure the environment.
  // - MakeResponse: Returns an identifier for the created environment and possibly other initial setup information.
  rpc make(MakeRequest) returns (MakeResponse) {}

  // Advances the environment by one timestep using the action provided in the request.
  // - StepRequest: Includes the action to be taken in the current state of the environment.
  // - StepResponse: Returns the new state of the environment, reward received, and a flag indicating if the episode has ended.
  rpc step(StepRequest) returns (StepResponse) {}

  // Resets the state of the environment, returning it to its initial conditions.
  // - ResetRequest: May include parameters for resetting to specific scenarios.
  // - ResetResponse: Provides the initial observation of the reset environment.
  rpc reset(ResetRequest) returns (ResetResponse) {}

  // Performs any necessary cleanup before the environment is closed.
  // - CloseRequest: May be empty or include specific closing instructions.
  // - CloseResponse: Acknowledges the environment has been successfully closed.
  rpc close(CloseRequest) returns (CloseResponse) {}

  // Generates a sample action from the environment's action space.
  // - ActionSpaceSampleRequest: May be empty or specify particular sampling criteria.
  // - ActionSpaceSampleResponse: Provides a sample action from the action space.
  rpc action_space_sample(ActionSpaceSampleRequest) returns (ActionSpaceSampleResponse) {}

  // Retrieves information about the environment's action space.
  // - ActionSpaceInfoRequest: May be empty or include parameters for the information request.
  // - ActionSpaceInfoResponse: Returns detailed information about the action space.
  rpc action_space_info(ActionSpaceInfoRequest) returns (ActionSpaceInfoResponse) {}

  // Retrieves information about the environment's observation space.
  // - ObservationSpaceInfoRequest: May be empty or include parameters for the information request.
  // - ObservationSpaceInfoResponse: Returns detailed information about the observation space.
  rpc observation_space_info(ObservationSpaceInfoRequest) returns (ObservationSpaceInfoResponse) {}

  // Sets the current scenario for the environment.
  // - SetScenarioRequest: Includes parameters defining the scenario to set.
  // - SetScenarioResponse: Acknowledges the scenario has been set.
  rpc set_scenario(SetScenarioRequest) returns (SetScenarioResponse) {}

  // Retrieves the current scenario of the environment.
  // - GetScenarioRequest: May be empty if simply retrieving the current scenario.
  // - GetScenarioResponse: Returns details of the current scenario.
  rpc get_scenario(GetScenarioRequest) returns (GetScenarioResponse) {}

  // Sets the render mode of the environment.
  // - SetRenderModeRequest: Includes parameters for the desired render mode.
  // - SetRenderModeResponse: Confirms the render mode has been set.
  rpc set_render_mode(SetRenderModeRequest) returns (SetRenderModeResponse) {}

  // Retrieves the current render mode of the environment.
  // - GetRenderModeRequest: May be empty if simply querying the current mode.
  // - GetRenderModeResponse: Returns the current render mode.
  rpc get_render_mode(GetRenderModeRequest) returns (GetRenderModeResponse) {}

  // Retrieves the current render of the environment.
  // - GetRenderRequest: May include parameters specifying the render details.
  // - GetRenderResponse: Provides the current render of the environment.
  rpc get_render(GetRenderRequest) returns (GetRenderResponse) {}
}

After making sure that your simulator is compatible with the Composabl platform, you can proceed to the next section.

Docker

• Go to the folder where your simulator is located. We can navigate to the simulator folder and see what is inside it.

• After that, ensure that you have Docker installed. You can check if Docker is installed by running the following command:

  Copy

  docker --version

DockerHub

• To log in to Docker Hub, run the following command:

  Copy

  docker login

  You will be prompted to enter your Docker Hub username and password. After that, you should see a message saying that you are logged in.

• Now, we can build the Docker image of the simulator. To do so, run the following command:

  Copy

  docker build -t <your-docker-hub-username>/<simulator-name> .

  This command will build the Docker image of the simulator. The -t flag is used to tag the image with the name <your-docker-hub-username>/<simulator-name>. The . at the end of the command indicates that the Dockerfile is in the current directory.

• After building the image, you can check if it was built successfully by running the following command:

  Copy

  docker images

  And then push the image to Docker Hub:

  Copy

  docker push <your-docker-hub-username>/<simulator-name>

Composabl UI

After that, you can go to the Composabl UI and upload your simulator. To do so, follow the steps below:

• Then, on the left sidebar, click on the "Simulators" tab. You should see a list of simulators that are already available on the platform.

  On the top right corner, you should see a button to "New Simulator". Click on it.

• A pop-up will appear, asking you to select between "Internal" and "External" simulators. Select "External".

• Then, you can fill the Title and Description of the simulator. We suggest smaller names for the Title and a more detailed description for the Description.

• After clicking in next step, a brief tutorial will open up. Take care to read it and then click on "Next Step" again.

• After that, you can fill the Docker Image field with the name of the Docker image you pushed to Docker Hub <your-docker-hub-username>/<simulator-name>.

  If the image is public, no more fields are needed. If the image is private, you need to fill the Docker Username and Docker Password fields with your Docker Hub username and password, respectively. Then, click on "Validate and next step".

Conclusion

If you've followed all the steps correctly, you should have successfully uploaded your simulator to the Composabl UI. You can now use your simulator to train agents and run simulations on the platform. If you have any questions or need help, feel free to reach out to us.

In this tutorial, we will cover how to connect to the agent system runtime, load a pre-trained agent system, run inference, and visualize the results in a production-like environment. The provided script, agent_inference.py, is a key component that demonstrates connecting to the Composabl agent system runtime, initializing the environment, and plotting agent system operation results.

Step 1: Understanding agent_inference.py

The script agent_inference.py connects to the runtime, loads a pre-trained agent system, connects to a local simulation, collects sensor data from the sim and plots the results. Here is an outline of the core steps in the process:

1. Start Runtime and Load Agent System: The script initializes the trainer and loads a pre-trained agent system from a model folder.

2. Set Up the Simulation Environment: It connects to a simulation environment.

3. Run Inference: The pre-trained agent system interacts with the simulation to perform inference (decisions), collecting observations and giving actions at each step.

4. Collect Data and Plot Results: Sensor data and actions are collected in a Pandas DataFrame, and the results are plotted using Matplotlib to visualize how the agent system is performing over time in a production like environment.

Step 2: Connecting to the Runtime and Loading the Agent System

The first task is to connect to the Composabl runtime and load the pre-trained agent system. This is accomplished using the Trainer and Agent classes. The agent system's model is loaded from the directory where the model was saved during training.

Here:

• Trainer(config) initializes the runtime with a configuration file.

• Agent.load(PATH_CHECKPOINTS) loads the saved agent from the specified checkpoint directory.

• trainer._package(agent) prepares the agent for inference by packaging it.

Step 3: Connecting to the Simulation Environment

Next, we connect the agent system to the simulation environment. The make() function creates a connection to the local simulator, and the environment is initialized.

Here:

• The simulator is configured to run locally (localhost:1337) and you have to start it locally and manually before.

• The environment is initialized with sim.init(), and the agent system is connected to it.

Step 4: Setting the Scenario and Running Inference

After connecting to the simulator, you need to set up the specific scenario that the agent system will operate in. This scenario determines the environment's initial state.

With the environment set, the agent can now run inference for a set number of iterations. At each iteration, the agent observes the environment, takes an action, and collects the results (observations and rewards). This is done in a loop.

In each iteration:

• The agent system performs an action based on the current observations.

• The environment advances one step with sim.step(action), and the agent receives a new observation and reward.

• Sensor data and actions are logged into a Pandas DataFrame for later analysis.

Step 5: Saving Data and Plotting Results

Once the inference loop is complete, the collected data is saved, and the results are visualized. The results are plotted using Matplotlib.

This code generates three subplots:

1. Temperature Controller (Tc) over time.

2. Temperature (T) and Reference Temperature (Tref) over time.

3. Concentration (Ca) and Reference Concentration (Cref) over time.

The plots provide a visual representation of the agent system's performance during the simulation. Finally, the figure is saved as inference_figure.png in the benchmarks directory.

Step 6: Running the Script

To run the script, execute the agent_inference.py in your terminal.

Conclusion

In this tutorial, we demonstrated how to:

• Connect a pre-trained Composabl agent system to a runtime and simulation environment.

• Set up a scenario and run inference.

• Collect observations and actions, and plot the results using Matplotlib.

By following these steps, you can visualize the performance of your agent system and gain insights into how it interacts with the environment over time.

The Composabl benchmarking feature allows you to compare the performance of different multi-agent systems against key performance indicators (KPIs). This tool helps you evaluate your agent system's effectiveness, track its improvements, and calculate potential return on investment (ROI).

Benchmarks are generated only after training is complete. You won't see benchmark data while training progresses, even for long-running jobs. If a training job fails, no benchmark data will be generated.

Note: Benchmarking does not influence agent system training, goals, or rewards. It is purely an analysis tool that helps you answer the question: "Based on how this agent system was trained and how we believe it should perform in real life, how much money is this agent system generating?"

Getting Started

Define the KPI for Your Project

The key performance indicator (KPI) for your project is the top-level objective that you will use to compare and evaluate your multi-agent systems' performance.

To define the KPI for your project, choose the sensor variable that represents the metric that best indicates performance success. For most processes, this is a metric with a clear business impact, such as product yield or energy use.

Setting up the KPI

To set up the KPI:

1. Navigate to the Benchmark page from the main dashboard

2. Click the "Settings" button in the top right of the KPI Performance Metrics section

3. In the "Set up KPI" section, configure the following:

   • Sensor Name: Select a sensor or preceptor variable to track (e.g., "Eps_Yield")

   • Target Value: Set the desired target for this KPI (e.g., 0.33)

   • Benchmark Unit: Select the unit of measurement (e.g., "% of Max Theoretical Yield"). Note that this is for UI readability only and doesn't affect calculations.

Setting up ROI Calculations

Return on Investment (ROI) calculations help quantify the financial impact of your agent systems' performance. The ROI is calculated based on the monetary value of the difference between your target value and the actual value of the KPI generated by that agent system, minus training costs.

1. In the Settings modal, navigate to the "Enter ROI criteria" section

2. Define the conditions for calculating ROI:

   • When: Select the metric to track (e.g., "Eps_Yield")

   • Condition: Choose how the metric changes (e.g., "increases by")

   • Percentage: Enter the percentage change that matters (e.g., 2%)

   • Value: Set the financial value of this change (e.g., $1,000,000)

   • Period: Select the time period (e.g., "per year")

For example: "When Eps_Yield increases by 1% of max theoretical yield, it is worth $1,000,000 per year."

You can change KPI and ROI settings at any time after training completes, and the benchmark results will be automatically recalculated. This allows you to explore different business scenarios without retraining your agent systems.

Reading the Benchmarking Dashboard

Understanding the Overview Panel

At the top of the benchmarking page, you'll find an Overview panel containing three key metrics:

• Highest performing Agent System: Displays the outcome of the highest performing agent system

• Lowest performing Agent System: Displays the outcome of the lowest performing agent system

• ROI: Shows the calculated financial return based on your KPI settings

Each metric displays both the value and the relevant unit (e.g., "% of Max Theoretical Yield").

Bar Graph Visualization

The bar graph in the KPI Performance Metrics section provides a visual comparison of your agent systems:

• Each bar represents an agent system's performance on your selected KPI (all bars correspond to the same KPI)

• The height of the bar indicates the specific value of the KPI for that agent system (Y-axis)

• Horizontal dashed lines show benchmark averages

• The actual value of the KPI for each agent system is shown above the respective bar

Performance Ranking Table

Below the graph is a detailed table showing:

• Agent System Number: Sequential ID for each agent system

• Agent System Name: Name or description of the agent system

• ROI: Calculated financial return based on the agent system's performance

  • Green arrows (↑) indicate positive ROI with the percentage and absolute value

  • Red arrows (↓) indicate negative ROI with the percentage and absolute value

• Eps_Yield (variable) per % of Max Theoretical Yield: Performance metric with indicators for highest agent system performers

Based on the notes provided in the image, I'll help you create a tutorial on accessing the agent system runtime after deploying it to Docker. This tutorial will explain the steps for building the Docker container, deploying the agent system, and accessing the runtime for inference or interaction.

Tutorial: Accessing the Agent System Runtime After Deploying to Docker

Once you have packaged and deployed your agent system inside a Docker container, the next step is accessing its runtime for operations like model inference. This tutorial will guide you through the process of building and running the Docker container and then connecting to the agent system's runtime for further interactions.

Step 1: Preparing the Dockerfile and Environment

To deploy the agent system to Docker, we need to first create a Dockerfile. The Dockerfile will package the necessary runtime, model, and environment for the agent system.

1. Dockerfile Setup: Your Dockerfile should contain the following key components:

   • Base Image: Use a Python base image (or any base that supports the necessary libraries).

   • Copy Model Files: Copy the pre-trained model (e.g., .gz file) to the container.

   • Install Dependencies: Install any required Python libraries (like OHTTP or other packages for the agent system).

Here’s an example Dockerfile:

Step 2: Building the Docker Image

1. Building the Image: You can build the Docker image by running the following command in the terminal. This will take the Dockerfile and the associated files (like the pre-trained model) and create an image.

• The -t flag allows you to tag the image (my-agent-runtime), which makes it easier to reference later.

• Make sure that the model file (agent.gz) and all relevant scripts are reachable within the Docker context (i.e., the directory from which you are building).

1. Checking the Image: Once the build is complete, you can verify that the image was created successfully by running:

Step 3: Running the Docker Container

Now that the image is built, the next step is to run it in a container. You will run the Docker container in an interactive mode to access the runtime.

• -it: Runs the container interactively.

• -p 8000:8000: Maps port 8000 from the container to port 8000 on your local machine so that you can access the HTTP server for the agent system runtime.

The HTTP server should now be up and running within the container, ready to handle model inference or other tasks.

Step 4: Accessing the Agent System Runtime

With the Docker container running, you can now connect to the agent system's runtime. The runtime will be an HTTP server, as mentioned in your notes. You can access it through a POST request for model inference or other operations.

1. Sending Requests to the Agent System: You can send a POST request to the running server using a tool like curl, Postman, or any Python HTTP library (such as requests).

Here’s an example using curl:

This request will:

• POST data to the /infer endpoint on localhost:8000, which is being forwarded from the Docker container.

• The agent system will handle the request, infer the model, and return the result.

1. Interacting with the Agent System: If you prefer to interact with the agent system directly, you can also enter the container’s interactive mode and run commands.

This will open a shell inside the running Docker container, allowing you to execute any runtime commands manually.

Step 5: Automating the Process

For convenience, you can automate the entire process of building the image, running the container, and interacting with the agent system by creating a script.

Here’s a basic example of an automation script:

Save this as run_agent.sh, and then execute it:

This script will:

• Build the Docker image.

• Run the container, mapping the necessary port and exposing the HTTP server for inference.

Step 6: Troubleshooting and Debugging

If the container fails to start, or if the server doesn't respond, you can debug the container by checking the logs:

This command displays the output of the running container, which can help diagnose issues such as missing dependencies or server errors.

Conclusion

In this tutorial, we walked through the process of:

• Building a Docker image with your agent system and its runtime.

• Running the Docker container interactively to expose the agent’s HTTP server.

• Accessing the agent system runtime by sending HTTP requests for inference or other tasks.

By following these steps, you can deploy and interact with your Composabl agent system in a Dockerized environment.

Overview

Due to the many options available to you for installing Kubernetes clusters, this document will not go into the specifics of setting up the cluster. Rather, it will provide you with guidance and requirements for your cluster.

Nodes

Depending on whether you want to use GPUs or not, you need the following nodes:

Nodes that are always required:

1. "main": This means nodes to run the control plane. The Composabl controller does not interact with these nodes, so they should be provisioned as recommended by the Kubernetes distribution you use.

2. "composabl": This node or nodes will be where the Composabl controller and Historian software are scheduled.

3. "envrunners": These nodes will handle training workloads. If you're not using GPUs, all training will be done on these nodes. If you are, these nodes will manage the communication with the simulators, and can be reduced in size

4. "simscpu": These nodes are where the simulators will be scheduled. Sizing depends on the simulator.

If you want to use GPU training, you need the following nodepool: 5. "learners": These nodes with GPUs will accelerate the learning step of the training process.

If your simulator can be accelerated using GPU, you can add the final node pool: 6. "simsgpu": These will run simulators, assigning a GPU to them.

1. Sizing

1. main: As required by your Kubernetes distribution

2. composabl: In total, 16GB of memory and 4 CPU - with 1 node at least 8GB of memory.

3. envrunners: If not using GPUs, we recommend 8 CPU and 8 or 16 GB of memory. In any case, the number of simulators that can be managed by each envrunner instance depends on the number of CPU

4. simscpu: The sizing of these nodes depends on the resource requirements of your simulator

5. learners: These nodes should have 1 Nvidia GPU. Other resources can be limited - 2 CPU and 8GB of memory is sufficient

6. simsgpu: As with simscpu, depends on the simulator requirements.

2. Labels

All groups of nodes must be labeled accordingly. The name, as given in the sizing guide is the name you should set as the agentpool label.

You may be able to define this during your cluster setup, but if not, you can use the following commands:

Replace the values in between <> with the name of the nodes you'd like to assign to a specific pool.

Storage

The components also need access to (semi)persistent, shared storage. This section will detail the types and amount of storage needed.

It needs the following PersistentVolumeClaims in the composabl-train namespace:

1. pvc-controller-data with a size of ±1Gi and ReadWriteOnce (or better) accessMode When using Azure, you will need to set the nobrl mountOption for this PVC, as this is required for the Composabl controller to function.

2. pvc-training-results with a suitable size - this is where your final agent system data will be stored before it is uploaded to the No-code application. It needs accessmode to be ReadWriteMany (RWX). A good initial size is to match historian-tmp.

3. historian-tmp is used as temporary storage for historian data. It needs to have an accessMode of ReadWriteOnce and the size will depend on the length of your training sessions. We recommend starting with 5Gi.

The size of pvc-training-results and historian-tmp is dependent on the amount and size of training jobs you want to run simultaneously on your cluster. If you plan on running long-lived training sessions with many cycles, you may want to increase the capacity for both,

Private image registry

If you want to use a private registry for simulator images, you will need to set up this private registry yourself, and make sure the cluster is able to pull images from this registry.

Next steps

In this tutorial, we will walk through how to set up a skill agent in Composabl that integrates with a third-party API. This type of integration allows your agent system to communicate with external systems, such as machine performance APIs, and use the data to make informed decisions.

We will create a programmed skill agent that connects to a mock third-party API, process its response, and return an action based on the data received. This tutorial will also touch on orchestrating this skill agent within your agent system.

Step 1: Defining the Programmed Skill Agent

A programmed skill agent in Composabl is created by specifying the logic for interacting with the external API and processing the response. In this case, we will create a simple API connection to a fake endpoint that returns data about machine performance. The agent system will act based on the information received.

1.1. Creating the API Integration Skill Agent

We’ll define a programmed skill agent for making the API request. Here's an example of how to define the skill agent using a controller function that calls the API and processes the response.

In this example:

• The compute_action() method sends observation data (e.g., from sensors) to a third-party API.

• The _call_api() function makes the API call and handles any errors that might occur.

• The _process_response() function processes the response from the third-party API and determines the appropriate action for the agent system to take based on the data.

Step 2: Adding the Programmed Skill Agent to the Agent System

2.1. Adding the Skill Agent to Composabl UI

Once the skill agent is defined, you can add it to your agent system in the UI using the methods below:

1. Create a new Skill Agent using the Composabl CLI with a given name and description and implementation type, that in this case will be a controller. The name will be "third_party_api_skill"

1. Change the controller.py code to use the class that you created: ThirdPartyAPISkill(). Change the pyproject.toml file to include your class ThirdPartyAPISkill in the entrypoint and its name:

1. Publish the Skill Agent to the UI

Select your organization and project that you want to publish it to.

Reference: https://docs.composabl.io/changelog/0-8-0.html

2.2. Adding the Skill Agent to Composabl SDK

Once the skill agent is defined, you can add it to your agent system using the add_skill() SDK method. This allows the agent system to execute the API connection skill agent when necessary.

Here’s how to add the ThirdPartyAPISkill to the skill agent:

By importing and creating the class with SkillController, you are indicating that this skill agent is programmed and does not require training. It will use predefined logic to interact with the third-party API and make decisions based on the data returned.

Conclusion

By following these steps, you’ve successfully defined and integrated a programmed skill agent that communicates with a third-party API into your Composabl agent system. The agent system can now take actions based on external data and dynamically respond to scenarios.

This approach allows agent systems to interface with a wide range of external systems, from monitoring equipment to adjusting machine settings, all through programmable skill agents.

Orchestration of skill agents through selectors ensures the agent system executes the correct skill agents at the right time, whether the skill agents are learned or programmed.

Introduction

To train Composabl agent systems on a cluster, you need to install the Composabl controller onto a compatible Kubernetes cluster.

This document will guide you through setting up this cluster using various options:

• Azure: Set up a cluster through our Azure Marketplace Composabl Training Cluster offer

• Automated: Set up a compatible Kubernetes cluster on Azure through the use of Pulumi (IaaS)

• Manually: Set up a compatible Kubernetes cluster through any tool and infrastructure of your choice

Components

The entire Composabl controller installation consists of two main components:

• Controller: This component connects to the Composabl No-code ui, and allows you to initiate training on your cluster.

• Historian: The historian component records detailed statistics regarding the training process of your agent system. After training, these will be available alongside the agent system in the no-code UI

RBAC

The installation of Composabl will introduce 2 namespaces, composabl-train and composabl-sims, for training workers and simulators respectively.

For normal operation, a series of permissions are required by the ServiceAccounts of the Composabl installation.

Cluster Setup

We currently have the following guides to setting up your own cluster:

About Cluster Training

Composabl offers two options for cluster training:

• Use Composabl's Training as a Service offering to train on our clusters

• Use your own compute clusters through Azure, AWS, or another provider

Ensure that Your Agent System is Ready

Before you submit your job for training on a cluster, make sure that your agent is fully configured and all the parameters have been set. That means checking all the agent components:

• Goals

• Perceptors

• Orchestrators, including goals for learned selectors and scenarios

• Skill agents, including goals for learned skills

• Scenarios, including scenario flows

Any component of the agent with a warning sign is not fully configured and not ready for training. Go back to edit that agent component and make sure that all of the fields are filled out.

Choose the Right Cluster

You can train on your own cluster or on Composabl’s clusters using training as a service (TaaS) credits. If you want to use Composabl’s clusters, ensure that you have credits available.

To train on your own cluster, make sure that you have set your cluster up and installed Composabl successfully.

Click Train and then choose the cluster option in the menu. You will then have the option to configure your training session.

Configure Your Training Session

Training session configuration options are the same whether you’re using TaaS or training on your own cluster.

Set the Number of Training Cycles

A training cycle is a complete pass through the entire task, with the agent system continuing until it reaches success or some other stop criteria. Your agent system will train each skill one at a time for the selected number of training cycles, starting from the bottom of the agent system design.

A training cycle involves about 1,000 agent decisions. Depending on the complexity of the task, agent systems may need to complete anywhere between 100 and several thousand training cycles to become proficient.

Set the Number of Simulators

You can run multiple simulators in parallel to speed up training. If you run more than one simulator during a training, the number of training cycles selected will be multiplied by the number of simulators, so 5 training cycles with 3 simulators selected would lead to 15 training cycles total.

You can use the Advanced Configuration to choose how powerful each machine running a simulator should be. If you choose Small, each training cycle selected will result in one training cycle completed. If you choose GPU, you will get 4 training cycles for each training cycle.

More training cycles running simultaneously will speed up training, but also increase costs. How long your training takes also depends on the complexity of your agent system and your simulator.

Start Training

When you have configured your settings correctly, click Start Training.

You will then be taken to the Training Sessions page. There you can follow the agent system training progress by viewing the real-time plots or the console output.

Note that it will take a few minutes for the visualization to begin.