Composabl is the platform for creating industrial-strength multi-agent AI systems that make high-impact decisions in the real world.

With Composabl, you can convert expert knowledge about how a process works into a team of agents with specialized skills that work together to allow the system to make the right decision in every situation. These skill agents can be either programmed or learned through advanced AI techniques and orchestrated so that the multi-agent system performs effectively in every part of the process and under any conditions. For skill agents that learn by practicing, Composabl trains the agents in realistic scenarios until the agent can succeed at the task and outperform the alternatives.

Quick Links

Find articles on key topics with these links.

Try Out a Tutorial

Integrate with Composabl

You can use any model or Python algorithm with Composabl for training agents, adding perception, analysis, and communication, and making decisions. See how to configure different types of modules and then publish them to the Composabl no-code UI for agent design, training, and deployment.

Create Modular Skill Agents

Composabl multi-agent systems are built on modular skills that break down a task into separate parts. Learn how to create skill agents to train with deep reinforcement learning.

Deploy Multi-Agent Systems

Once Composabl agentic systems are designed and trained, you can export them to the Composabl runtime to connect with your system. Learn how to deploy an agent within the runtime container and how to use Composabl's tools to analyze agent behavior during both training and deployment.

Install the Composabl CLI Below, you can find the commands available in the Composabl CLI.

Login

Agent

Sim (composabl sim)

Perceptor (composabl perceptor)

Skill (composabl skill)

Selector (composabl selector)

Job (composabl job)

Controller: A skill agent that makes decisions based on programmed algorithms. Also called a programmed skill agent.

Decisions: The outputs of a Composabl system - its instructions for controlling the system

Design patterns: Common structures that can be used to quickly create multi-agent systems

Episode: An entire run through the task

Functional Pattern: A machine teaching design pattern used to orchestrate skills in sequences

Iteration: One decision during agent training or performance

Learned skill agent: A skill agent that uses DRL to make decisions and learn by practicing in simulation

Orchestration: Arranging agents as modular building blocks to work together to complete tasks

Orchestrator: A special type of skill agent that assigns decision-making control to the right decision-making skill agent based on current conditions

Perceptor: An ML model or other algorithm that interprets sensor data

Plan-Execute Pattern: A machine teaching design pattern used to orchestrate skill agents in pairs that work together to make decisions

Programmed skill agent: A skill agents that makes decisions based on programmed algorithms. Also called controllers.

Project: A collection of agents for the same use case that share the same simulator and top-level goal

Scenarios: Conditions that are associated with specific skills

Sensors: The part of the system that takes in information from the simulator or real environment - the eyes, ears, and other senses of the system

Simulator: The virtual environment where agents practice and improve performance

Skill agents: Modules within a multi-agent system that make decisions to complete all or part of a task

Strategy Pattern: A machine teaching design pattern used to orchestrate skill agents in hierarchies

Teacher: An algorithm that creates a skill that uses DRL to learn to make decisions

Install the Composabl SDK

A Two-Part Platform

Composabl is a two-part platform with a no-code UI and a Python SDK and CLI. The interplay of these parts gives Composabl its combination of usability and power.

The two parts enable teams to work together effectively. People and roles who primarily use code, such as data scientists and controls engineers, use the SDK to create modules like ML models and deep reinforcement learning skills. Then, subject matter experts, process engineers, and others can use the no-code interface to easily create teams of agents from these modular building blocks and train them to succeed.

We designed the platform this way because, for complex, high-value processes, there will be some tasks that can only be done through coding, and some team members who prefer to work in code, and other tasks that are better performed through a visual interface. Both parts of the platform work together.

Most users use both parts of the platform to some extent but spend more time in the no-code UI or the SDK, depending on their expertise and role. How you will use the platform depends on your role and what part of the process you are working on.

Set Up Your Environment

The first thing you will need to do after you login to your account is to connect a simulation to Composabl.

Access Composabl

You can access Composabl Via a no-code UI and an SDK. They work together to enable you to build, train, and deploy autonomous agents.

No-Code UI

Python CLI and SDK Instalation

Installing the SDK

pip install -U composabl

Development

pip install -U composabl-dev

Note: The Composabl CLI and SDK require Python version 3.10.x

Workflow Steps

• Step 1: UI | Create your first use case

• Step 2: UI | Set performance goals

• Step 3: UI and SDK | Create agents

  • Create skill agents to train with rewards using deep reinforcement learning in the UI, or the SDK.

  • Create or package ML models to import to UI with the SDK to add advanced perception to agents.

  • Create or package LLMs to import with the SDK to the UI to add natural language to agents.

  • Create or package controllers and optimization algorithms to import to the UI with the SDK to add programmed decision-making to agents.

• Step 4: SDK | Publish to the UI with one CLI command

• Step 5: UI | Orchestrate modular components together to create teams of agents in the UI

• Step 6: UI | Train your teams of agents at scale with one click using the UI

• Step 7: Notebook | Export Historian training data and perform detailed analysis

• Step 8: UI and SDK | Export trained multi-agent systems and connect them to the Composabl runtime for deployment

The examples and code samples in this documentation refer to our sample use cases. These examples are real-world use cases with complex goals and constraints. In each case, the Composabl team has built agents that exceed the benchmark control technology by orders of magnitude.

Industrial Mixer

About the Use Case

Learn more: Read the Composabl whitepaper about the production scheduling use case.

The industrial mixer use case is a realistic case study of a chemical process control agent controlling a continuous stirred tank chemical reaction. The agent controls the temperature in a tank where a chemical reaction occurs to create a product.

As the chemicals are stirred together in the tank, the reaction produces heat at a nonlinear, unpredictable rate. If the tank isn’t cooled enough, it can reach dangerous temperatures, a condition called thermal runaway. If it’s cooled too much, not enough product will be produced. The agent needs to balance these two goals, keeping the tank at the right temperature at every moment to optimize production while ensuring safety.

Explore Agent Components

Access perceptors, skills, and selectors for this use case.

Production Scheduling

About the Use Case

Learn more: Read the Composabl whitepaper about the production scheduling use case.

The production scheduling use case is an complex production planning problem set in an industrial bakery. The agent must determine the right amount of cookies, cakes, and cupcakes to make each day, directing teams of workers and equipment and responding to fluctuations in costs, pricing, and demand.

The case study, developed in partnership with Composabl partner Rovisys, requires the agent to make a choice every minute between 24 possible combinations of equipment, task, employee and product, over the course of a 400-decision day, with the ultimate goal of maximizing profit.

Along with the strategy pattern, and the perception pattern, the plan-execute pattern is one of the major design patterns of machine teaching. In this pattern, the skills work together in a skill group, with the first skill determining what the action should be and the second skill determining how to achieve it.

What is special about this agent is that it combines DRL and MPC, the technologies from the two single-skilled agents — the worst performers — to create the best performing agent.

In this example, the DRL skill first uses its powers of learning and experimentation to determine the goal temperature for the cooling jacket — the set point. It then passes this information on to the MPC skill, which uses its powers of control and execution to direct the agent on what action to take to achieve the desired temperature.

Remember how the strategy pattern is like a math class where each student solves the problems they are best at, as assigned by the teacher? In the plan-execute pattern, the students work in groups to solve problems together. Let’s say Student A is good at translating word problems into equations, while Student B is good at solving equations. Student A works on each problem first, and then passes it over to Student B, who produces the solution. No teacher is needed here, because the students divide each problem the same way.

Let's get started configuring this agent!

1. Publish the MPC Skill to Your Project

This agent has two skills called control_full_reaction and mpc-skill-group. We have already published control_full_reaction to our project, so we only need to publish mpc-skill-group to build this agent in the Agent Builder UI. To publish mpc-skill-group to your project you will need to open up your favorite code editor and terminal. In your terminal, navigate to the skills folder of the Industrial Mixer Repo and use this command with the Composabl CLI.

composabl skill publish mpc-skill-group

Return to the agent builder studio and refresh the page. The skills will appear in the skills menu on the left of your page.

Explore the Code Files

All skills, perceptors, and selectors have a minimum of two files in them. A Python file contains the code that the agent will use, and a config file.

1. pyproject.toml, a config file with the following information.

2. A Python file. For this agent, we use controller class with the following code and explanations in comments inline.

File Structure

See the Code

MPC Skill Group Controller Skill

pyproject.toml

[project]
name = "MPC Skill Group"
version = "0.1.0"
description = "MPC prepared for Skill Group"
authors = [{ name = "John Doe", email = "john.doe@composabl.com" }]
dependencies = [
    "composabl-core",
    "scipy",
    "casadi==3.6.6",
    "do_mpc==4.6.5"
]

[composabl]
type = "skill-controller"
entrypoint = "mpc_skill_group.controller:Controller"

controller.py

from random import randint
from typing import Dict, List

from composabl_core import SkillController
######
import os
import math
import numpy as np
import do_mpc
import numpy as np
from casadi import *
from scipy import interpolate
from math import exp

# time step (seconds) between state updates
Δt = 1

π = math.pi

class Controller(SkillController):
    def __init__(self, *args, **kwargs):
        """
        Initialize the Controller skill with default values.
        
        Args:
            *args: Variable length argument list.
            **kwargs: Arbitrary keyword arguments.
        """
        # Initialize a counter to track the number of actions computed
        self.count = 0

    async def compute_action(self, obs, action):
        """
        Compute the control action (ΔTc) based on current observations using Model Predictive Control (MPC).
        
        Args:
            obs (list or dict): Current sensor observations.
                If list, expected order: ['T', 'Tc', 'Ca', 'Cref', 'Tref']
            action: The previous action taken (not directly used but considered for ΔTc calculation).
        
        Returns:
            list: A list containing the computed change in Tc (ΔTc).
                  Example: [ΔTc]
        """
        # Convert observations to dictionary if they are provided as a list
        if type(obs) == list:
            obs = {
                'T': obs[0],
                'Tc': obs[1],
                'Ca': obs[2],
                'Cref': obs[3],
                'Tref': obs[4]
            }
        # else:
        #     # Uncomment if you need to ensure all values are floats
        #     for key, value in obs.items():
        #         obs[key] = float(value)

        # Handle action input: ensure it's a float value
        if type(action) == list or type(action) == np.ndarray:
            action = action[0]
        elif type(action) == dict:
            assert type(action['action']) == float
            action = float(action['action'])
        else:
            action = float(action)

        # Initialize noise variable (currently set to 0; can be modified for stochasticity)
        noise = 0

        # Extract and convert sensor readings to float
        CrSP = float(obs['Cref'])    # Reference concentration
        Ca0 = float(obs['Ca'])       # Actual concentration at current step
        T0 = float(obs['T'])         # Temperature at current step
        Tc0 = float(obs['Tc']) + action  # Cooling liquid temperature adjusted by action

        # Define constants for the CSTR model
        F = 1          # Volumetric flow rate (m³/h)
        V = 1          # Reactor volume (m³)
        k0 = 34930800  # Pre-exponential nonthermal factor (1/h)
        E = 11843      # Activation energy per mole (kcal/kmol)
        R = 1.985875   # Boltzmann's ideal gas constant (kcal/(kmol·K))
        ΔH = -5960     # Heat of reaction per mole (kcal/kmol)
        phoCp = 500    # Density multiplied by heat capacity (kcal/(m³·K))
        UA = 150       # Overall heat transfer coefficient multiplied by tank area (kcal/(K·h))
        Cafin = 10     # Inlet concentration (kmol/m³)
        Tf = 298.2     # Feed temperature (K)

        # --- MPC MODEL SETUP ---
        model_type = 'continuous'  # Define model type: 'discrete' or 'continuous'
        model = do_mpc.model.Model(model_type)

        # Define state variables
        Ca = model.set_variable(var_type='_x', var_name='Ca', shape=(1,1))  # Concentration
        T = model.set_variable(var_type='_x', var_name='T', shape=(1,1))    # Temperature

        # Define measurements (if any) with optional measurement noise
        model.set_meas('Ca', Ca, meas_noise=True)
        model.set_meas('T', T, meas_noise=True)

        # Define control input
        Tc = model.set_variable(var_type='_u', var_name='Tc')  # Cooling liquid temperature

        # Define time-varying parameters (TVPs)
        model.set_variable(var_type='_tvp', var_name='Caf')   # Inlet concentration (kmol/m³)
        model.set_variable(var_type='_tvp', var_name='Tref')  # Reference temperature (K)

        # Define model equations (right-hand side)
        model.set_rhs('Ca', (F/V * (Cafin - Ca)) - (k0 * exp(-E/(R*T))*Ca))
        model.set_rhs('T', (F/V *(Tf-T)) - ((ΔH/phoCp)*(k0 * exp(-E/(R*T))*Ca)) - ((UA /(phoCp*V)) *(T-Tc)))

        # Finalize model setup
        model.setup()

        # --- CONTROLLER SETUP ---
        mpc = do_mpc.controller.MPC(model)
        setup_mpc = {
            'n_horizon': 20,       # Prediction horizon
            'n_robust': 1,         # Number of robust steps
            'open_loop': 0,        # Open-loop setting
            't_step': Δt,          # Time step (seconds)
            'store_full_solution': True  # Store full solution
        }

        mpc.set_param(**setup_mpc)

        # Suppress IPOPT solver output for cleaner logs
        surpress_ipopt = {'ipopt.print_level':0, 'ipopt.sb': 'yes', 'print_time':0}
        mpc.set_param(nlpsol_opts = surpress_ipopt)

        # Scaling for states and inputs to improve numerical stability
        mpc.scaling['_x', 'T'] = 100
        mpc.scaling['_u', 'Tc'] = 100

        # --- OBJECTIVE FUNCTION ---
        _x = model.x
        _tvp = model.tvp
        _u = model.u

        # Define terminal and stage cost
        mterm = ((_x['Ca'] - CrSP))**2  # Terminal cost
        lterm = ((_x['Ca'] - CrSP))**2  # Stage cost

        mpc.set_objective(mterm=mterm, lterm=lterm)

        # Define control input penalties to discourage large control actions
        mpc.set_rterm(Tc=1.5)  # Input penalty for Tc

        # --- CONSTRAINTS ---
        # Bounds for state variables
        mpc.bounds['lower', '_x', 'Ca'] = 0.1   # Minimum concentration
        mpc.bounds['upper', '_x', 'Ca'] = 12    # Maximum concentration

        mpc.bounds['upper', '_x', 'T'] = 400    # Maximum temperature
        mpc.bounds['lower', '_x', 'T'] = 100    # Minimum temperature

        # Bounds for control inputs
        mpc.bounds['lower', '_u', 'Tc'] = 273   # Minimum cooling temperature (K)
        mpc.bounds['upper', '_u', 'Tc'] = 322   # Maximum cooling temperature (K)

        # --- TIME-VARYING PARAMETERS (TVPs) SETUP ---
        # Define templates for TVPs
        tvp_temp_1 = mpc.get_tvp_template()
        tvp_temp_1['_tvp', :] = np.array([8.5698])

        tvp_temp_2 = mpc.get_tvp_template()
        tvp_temp_2['_tvp', :] = np.array([2])

        tvp_temp_3 = mpc.get_tvp_template()
        tvp_temp_3['_tvp', :] = np.array([2])

        # Define a function to update TVPs based on current time
        def tvp_fun(t_now):
            p1 = 22    # Time step 1
            p2 = 74    # Time step 2
            time = 90  # Total time

            # Define concentration and temperature equilibrium points
            ceq = [8.57, 6.9275, 5.2850, 3.6425, 2]
            teq = [311.2612, 327.9968, 341.1084, 354.7246, 373.1311]

            # Interpolate concentration and temperature based on current time
            C = interpolate.interp1d([0, p1, p2, time], [8.57, 8.57, 2, 2])
            T_ = interpolate.interp1d([0, p1, p2, time], [311.2612, 311.2612, 373.1311, 373.1311])

            if t_now < p1:
                return tvp_temp_1
            elif p1 <= t_now < p2:
                y = float(C(t_now))
                tvp_temp_3['_tvp', :] = np.array([y])
                return tvp_temp_3
            else:
                return tvp_temp_2

        mpc.set_tvp_fun(tvp_fun)

        # Finalize MPC setup
        mpc.setup()

        # --- ESTIMATOR SETUP ---
        estimator = do_mpc.estimator.StateFeedback(model)

        # --- SIMULATOR SETUP ---
        simulator = do_mpc.simulator.Simulator(model)
        params_simulator = {
            't_step': Δt  # Time step (seconds)
        }

        simulator.set_param(**params_simulator)

        # Define templates for simulator parameters
        p_num = simulator.get_p_template()
        tvp_num = simulator.get_tvp_template()

        # Define functions for TVPs and uncertain parameters in the simulator
        def tvp_fun_sim(t_now):
            return tvp_num

        def p_fun_sim(t_now):
            return p_num

        simulator.set_tvp_fun(tvp_fun_sim)
        simulator.set_p_fun(p_fun_sim)

        # Finalize simulator setup
        simulator.setup()

        # --- INITIAL STATE SETUP ---
        # Initialize states for MPC, simulator, and estimator
        x0 = simulator.x0
        x0['Ca'] = Ca0
        x0['T'] = T0

        u0 = simulator.u0
        u0['Tc'] = Tc0

        mpc.x0 = x0
        simulator.x0 = x0
        estimator.x0 = x0

        mpc.u0 = u0
        simulator.u0 = u0
        estimator.u0 = u0

        # Set initial guess for MPC
        mpc.set_initial_guess()

        # --- MPC CONTROL LOOP ---
        # Simulate N steps (currently set to 1)
        u0_old = 0
        time_steps = 1
        for k in range(time_steps):
            if k > 1:
                u0_old = u0[0][0]

            # Make a control step using MPC
            u0 = mpc.make_step(x0)

            # Enforce a maximum change of ±10 on the control input
            if k > 1:
                if u0[0][0] - u0_old > 10:
                    u0 = np.array([[u0_old + 10]])
                elif u0[0][0] - u0_old < -10:
                    u0 = np.array([[u0_old - 10]])
            else:
                if u0[0][0] - Tc0 >= 10:
                    u0 = np.array([[Tc0 + 10]])
                elif u0[0][0] - Tc0 <= -10:
                    u0 = np.array([[Tc0 - 10]])

            # Add Gaussian noise to the measurements
            error_var = noise
            σ_max1 = error_var * (8.5698 - 2)
            σ_max2 = error_var * (373.1311 - 311.2612)
            mu = 0
            v0 = np.array([
                mu + σ_max1 * np.random.randn(1, 1)[0],
                mu + σ_max2 * np.random.randn(1, 1)[0]
            ])

            # Simulate the next step with the control input and noise
            y_next = simulator.make_step(u0, v0=v0)  # MPC simulation step

            # Reshape state values for consistency
            state_ops = y_next.reshape((1, 2))

            # --- BENCHMARK SETUP ---
            p1 = 22
            p2 = 74
            ceq = [8.57, 6.9275, 5.2850, 3.6425, 2]
            teq = [311.2612, 327.9968, 341.1084, 354.7246, 373.1311]

            # Interpolate reference concentration and temperature
            C = interpolate.interp1d([0, p1, p2, time_steps], [8.57, 8.57, 2, 2])
            T_ = interpolate.interp1d([0, p1, p2, time_steps], [311.2612, 311.2612, 373.1311, 373.1311])

            # Update reference concentrations and temperatures based on current step
            if k < p1:
                Cref = 8.5698
                Tref = 311.2612
            elif p1 <= k < p2:
                y = float(C(k))
                y2 = float(T_(k))
                Cref = y
                Tref = y2
            else:
                Cref = 2
                Tref = 373.1311

            # Update the estimator with the new measurements
            x0 = estimator.make_step(y_next)  # Update state estimates

        # Increment the action counter
        self.count += 1

        # Compute the change in Tc (ΔTc) based on the new control input
        newTc = u0[0][0]
        dTc = float(newTc) - float(obs['Tc'])

        # Return the computed ΔTc as a list
        return [dTc]

    async def transform_sensors(self, obs):
        """
        Process and potentially modify sensor observations before they are used.

        Args:
            obs (dict): Current sensor observations.

        Returns:
            dict: Transformed sensor observations.

        Note:
            - Currently, this method returns the observations unchanged.
            - This can be customized to apply transformations if needed.
        """
        # Currently, no transformation is applied to sensors
        return obs

    async def filtered_sensor_space(self):
        """
        Define which sensors are relevant for this perceptor.

        Returns:
            list: Names of the sensors to be used.

        Note:
            - Specifies a list of sensor names that this perceptor will utilize.
            - Helps in focusing the perceptor's operations on relevant data.
        """
        # Specify the sensors that this perceptor will use
        return ['T', 'Tc', 'Ca', 'Cref', 'Tref', 'Conc_Error', 'Eps_Yield', 'Cb_Prod']

    async def compute_success_criteria(self, transformed_obs, action):
        """
        Determine whether the success criteria have been met.

        Args:
            transformed_obs (dict): Transformed sensor observations.
            action: The action taken.

        Returns:
            bool: True if success criteria are met, False otherwise.

        Behavior:
            - Currently always returns False.
            - Can be implemented with logic to check if certain conditions are satisfied.
        """
        # Placeholder for success criteria logic
        return False

    async def compute_termination(self, transformed_obs, action):
        """
        Determine whether the training episode should terminate.

        Args:
            transformed_obs (dict): Transformed sensor observations.
            action: The action taken.

        Returns:
            bool: True if the episode should terminate, False otherwise.

        Behavior:
            - Currently always returns False.
            - Can be implemented with logic to terminate based on certain conditions.
        """
        # Placeholder for termination condition logic
        return False

2. Build the Strategy Pattern Agent in the Agent Builder UI

First drag the skill control_full_reaction from the hand side of the page to the skill container. Once it's there drag the mpc-skill-group and make sure that it is dropped below the control_full_reaction skill and not beside it.

3. Run Your Training Session

Now, we are ready to train your agent and see the results. We suggest you run 50 training cycles.

4. View Results

When the training has been completed, you can view your results in the training sessions tab in the UI. This will show you information on how well the agent is learning.

Analyzing the Plan-Execute Pattern Agent’s Performance

Conversion rate: 95% Thermal runaway risk: Very low

We tested this fully trained agent and plotted the results.

This agent is the best performer of the group. Combining two imperfect technologies together with Machine Teaching produces much better results than either technology achieves alone.

The DRL agent is a very simple agent design with only one skill. This agent does not use machine teaching to decompose the task into skills that can be trained separately. Instead, the entire reaction is controlled by a skill trained with deep reinforcement learning.

Let's get started configuring this agent!

1. Publish the DRL Skill to Your Project

This agent has a single skill called Control Full Reaction. To publish that skill to your project you will need to open up your favorite code editor and terminal. In your terminal, navigate to the skills folder and use the command with the Composable CLI.

Return to the agent builder studio and refresh the page. You will see the skill in the skills menu on the left of your page.

Explore the Code Files

All skills, perceptors, and selectors have a minimum of two files in them. A Python file contains the code that the agent will use, and a config file.

1. pyproject.toml, a config file with the following information.

3. Add the Skill to your DRL Agent

Drag the skill control_reaction that you can now see on the left-hand side of your project onto the Skill Layer.

4. Run Your Training Session

Now, we are ready to train your agent and see the results. We suggest you run 50 training cycles.

5. View Results

When the training has been completed, you can view your results in the training sessions tab in the UI. This will show you information on how well the agent is learning.

You will most likely see a steep learning curve as the agent experiments with different control strategies and learns from the results. When the learning curve plateaus, that usually means that the skill is trained.

Analyze the DRL Agent's Performance

Conversion rate: 90% Thermal runaway risk: Low

We tested this fully trained agent and plotted the results.

The DRL agent performs well. Its relatively thin shadow means that it performs consistently over different conditions and it stays within the safety threshold almost every time

This agent controls the initial steady state very well, staying right on the benchmark line. But during the transition, the DRL agent goes off the benchmark line quite a bit. It doesn't notice right away when the transition phase begins, staying too long in the lower region of the graph, and then overcorrecting. That's because DRL works by experimentation, teaching itself how to get results by exploring every possible way to tackle a problem. It has no prior knowledge or understanding of a situation and relies entirely on trial and error. That means that it is potentially well suited to complex processes – like the transition phase - that can’t easily be represented mathematically.

But its behavior is erratic because it can’t distinguish between the phases. The DRL agent’s skills do better than the traditional automation benchmark but still leave some room for improvement.

This tutorial will take you through the process of building agents for a realistic chemical manufacturing use case.

1. Learn about the use case (this page)

7. Experiment with your own designs

About the Use Case

Why an industrial mixer?

Mixing and blending tasks are great use cases for intelligent agents because they are both complex and high-value. In the example in this tutorial, improvements in process could lead to millions in ROI.

Use Case Overview

In the industrial mixer use case, raw materials are stirred together inside a tank together, undergoing a reaction that produces the desired end product.

The goal of the process is to convert as much of the raw material as possible. But as the chemicals mix and the conversion occurs, the tank heats up. If the temperature gets too high, a condition called “thermal runaway” occurs, causing explosions and fires.

To produce as much chemical as possible, the operator must constantly adjust the temperature in the tank, keeping it high enough to allow productivity but low enough to avoid any thermal runaway risk.

As in all machine teaching use cases, this process can be summarized in the form of two separate goals that must be balanced against each other:

The process is controlled by adjusting the mixture's temperature in the tank using a "jacket" filled with coolant. Lowering the coolant temperature in the jacket lowers the temperature in the tank, decreasing the risk of thermal runaway.

However, cooling the tank can also reduce yield. By how much? The answer varies unpredictably – temperature changes affect chemical concentration differently at different parts of the reaction. That nonlinear relationship between temperature and yield is why this is a nuanced process that benefits so much from intelligent automation.

Simulating a Complex Reaction

The industrial mixer simulator uses principles of chemistry to model the behavior of the system in real life.

Benchmark

The current automation solution is a linear MPC controller. As the current solution, this agent's performance is the benchmark for the other designs.

The image below shows an MPC controller represented in the visual system of agent designs. The agent takes in sensor information about the temperature in the tank and the concentrations of the chemicals.

It passes that information to the skills layer of the agent. The skills layer contains a single programmed skill: control reactor. This skill uses a mathematical model to determine the desired temperature set point for the tank. It also determines the control actions to take to achieve that temperature using the cooling jacket, and outputs those actions as decisions.

In simulation, this agent's conversion rate was 82%. That means that 82% of the reagents were turned into product, with 18% waste, but the risk of thermal runaway is high.

Like all technologies, model predictive control has a “personality,” a unique set of strengths and weaknesses that can be seen reflected in this performance. MPC is a rule follower. It is a way of controlling a process using mathematical relationships that can be fully mapped and programmed by human engineers. It works well in situations that are straightforward and linear.

The agent does a good job at the start, in the first steady state. However, as the transition phase begins, the agent’s performance starts to fail. The wide shadow shows that this agent can’t adapt to the unpredictable conditions in the transition. Its performance becomes dangerously inconsistent, potentially allowing the temperature to exceed the thermal runaway checkpoint at nearly every point in the reaction.

When an MPC controller is used to control this process in the real world, a human operator needs to step in and take over control before the automated system lets the temperature cross the thermal runaway threshold.

Get Started

The following tutorials will walk you through the anatomy of some multi-agent systems that take different approaches to learning how to control the temperature of a mixer more effectively to maximize yield and avoid thermal runaway.

This getting started guide takes you through the steps you will need to go through before you can started building your teams of agents.

Prerequisites

• A Composabl account

Create Your Use Case

Select the simulator you will use for the project and the assigned team. If you haven't created a team yet, you will need to do that before you create your first project.

Then, you can use the AI prompt to explain what you are attempting to achieve with your project and have it bootstrap based on your description, or you can tell the agent AI what to name your project. I'm going to ask them to call it Chemical Process Control.

Click Create Project and Finish

Set Project Goals

To finish setting up your project, set your top-level project goals.

Now, we will set the goals for this example. In this project, we want to:

• Maximize yield (make as much product as we can)

• Avoid thermal runaway (prevent dangerous overheating)

Goal Title and Description

Fill out the goal title and description for the top level goals.

Goal Title: Chemical Process Control Goals

Description: The goal for this project is to maximize yield and avoid thermal runaway by keeping the temperature below 400 Kelvin

Logical Structure

We are going to add two conditions to represent the two goals.

1. To maximize yield, the objective is Maximize and the variable is Eps_Yield

2. To avoid thermal runaway, the objective is Avoid, the variable is T, and the target is 400

Now you can save your changes.

Now that you have your project created and your goals set you're ready to start creating agents!

The simulator is the part of the agent ecosystem that tells you what will happen when you take a certain action. Whether based on historical data, physical principles and math, or other methods, the simulation environment models the real system and allows your agents to train.

Explore Composabl's Simulators

To look under the hood and see how a simulator works, explore Composabl's public Python simulators hosted on Docker Hub.

You can build and train agents using these simulations to explore Composabl, educate yourself and your team, and create demonstrations and POCs. These are relatively simple simulations that can train agents quickly.

The best way to access these simulations is through the CLI.

To see the options available:

To connect to one of the simulators:

Simulation Help

If you have a simulator, this section of the documentation will explain how to connect it to Composabl so that you can use it to train agents.

If you don’t have a simulator, you may want to work with a Composabl partner to help you develop and connect one.

Multi-agent systems have structure, just like our brains. Different parts perform different functions.

This modularity is a key to building high performing agentic systems quickly and effectively. Most complex systems, from factories to rockets to software programs, are broken into modular pieces to separate concerns, reuse design patterns, and integrate pre-built components. Like these systems, decision-making AI also works best when decisions are separated into modular concerns. Modular structure makes intelligent agents easier to build, test, and maintain.

How Agentic Systems Make Decisions

Multi-agent systems work through a feedback loop. The system gets information from the simulation or real environment and then makes a decision to adjust one or more of its control variables in some way. This creates a change that is reflected in new sensor information that the agent can then process and use to decide on the next adjustment.

While agents are being built and trained, they are connected to simulators that give them feedback about how their actions affect the system they are learning to control. Once an agent is deployed, the simulator is replaced by an equivalent connection to the real system.

The simulator and the multi-agent system function in a continuous feedback loop, with the system outputting actions that affect the simulation environment, and the simulator returning new information to the agent based on every decision.

Sensors are the part of the multi-agent system that take in information from the simulator or the real environment about conditions and about the results of the agent's actions. They are the equivalent of human senses – the eyes and ears. Sensor readings come from specific variables in the simulation or control system that measure various aspects of the state space.

Decisions are the system's outputs, adjusting one or more of the control variables to control the process dynamically throughout the performance of the task. Each decision during training is known as an iteration, while a series of iterations to complete the task is called an episode.

Inside the Multi-Agent System

The team of agents is organized in layers, with information and decisions passing through each layer in sequence.

Skill Agents

The skills layer, or decision-making layer, is the heart of a multi-agent system. This is where the system makes its control decisions. When a specific skill agent is activated, it will determine the control action the system should take.

Multi-agent systems contain multiple agents orchestrated together according to design patterns. Skill agents can work together in groups, sequences, hierarchies, or coordinated teams. This modularity allows the agent to learn more efficiently and perform better.

You can imagine skill agents being like students on a math team who are working together to complete a set of problems. Each student performs best solving a particular kind of problem: one is good at fractions and one at decimals. Depending on the type of problem, the appropriate student will use their expertise to solve the problem and produce the answer for the team. Sometimes one student might handle the entire problem, and for other problems more than one student might need to work together.

Just as different students’ capabilities make them able to solve particular problems, different skill agents may make use of different technologies. Some types of decisions are best approached through agents that can be programmed with mathematical calculations, rules, or optimization algorithms. Others that are more complex and nonlinear can be trained using deep reinforcement learning.

Examples of Skills

For an HVAC system regulating temperature in an office building:

• Control the system during the day

• Control the system at night

For a factory where responses are needed to different types of alarms:

• Handle safety critical alarms (programmed with rules)

• Handle simple alarms (programmed with calculations)

• Handle complex alarms (learned with reinforcement learning)

For a drone autopilot:

• Stabilize

• Get to landing zone

• Land

• Avoid obstacles

For a robotic arm used to grab and stack objects:

• Reach (extend the robot arm from the "elbow" and "wrist")

• Move (move the arm laterally using the "shoulder)

• Orient (turn the "wrist" to position the "hand")

• Grasp (Manipulate the "fingers" to clamp down)

• Stack (Move laterally while grasping)

Orchestrators

Orchestrator skills are the supervisors for your agent. In the math class analogy, the orchestrator would be like the teacher. The teacher assesses the type of problem and assigns the right student.

In a multi-agent system, an orchestrator uses information from the sensors and perceptors to understand the scenario and then determine which skill agent is needed. Once the skill agent is called into service, it makes the decision.

For example, for HVAC control, an orchestrator would determine whether day or night control is needed, and then pass control to the appropriate skill agent. In the safety alarm example, the orchestrator determines the type of alarm and then passes the decision to the right skill agent. In the drone and robotic arm examples, the skills need to be performed in sequence. In these cases, the orchestrator assigns the appropriate skill agent as needed for each step in the process.

Perceptors

A perception layer is an optional enhancement layer. Perceptors process and translate sensor data into a format that can be used to make decisions. In other words, the perception layer inputs the sensor variables and outputs new variables deduced by calculation, machine learning, or other programming.

For example, if we design a team of agents as an autopilot for a drone, we might have sensor variables that measure pitch, yaw, roll (position of the drone), velocity in each of those three directions, and acceleration in each of those three directions. But what about stability? Stability is an important characteristic to understand while flying a drone, but there is no sensor variable that describes stability. It is too complex to be captured by a single sensor.

The perception layer allows us to create a variable for stability. It can be calculated using dynamics equations or trained with supervised machine learning. The new variable then becomes accessible to the rest of the multi-agent system along with the other sensor variables.

Examples of Perceptors

• Computer Vision: A camera sensor passes image or video feeds into a perceptor module that identifies object types and locations

• Auditory Perception: A microphone sensor passes machine sounds to a perceptor module that identifies which state the machine is in based on the sounds that it is making

• Prediction: A perceptor module inputs quality measurements and past agent actions and predicts whether current actions will lead to acceptable quality measurements

• Anomaly Detection: A perceptor modules inputs market variables and detects when the market is changing regimes.

• Classification and Clustering: A perceptor module inputs machine and process data and classifies which of several conditions a manufacturing line is currently in.

Skills are the foundational building blocks for your intelligent agent. They take action to achieve goals in key scenarios where your agent needs to succeed. To build an agent with Machine Teaching, you will create multiple skills and then orchestrate them together.

You can use three different types of skill within Composabl.

For learned skills you can use either the UI or the SDK successfully to create teachers, but the SDK includes some fine-tuning options that are not available in the UI.

Controllers for programmed skills can be created only through the SDK. They can then be published to the UI for use in agents.

You can use either the UI or the SDK to create selectors.

The strategy pattern agent performs well, but it's not perfect in avoiding thermal runaway. One good way to address that is to add a perception layer.

Perceptors are special skills that process and interpret sensor data before passing it to the rest of the agent. To improve the strategy pattern's performance on temperature control, you can add a perception layer that uses machine learning to predict thermal runaway. This ML model is trained to interpret the sensor data and check for conditions that might indicate an elevated risk of thermal runaway, and then pass that information to the selector along with the rest of the sensor data.

Think of the perception layer as an additional set of senses that helps the agent predict when something might go wrong, like a teacher monitoring the class for early signs of trouble.

Let's get started configuring this agent!

1. Add the Perceptor Skill to Your Project

This agent has a perceptor skill called thermal_runaway_predictor. To publish it to your project you will need to open up your favorite code editor and terminal. In your terminal, navigate to the perceptors folder and use this command with the Composabl CLI.

composabl perceptor publish thermal_runaway_predictor

Return to the agent builder studio and refresh the page. The skill will appear in the skills menu on the left side of your page.

Explore the Code Files

All skills, perceptors, and selectors have a minimum of two files in them. A Python file contains the code that the agent will use, and a config file. Perceptors have some more files to load in ML models and other python packages.

File Structure

Thermal Runaway Perceptor

pyproject.toml

[project]
name = "Thermal Runaway Predictor - ML 1.2.2"
version = "0.1.0"
description = "ML thermal runaway predictor"
authors = [{ name = "John Doe", email = "john.doe@composabl.com" }]
dependencies = [
    "composabl-core",
    "scikit-learn==1.2.2"
]

[composabl]
type = "perceptor"
entrypoint = "thermal_runaway_predictor.perceptor:ThermalRunawayPredict"

# Include additional data files
[tool.setuptools.packages.find]
where = ["thermal_runaway_predictor"]

[tool.setuptools.package-data]
"*" = ["*.json", "*.pkl"]

thermal_runaway_predictor.py

from composabl_core import PerceptorImpl

#######
import os
import pickle

# Determine the directory where the current script is located
path = os.path.dirname(os.path.realpath(__file__))

class ThermalRunawayPredict(PerceptorImpl):
    def __init__(self, *args, **kwargs):
        """
        Initialize the ThermalRunawayPredict perceptor with default values and load the machine learning model.
        
        Args:
            *args: Variable length argument list.
            **kwargs: Arbitrary keyword arguments.
        """
        # Initialize the prediction output variable
        self.y = 0
        
        # Initialize a flag to indicate thermal runaway status
        self.thermal_run = 0
        
        # Load the pre-trained machine learning model from a pickle file
        # The model is expected to be located in the 'ml_models' directory relative to the script's path
        model_path = os.path.join(path, "ml_models", "ml_predict_temperature_122.pkl")
        try:
            with open(model_path, 'rb') as model_file:
                self.ml_model = pickle.load(model_file)
        except FileNotFoundError:
            print(f"Machine learning model not found at {model_path}. Please ensure the model file exists.")
            self.ml_model = None
        except Exception as e:
            print(f"An error occurred while loading the ML model: {e}")
            self.ml_model = None
        
        # Initialize a list to store historical ML predictions if needed
        self.ML_list = []
        
        # Initialize the last recorded 'Tc' value to compute its change (ΔTc)
        self.last_Tc = 0

    async def compute(self, obs_spec, obs):
        """
        Compute the thermal runaway prediction based on current sensor observations.
        
        Args:
            obs_spec: Observation specification (not used in this implementation).
            obs: Current sensor observations. Can be a list or a dictionary.
        
        Returns:
            dict: A dictionary containing the thermal runaway prediction.
                  Example: {"thermal_runaway_predict": 1}
        """
        # Ensure that 'obs' is a dictionary. If not, convert it using predefined sensor keys.
        if not isinstance(obs, dict):
            # Define the expected sensor keys
            obs_keys = ['T', 'Tc', 'Ca', 'Cref', 'Tref', 'Conc_Error', 'Eps_Yield', 'Cb_Prod']
            # Convert the list to a dictionary by zipping it with the sensor keys
            obs = dict(zip(obs_keys, obs))
            print("Converted 'obs' to dictionary format using predefined sensor keys.")
        
        # Calculate the change in 'Tc' (ΔTc) since the last observation
        if self.last_Tc == 0:
            # If this is the first observation, assume an initial ΔTc of 5
            self.ΔTc = 5
        else:
            # Compute ΔTc as the difference between current 'Tc' and the last recorded 'Tc'
            try:
                current_Tc = float(obs['Tc'])
                self.ΔTc = current_Tc - self.last_Tc
            except (KeyError, ValueError, TypeError) as e:
                # Handle cases where 'Tc' is missing or cannot be converted to float
                print(f"Error accessing or converting 'Tc': {e}")
                self.ΔTc = 0  # Default to 0 if there's an error
        
        # Initialize the prediction output
        y = 0
        
        # Check if the current temperature 'T' exceeds or equals 340
        try:
            current_T = float(obs['T'])
        except (KeyError, ValueError, TypeError) as e:
            print(f"Error accessing or converting 'T': {e}")
            current_T = 0  # Default to 0 if there's an error
        
        if current_T >= 340:
            # Prepare the feature vector for the ML model
            try:
                Ca = float(obs['Ca'])
                Cref = float(obs['Cref'])
            except (KeyError, ValueError, TypeError) as e:
                print(f"Error accessing or converting 'Ca' or 'Cref': {e}")
                Ca = 0
                Cref = 0
            
            # Feature vector: [Ca, T, Tc, ΔTc]
            X = [[Ca, current_T, self.ΔTc]]
            
            # If the ML model was loaded successfully, make a prediction
            if self.ml_model:
                try:
                    # Predict the probability of thermal runaway
                    y_proba = self.ml_model.predict_proba(X)
                    
                    # Get the predicted class label (e.g., 0 or 1)
                    y = self.ml_model.predict(X)[0]
                    
                    # Optionally, use the probability to adjust prediction confidence
                    # For example, set y=1 only if the probability of class 1 is >= 0.3
                    if y_proba[0][1] >= 0.3:
                        y = 1
                    else:
                        y = 0
                except Exception as e:
                    print(f"Error during ML model prediction: {e}")
                    y = 0
            else:
                print("ML model is not loaded. Cannot make predictions.")
                y = 0
        
        # Update the last recorded 'Tc' with the current value for the next computation
        try:
            self.last_Tc = float(obs['Tc'])
        except (KeyError, ValueError, TypeError) as e:
            print(f"Error accessing or converting 'Tc' for updating last_Tc: {e}")
            self.last_Tc = self.last_Tc  # Keep the previous value if there's an error
        
        # Optionally, store the prediction in ML_list for historical tracking
        self.ML_list.append(y)
        
        # Update the prediction output variable
        self.y = y
        
        # Return the prediction as a dictionary
        return {"thermal_runaway_predict": y}

    def filtered_sensor_space(self, obs):
        """
        Define which sensors are relevant for this perceptor.
        
        Args:
            obs: Current sensor observations (not used in this implementation).
        
        Returns:
            list: Names of the sensors to be used.
        """
        # Specify the sensors that this perceptor will use
        return ['T', 'Tc', 'Ca', 'Cref', 'Tref', 'Conc_Error', 'Eps_Yield', 'Cb_Prod']

2. Copy the Strategy Pattern Agent, name it Strategy Pattern with Perceptor, and add the Perceptor Skill to your Strategy Pattern Agent

Drag the Perceptor thermal_runaway_predictor that you can now see on the left-hand side of your project onto the perception layer.

3. Run Your Training Session

Now, we are ready to train your agent and see the results. We suggest you run 50 training cycles.

4. View Results

When the training has been completed, you can view your results in the training sessions tab in the UI. This will show you information on how well the agent is learning.

The agent training results will be a little bit different from the strategy pattern alone. That's because the thermal runaway predictor is making a difference in how the agent performs.

Analyzing the Strategy Pattern Agent’s Performance with Perception

Conversion rate: 92% Thermal runaway risk: Very low

We tested this fully trained agent and plotted the results.

Adding perception improves agent temperature control performance.

The red lines on the graph show where the perceptors helped the agent make adjustments to avoid thermal runaway. This agent gets the same yield as the strategy pattern agent, but the improved temperature control has reduced the thermal runaway incidents from low to 0.

In this tutorial, we will learn how to upload simulators via the Composabl CLI as well as your custom, adapted simulators to the Composabl UI.

Upload simulators via Composabl CLI

To upload simulators that already follows the Composabl simulation specification all you need to do is the fallowing command from the simulation folder:

• After that, you can go to the Composable editor and connect that sim to any project.

Upload Third-Party Simulators via Docker

Prerequisites

If your simulator isn't already compatible with the Composabl platofrm you will need to create gRPC bindings and upload it as a Docker image in the Composabl editor. You can follow along with these Composabl API patterns and the following structure:

Going through the files:

• docker/entrypoint.sh: This file is the entrypoint of your Docker container.

• src/exceptions/invalid_usage.py: This file contains the exception class InvalidUsage that is used to raise exceptions in the simulator.

• src/__init__.py: This file is the initialization file of the module. No code is needed here, but for Python to recognize the folder as a module, this file is necessary.

• src/main.py: This file is the main file of the simulator. It uses the composabl_core.networking module to expose the simulator to the Composabl platform. This file is also available zipped along with this tutorial.

• src/server_impl.py: This file contains the implementation of the server that will be used to run the simulator.

• src/sim.py: This file contains your implementation of the simulator itself. Usually, a Env (inheriting from gym.Env) class is implemented here, and it is used to run the simulator.

• Dockerfile: This file is the Dockerfile that will be used to build the Docker image of your simulator.

• requirements.txt: This file contains the Python packages that are necessary to run your simulator. It is used to install the necessary packages in the Docker image.

gRPC Spec for Implementation

Under the hood, the Composabl SDK uses gRPC to communicate with the Composabl platform.

To create a simulator that works with the Composabl API, you have to implement the following gRPC methods:

After making sure that your simulator is compatible with the Composabl platform, you can proceed to the next section.

Docker

• Go to the folder where your simulator is located. We can navigate to the simulator folder and see what is inside it.

• After that, ensure that you have Docker installed. You can check if Docker is installed by running the following command:

DockerHub

• To log in to Docker Hub, run the following command:

  You will be prompted to enter your Docker Hub username and password. After that, you should see a message saying that you are logged in.

• Now, we can build the Docker image of the simulator. To do so, run the following command:

  This command will build the Docker image of the simulator. The -t flag is used to tag the image with the name <your-docker-hub-username>/<simulator-name>. The . at the end of the command indicates that the Dockerfile is in the current directory.

• After building the image, you can check if it was built successfully by running the following command:

  And then push the image to Docker Hub:

Composabl UI

After that, you can go to the Composabl UI and upload your simulator. To do so, follow the steps below:

• Then, on the left sidebar, click on the "Simulators" tab. You should see a list of simulators that are already available on the platform.

  On the top right corner, you should see a button to "New Simulator". Click on it.

• A pop-up will appear, asking you to select between "Internal" and "External" simulators. Select "External".

• Then, you can fill the Title and Description of the simulator. We suggest smaller names for the Title and a more detailed description for the Description.

• After clicking in next step, a brief tutorial will open up. Take care to read it and then click on "Next Step" again.

• After that, you can fill the Docker Image field with the name of the Docker image you pushed to Docker Hub <your-docker-hub-username>/<simulator-name>.

  If the image is public, no more fields are needed. If the image is private, you need to fill the Docker Username and Docker Password fields with your Docker Hub username and password, respectively. Then, click on "Validate and next step".

Conclusion

If you've followed all the steps correctly, you should have successfully uploaded your simulator to the Composabl UI. You can now use your simulator to train agents and run simulations on the platform. If you have any questions or need help, feel free to reach out to us.

Controllers are programmed skills used in Composabl agents. These may be optimizers, PID or MPC controllers, heuristics, or any other Python algorithms that are useful to control parts of a process. Configure controllers in the SDK and then publish them to the UI to use in agents.

Create a New skill

To create a skill in the Python SDK, begin by logging in to the SDK by typing Composabl login from the CLI.

Then type composabl skill new.

Give the skill a name and a description in response to the prompts that follow. Choose your skill type ascontroller (a programmed module like an optimization algorithm or MPC controller).

Specify the folder where you’d like to create the skill.

The Composal SDK will create a folder and Python controller.py() file from the template.

The Python Controller Class

The Python controller class offers several functions that you can use to build your algorithms or rule-based controllers in Composabl.

Functions for Training

Controllers don't need training, since they are based on programming rather than learning, but they include functions that connect them to the rest of the agent during training.

Initialize your algorithm: the __init__ Function

The __init__ function can be used to initialize your Algorithm and define initial configuration variables, this is called once when the Runtime starts. Let's supose that you want to use a MPC algorithm:

Process Observation to Compute Action: the compute_action Function

The compute_action function will process the observation and returns an action. This function returns a number that represents the action that will be sent to the simulation.

The compute_termination function tells the Composabl platform when to terminate a practice episode and start over with a new practice scenario (episode). From a controller perspective, it makes most senses to terminate an episode when the agent succeeds, fails, or is pursuing a course of action that you do not find likely to succeed. This function returns a Boolean flag (True or False) whether to terminate the episode. You can calculate this criteria however seems best.

The success_criteria function provides a definition of skill success and a proxy for how completely the agent has learned the skill. The platform uses the output of this function (True or False) to calculate when to stop training one skill and move on to training the next skill. It is also used to determine when to move to the next skill in a fixed order sequence. The agent cannot move from one skill in a fixed order sequence to the next, until the success criteria for one skill is reached.

Here are some examples of success criteria definition:

• A simple but naive success criteria might return True if a certain sensors or KPI value for an episode or scenario crosses a threshold, but False if it does not.

• A complex success criteria might compare a benchmark controller or another agent to the agent across many key variables and trials. It returns True if the agent beats the benchmark on this criteria, but False otherwise.

Functions to Manage Information Inside Agents

As information passes through perceptors, skills, and selectors in the agent, sometimes it needs to change format along the way. You can use three teaching functions to transform sensor and action variables inside agents: transform_ sensors, transform_action, and filtered_ sensor _space.

To transform sensor variables, use the transform_sensor function to calculate changes to specific sensors, then return the complete set of sensor variables (the observation space).

Two of the most common reasons for transforming sensor variables are conversion and normalization. For example, if a simulator reports temperature values in Fahrenheit, but the agent expects temperature values in Celsius, use the transform_sensor function to convert between the two.

Normalization is when you transform variables into different ranges. For example, one sensor variable in your agent might have very large values (in the thousands), but another variable might have small values (in the tenths), so you might use the transform_sensor function to transform these disparate sensor values to a range from 0 to 1 so that they can better be compared and used in the agent.

You may want to transform action variables for the same reasons as sensor variables.

Use the filtered_sensor_space function to pare down the list of sensor variables you need for a particular skill. Pass only the information that a skill or module needs in order to learn or perform well.

Scenarios are situations where your agent needs to behave differently to succeed.

Scenarios are created by carving out sections of the simulation space, as defined by specific configurations of variables and variable ranges. This allows you to train skills only in the scenarios where they will need to perform, leading to more efficient training and better performance. Selectors learn to recognize scenarios and pass control to the appropriate specialist skill to handle the scenario’s demands.

Types of Scenarios

How you define scenarios depends on the type of sensor variables you are working with.

Discrete variables are categories that describe a particular scenario. For each scenario, there is likely a perceptor in the agent (a machine learning model for example) that inputs the sensors, processes the sensor values, and outputs the discrete variable category.

Continuous variables are numbers. When they are used to define a scenario that one number value defines the section of the space that your agent will explore for decisions.

Sometimes a scenario is better defined by a range of continuous values than by a single continuous value. In that case, the scenario would be defined by a continuous variable range.

Here are some examples of how scenarios can be defined in different ways:

Scenarios Example

Let’s say that we are creating an agent to plan the operations of a restaurant with varying demand for three different recipes. We could create three different scenarios:

• Low demand: 30 recipe A, 20 recipe B, and 10 recipe C

• Normal demand: 60 recipe A, 45 recipe B, and 10 recipe C

• High demand: 100 recipe A, 50 recipe B, and 25 recipe C

Once you defined these scenarios, you could set the agent to train in the low demand scenario until the success criteria is reached. Then it would move to normal demand, and then to high demand. We would expect the agent to learn faster as it progressed through the scenarios, with cumulative knowledge building over time.

Set Up Scenarios in the UI

Add Scenarios to Projects

Add scenarios to your project by clicking on Scenarios from the lefthand menu to open the Scenarios page.

Click Add scenario to create a new scenario. Give your scenario a name and then click Add condition to configure it. You will then have the option to choose from any of the available sensor variables and apply conditions to them.

• For continuous variables, select Is and enter the exact value.

• For continuous variable ranges, select Is between and enter the range.

• For discrete variables, select Is element of and enter the possible values.

Create Scenario Flows

In addition to identifying scenarios, Composabl also lets you teach your agents about how scenarios relate to each other. To do this, you can build scenario flows to establish sequences of scenarios for the agent to practice.

Without scenario flows, the training platform will connect scenarios at random. But sometimes it is important for agents to practice scenarios in a specific order. For example, it might be important to practice flying a drone in high wind and then landing in the same conditions. Or an agent managing a production line might need to practice going from a scenario in which raw materials are scarce to one in which material costs rapidly rise. By creating a scenario flow, you can ensure that a given sequence of scenarios will be included in training.

Once you have scenarios built, create sequential sections of the task by clicking Add section.

After creating all the sections you need, drag and drop the scenarios into the sections.

Create as many flows as you need to capture the different scenario sequences your agent will need to navigate.

Add Scenarios to Skills

Scenarios are added to agents as part of configuring skills. Scenarios are added to skills so that skills know what specific conditions they need to master. Not all scenarios are relevant to all skills. For example, a drone landing skill doesn’t need to practice taking off in windy weather.

The configurations you set up when building the scenario flows will appear in the skill configuration modal. For each section of the process, as defined by the flows, tell the selector which scenarios it should apply by checking the boxes next to each scenario.

A project is a collection of agents for the same use case. The best way to use Composabl is to build multiple agents within the same project using different design patterns and variations. This allows you to iterate and improve your agents to get the best possible performance.

All agents within a project share the same goals and the same simulator.

Create a Project in the UI

To create a project, click on New Project in the upper right hand corner of your dashboard.

You'll be prompted to enter the simulator associated with your project and then choose your team.

Use the AI Project Creation Assistant

You also have the option to use Composabl's AI assistant to help with project set up. This specialized AI assistant is designed to help solve the "cold start" problem and get you started with designing your agent more quickly.

Based on your written description of your project, the AI will automatically create customized goals and skills within your project. You can then access those goals and skill in the Agent Builder studio and edit and adjust them as needed.

When prompting the assistant, use as much detail as you can. You can tell the AI:

• What the problem or use case is

• What equipment or process the agent will control

• What the overall goal is

• Anything you know about phases of the process or different scenarios

You can integrate your simulator with the Composabl SDK by using the ServerComposabl class. This class provides the necessary methods for the simulator to interact with the Composabl SDK.

The methods of the ServerComposabl class allow the Composabl SDK to automatically take care of serializing and deserializing the different requests and responses.

To conform your simulator to the Composabl SDK, you must define a server implementation class that defines methods of how to talk with the Composabl SDK.

Set Up the Simulation Environment Instance

Make

Make is a request to create a new instance of the environment with the specifications requested.

• string env_Id; Identifier for the type of environment to create.

• dictionary env_Init; Initial configuration for the environment, as defined within the runtime configuration (link to section about how to define runtime configuration parameters)

Sensor_space_info

Sensor_space_info provides details about the environment’s sensor space.

Action_space_info

Action_space_info defines the agent's action space.

Action_space_sample

The action_space_sample function returns an element of the simulator’s action space.

Run the Simulation Environment Instance

Reset

Reset is a request to reset the environment, and returns the first observation of the newly reset environment.

• observation Initial observation of the environment.

• Dictionary info Additional information about the reset environment.

Step

Step provides the agent action to be applied to the environment. The return structure is as follows:

• observation; The observation following the action.

• float reward The reward received after taking the action.

• bool terminated Whether the episode has ended.

• bool truncated Whether the episode was truncated before a natural conclusion.

• Dictionary info Additional information about the step.

Close

Close denotes the simulator is done being used and may perform any necessary cleanups required.

Set_Scenario

Get_Scenario

Get_scenario returns the scenario that the simulation is currently running.

Create Visualizations

Get_Render

Get_render provides the current rendered image of the environment, either as a numpy array or a string.

In this tutorial, we will walk through how to integrate a trained machine learning (ML) model into your Composabl agent as a Perceptor. A perceptor allows your agent to interpret data from sensors, process it using a machine learning model, and output new variables that will help the agent make better decisions.

The goal is to publish a pre-trained ML model as a perceptor that adds a new layer of perception to your agent, enabling it to process sensor data in a more advanced way. This could be useful in a variety of scenarios, such as predictive maintenance, anomaly detection, or autonomous decision-making.

Step 1: Understanding the Perceptor

A Perceptor in Composabl is a module in the perception layer that inputs sensor data, processes it (potentially using an ML model), and outputs new variables that are automatically added to the list of available sensors.

For this example, let’s assume we are building a perceptor that uses a trained machine learning model to predict thermal runaway in a system.

Step 2: Setting Up the Trained Model

We will use a pre-trained ML model stored as a pickle file to predict thermal runaway based on certain temperature and chemical sensor readings. Here’s how to set up the trained ML model for use as a perceptor.

1. Store the ML Model: Assume the ML model has been trained and saved as a .pkl file. For this example, the model is stored in the path: ml_models/ml_predict_temperature.pkl.

2. Load the ML Model in the Perceptor: In the perceptor class, we will load the model and define how it processes the sensor data.

Step 3: Creating the Perceptor

Now, we’ll create the perceptor using the trained ML model to process the sensor data and predict thermal runaway events. The perceptor will be responsible for calling the model and returning the prediction as a new sensor variable.

We can start by creating the preceptor by using the Composable CLI with the following command:

The new preceptor will have the following file structure:

3.1. Configuring your pyproject.toml file

3.2. Implementing the Perceptor in the perceptor.py file

Here’s the Python code to create the perceptor:

In this perceptor:

• We load the trained machine learning model from a pickle file.

• The compute() method takes in sensor data (e.g., temperature, chemical concentrations), processes it, and uses the ML model to predict whether a thermal runaway event will occur.

• The perceptor outputs the prediction as a new sensor variable, thermal_runaway_predict.

3.2. Adding the Perceptor to Your Agent

Once the perceptor is defined, you can login to the Composabl editor and add it to your agent.

Conclusion

In this tutorial, we covered how to publish a trained ML model as a perceptor in Composabl. This allows the agent to integrate more advanced decision-making by processing raw sensor data through a machine learning model and outputting predictions as new sensor variables. This method can be applied in various domains, such as predictive maintenance, anomaly detection, and control systems.

Skills can be arranged in sequences or hierarchies, in skill groups, or as coordinated skills that output multiple decisions together. The orchestration structures reflect common design patterns that can be used to accelerate the design and creation of agents.

Orchestrate Skills in Hierarchies and Sequences

For some agent designs, the task will be broken down into different skills that each control the system under certain conditions. For these agents, a special skill called a selector chooses the right skill at the right time to accomplish the task. Selectors are the specialized supervisor skills that orchestrate the skills together, determining which skill to activate based on the conditions the system needs to respond to.

To add a selector to an agent, drag the selector into your agent above the skills.

You will then be prompted to configure the selector.

You will also be prompted to choose between an additional set of options that correspond to two separate Machine Teaching design patterns.

• Fixed-order sequence: perform the skills in a set order. This is used in the Functional Pattern, a design pattern that is useful for tasks that involve fixed sequences of actions.

• Variable order sequence: perform the skills in any order based on the selector’s determination. This is used in the Strategy Pattern, a design pattern that is useful for tasks that require different control strategies to be used in different situations or conditions.

Orchestrate Skills in Groups

Unlike agent designs that use a selector to assign control to skills one at a time, agents with skill groups use skills working together to make decisions.

Skill groups always consist of two skills. To create a skill group, simply drag the second skill under the first, and a skill group will automatically be created.

Skill groups are used for the Plan-Execute Pattern, where one skill determines what the action should be and a second skill then “turns the knobs” to implement the decision.

In the industrial mixer example, the DRL skill is able to train effectively because the actions of the MPC controller are predictable. That means that it can practice and learn knowing that variations in performance are due to its own actions.

In agents with multiple DRL skills arranged in plan-execute patterns, Composabl will always train the skills from the bottom to the top. In other words, the execute skill will have to achieve competence before the plan skill will start training. That allows each skill to effectively interpret the feedback from the system without confusion from each other.

Orchestrate Coordianted Skills

Some tasks require multiple skills to work together on a single decision, but in parallel rather than in sequence. Agents for these tasks use coordinated skills that learn to take action together toward a goal. Also known as Multi-Agent Training, coordinated skills are trained using a coach, rather than a teacher.

Examples of Coordinated Skills

Traffic Optimization: Enhancing traffic flow and safety by teaching individual vehicles to navigate optimally and cooperate with each other.

Collaborative Robotics: Enabling robots to work together on tasks such as assembly in manufacturing or coordination in logistics.

Smart Grids: Optimizing energy distribution by having agents represent power plants, storage, and consumers to improve efficiency and stability.

Multiplayer Games: Creating adaptive and intelligent NPCs that can offer dynamic challenges to players in competitive or cooperative game settings.

Communication Networks: Improving network performance by optimizing resource allocation and traffic routing through agents representing network components.

Environmental Management: Balancing economic, ecological, and social goals in land use and resource management by simulating stakeholders as agents.

Healthcare Logistics: Strategizing resource allocation and treatment plans in scenarios like pandemics by considering the actions of hospitals, pharmacies, and patients as agents.

Supply Chain Optimization: Minimizing costs and delivery times in supply chains by coordinating agents representing various stages of the supply chain process.

Orchestrate Coordinated Skills with the SDK

Coordinated skills are not yet available in the UI. In the SDK, we have expanded the API to integrate Coordinated Skills through the add_coordinated_skill method on your agent. This method accepts a new class that gets configured named CoordinatedSkill, just as with the Teacher or Controller classes we implement this class by inheriting from the Coach class.

The coordinated skill will now take the incoming observation and action spaces and pass it to the sub-skills as a shared environment observation and action taking. The sub-skills will then return their own observations and actions, which will be passed back to the coordinated skill. The coordinated skill will then return the combined observations and actions to the agent.

The performance goal is the most important KPI or metric you will use to evaluate your agent’s success.

This goal directs your agent as it trains. The AI learning technology within the agent will reward the agent when it gets closer to the goal, helping it to improve.

Set a Top-Level Goal in the UI

To edit a goal created by the copilot, or to create a new goal, follow these steps.

1. Navigate to the project page

2. Click Set up goal and enter a name and description for your goal.

3. Click Add condition to define the goal.

Use the dropdown menus to select the variables and then define the parameters for each goal you want to include.

Configure Goals

Goals apply to one of the sensor variables, and are defined using one of five possible directives:

• Avoid: Keep the variable from reaching a specified value

• Maximize: Maximize the value of the variable

• Minimize: Minimize the value of the variable

• Approach: Get the value to the target range as quickly as possible

• Maintain: Keep the variable at a specified value

Advanced Goal Settings

You can also use advanced goal settings to fine-tune your goals. Access the advanced settings by clicking on the settings icon associated with one of the goals. You'll then see additional options to configure your goal.

• Tolerance only applies to the three objective types that include a target, Avoid, Approach, and Maintain. This setting allows you to tell the agent to accept a range of values around the target as successful performance. You might use this to prevent the agent from using too much compute power trying to get from good enough to perfect.

• Stop value allows you to tell the agent to end the training episode when the variable reaches the target value. This could be because the agent has succeeded and the process is complete, or because it has failed and needs to try again.

• Stop steps allows you to tell the agent to end the training episode after a certain number of iterations.

• Boundaries are for normalizing rewards. This is useful when the problem has variables that are very different, which can otherwise make it difficult for the trainer to calculate reward. For example, one sensor variable in your agent might have very large values (in the thousands), but another variable might have small values (in the tenths), so you might use boundaries to allow the agent to better compare the two variables.

• Scale allows you to provide relative weights to goals to account for goals that have different levels of importance or priority. This is very difficult to get right with Machine Teaching and should be handled with care.

In this tutorial, we will walk through how to set up a skill in Composabl that integrates with a third-party API. This type of integration allows your agent to communicate with external systems, such as machine performance APIs, and use the data to make informed decisions.

We will create a programmed skill that connects to a mock third-party API, process its response, and return an action based on the data received. This tutorial will also touch on orchestrating this skill within your agent.

Step 1: Defining the Programmed Skill

A programmed skill in Composabl is created by specifying the logic for interacting with the external API and processing the response. In this case, we will create a simple API connection to a fake endpoint that returns data about machine performance. The agent will act based on the information received.

1.1. Creating the API Integration Skill

We’ll define a programmed skill for making the API request. Here's an example of how to define the skill using a controller function that calls the API and processes the response.

import requests 
from composabl import SkillController

# Define the programmed skill 
class ThirdPartyAPISkill(SkillController): 

    def __init__(self, *args, **kwargs):
        self.api_url = "https://api.example.com/machine-status" 

    async def compute_action(self, obs, action):
        # Send sensor data to the third-party API 
        response = self._call_api(obs) 
        # Process the response and return an action 
        action = self._process_response(response) 
        return action
  
    def _call_api(self, observation): 
        try: 
            response = requests.post( 
                self.api_url,  
                json=observation,  
                headers={'Content-Type': 'application/json'} 
            ) 
            response.raise_for_status() 
            return response.json() 

        except requests.RequestException as e: 
            print(f"API call failed: {e}") 
            return None 

    def _process_response(self, response): 
        if not response:
            # Default action 
            return 0.0

        action = float(response.get("action"))
        reason = response.get("reason", "No reason provided") 

        print(f"Action: {action} - Reason: {reason}") 
        return action

    async def transform_sensors(self, obs):
        return obs

    async def filtered_sensor_space(self):
        return ['sensor1', 'sensor2', 'sensor3']

    async def compute_success_criteria(self, transformed_obs, action):
        return False

    async def compute_termination(self, transformed_obs, action):
        return False

In this example:

• The compute_action() method sends observation data (e.g., from sensors) to a third-party API.

• The _call_api() function makes the API call and handles any errors that might occur.

• The _process_response() function processes the response from the third-party API and determines the appropriate action for the agent to take based on the data.

Step 2: Adding the Programmed Skill to the Agent

2.1. Adding the Skill to Composabl UI

Once the skill is defined, you can add it to your agent in the UI using the methods below:

1. Create a new Skill using the Composabl CLI with a given name and description and implementation type, that in this case will be a controller. The name will be "third_party_api_skill"

composabl skill new

1. Change the controller.py code to use the class that you created: ThirdPartyAPISkill(). Change the pyproject.toml file to include your class ThirdPartyAPISkill in the entrypoint and its name:

[project]
name = "Third Party API Skill"

entrypoint = "third_party_api_skill.controller:ThirdPartyAPISkill"

1. Publish the Skill to the UI

composabl login

composabl skill publish third_party_api_skill

Select your organization and project that you want to publish it to.

Reference: https://docs.composabl.io/changelog/0-8-0.html

2.2. Adding the Skill to Composabl SDK

Once the skill is defined, you can add it to your agent using the add_skill() SDK method. This allows the agent to execute the API connection skill when necessary.

Here’s how to add the ThirdPartyAPISkill to the agent:

# Define and add the third-party API skill 
third_party_skill = Skill("third_party_api", ThirdPartyAPISkill) 
agent.add_skill(third_party_skill) 

By importing and creating the class with SkillController, you are indicating that this skill is programmed and does not require training. It will use predefined logic to interact with the third-party API and make decisions based on the data returned.

Conclusion

By following these steps, you’ve successfully defined and integrated a programmed skill that communicates with a third-party API into your Composabl agent. The agent can now take actions based on external data and dynamically respond to scenarios.

This approach allows agents to interface with a wide range of external systems, from monitoring equipment to adjusting machine settings, all through programmable skills.

Orchestration of skills through selectors ensures the agent executes the correct skills at the right time, whether the skills are learned or programmed.

Perceptors use the SDK and CLI workflow.

To access a template for a perceptor, type composabl perceptor new into the CLI. Composabl will then generate a perceptor template that you can populate with your information.

In this simple perceptor example we calculate the perceptor outputs that will be added as new sensor variables and we create a list of perceptors that comprise the perception layer.

python
class DeltaCounter():
    def __init__(self):
        self.key = "state1"
        self.previous_value = None

    def compute(self, sensors):
        if self.previous_value is None:
            self.previous_value = sensors[self.key]
            return {"delta_counter": 0, "state2": 0}

        delta = sensors ["state1"] - self.previous_value
        self.previous_value = sensors["state1"]
        return {"delta_counter": delta, "state2": 0}

    def filtered_sensor_space(self, sensors):
        return ["state1"]

delta_counter = Perceptor(["delta_counter", "state2"], DeltaCounter, "the change in the counter from the last two steps")

Analyzing agent behavior is a key part of using Composabl. This capability is key for:

• Building agents that beat performance benchmarks

• Monitoring the performance of your agents and simulators in real-time

• Analyzing your agents and simulators to understand what went wrong in case of problems

• Creating dashboards and reports to share with your team or customers

Composabl offers multiple options for analyzing agent behavior. For high-level summaries, you can use the benchmark reporting features in the UI. For more in-depth information, you can use the Historian in the SDK, which creates a database of all the information gathered during training and deployment. You can also design an LLM module in your agent to provide customizable communications capacity using natural language.

In this tutorial, we will explore how to use the historian to validate the trained AI agent in Composabl and training logs. The historian stores historical time-series data in an optimized format (parquet) - https://www.databricks.com/glossary/what-is-parquet, which helps in evaluating how the agent is performing over training.

Step 1: Accessing the Historian Data

The historian file stores time-series data essential for validating agent training. There are several ways to access and store the historian data, but the recommended format is as a delta file (parquet).

1. Understanding the Format:

   • The historian data is typically large, around 500 megabytes for standard operations. It is stored in a Delta Lake file format, optimized for time-series data and supporting efficient queries.

   • This file can be downloaded as XML or another format (e.g., CSV or XLS) but is most efficient in Delta Lake for handling larger datasets.

2. Downloading the Historian File:

   • From the Composabl UI, download the historian file. This file will likely come in a compressed format (e.g., .gz).

   • After extracting it, you should see the delta file containing time-series data.

Step 2: Setting Up for Validation

1. Unpacking the Historian File:

   • If the historian file is compressed (e.g., .gz), unpack the file using a tool like gzip:

     Copy

     
     gunzip -k historian_file.gz 
     

   • Once unzipped, you’ll see a 10 MB+ delta file with historical time-series data.

2. Understanding the Delta File:

   • The delta file is optimized for fast reads and writes of time-series data.

   • It supports an append-only structure, which ensures that each new piece of data can be added efficiently without modifying the existing data.

Step 3: Querying the Historian Data

1. Setting Up a Query Environment:

   • To validate your agent’s training, you’ll need to set up an environment that allows you to query the delta file. Delta Lake integrates well with systems like Apache Spark, but for simple querying, you can use tools like pandas in Python.

2. Querying for Agent Training Logs:

   • Extract and analyze relevant historical data from the delta file. Here's a simple Python example for querying the delta file using pandas:

   Copy

   
   import pandas as pd 
   
   
   
   # Load the historian delta file 
   
   df = pd.read_parquet('historian_delta_file.parquet') 
   
   df = df.sort_values(by=['timestamp'])
   
   df_data = df[df['category_sub'].isin(['step', 'skill-training','skill-training-cycle'])]
   #filter df with composabl_obs on "data" col only
   df_data = df_data[(df_data['data'].str.contains('composabl_obs')) | (df_data['category_sub'].str.contains('skill-training')) | (df_data['category_sub'].str.contains('skill-training-cycle'))]
   
   #df_data['data'] = df_data['data'].apply(lambda x: x if 'composabl_obs' in x else None)
   def convert_to_dict(x):
      try:
         return json.loads(x)
      except:
         try:
               return ast.literal_eval(x)
         except:
               return None
   
   df_data['data'] = df_data['data'].apply(lambda x: convert_to_dict(x))
   
   df_data['skill_name'] = df_data['data'].apply(lambda x: x['name'] if 'is_done' in x else None)
   df_data['skill_name'] = df_data['skill_name'].fillna(method='bfill')
   
   df_data['reward'] = df_data['data'].apply(lambda x: x['teacher_reward'] if 'composabl_obs' in x else None)
   
   df_data['obs'] = df_data['data'].apply(lambda x: x['composabl_obs'] if 'composabl_obs' in x else None)
   
   #df_data['done'] = df_data['data'].apply(lambda x: x["teacher_terminated"] if "teacher_terminated" in x else None)
   df_data['cycle'] = df_data['data'].apply(lambda x: x['cycle'] if 'cycle' in x else None)
   df_data['cycle'] = df_data['cycle'].fillna(method='bfill')
   
   df_data = df_data[df_data['category_sub'] == 'step']
   
   print(df_data)
   
   # group by runs
   df_group = df_data.groupby(['run_id','skill_name','cycle'])['reward'].mean()
   
   # Process observation data
   df_obs = pd.DataFrame(data=[[v[0] for v in list(x.values())] for x in df_data['obs'].values], columns=[list(df_data['obs'][0].keys())])
   
   df_obs['cycle'] = df_data['cycle']
   df_obs['run_id'] = df_data['run_id']
   df_obs['skill_name'] = df_data['skill_name']
   df_obs.columns = [x[0] for x in list(df_obs.columns)]
   
   # Episode Reward by Run Id
   for run_id in list(set([x[0] for x in df_group.index])):
      for skill in list(set([x[1] for x in df_group.index])):
         #df_group[run_id].plot(subplots=True, title=run_id)
         plt.plot(df_group[run_id][skill])
         plt.ylabel(f'Mean Episode Reward')
         plt.xlabel(f'Cycle')
         plt.title(f'{run_id} - {skill}')
   
         plt.show()
   

Key Benefits of Using the Historian for Validation:

• Optimized Data Handling: The Delta Lake format is designed for fast querying, making it ideal for time-series data.

• Efficient Storage: The append-only nature ensures that new data can be added without overwriting or modifying existing data, making it easy to track data over time.

• Continuous Monitoring: By continuously adding data to the historian, you can validate your agent's long-term impact on machine performance, uptime, and safety.

Once you have created new agent components in the SDK or configured existing algorithms and models, you can publish them with a simple CLI workflow to make them available in the UI to drag and drop into agent designs.

Agent components are published into projects. They will then be available to use for all the agents you create for that project.

Publish Skills and Selectors

1. Log in to Composabl: composabl login.The system will redirect you to the UI to enter your credentials and log in.

2. Return to the command line and navigate to the folder containing the skill or selector you want to publish.

3. Publish the skill or selector: composabl skill publish or composabl selector publish.

4. Select your organization from the dropdown menu.

5. Select your project from the dropdown menu. Save your new skill or selector there.

6. Your skill or selector will begin publishing. When the process is complete, go to the UI, navigate to the Agent Builder Studio, and refresh your browser to see your new skill or selector in the sidebar.

Publish Perceptors

1. Log in to Composabl: composabl login.The system will redirect you to the UI to enter your credentials and log in.

2. Navigate to the perceptors folder (one level above the individual perceptor folder).

3. Publish the perceptor: composabl perceptor publish perceptor_name

4. Select your organization from the dropdown menu.

5. Select your project from the dropdown menu. Save your new skill or selector there.

6. Your skill or selector will begin publishing. When the process is complete, go to the UI, navigate to the Agent Builder Studio, and refresh your browser to see your new skill or selector in the sidebar.

Adding perception modules to your agent can provide more rich, complex, condensed, and nuanced information to the decision-making parts of the agent. For example, you might include a computer vision model in your perception layer that inputs images or video from a camera and outputs classifications of objects that it identifies. You can also add large language models as perceptors to take in and interpret information in natural language.

Each module in the perception layer for a Composabl agent inputs the sensor variables, processes those variables in some way, and outputs one or more new variables that the platform will automatically add to the list of sensors.

Perceptors can use any supported Python function or library to calculate outputs. They can even call machine learning and large language models or their APIs.

The next three pages explain how to use the SDK and CLI workflow to create new perceptors or configure existing models as perceptors to use in Composabl agents.

Add Perceptors to Agents

Just like skills, perceptors can be dragged and dropped into agents using the UI. Perceptors will always be situated in the Perception layer that comes before selectors and skills. That’s because perception needs to be applied to the sensor inputs to create new variables that are then passed to the skills layer for the agent to use in decision-making.

You can use the Composabl UI to create skills that learn with deep reinforcement learning. When you create a teacher in the UI, you select goals. Composabl then turns these goals into reward functions to train the skill.

Create Learned Skills with Goals in the UI

To create a skill from the UI, click New skill from the skills panel and name your skill in the modal that pops up. You will then be prompted to configure your skill.

If you want your skill to be taught with deep reinforcement learning, select Teacher. You’ll then be prompted to add goals to your skill for training.

Configure Goals

Each skill in your agent succeeds as it approaches a specific goal. The goals of each skill should be clean and simple. If your agent is designed well, based on a good breakdown of the task into skills, each skill will have a clear goal.

Goals apply to one of the sensor variables, and are defined using one of five possible directives:

• Avoid: Keep the variable from reaching a specified value

• Maximize: Maximize the value of the variable

• Minimize: Minimize the value of the variable

• Approach: Get the value to the target range as quickly as possible

• Maintain: Keep the variable at a specified value

For example, for the industrial mixer, we want to maximize the concentration of the product, Ca. We also want to avoid temperature, T, getting above 400 degrees Kelvin.

Advanced Goal Settings

You can also use advanced goal settings to fine-tune your goals. Access the advanced settings by clicking on the settings icon associated with one of the goals.

You'll then see additional options to configure your goal.

• Tolerance only applies to the three objective types that include a target, Avoid, Approach, and Maintain. This setting allows you to tell the agent to accept a range of values around the target as successful performance. You might use this to prevent the agent from using too much compute power trying to get from good enough to perfect.

• Stop value allows you to tell the agent to end the training episode when the variable reaches the target value. This could be because the agent has succeeded and the process is complete, or because it has failed and needs to try again.

• Stop steps allows you to tell the agent to end the training episode after a certain number of iterations.

• Boundaries are for normalizing rewards. This is useful when the problem has variables that are very different, which can otherwise make it difficult for the trainer to calculate reward. For example, one sensor variable in your agent might have very large values (in the thousands), but another variable might have small values (in the tenths), so you might use boundaries to allow the agent to better compare the two variables.

• Scale allows you to provide relative weights to goals to account for goals that have different levels of importance or priority. This is very difficult to get right with Machine Teaching and should be handled with care.

The Training Sessions page allows you to view agents' training in real time and analyze their performance in training.

When you begin a training session, the graphs for each trained skill will begin to generate. You can watch your skills learn by viewing the graphs, or you can click on the Console Output tab for detailed information about each training decision.

The shape of the curve can help you understand how your agent is learning. When the curve plateaus, that usually means that the skill has been succesfully trained and will not learn more. If the curve shows jagged ups and downs, then the skill isn't performing consistently and has more learning to do. Sometimes this is a sign that you should go back and adjust the training settings.

The training sessions page shows a list of all the training sessions for a project in a menu on the left of the screen, allowing you to jump between different agents, as well as different training sessions for the same agent.

About Cluster Training

Composabl agents use Kubernetes clusters to train at scale. A cluster is a collection of computers that work on large tasks simultaneously. This provides enough compute to complete large training tasks as efficiently as possible.

Composabl offers two options for cluster training:

• Use Composabl's Training as a Service offering to train on our clusters

• Use your own compute clusters through Azure, AWS or another provider

Ensure that Your Agent is Ready

Before you submit your job for training on a cluster, make sure that your agent is fully configured and all the parameters have been set. That means checking all the agent components:

• Goals

• Perceptors

• Selectors, including goals for learned selectors and scenarios

• Skills, including goals for learned skills

• Scenarios, including scenario flows

Any component of the agent with a warning sign is not fully configured and not ready for training. Go back to edit that agent component and make sure that all of the fields are filled out.

Choose the Right Cluster

You can train on your own cluster or on Composabl’s clusters using training as a service (TaaS) credits. If you want to use Composabl’s clusters, ensure that you have credits available.

To train on your own cluster, make sure that you have set your cluster up and installed Composabl successfully.

Click Train and then choose the cluster option in the menu. You will then have the option to configure your training session.

Configure Your Training Session

Training session configuration options are the same whether you’re using TaaS or training on your own cluster.

Set the Number of Training Cycles

A training cycle is a complete pass through the entire task, with the agent continuing until it reaches success or some other stop criteria. Your agent will train each skill one at a time for the selected number of training cycles, starting from the bottom of the agent design.

A training cycle typically involves about 1,000 agent decisions. Depending on the complexity of the task, agents may need to complete anywhere between 100 and several thousand training cycles to become proficient.

Set the Number of Simulators

You can run multiple simulators in parallel to speed up training. If you run more than one simulator during a training, the number of training cycles selected will be multiplied by the number of simulators, so 5 training cycles with 3 simulators selected would lead to 15 training cycles total.

You can use the Advanced Configuration to choose how powerful each machine running a simulator should be. If you choose Small, each training cycle selected will result in one training cycle completed. If you choose GPU, you will get 4 training cycles for each training cycle.

More training cycles running simultaneously will speed up training but also increase costs. How long your training takes also depends on the complexity of your agent and your simulator.

Start Training

When you have configured your settings correctly, click Start Training.

You will then be taken to the Training Sessions page. There you can follow the agent training progress by viewing the real-time plots or the console output.

Note that it will take a few minutes for the visualization to begin.

The Composabl benchmarking feature allows you to compare the performance of different multi-agent systems against key performance indicators (KPIs). This tool helps you evaluate your agent system's effectiveness, track their improvements, and calculate potential return on investment (ROI).

Benchmarks are generated only after training is complete. You won't see benchmark data while training progresses, even for long-running jobs. If a training job fails, no benchmark data will be generated.

Note: Benchmarking does not influence agent system training, goals, or rewards. It is purely an analysis tool that helps you answer the question: "Based on how this agent system was trained and how we believe it should perform in real life, how much money is this agent system generating?"

Getting Started

Define the KPI for Your Project

The key performance indicator (KPI) for your project is the top-level objective that you will use to compare and evaluate your multi-agent systems' performance.

To define the KPI for your project, choose the sensor variable that represents the metric that best indicates performance success. For most processes, this is a metric with a clear business impact, such as product yield or energy use.

Setting up the KPI

To set up the KPI:

1. Navigate to the Benchmark page from the main dashboard

2. Click the "Settings" button in the top right of the KPI Performance Metrics section

3. In the "Set up KPI" section, configure the following:

   • Sensor Name: Select a sensor or preceptor variable to track (e.g., "Eps_Yield")

   • Target Value: Set the desired target for this KPI (e.g., 0.33)

   • Benchmark Unit: Select the unit of measurement (e.g., "% of Max Theoretical Yield"). Note that this is for UI readability only and doesn't affect calculations.

Setting up ROI Calculations

Return on Investment (ROI) calculations help quantify the financial impact of your agent systems' performance. The ROI is calculated based on the monetary value of the difference between your target value and the actual value of the KPI generated by that agent system, minus training costs.

1. In the Settings modal, navigate to the "Enter ROI criteria" section

2. Define the conditions for calculating ROI:

   • When: Select the metric to track (e.g., "Eps_Yield")

   • Condition: Choose how the metric changes (e.g., "increases by")

   • Percentage: Enter the percentage change that matters (e.g., 2%)

   • Value: Set the financial value of this change (e.g., $1,000,000)

   • Period: Select the time period (e.g., "per year")

For example: "When Eps_Yield increases by 1% of max theoretical yield, it is worth $1,000,000 per year."

You can change KPI and ROI settings at any time after training completes, and the benchmark results will be automatically recalculated. This allows you to explore different business scenarios without retraining your agent systems.

Reading the Benchmarking Dashboard

Understanding the Overview Panel

At the top of the benchmarking page, you'll find an Overview panel containing three key metrics:

• Highest performing Agent System: Displays the outcome of the highest performing agent system

• Lowest performing Agent System: Displays the outcome of the lowest performing agent system

• ROI: Shows the calculated financial return based on your KPI settings

Each metric displays both the value and the relevant unit (e.g., "% of Max Theoretical Yield").

Bar Graph Visualization

The bar graph in the KPI Performance Metrics section provides a visual comparison of your agent systems:

• Each bar represents an agent system's performance on your selected KPI (all bars correspond to the same KPI)

• The height of the bar indicates the specific value of the KPI for that agent system (Y-axis)

• Horizontal dashed lines show benchmark averages

• The actual value of the KPI for each agent system is shown above the respective bar

Performance Ranking Table

Below the graph is a detailed table showing:

• Agent System Number: Sequential ID for each agent system

• Agent System Name: Name or description of the agent system

• ROI: Calculated financial return based on the agent system's performance

  • Green arrows (↑) indicate positive ROI with the percentage and absolute value

  • Red arrows (↓) indicate negative ROI with the percentage and absolute value

• Eps_Yield (variable) per % of Max Theoretical Yield: Performance metric with indicators for highest agent system performers

Deploying an agent means exporting the trained agent, loading it into your production environment, and then asking the agent for decisions. In Composabl, the export is a json file called agent.json that contains all you need to deploy your agent. You can load the agent file to use in your IT infrastructure with many ways.

This document will show how you can deploy your agent as an API using Python and Flask.

Step 1: Accessing and Preparing the Files

To deploy the agent as an API, we need to extract the agent.json model, get the agent_api.py script to start the API, requirements.txt to install packages. You can find a sample for these files in our GitHub repo: https://github.com/Composabl/examples.composabl.io/tree/main/deploy_agent

This is the structure needed for the API:

*** How to extract the agent.json ***

1. Log into the Composabl UI (https://app.composabl.com/onboarding), train your agent, and navigate to the training sessions section.

2. Check the status of the agent:

   • Green status (Done) indicates finished training.

3. Download the agent essential file:

   • The agent file (a .gz compressed file).

   • Extract the agent file agent.json to the model folder

Step 2: Get the API python file

Step 3: Install requirement packages

Run the following in your terminal:

It will install these packages:

• composabl

• flask[async]

• numpy

Step 4: Export your license and start the API

To start Composabl API, you will need to export your license as an environment variable and then use Python to start the Flask API with your agent.

Step 5: Test your API

After running the API, you can test it by opening the terminal and run the script below:

In the POST request, we pass the use case "observation" with sensor variables and their values to receive an action from the agent. The code above is related to the agent.json demo for Chemical Process Control.

Once you have completed training an agent and are ready to deploy it you can download it from the UI.

1. Go to the Training tab in your project.

2. Select the training session for the agent you want to deploy.

3. Click on the Artifacts drop down in the upper right of the screen.

4. Click Inference Model and you will down load a file called agent.json.

Based on the notes provided in the image, We'll help you create a tutorial on accessing the agent runtime after deploying it to Docker. This tutorial will explain the steps for building the Docker container, deploying the agent, and accessing the runtime for inference or interaction.

Tutorial: Accessing the Agent Runtime After Deploying to Docker

Once you have packaged and deployed your agent inside a Docker container (https://docs.composabl.com/deploy-agents/deploy-an-agent-in-a-container), the next step is accessing its runtime for operations like model inference. This tutorial will guide you through the process of building and running the Docker container and then connecting to the agent's runtime for further interactions.

Step 1: Preparing the Dockerfile and Environment

To deploy the agent to Docker, we need to first create an image from the Dockerfile (https://docs.composabl.com/deploy-agents/deploy-an-agent-in-a-container). The Dockerfile will package the necessary runtime, model, and environment for the agent.

Step 2: Building the Docker Image

1. Building the Image: You can build the Docker image by running the following command in the terminal. This will take the Dockerfile and the associated files (like the pre-trained model) and create an image.

• The -t flag allows you to tag the image (composabl_agent_api), which makes it easier to reference later.

• Make sure that the model file (agent.json) and all relevant scripts are reachable within the Docker context (i.e., the directory from which you are building).

1. Checking the Image: Once the build is complete, you can verify that the image was created successfully by running:

Step 3: Running the Docker Container

Now that the image is built, the next step is to run it in a container. You will run the Docker container in an interactive mode to access the runtime.

• -it: Runs the container interactively.

• -p 8000:8000: Maps port 8000 from the container to port 8000 on your local machine so that you can access the HTTP server for the agent runtime.

• -e COMPOSABL_LICENSE="<your_license>" : is exporting the environment variable and linking to your composabl license

The HTTP server should now be up and running within the container, ready to handle model inference or other tasks.

Step 4: Accessing the Agent Runtime

With the Docker container running, you can now connect to the agent's runtime. The runtime will be an HTTP server, as mentioned in your notes. You can access it through a POST request for model inference or other operations.

1. Sending Requests to the Agent: You can send a POST request to the running server using a tool like curl, Postman, or any Python HTTP library (such as requests).

Here’s an example using curl:

This request will:

• POST data to the /predict endpoint on localhost:8000, which is being forwarded from the Docker container.

• The agent will handle the request, infer the model, and return the action as a result.

Conclusion

In this tutorial, we walked through the process of:

• Building a Docker image with your agent and its runtime.

• Running the Docker container interactively to expose the agent’s HTTP server.

• Accessing the agent runtime by sending HTTP requests for inference or other tasks.

By following these steps, you can deploy and interact with your Composabl agent in a Dockerized environment.s

Based on the notes provided in the image, I'll help you create a tutorial on accessing the agent runtime after deploying it to Docker. This tutorial will explain the steps for building the Docker container, deploying the agent, and accessing the runtime for inference or interaction.

Tutorial: Accessing the Agent Runtime After Deploying to Docker

Once you have packaged and deployed your agent inside a Docker container, the next step is accessing its runtime for operations like model inference. This tutorial will guide you through the process of building and running the Docker container and then connecting to the agent's runtime for further interactions.

Step 1: Preparing the Dockerfile and Environment

To deploy the agent to Docker, we need to first create a Dockerfile. The Dockerfile will package the necessary runtime, model, and environment for the agent.

1. Dockerfile Setup: Your Dockerfile should contain the following key components:

   • Base Image: Use a Python base image (or any base that supports the necessary libraries).

   • Copy Model Files: Copy the pre-trained model (e.g., .gz file) to the container.

   • Install Dependencies: Install any required Python libraries (like OHTTP or other packages for the agent).

Here’s an example Dockerfile:

Step 2: Building the Docker Image

1. Building the Image: You can build the Docker image by running the following command in the terminal. This will take the Dockerfile and the associated files (like the pre-trained model) and create an image.

• The -t flag allows you to tag the image (my-agent-runtime), which makes it easier to reference later.

• Make sure that the model file (agent.gz) and all relevant scripts are reachable within the Docker context (i.e., the directory from which you are building).

1. Checking the Image: Once the build is complete, you can verify that the image was created successfully by running:

Step 3: Running the Docker Container

Now that the image is built, the next step is to run it in a container. You will run the Docker container in an interactive mode to access the runtime.

• -it: Runs the container interactively.

• -p 8000:8000: Maps port 8000 from the container to port 8000 on your local machine so that you can access the HTTP server for the agent runtime.

The HTTP server should now be up and running within the container, ready to handle model inference or other tasks.

Step 4: Accessing the Agent Runtime

With the Docker container running, you can now connect to the agent's runtime. The runtime will be an HTTP server, as mentioned in your notes. You can access it through a POST request for model inference or other operations.

1. Sending Requests to the Agent: You can send a POST request to the running server using a tool like curl, Postman, or any Python HTTP library (such as requests).

Here’s an example using curl:

This request will:

• POST data to the /infer endpoint on localhost:8000, which is being forwarded from the Docker container.

• The agent will handle the request, infer the model, and return the result.

1. Interacting with the Agent: If you prefer to interact with the agent directly, you can also enter the container’s interactive mode and run commands.

This will open a shell inside the running Docker container, allowing you to execute any runtime commands manually.

Step 5: Automating the Process

For convenience, you can automate the entire process of building the image, running the container, and interacting with the agent by creating a script.

Here’s a basic example of an automation script:

Save this as run_agent.sh, and then execute it:

This script will:

• Build the Docker image.

• Run the container, mapping the necessary port and exposing the HTTP server for inference.

Step 6: Troubleshooting and Debugging

If the container fails to start, or if the server doesn't respond, you can debug the container by checking the logs:

This command will display the output of the running container, which can help diagnose issues like missing dependencies or server errors.

Conclusion

In this tutorial, we walked through the process of:

• Building a Docker image with your agent and its runtime.

• Running the Docker container interactively to expose the agent’s HTTP server.

• Accessing the agent runtime by sending HTTP requests for inference or other tasks.

By following these steps, you can deploy and interact with your Composabl agent in a Dockerized environment._

In this tutorial, we will cover how to connect to the agent runtime, load a pre-trained agent, run inference, and visualize the results in a production like evironment. The provided script, agent_inference.py, is a key component that demonstrates connecting to the Composabl agent runtime, initializing the environment, and plotting agent operation results.

Step 1: Understanding agent_inference.py

The script agent_inference.py connects to the runtime, loads a pre-trained agent, connects to a local simulation, collects sensor data from the sim and plots the results. Here is an outline of the core steps in the process:

1. Start Runtime and Load Agent: The script initializes the trainer and loads a pre-trained agent from a model folder.

2. Set Up the Simulation Environment: It connects to a simulation environment.

3. Run Inference: The pre-trained agent interacts with the simulation to perform inference (decisions), collecting observations and giving actions at each step.

4. Collect Data and Plot Results: Sensor data and actions are collected in a Pandas DataFrame, and the results are plotted using Matplotlib to visualize how the agent is performing over time in a production like environment.

Step 2: Connecting to the Runtime and Loading the Agent

The first task is to connect to the Composabl runtime and load the pre-trained agent. This is accomplished using the Trainer and Agent classes. The agent's model is loaded from the directory where the model was saved during training.

Here:

• Trainer(config) initializes the runtime with a configuration file.

• Agent.load(PATH_CHECKPOINTS) loads the saved agent from the specified checkpoint directory.

• trainer._package(agent) prepares the agent for inference by packaging it.

Step 3: Connecting to the Simulation Environment

Next, we connect the agent to the simulation environment. The make() function creates a connection to the local simulator, and the environment is initialized.

Here:

• The simulator is configured to run locally (localhost:1337) and you have to start it locally and manually before.

• The environment is initialized with sim.init(), and the agent is connected to it.

Step 4: Setting the Scenario and Running Inference

After connecting to the simulator, you need to set up the specific scenario that the agent will operate in. This scenario determines the environment's initial state.

With the environment set, the agent can now run inference for a set number of iterations. At each iteration, the agent observes the environment, takes an action, and collects the results (observations and rewards). This is done in a loop.

In each iteration:

• The agent performs an action based on the current observations.

• The environment advances one step with sim.step(action), and the agent receives a new observation and reward.

• Sensor data and actions are logged into a Pandas DataFrame for later analysis.

Step 5: Saving Data and Plotting Results

Once the inference loop is complete, the collected data is saved, and the results are visualized. The results are plotted using Matplotlib.

This code generates three subplots:

1. Temperature Controller (Tc) over time.

2. Temperature (T) and Reference Temperature (Tref) over time.

3. Concentration (Ca) and Reference Concentration (Cref) over time.

The plots provide a visual representation of the agent's performance during the simulation. Finally, the figure is saved as inference_figure.png in the benchmarks directory.

Step 6: Running the Script

To run the script, execute the agent_inference.py in your terminal.

Conclusion

In this tutorial, we demonstrated how to:

• Connect a pre-trained Composabl agent to a runtime and simulation environment.

• Set up a scenario and run inference.

• Collect observations and actions, and plot the results using Matplotlib.

By following these steps, you can visualize the performance of your agent and gain insights into how it interacts with the environment over time.

The Composabl SDK offers a suite of advanced tools to train skills using deep reinforcement learning. Using the Python teacher class, you can fine-tune the rewards for your skills. Once you have configured a skill with the SDK, you can publish it to the UI to use in agent designs.

Create a New Skill

To create a skill in the Python SDK, begin by logging in to the SDK by typing Composabl login from the CLI.

Then type composabl skill new.

Give the skill a name and a description in response to the prompts that follow. Choose whether your skill should be a teacher (learned with AI), controller (a programmed module like an optimization algorithm or MPC controller), or coach.

Specify the folder where you’d like to create the skill.

The Composal SDK will create a folder and Python teacher file from the template.

The Python Teacher Class

The Python teacher class offers several functions that you can use to fine-tune the training of your skills.

Functions for Training

Train with Rewards: the compute_reward Function

The compute_termination function tells the Composabl platform when to terminate a practice episode and start over with a new practice scenario (episode). From a teaching perspective, it makes most senses to terminate an episode when the agent succeeds, fails, or is pursuing a course of action that you do not find likely to succeed. This function returns a Boolean flag (True or False) whether to terminate the episode. You can calculate this criteria however seems best.

The success_criteria function provides a definition of skill success and a proxy for how completely the agent has learned the skill. The platform uses the output of this function (True or False) to calculate when to stop training one skill and move on to training the next skill. It is also used to determine when to move to the next skill in a fixed order sequence. The agent cannot move from one skill in a fixed order sequence to the next, until the success criteria for one skill is reached.

Here are some examples of success criteria definition:

• A simple but naive success criteria might return True if the average reward for an episode or scenario crosses a threshold, but False if it does not.

• A more complex success criteria might calculate root mean squared error (RMSE) for key variables across the episode and return True if the error is less than a customer specified benchmark, but False otherwise.

• A complex success criteria might compare a benchmark controller or another agent to the agent across many key variables and trials. It returns True if the agent beats the benchmark on this criteria, but False otherwise.

Train with Goals

Training with goals lets you use a predefined reward structure rather than configuring the rewards individually. When you use a goal, your agent will inherit the compute reward, compute termination, and compute success functions from the goal. (You will still have the option to further customize those functions as needed.)

The five goal types you can use are:

• AvoidGoal

• MaximizeGoal

• MinimizeGoal

• ApproachGoal

• MaintainGoal

Goals are added using specialized teacher classes rather than the general teacher class that you would otherwise use to teach skills. For example, for a skill named Balance that you wanted to train with a goal to maintain a specific orientation, you would use the MaintainGoal teacher class.

The parameters you can use for goals are:

You can also use more than one goal for a single skill using the CoordinatedGoal teacher class. This is useful when your agent needs to behave in a way that creates a balance between two goals that are both important.

Functions to Guide Agent Behavior with Rules

Just like rules guide training and behavior for humans, providing rules for the agent to follow can guide agent decision-making more quickly to success. Rules guide the behavior of an agent based on expertise and constraints.

Add Rules: the compute_action_mask Function

The compute_action_mask teaching function expresses rules that trainable agents must follow.

The compute_action_mask teaching function works only for discrete action spaces (where the actions are integers or categories), not for continuous action spaces (where decision actions are decimal numbers). If you specify a mask for a skill whose actions are continuous, the platform will ignore the action mask.

The function returns a list of 0 and 1 values. Zero means that the action is forbidden by the rule. One means that the action is allowed by the rule. The function may change returned value after each decision. This allows complex logic to express nuanced rules.

In the example above, the first action is forbidden for the next decision, but the second and third actions are allowed. The logic in the skill itself (whether learned or programmed) will choose between the allowed second and third actions.

All selectors have a discrete action space (they choose which child skill to activate), so you can always apply the compute_action_mask function to teach them.

Functions to Manage Information Inside Agents

As information passes through perceptors, skills, and selectors in the agent, sometimes it needs to change format along the way. You can use three teaching functions to transform sensor and action variables inside agents: transform_ sensors, transform_action, and filtered_ sensor _space.

To transform sensor variables, use the transform_sensor function to calculate changes to specific sensors, then return the complete set of sensor variables (the observation space).

Two of the most common reasons for transforming sensor variables are conversion and normalization. For example, if a simulator reports temperature values in Fahrenheit, but the agent expects temperature values in Celsius, use the transform_sensor function to convert between the two.

Normalization is when you transform variables into different ranges. For example, one sensor variable in your agent might have very large values (in the thousands), but another variable might have small values (in the tenths), so you might use the transform_sensor function to transform these disparate sensor values to a range from 0 to 1 so that they can better be compared and used in the agent.

You may want to transform action variables for the same reasons as sensor variables.

Use the filtered_sensor_space function to pare down the list of sensor variables you need for a particular skill. Pass only the information that a skill or module needs in order to learn or perform well.