Orchestrate Skills

Skills can be arranged in sequences or hierarchies, in skill groups, or as coordinated skills that output multiple decisions together. The orchestration structures reflect common design patterns that can be used to accelerate the design and creation of agents.

Orchestrate Skills in Hierarchies and Sequences

For some agent designs, the task will be broken down into different skills that each control the system under certain conditions. For these agents, a special skill called a selector chooses the right skill at the right time to accomplish the task. Selectors are the specialized supervisor skills that orchestrate the skills together, determining which skill to activate based on the conditions the system needs to respond to.

To add a selector to an agent, drag the selector into your agent above the skills.

You will then be prompted to configure the selector.

Selectors can be trained with DRL just like the other skills, and you set up goals for them the same way. Click on Configure Selector to be taken to the menu to set your selector’s goals. Note that the goals of the agent’s top-level selector should be the same as the goals for the agent as a whole.

You will also be prompted to choose between an additional set of options that correspond to two separate Machine Teaching design patterns.

• Fixed-order sequence: perform the skills in a set order. This is used in the , a design pattern that is useful for tasks that involve fixed sequences of actions.

• Variable order sequence: perform the skills in any order based on the selector’s determination. This is used in the , a design pattern that is useful for tasks that require different control strategies to be used in different situations or conditions.

Orchestrate Skills in Groups

Unlike agent designs that use a selector to assign control to skills one at a time, agents with skill groups use skills working together to make decisions.

Skill groups always consist of two skills. To create a skill group, simply drag the second skill under the first, and a skill group will automatically be created.

Skill groups are used for the , where one skill determines what the action should be and a second skill then “turns the knobs” to implement the decision.

A common plan-execute design is the set point pattern, where a DRL skill determines what the set point of a system should be, and then a traditional automation technology (a controller that uses model predictive control or optimization) determines what actions are necessary to get the system to the set point. For example, in the industrial mixer use case, the DRL plan skill decides what temperature the liquid in the tank should be. Then the MPC execute skill determines what temperature of coolant needs to be flowed around the tank to achieve the desired set point.

In the industrial mixer example, the DRL skill is able to train effectively because the actions of the MPC controller are predictable. That means that it can practice and learn knowing that variations in performance are due to its own actions.

In agents with multiple DRL skills arranged in plan-execute patterns, Composabl will always train the skills from the bottom to the top. In other words, the execute skill will have to achieve competence before the plan skill will start training. That allows each skill to effectively interpret the feedback from the system without confusion from each other.

Orchestrate Coordianted Skills

Some tasks require multiple skills to work together on a single decision, but in parallel rather than in sequence. Agents for these tasks use coordinated skills that learn to take action together toward a goal. Also known as Multi-Agent Training, coordinated skills are trained using a coach, rather than a teacher.

Examples of Coordinated Skills

Traffic Optimization: Enhancing traffic flow and safety by teaching individual vehicles to navigate optimally and cooperate with each other.

Collaborative Robotics: Enabling robots to work together on tasks such as assembly in manufacturing or coordination in logistics.

Smart Grids: Optimizing energy distribution by having agents represent power plants, storage, and consumers to improve efficiency and stability.

Multiplayer Games: Creating adaptive and intelligent NPCs that can offer dynamic challenges to players in competitive or cooperative game settings.

Communication Networks: Improving network performance by optimizing resource allocation and traffic routing through agents representing network components.

Environmental Management: Balancing economic, ecological, and social goals in land use and resource management by simulating stakeholders as agents.

Healthcare Logistics: Strategizing resource allocation and treatment plans in scenarios like pandemics by considering the actions of hospitals, pharmacies, and patients as agents.

Supply Chain Optimization: Minimizing costs and delivery times in supply chains by coordinating agents representing various stages of the supply chain process.

Orchestrate Coordinated Skills with the SDK

Coordinated skills are not yet available in the UI. In the SDK, we have expanded the API to integrate Coordinated Skills through the add_coordinated_skill method on your agent. This method accepts a new class that gets configured named CoordinatedSkill, just as with the Teacher or Controller classes we implement this class by inheriting from the Coach class.

The coordinated skill will now take the incoming observation and action spaces and pass it to the sub-skills as a shared environment observation and action taking. The sub-skills will then return their own observations and actions, which will be passed back to the coordinated skill. The coordinated skill will then return the combined observations and actions to the agent.

python
# ####################################################################################################
# Define the Coordinated Coach
# ####################################################################################################
class CoordinatedCoach(Coach):
    def __init__(self):
        self.counter = 0

    def compute_reward(self, transformed_ sensors, action, sim_reward):
        """
        Computes the reward for the given transformed observation and action
        :param transformed_ sensors: The transformed observation
        :param action: The actions dict
        :param sim_reward: The reward from the simulation
        :return: The reward, as a dictionary, with each key the sub-skill name and the value the reward
        """
        self.counter += 1
        return 1

    def compute_success_criteria(self, transformed_ sensors, action):
        # keep the episodes short to make testing quicker
        return self.counter > 100

    def compute_termination(self, transformed_ sensors, action):
        # keep the episodes short to make testing quicker
        return self.counter > 150

    def transform_action(self, composabl_ sensors, action):
        return action


# ####################################################################################################
# Construct your Agent
# ####################################################################################################
s1 = Skill("skill1", IncrementTeacher)
s2 = Skill("skill2", IncrementTeacher)

a = Agent()
a.add_coordinated_skill(CoordinatedSkill(
  "my-coordinated-skill",
  CoordinatedCoach,
  [s1, s2]
))