21 KiB

Raw Blame History

layout	title	nav_order
default	Workflow tutorial	20

Workflow tutorial

This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated GitHub issue.
{: .warning}

You can automate the setup of common use cases, such as conversational chat, using a Chain-of-Thought (CoT) agent. An agent orchestrates and runs ML models and tools. A tool performs a set of specific tasks. This page presents a complete example of setting up a CoT agent. For more information about agents and tools, see Agents and tools

The setup requires the following sequence of API requests, with provisioned resources used in subsequent requests. The following list provides an overview of the steps required for this workflow. The step names correspond to the names in the template:

Deploy a model on the cluster
- create_connector_1: Create a connector to an externally hosted model.
- register_model_2: Register a model using the connector that you created.
- deploy_model_3: Deploy the model.
Use the deployed model for inference
- Set up several tools that perform specific tasks:
  - math_tool: Set up a math tool.
  - ml_model_tool: Set up a machine learning (ML) model tool.
- Set up one or more agents that use some combination of the tools:
  - sub_agent: Create an agent that uses the math tool.
- Set up tools representing these agents:
  - agent_tool: Wrap the sub_agent so that you can use it as a tool.
- root_agent: Set up a root agent that may delegate the task to either a tool or another agent.

The following sections describe the steps in detail. For the complete workflow template, see Complete YAML workflow template.

Workflow graph

The workflow described in the previous section is organized into a template. Note that you can order the steps in several ways. In the example template, the ml_model_tool step is specified right before the root_agent step, but you can specify it at any point after the deploy_model_3 step and before the root_agent step. The following diagram shows the directed acyclic graph (DAG) that OpenSearch creates for all of the steps in the order specified in the template.

{:style="width: 100%; max-width: 600px;" class="img-centered"}

1. Deploy a model on the cluster

To deploy a model on the cluster, you need to create a connector to the model, register the model, and deploy the model.

create_connector_1

The first step in the workflow is to create a connector to an externally hosted model (in the following example, this step is called create_connector_1). The content of the user_inputs field exactly matches the ML Commons Create Connector API:

nodes:
- id: create_connector_1
  type: create_connector
  user_inputs:
    name: OpenAI Chat Connector
    description: The connector to public OpenAI model service for GPT 3.5
    version: '1'
    protocol: http
    parameters:
      endpoint: api.openai.com
      model: gpt-3.5-turbo
    credential:
      openAI_key: '12345'
    actions:
    - action_type: predict
      method: POST
      url: https://${parameters.endpoint}/v1/chat/completions

When you create a connector, OpenSearch returns a connector_id, which you need in order to register the model.

register_model_2

When registering a model, the previous_node_inputs field tells OpenSearch to obtain the required connector_id from the output of the create_connector_1 step. Other inputs required by the Register Model API are included in the user_inputs field:

- id: register_model_2
  type: register_remote_model
  previous_node_inputs:
    create_connector_1: connector_id
  user_inputs:
    name: openAI-gpt-3.5-turbo
    function_name: remote
    description: test model

The output of this step is a model_id. You must then deploy the registered model to the cluster.

deploy_model_3

The Deploy Model API requires the model_id from the previous step, as specified in the previous_node_inputs field:

- id: deploy_model_3
  type: deploy_model
  # This step needs the model_id produced as an output of the previous step
  previous_node_inputs:
    register_model_2: model_id

When using the Deploy Model API directly, a task ID is returned, requiring use of the Tasks API to determine when the deployment is complete. The automated workflow eliminates the manual status check and returns the final model_id directly.

Ordering steps

To order these steps in a sequence, you must connect them by an edge in the graph. When a previous_node_input field is present in a step, OpenSearch automatically creates a node with source and dest fields for this step. The output of the source is required as input for the dest. For example, the register_model_2 step requires the connector_id from the create_connector_1 step. Similarly, the deploy_model_3 step requires the model_id from the register_model_2 step. Thus, OpenSearch creates the first two edges in the graph as follows in order to match the output with the required input and raise errors if the required input is missing:

edges:
- source: create_connector_1
  dest: register_model_2
- source: register_model_2
  dest: deploy_model_3

If you define previous_node_inputs, then defining edges is optional. {: .note}

2. Use the deployed model for inference

A CoT agent can use the deployed model in a tool. This step doesn’t strictly correspond to an API but represents a component of the body required by the Register Agent API. This simplifies the register request and allows reuse of the same tool in multiple agents. For more information about agents and tools, see Agents and tools.

math_tool

You can configure other tools to be used by the CoT agent. For example, you can configure a math tool as follows. This tool does not depend on any previous steps:

- id: math_tool
  type: create_tool
  user_inputs:
    name: MathTool
    type: MathTool
    description: A general tool to calculate any math problem. The action input
      must be a valid math expression, like 2+3
    parameters:
      max_iteration: 5

sub_agent

To use the math tool in the agent configuration, specify it as one of the tools in the previous_node_inputs field of the agent. You can add other tools to previous_node_inputs as necessary. The agent also needs a large language model (LLM) in order to reason with the tools. The LLM is defined by the llm.model_id field. This example assumes that the model_id from the deploy_model_3 step will be used. However, if another model is already deployed, the model_id of that previously deployed model could be included in the user_inputs field instead:

- id: sub_agent
  type: register_agent
  previous_node_inputs:
    # When llm.model_id is not present this can be used as a fallback value
    deploy-model-3: model_id
    math_tool: tools
  user_inputs:
    name: Sub Agent
    type: conversational
    description: this is a test agent
    parameters:
      hello: world
    llm.parameters:
      max_iteration: '5'
      stop_when_no_tool_found: 'true'
    memory:
      type: conversation_index
    app_type: chatbot

OpenSearch will automatically create the following edges so that the agent can retrieve the fields from the previous node:

- source: math_tool
  dest: sub_agent
- source: deploy_model_3
  dest: sub_agent

agent_tool

You can use an agent as a tool for another agent. Registering an agent produces an agent_id in the output. The following step defines a tool that uses the agent_id from the previous step:

- id: agent_tool
  type: create_tool
  previous_node_inputs:
    sub_agent: agent_id
  user_inputs:
    name: AgentTool
    type: AgentTool
    description: Agent Tool
    parameters:
      max_iteration: 5

OpenSearch automatically creates an edge connection because this step specifies the previous_node_input:

- source: sub_agent
  dest: agent_tool

ml_model_tool

A tool may reference an ML model. This example gets the required model_id from the model deployed in a previous step:

- id: ml_model_tool
  type: create_tool
  previous_node_inputs:
    deploy-model-3: model_id
  user_inputs:
    name: MLModelTool
    type: MLModelTool
    alias: language_model_tool
    description: A general tool to answer any question.
    parameters:
      prompt: Answer the question as best you can.
      response_filter: choices[0].message.content

OpenSearch automatically creates an edge in order to use the previous_node_input:

- source: deploy-model-3
  dest: ml_model_tool

root_agent

A conversational chat application will communicate with a single root agent that includes the ML model tool and the agent tool in its tools field. It will also obtain the llm.model_id from the deployed model. Some agents require tools to be in a specific order, which can be enforced by including the tools_order field in the user inputs:

- id: root_agent
  type: register_agent
  previous_node_inputs:
    deploy-model-3: model_id
    ml_model_tool: tools
    agent_tool: tools
  user_inputs:
    name: DEMO-Test_Agent_For_CoT
    type: conversational
    description: this is a test agent
    parameters:
      prompt: Answer the question as best you can.
    llm.parameters:
      max_iteration: '5'
      stop_when_no_tool_found: 'true'
    tools_order: ['agent_tool', 'ml_model_tool']
    memory:
      type: conversation_index
    app_type: chatbot

OpenSearch automatically creates edges for the previous_node_input sources:

- source: deploy-model-3
  dest: root_agent
- source: ml_model_tool
  dest: root_agent
- source: agent_tool
  dest: root_agent

For the complete DAG that OpenSearch creates for this workflow, see the workflow graph.

Complete YAML workflow template

The following is the final template including all of the provision workflow steps in YAML format:

YAML template

{: .text-delta}

# This template demonstrates provisioning the resources for a 
# Chain-of-Thought chat bot
name: tool-register-agent
description: test case
use_case: REGISTER_AGENT
version:
  template: 1.0.0
  compatibility:
  - 2.12.0
  - 3.0.0
workflows:
  # This workflow defines the actions to be taken when the Provision Workflow API is used
  provision:
    nodes:
    # The first three nodes create a connector to a remote model, registers and deploy that model
    - id: create_connector_1
      type: create_connector
      user_inputs:
        name: OpenAI Chat Connector
        description: The connector to public OpenAI model service for GPT 3.5
        version: '1'
        protocol: http
        parameters:
          endpoint: api.openai.com
          model: gpt-3.5-turbo
        credential:
          openAI_key: '12345'
        actions:
        - action_type: predict
          method: POST
          url: https://${parameters.endpoint}/v1/chat/completions
    - id: register_model_2
      type: register_remote_model
      previous_node_inputs:
        create_connector_1: connector_id
      user_inputs:
        # deploy: true could be added here instead of the deploy step below
        name: openAI-gpt-3.5-turbo
        description: test model
    - id: deploy_model_3
      type: deploy_model
      previous_node_inputs:
        register_model_2: model_id
    # For example purposes, the model_id obtained as the output of the deploy_model_3 step will be used
    # for several below steps.  However, any other deployed model_id can be used for those steps.
    # This is one example tool from the Agent Framework.
    - id: math_tool
      type: create_tool
      user_inputs:
        name: MathTool
        type: MathTool
        description: A general tool to calculate any math problem. The action input
          must be a valid math expression, like 2+3
        parameters:
          max_iteration: 5
    # This simple agent only has one tool, but could be configured with many tools
    - id: sub_agent
      type: register_agent
      previous_node_inputs:
        deploy-model-3: model_id
        math_tool: tools
      user_inputs:
        name: Sub Agent
        type: conversational
        description: this is a test agent
        parameters:
          hello: world
        llm.parameters:
          max_iteration: '5'
          stop_when_no_tool_found: 'true'
        memory:
          type: conversation_index
        app_type: chatbot
    # An agent can be used itself as a tool in a nested relationship
    - id: agent_tool
      type: create_tool
      previous_node_inputs:
        sub_agent: agent_id
      user_inputs:
        name: AgentTool
        type: AgentTool
        description: Agent Tool
        parameters:
          max_iteration: 5
    # An ML Model can be used as a tool
    - id: ml_model_tool
      type: create_tool
      previous_node_inputs:
        deploy-model-3: model_id
      user_inputs:
        name: MLModelTool
        type: MLModelTool
        alias: language_model_tool
        description: A general tool to answer any question.
        parameters:
          prompt: Answer the question as best you can.
          response_filter: choices[0].message.content
    # This final agent will be the interface for the CoT chat user
    # Using a flow agent type tools_order matters
    - id: root_agent
      type: register_agent
      previous_node_inputs:
        deploy-model-3: model_id
        ml_model_tool: tools
        agent_tool: tools
      user_inputs:
        name: DEMO-Test_Agent
        type: flow
        description: this is a test agent
        parameters:
          prompt: Answer the question as best you can.
        llm.parameters:
          max_iteration: '5'
          stop_when_no_tool_found: 'true'
        tools_order: ['agent_tool', 'ml_model_tool']
        memory:
          type: conversation_index
        app_type: chatbot
    # These edges are all automatically created with previous_node_input
    edges:
    - source: create_connector_1
      dest: register_model_2
    - source: register_model_2
      dest: deploy_model_3
    - source: math_tool
      dest: sub_agent
    - source: deploy_model_3
      dest: sub_agent
    - source: sub_agent
      dest: agent_tool
    - source: deploy-model-3
      dest: ml_model_tool
    - source: deploy-model-3
      dest: root_agent
    - source: ml_model_tool
      dest: root_agent
    - source: agent_tool
      dest: root_agent

Complete JSON workflow template

The following is the same template in JSON format:

JSON template