SageMaker Pipeline

Amazon SageMaker Pipelines is a purpose-built continuous integration and continuous delivery (CI/CD) service for machine learning workflows. It enables data scientists and ML engineers to define, automate, and manage end-to-end machine learning workflows as a series of interconnected steps including data processing, model training, evaluation, and deployment.

The SageMaker Pipeline Action Agent allows XMPro Data Streams to integrate with Amazon SageMaker Pipelines by triggering pipeline executions and monitoring their status. The Agent supports two authentication methods: direct AWS access keys or Azure AD federation via AWS STS AssumeRoleWithWebIdentity, enabling secure cross-cloud integration scenarios. This allows organizations to incorporate ML inference workflows into their real-time operational data streams, triggering model retraining, batch predictions, or data processing pipelines

Details for examples and its configuration can be found in the How to Use section.

Pre-requisites

The following are required to use this Agent depending on the Authentication method used:

General Requirements:

• An active AWS account with access to Amazon SageMaker

• At least one SageMaker Pipeline defined in the target AWS region

• IAM permissions for SageMaker Pipeline operations. The following IAM policy provides the minimum required permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "SageMakerPipelineOperations",
      "Effect": "Allow",
      "Action": [
        "sagemaker:ListPipelines",
        "sagemaker:DescribePipeline",
        "sagemaker:ListPipelineVersions",
        "sagemaker:StartPipelineExecution",
        "sagemaker:DescribePipelineExecution"
      ],
      "Resource": "*"
    }
  ]
}

Note: Additional permissions may be required depending on what the pipeline does internally, such as access to S3 buckets, Lambda functions, or other AWS services.

When using AWS Access Key Authentication:

• Access Key and Secret Access Key for the IAM user

• The IAM user must have the SageMaker Pipeline permissions listed above attached directly or via an IAM group/role

When using Azure AD Federation Authentication:

This method uses Azure AD to obtain temporary AWS credentials via STS AssumeRoleWithWebIdentity, enabling secure cross-cloud authentication without storing long-lived AWS credentials.

Azure AD Requirements:

• An Azure AD Tenant Id

• An App Registration in Azure AD with:

  • A Client ID (Application ID)

  • A Client Secret

  • A configured scope for token requests. Use api://{clientId}/.default if your App Registration has an Application ID URI configured, or {clientId}/.default if it does not.

AWS Requirements:

• An OIDC Identity Provider configured in IAM to trust Azure AD

• An IAM Role configured to trust the Azure AD identity provider

• The IAM Role must have the SageMaker Pipeline permissions listed above

• The IAM Role trust policy must allow sts:AssumeRoleWithWebIdentity from the Azure AD issuer

Setting up the OIDC Identity Provider:

1. In AWS IAM, create a new Identity Provider with:

   • Provider type: OpenID Connect

   • Provider URL: https://sts.windows.net/<AZURE_TENANT_ID>/

   • Audience: Your Azure AD Client ID

Example IAM Role Trust Policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/sts.windows.net/<AZURE_TENANT_ID>/"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "sts.windows.net/<AZURE_TENANT_ID>/:aud": "<AZURE_CLIENT_ID>"
        }
      }
    }
  ]
}

For detailed instructions on setting up Azure AD federation with AWS, see AssumeRoleWithWebIdentity docs.

Current Version

• ​Request the SageMaker Pipeline Action Agent v2.0 (Last Update: 21 Apr 2026)

Release Notes

Version	Date	Description
2.0	07 May 2026	Support Event-Level Streaming (when using XMPro Platform v4.6+) for reduced memory usage and earlier downstream processing.
1.0	16 Apr 2026	Initial Release.