Configuration

This section explains each of the properties in the configuration blade.

General

PropertyDescription

Collection

This associates the Agent to a specific Collection (default to that of the current Data Stream).

Refer to Collections and Stream Hosts to understand more about Collections.

Upload schedule

PropertyDescription

File upload interval (seconds)

A timespan to specify the frequency at which the Agent uploads data to Azure Data Lake. A file upload interval of at least one minute is recommended to ensure that the Agent neither fails to write the events nor overwrites existing events to the Data Lake Storage.

Max Retries

If the upload fails, the number of times to retry before abandoning the upload attempt. Uploads will resume on the next interval.

Authentication

PropertyDescription

Use Variables

Tick to select saved Variables for the settings in this section; or enter the text manually.

Subscription Id

The Id of the Azure Subscription.

Tenant Id

The Id of the Azure Tenant to which the App Registration belongs.

Client Id

The Id of the Azure App Registration to authenticate against.

Client Secret

The secret of the Azure App Registration to authenticate against.

Data Lake

PropertyDescription

Resource Group

The name of the Azure Resource group in which the Azure Storage Account instance is located. Resource Groups will be searched for and listed using the Authentication settings, but the name can be entered manually if necessary.

Storage Account

The name of the Azure Storage Account instance to use. Storage Account instances will be searched for - filtered by Premium tier - and listed using the Authentication settings, but the name can be entered manually if necessary.

Blob Container

The name of the Blob Container within the Azure Storage Account in which to store data.

Create a new Storage Account

This link opens a sub-page that allows you to create a new Premium-tier Azure Storage Account - if the App Registration has been given management access.

Refer Azure resource creation on how to enable this link.

PropertyDescription

Resource Group

The name of the Resource Group where the new Storage Account instance is created. If it does not exist, it will be created.

Resource Location

The location the new Storage Account instance will be hosted in.

Name

The name of the new Storage Account instance.

If the Storage Account instance already exists, no changes will be made.

Once values are selected, click the Create Storage Account button to create the Storage Account. The operation may take several minutes. After it is finished, click Apply to set the details for the new Storage Account on the main page.

Create a new Blob Container

This link opens a sub-page that allows you to create a new Container within the Azure Storage Account instance.

Refer Azure resource creation on how to enable this link.

PropertyDescription

Container Name

The name of the new Container.

If the container already exists, no changes will be made.

Once values are selected, click the Create Container button to create the Container. The operation may take several minutes. After it is finished, click Apply to set the details for the new Container on the main page.

File Details

PropertyDescription

Columns to output

The list of columns to output to Azure Data Lake. If left blank, all columns will be outputted.

File Format

The format of the destination file created in Azure Data Lake. Supported formats are:

  • JSON (.json)

  • CSV (.csv)

  • Parquet (.parquet)

Date Format

The pattern to use when formatting the date for use in the destination file name. Available options are:

  • yyyy-MM-dd

  • yyyy/MM/dd

  • yyyyMMdd

Time Format

The pattern to use when formatting the time for use in the destination file name.

  • HH-mm-ss

  • HHmmss

Custom File Name

The path and file name to use for the destination file in Azure Data Lake.

If left blank, the default path will be used: {stream-agent-id}\{date}\{time}.{ext}

The file name should not include the file extension.

The file name must include the {date} and {time} tags which will be substituted with the data and time values, formatted according to the above configured date and time formats.

Additional tags can be included using the same format. (e.g {tag-name} ) If additional tags are detected in the path, they will be added to the File Name Tag Map grid below where each tag can be mapped to a value from the input data.

File Name Tag Map

PropertyDescription

Tag

The tag name

Map

The column value that is mapped to the tag. All instances of the tag will be replaced by the value of the mapped column.

Before configuring the Action Agent, please ensure that its input endpoint is connected to a parent Agent which will be sending data to it.

Endpoints

NameDescription

Input

This endpoint is used to receive events from the parent Agent.

Output

Events received from the parent Agent are made available to this endpoint with unchanged attributes and data.

Last updated