Comment on page

Configuration

This section explains each of the properties in the configuration blade. 
General
Property
Description
Collection
This associates the Agent to a specific Collection (default to that of the current Data Stream).
Refer to Collections and Stream Hosts to understand more about Collections.
Upload schedule
Property
Description
File upload interval (seconds)
A timespan to specify the frequency at which the Agent uploads data to Azure Data Lake.

A file upload interval of at least one minute is recommended to ensure that the Agent neither fails to write the events nor overwrites existing events to the Data Lake Storage.
Max Retries
If the upload fails, the number of times to retry before abandoning the upload attempt. Uploads will resume on the next interval.
Authentication
Property
Description
Use Variables
Tick to select saved Variables for the settings in this section; or enter the text manually. 
Subscription Id
The Id of the Azure Subscription.
Tenant Id
The Id of the Azure Tenant to which the App Registration belongs.
Client Id
The Id of the Azure App Registration to authenticate against.
Client Secret
The secret of the Azure App Registration to authenticate against.
Data Lake
Property
Description
Resource Group
The name of the Azure Resource group in which the Azure Storage Account instance is located.

Resource Groups will be searched for and listed using the Authentication settings, but the name can be entered manually if necessary.
Storage Account
The name of the Azure Storage Account instance to use.

Storage Account instances will be searched for - filtered by Premium tier - and listed using the Authentication settings, but the name can be entered manually if necessary.
Blob Container
The name of the Blob Container within the Azure Storage Account in which to store data.
Create a new Storage Account
This link opens a sub-page that allows you to create a new Premium-tier Azure Storage Account - if the App Registration has been given management access. 
Refer Azure resource creation on how to enable this link.
Property
Description
Resource Group
The name of the Resource Group where the new Storage Account instance is created. If it does not exist, it will be created.
Resource Location
The location the new Storage Account instance will be hosted in.
Name
The name of the new Storage Account instance.
If the Storage Account instance already exists, no changes will be made.
Once values are selected, click the Create Storage Account button to create the Storage Account. The operation may take several minutes. After it is finished, click Apply to set the details for the new Storage Account on the main page.
Create a new Blob Container
This link opens a sub-page that allows you to create a new Container within the Azure Storage Account instance.
Refer Azure resource creation on how to enable this link.
Property
Description
Container Name
The name of the new Container.
If the container already exists, no changes will be made.
Once values are selected, click the Create Container button to create the Container. The operation may take several minutes. After it is finished, click Apply to set the details for the new Container on the main page.
File Details
Property
Description
Columns to output
The list of columns to output to Azure Data Lake.
If left blank, all columns will be outputted.
File Format
The format of the destination file created in Azure Data Lake. 
Supported formats are: 
JSON (.json)
CSV (.csv)
Parquet (.parquet)
Date Format
The pattern to use when formatting the date for use in the destination file name.
Available options are:
yyyy-MM-dd
yyyy/MM/dd
yyyyMMdd

Time Format
The pattern to use when formatting the time for use in the destination file name.
HH-mm-ss
HHmmss
Custom File Name
The path and file name to use for the destination file in Azure Data Lake.
If left blank, the default path will be used:
{stream-agent-id}\{date}\{time}.{ext}
​
The file name should not include the file extension.
​
The file name must include the {date} and {time} tags which will be substituted with the data and time values, formatted according to the above configured date and time formats.
​
Additional tags can be included using the same  format. (e.g  {tag-name} )
If additional tags are detected in the path, they will be added to the File Name Tag Map grid below where each tag can be mapped to a value from the input data.

File Name Tag Map
Property
Description
Tag
The tag name
Map
The column value that is mapped to the tag. All instances of the tag will be replaced by the value of the mapped column.
Before configuring the Action Agent, please ensure that its input endpoint is connected to a parent Agent which will be sending data to it.
Endpoints
Name
Description
Input
This endpoint is used to receive events from the parent Agent.
Output
Events received from the parent Agent are made available to this endpoint with unchanged attributes and data.

Property	Description
Collection	This associates the Agent to a specific Collection (default to that of the current Data Stream).

Property	Description
File upload interval (seconds)	A timespan to specify the frequency at which the Agent uploads data to Azure Data Lake. A file upload interval of at least one minute is recommended to ensure that the Agent neither fails to write the events nor overwrites existing events to the Data Lake Storage.
Max Retries	If the upload fails, the number of times to retry before abandoning the upload attempt. Uploads will resume on the next interval.

Property	Description
Use Variables	Tick to select saved Variables for the settings in this section; or enter the text manually.
Subscription Id	The Id of the Azure Subscription.
Tenant Id	The Id of the Azure Tenant to which the App Registration belongs.
Client Id	The Id of the Azure App Registration to authenticate against.
Client Secret	The secret of the Azure App Registration to authenticate against.

Property	Description
Resource Group	The name of the Azure Resource group in which the Azure Storage Account instance is located. Resource Groups will be searched for and listed using the Authentication settings, but the name can be entered manually if necessary.
Storage Account	The name of the Azure Storage Account instance to use. Storage Account instances will be searched for - filtered by Premium tier - and listed using the Authentication settings, but the name can be entered manually if necessary.
Blob Container	The name of the Blob Container within the Azure Storage Account in which to store data.

Property	Description
Resource Group	The name of the Resource Group where the new Storage Account instance is created. If it does not exist, it will be created.
Resource Location	The location the new Storage Account instance will be hosted in.
Name	The name of the new Storage Account instance.

Property	Description
Container Name	The name of the new Container.

Property	Description
Columns to output	The list of columns to output to Azure Data Lake. If left blank, all columns will be outputted.
File Format	The format of the destination file created in Azure Data Lake. Supported formats are: JSON (.json) CSV (.csv) Parquet (.parquet)
Date Format	The pattern to use when formatting the date for use in the destination file name. Available options are: yyyy-MM-dd yyyy/MM/dd yyyyMMdd
Time Format	The pattern to use when formatting the time for use in the destination file name. HH-mm-ss HHmmss
Custom File Name	The path and file name to use for the destination file in Azure Data Lake. If left blank, the default path will be used: {stream-agent-id}\{date}\{time}.{ext} The file name should not include the file extension. The file name must include the {date} and {time} tags which will be substituted with the data and time values, formatted according to the above configured date and time formats. Additional tags can be included using the same format. (e.g {tag-name} ) If additional tags are detected in the path, they will be added to the File Name Tag Map grid below where each tag can be mapped to a value from the input data.

Property	Description
Tag	The tag name
Map	The column value that is mapped to the tag. All instances of the tag will be replaced by the value of the mapped column.

Name	Description
Input	This endpoint is used to receive events from the parent Agent.
Output	Events received from the parent Agent are made available to this endpoint with unchanged attributes and data.

How to use? - Previous

Example

Next - How to use?

Release Notes

Last modified 5mo ago