Configure Azure Storage Blob Modular Inputs for Splunk Add-on for Microsoft Cloud Services

Before you enable inputs, complete the previous steps in the configuration process:

• Configure a Storage Account in Microsoft Cloud Service
• Connect to your Azure Storage account with the Splunk Add-on for Microsoft Cloud Services

Configure your inputs on the Splunk platform instance responsible for collecting data for this add-on, usually a heavy forwarder. You can configure inputs using Splunk Web (recommended) or using the configuration files.

The Azure Storage Blob modular input for Splunk Add-on for Microsoft Cloud Services does not support the ingestion of gzip files. Only plaintext files are supported.

Since the format of the data in Azure Storage Blob channel varies (including text and binary data), the Splunk best practice is to leverage the options for sourcetypes to make the event data more effective. See Overview of Event Processing for more information.

Configure inputs using Splunk Web

Configure your inputs using Splunk Web on the Splunk platform instance responsible for collecting data for this add-on, usually a heavy forwarder.

1. In the Splunk Add-on for Microsoft Cloud Services, click Inputs.
2. Click Create New Input and select Azure Storage Blob.
3. Enter the Name, Storage Account, Container Name, Blob list, Interval, Index and Sourcetype using the inputs parameters table below.

Configure inputs using Configuration File

1. Create a file called inputs.conf under $SPLUNK_HOME/etc/apps/Splunk_TA_microsoft-cloudservices/local.
2. Configure Azure Storage Blob input with the following stanza:

   [mscs_storage_blob://<input_name>]]
   account = <value>
   application_insights = <value>
   blob_mode = <value>
   collection_interval = <value>
   container_name = <value>
   blob_list = <value>  
   exclude_blob_list = <value>
   decoding = <value>
   guids = <value>
   index = <value>
   log_type = <value>
   sourcetype = <value>
   disabled = <value>
   

Inputs Parameters

Each attribute in the following table corresponds to a field in Splunk Web.

Attribute	Corresponding field in Splunk Web	Description
mscs:storage:blob://<input_name>	Name	Enter a friendly name of your inputs.
account	Azure Storage Account	Select the storage account name you configured.
application_insights	Application Insights Check	Indicates whether the Azure storage blob ingests data from application_insights. If yes, value is 1, otherwise 0. Log type and GUIDs are required if enabled.
container_name	Container Name	Enter the container name under the storage account. You can only add one container name for each input.
blob_list	Blob List	Enter the Blob name or regex which you want to collect the data from. You can add multiple blob names separated by commas. If you leave this field empty, this add-on will collect all the blob lists under the Container Name you just configured. You can enter the specific blob list name, use wildcard or use regex expression in this field. If you want to collect data from a specific blob list, just enter the name of the blob list here, such as blob_name. You can use wildcard in this field, e.g. blob*, this add-on will collect data from the blob lists of the names starting from blob. And you can also use comma to separate multiple blob names, e.g. blob, name*. If you want to use regex, the syntax is JSON format: {"regex syntax":3}, 3 stands for regex file. If you want to enter the blob list which has both wildcard and regex, you can enter both separated by commas, for example, {"regex syntax" :3, blob* :2}, 2 stands for wildcard list. If you want to enter the blob list using all of the three expressions, you can use the syntax like {"regex syntax" :3, blob* :2, blob :1}, 1 stands for using a specific blob list name. Constraints: The blob name must be at least one character long but cannot be more than 1,024 characters. Blob names are case-sensitive. Reserved URL characters must be properly escaped. The number of path segments comprising the blob name cannot exceed 254.
blob_mode	NULL	Select blob mode from the following values: append: Makes the blob storage input retrieve only the incremental changes. random: Makes the blob storage input retrieve the entire blob again on an update. Default is random.
collection_interval	Interval	The number of seconds to wait before the Splunk platform runs the command again. The default is 3600 seconds.
decoding	Decoding	Specify the character set of the file, such as UTF-8 or UTF-32. If you leave this field blank, this add-on will use the default character set of the file.
exclude_blob_list	Excluded Blob List	Optional. Enter the Blob name or regex that you do not want to collect the data from. You can add multiple blob names separated by commas. The syntax of the Excluded Blob List is the same as Blob List. For example, if you do not want to include blobs from 2020, and your blob name is /y=2020/m=10/d=05/h=08/m=00/blob1.txt, then your regex can be .*y=2020\/.*.
guids	GUIDs	Indicates the guid identifier used for application insights data with format: <application insights resource name>_<instrumentation key>. Entered as comma separated values. Required if application_insights is enabled. Enter individual GUIDs (comma separated values).
index	Index	The index in which to store Azure Storage Blob data.
log_type	Log type	Filters the results to return only blobs whose names begin with the specified log type. Application Insights blob format: <container_name>/<guid>/<arbitrary_log_type_value>/<yyyy-mm-dd>/<hh>/<blob_file>. Only one log type value per input. Required if application_insights is enabled.
sourcetype	Sourcetype	The default is mscs:storage:blob. To simplify field extraction, enter one of the following predefined sourcetypes: mscs:storage:blob:json, or mscs:storage:blob:xml.

If there is a file match the syntax both in Blob List and Exclude Blob List, Exclude Blob List is in higher priority. For example, if there is a blob list name blob1, and it matches the syntax you set in Blob List and Exclude Blob List, this add-on will exclude this list because Exclude Blob List is in higher priority.

Configure ingestion mode

Configure ingestion mode by selecting a blob mode that aligns with the blob type that you selected while creating the blob in your Azure storage account.

1. On your Splunk platform deployment, navigate to the $SPLUNK_HOME/etc/apps/Splunk_TA_microsoft-cloudservices/local directory.
2. Open inputs.conf file with a text editor.
3. Navigate to the stanza of the blob storage input that you created.
4. Change the blob_mode to append or random, based on the following table:
   

   blob_type\ingestion_mode	Incremental	Full
   append	blob_mode is irrelevant. You will always receive incremental changes to your blob	N/A
   block or page	Set blob_mode = append. If you are using a block_blob to serve a use case where you are appending data to the blob and only want the incremental changes	blob_mode = random. Once a blob is complete or closed, the contents will be ingested to splunk

5. Save your changes.