Configure Azure Storage Blob modular inputs for the Splunk Add-on for Microsoft Cloud Services
Before you enable inputs, complete the previous steps in the configuration process:
- Configure a Storage Account in Microsoft Cloud Service
- Connect to your Azure Storage account with the Splunk Add-on for Microsoft Cloud Services
Configure your inputs on the Splunk platform instance responsible for collecting data for this add-on, usually a heavy forwarder. You can configure inputs using Splunk Web as a best practice, or you can use the configuration files.
Versions 5.0.0 and higher of the Splunk Add-on for Microsoft Cloud Services contain changes to the checkpoint mechanism for the Storage Blob input. See the upgrade steps in this manual for more information.
The Azure Storage Blob modular input for Splunk Add-on for Microsoft Cloud Services does not support the ingestion of gzip files. Only plaintext files are supported.
Since the format of the data in Azure Storage Blob channel varies, use source types to make the event data more effective. See Overview of Event Processing for more information.
Introduced Horizontal Scaling in the Splunk Add-on for Microsoft Cloud Services Version 5.0.0. Horizontal scaling provides functionality to collect data from the same Storage Container using multiple inputs in parallel to reduce data ingestion delays.
Analyze the user-case before opting for Horizontal Scaling. Horizontal Scaling is designed for containers containing a huge number of files. However, if there is a small number of large files in the container, then scaling up might be limited by the indexing rate of the environment.
Horizontal Scaling is not directly proportional to ingestion rate. For instance, if 1 input is capable of collecting the entire container's data in 1 hour, then creating 2 inputs will not necessarily make the net collection time to 30 mins, and 3 inputs will not necessarily bring it down to 20 minutes.
Scale the inputs incrementally and monitor the ingestion rate before scaling up the environment again. If the number of inputs starts filling up the indexing queue of Splunk, then the health of the environment might be adversely affected.
The horizontal scaling should only be used after the file based checkpoint for the input has been successfully migrated to KV Store. Otherwise, it may lead to data duplication.
- All input should use the same index.
- All Splunk instances should use the centralized KVStore. In Victoria stack, there is a centralized KVStore so this feature can be used there. If Splunk instances use a different KVStore, there will be data duplication. If one Heavy Forwarder uses its own KVStore and another Heavy Forwarder uses a different KVStore, and both Heavy Forwarders have their inputs collecting data from the same Storage Container, then there will be data duplication.
- There is a small chance of data duplication, up to 5%.
Configure inputs using Splunk Web
Configure your inputs using Splunk Web on the Splunk platform instance responsible for collecting data for this add-on, usually a heavy forwarder.
- In the Splunk Add-on for Microsoft Cloud Services, select Inputs.
- Select Create New Input and select Azure Storage Blob.
- Enter the Name, Storage Account, Container Name, Blob list, Interval, Index and Sourcetype using the Inputs parameters table.
Configure inputs using Configuration File
- Create a file called inputs.conf under
- Configure the Azure Storage Blob input with the following stanza:
[mscs_storage_blob://<input_name>]] account = <value> application_insights = <value> blob_mode = <value> collection_interval = <value> container_name = <value> prefix = <value> blob_list = <value> exclude_blob_list = <value> decoding = <value> guids = <value> index = <value> log_type = <value> sourcetype = <value> disabled = <value> read_timeout = <value>
Each attribute in the following table corresponds to a field in Splunk Web.
|Attribute||Corresponding field in Splunk Web||Description|
||Name||Enter a friendly name of your inputs. Do not use whitespaces in your input names. Name cannot contain any whitespace.|
||Azure Storage Account||Select the storage account name you configured. Name cannot contain any whitespace.|
||Application Insights Check||Indicates whether the Azure storage blob ingests data from |
||Container Name||Enter the container name under the storage account. You can only add one container name for each input.|
||Prefix||Specify the prefix string for the blobs. Input will only collect the data from the blobs whose names begin with the specified prefix.
For instance, if user wants to collect data from
||Blob List||Enter the Blob name, wildcard, or regular expression for the data you want to collect. You can add multiple blob names separated by commas. If you leave this field empty, this add-on collects all the blob lists under the Container Name you configured.
If you want to collect data from a specific blob list, enter the name of the blob list, such as
||NULL||Select the following blob:|
||Interval||The number of seconds to wait before the Splunk platform runs the command again. The default is 3600 seconds.|
||Decoding||Specify the character set of the file, such as UTF-8 or UTF-32. If you leave this field blank, this add-on uses the default character set of the file.|
||Excluded Blob List||Optional. Enter the Blob name or regular expressions for the data you do not want to collect. You can add multiple blob names separated by commas. The syntax of the Excluded Blob List is the same as Blob List.
||GUIDs||Indicates the GUID identifier used for application insights data with this format: <application insights resource name>_<instrumentation key>. Required if |
||Index||The index in which to store Azure Storage Blob data.|
||Log type||Filters the results to return only blobs whose names begin with the specified log type. Use the following application Insights blob format: |
||Sourcetype||The default is
||Read Timeout||Specify the maximum amount of time (in seconds) to wait for a response from the Azure Storage service when reading data. The default value is 60 seconds.|
If a file matches the syntax both in '''Blob List''' and '''Exclude Blob List''', '''Exclude Blob List''' takes priority. For example, if there is a blob list name '''blob1''', and it matches the syntax you set in '''Blob List''' and '''Exclude Blob List''', this add-on will exclude this list because '''Exclude Blob List''' is in higher priority.
Configure ingestion mode
Configure ingestion mode by selecting a blob mode that aligns with the blob type that you selected while creating the blob in your Azure storage account.
- On your Splunk platform deployment, navigate to the
- Open inputs.conf file with a text editor.
- Navigate to the stanza of the blob storage input that you created.
- Change the
random, based on the following table:
blob_type\ingestion_mode Incremental Full append blob_mode is irrelevant.
You always receive incremental changes to your blob.
N/A block or page
If you use a block_blob to append data to the blob and only want the incremental changes, set
blob_mode = append.
blob_mode = random.
After a blob is complete or closed, the contents are ingested to the Splunk platform.
- Save your changes.
Introduced the "Allow Storage Blob Deletion" option in the Configuration -> Advanced tab in the Version 5.0.0 of the Splunk Add-on for Microsoft Cloud Services. This option allows the deletion of checkpoint files from the Splunk environment after migration to KVStore. Enable this option after all the Storage Blob inputs are migrated to KVStore successfully and the System is stable.
Configure the Azure Storage Table modular Input for the Splunk Add-on for Microsoft Cloud Services
Configure Azure Virtual Machine metrics modular Input for Splunk Add-on for Microsoft Cloud Services
This documentation applies to the following versions of Splunk® Supported Add-ons: released