Troubleshooting Azure Activity Logs data in Data Manager

See the following sections for information on troubleshooting Azure Activity Logs data ingestion in Data Manager.

For troubleshooting issues that affect both Azure Active Directory and Azure Activity Logs, see the Troubleshoot Azure data ingestion in Data Manager topic in this manual.

Failed Events

The Azure Function performs a backup of events whenever it fails to send the data. These events get backed up as blobs in the Azure Storage account with the prefix splkactstr. Open the storage account on Azure Portal and navigate to Containers. Eventhub messages that could not be parsed get backed up in a blob with failed-to-parse in the name. Eventhub messaged that could not be sent to splunk due to some network error get backed up in a blob with failed-to-send in the name.

Enabling Diagnostics Settings Troubleshooting

Error	Tips
Not enough permissions to run script	Navigate to the relevant subscription in the Azure portal, and open the Access Control (IAM) page from the bar on the lefthand side. Under Role Assignments ensure that the user executing the script has an Owner role assigned for the subscription they would like to onboard.
Script takes a long time to execute	The PowerShell script to enable diagnostic settings may take more than a couple of minutes to run depending on the number of subscriptions to be onboarded. As long as the script prints outputs, it is executing as expected. If the script does not progress, it is safe to terminate it and try again. The script is idempotent in nature and will result in the same result if the same set of parameters are passed.
Error Message: The limit of 5 diagnostic settings was reached. To create new setting 'splunk-activity-logs-00000000-0000-0000-0000-000000000000', delete an existing one.	Azure only allows 5 diagnostic settings to be configured for each subscription's activity logs. If this error is seen, delete any unused diagnostic configuration and execute the script again.

Search for events and logs

Use the following searches to find events and logs. From the Splunk Cloud menu bar, click Apps > Search & Reporting.

If data ingestion is failing, but you see no errors in Data Manager, you can check for errors in the Azure logs by running the following in Splunk Web Search.

index=<user selected index> sourcetype="azure:monitor:activity"

Search for Azure events associated with a specific input ID.

index=<user selected index> datamanager_input_id=<input_id>