Troubleshoot Azure data ingestion in Data Manager

Troubleshooting tips include, but are not limited to, the following items that can assist throughout the onboarding process.

Prerequisite troubleshooting

You will get an error during onboarding if any of the following do not match what is configured during the prerequisites.

Message	Tips
Incorrect tenant ID	Verify the tenant id displayed in the Overview > Tenant information in the Azure portal.
Incorrect client ID	Verify the client ID of the app matches the ID that is registered in the Azure portal.
Incorrect client ID	Verify the client secret of the app matches the secret that is registered in the Azure portal.

Data ingestion troubleshooting

You will get an error during onboarding if any of the following do not match what is configured during the Prerequisites step of the Data Manager onboarding process.

Message	Tips
Invalid client permissions results in messages such as the following: 401, invalid_client, Invalid client secret is provided, the permission set () sent in the request does not include the expected permission.	Edit the input in Data Manger to provide valid credentials, and check again for incoming data. Alternatively, review the Prerequisites instructions again to add the correct permissions to the application and grant admin consent for all permissions in the Azure portal.
Invalid endpoint type leads to messages such as the following: 400 or Request is being redirected to XXX.	Select a different endpoint by editing the input in Data Manager, and check again for incoming data again.
Azure function throws Microsoft.Azure.EventHubs.ReceiverDisconnectedException	This exception is expected, and sometimes gets thrown by the platform when Event Hub partition ownership changes, especially when scaling up and down. If there are an excessive number of these exceptions within any time period, it is an indication of a bad partition processor machine or unstable network. For example, if processor A is processing events from a partition, and a processor B wants to process events from the same partition, processor A experiences a ReceiverDisconnectedException. There will be no data loss. A user might see some duplicate events.
Azure function throws Microsoft.Azure.EventHubs.Processor.LeaseLostException	This is an expected exception that sometimes gets thrown by the platform when Event Hub partition ownership changes.

ARM Template Deployment Troubleshooting

Error	Tips
Service principal does not exist	Review the prerequisite instructions to add the correct permissions to the application and grant admin consent for all permissions in the Azure portal.
Resource group or any other resource already exists	If the resource group or resource already exists, it may be possible the same input is being deployed again. Navigate to your Azure Portal, and verify that the resources exist. In the event a resource name collision occurs, a new Data Manager Input will need to be created and the existing deleted. Click Review Finish Setup and Monitor Data Input. On the Data Management page select the created data input. Select Data Input > Data Inpput Details > Delete. Follow the steps to delete any resources.
Not enough permissions to execute deployment command	Check and update the powershell execution policy using the Get-ExecutionPolicy and Set-ExecutionPolicy commands.
Not enough permissions to execute ARM template	The onboarding user must have the Owner role for the Azure subscription where the ARM template will be deployed, in order to create the data ingestion resources. If you do not have the subscription Owner role, and would like to perform the onboarding yourself, ask the subscription Owner to assign you the subscription Owner role.
Number of tags per resource is limited to 50	Check how many tags are applied to the resources in question in your Azure environment. Resource tags can be manually added in Azure to the resources created by Data Manager, this may not reflect in the Data Manager UI and can cause a resource tag limit error when trying to add new tags. The maximum number of tags per resource or resource group is limited to 50.

Error deploying ARM template: Required parameter WEBSITE_CONTENTSHARE is missing

When deploying the ARM template, you may receive the following error:

The deployment 'splunk-activity-logs-deploy-resources' failed with error(s). Showing 3 out of 3 error(s).
Status Message: Tenant ID, application ID, principal ID, and scope are not allowed to be updated. (Code:RoleAssignmentUpdateNotPermitted)
Status Message: Required parameter WEBSITE_CONTENTSHARE is missing. (Code: BadRequest)

WEBSITE_CONTENTSHARE is auto generated when the Azure Function is created. If an Azure Function already exists with the same name, it won't get created, and this error is thrown. Usually this is because there is a collision in the name of the Azure Function, possibly because one already exists that has the same Data Manager input id in the name.

Before trying to redeploy Azure resources using the ARM template, make sure to delete the old deployment and Resource Group for this Data Manager input, then run the deployment command. Or, create a new Data Manager input and use the new Data Manager input id.

Error deploying ARM template: "At least one resource deployment operation failed; The resource operation completed with terminal provisioning state 'Failed'."

When deploying the ARM template, you may receive the following error:

The deployment 'splunk-activity-logs-deploy-resources' failed with error(s). Showing 1 out of 1 error(s).
Status Message: At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details. (Code: DeploymentFailed)
 
Azure Portal Error for Microsoft.Web/sites/sourcecontrols:
{ "status": "Failed", "error": { "code": "ResourceDeploymentFailure", "message": "The resource operation completed with terminal provisioning state 'Failed'." }}

The Azure Function code needs to be fetched from Github to be deployed. Sometimes this fetch fails results in the above error.

Follow the steps in the Ensure Azure Function is deployed correctly section of this topic to ensure the Azure Function has been deployed properly.

Ensure Azure Function is deployed correctly

1. Find the Azure Function
   1. Navigate to portal.azure.com
   2. Navigate to the destination subscription.
   3. On the navigation panel, select Resource groups.
   4. Select the resource group for the SCDM input. The name is SplunkDMDataIngest-[Data manager input id]
   5. Select the Function App. The name will be suffixed with the data manager input id.
   6. On the navigation panel, select Functions.
2. Confirm the Azure Function is not deployed
   1. In the Functions section in the Function App, you may notice there are no functions and the Azure portal displays No results.
3. Redeploy the Azure Function
   1. In the same Function App as the previous steps, navigate to Configuration.
   2. Reveal the value of the WEBSITE_RUN_FROM_PACKAGE config. This should be a downloadable Splunk zip package URL link (follows the format https://downloads.splunk.com/*)
   3. If the package fails to download then the URL may be invalid. Reach out to Splunk support to obtain the correct download link.

Data management troubleshooting

If your status on the Data Management page is not Success or In Progress, and the status never changes when you click Refresh, you may have to delete the data input and start again.

For information about status messages, see Verify the data input for Azure in Data Manager.

Search for events and logs

Use the following searches to find events and logs. From the Splunk Cloud menu bar, click Apps > Search & Reporting.

If data ingestion is failing, but you see no errors in Data Manager, you can check for errors in the Azure logs by running the following in Splunk Web Search.

index=<user selected index> sourcetype="azure:*"

Search for Azure events associated with a specific input ID.

index=<user selected index> datamanager_input_id=<input_id>