Troubleshoot GCP data ingestion in Data Manager

Troubleshooting tips include, but are not limited to, the following items that can assist throughout the onboarding process.

For more troubleshooting tips, see the Troubleshooting in Data Manager and the Troubleshoot general problems chapters in this manual.

Data management troubleshooting

If your status on the Data Management page is not Success or In Progress, and the status never changes when you click Refresh, you may have to delete the data input and start again.

For information about status messages, see Verify the data input for GCP in Data Manager.

Search for events and logs

Use the following searches to find events and logs. From the Splunk Cloud menu bar, click Apps > Search & Reporting.

If data ingestion is failing, but you see no errors in Data Manager, you can check for errors in the GCP logs by running the following in Splunk Web Search.

index=<user selected index> sourcetype="google:gcp:pubsub:*"

Search for GCP events associated with a specific input ID.

index=<user selected index> source=gcp_cloud_logging_<input id>

Recover events from the dead-letter topic for a GCP input

To recover the events from the dead-letter topic perform the following steps.

1. Identify and fix the cause of your failed messages.
   
   Before triggering the recovery Dataflow job, the root cause of the delivery failures must be fixed. To identify and fix the root cause of your message failure, perform the steps contained in the Troubleshoot failed messages section of the Deploying production-ready log exports to Splunk using Dataflow topic in the Google Cloud documentation.


2. Replay your failed messages.
   
   After fixing the root cause of your failed messages, your GCP admin must trigger a Pub/Sub to Pub/Sub Dataflow job in order to replay the logs that are currently in dead-letter topic. This delivers the events from the dead-letter topic to the original Dataflow job that writes to the Splunk software, which ensures that they can be ingested to your Splunk Cloud deployment.
   To trigger a GCP Dataflow job that is used to deliver the messages to the primary GCP Dataflow job that was set up using Terraform to export logs to your Splunk Cloud platform, follow the steps outlined at the Replay failed messages section in the Deploying production-ready log exports to Splunk using Dataflow topic in the GCP documentation.
   
   Use the following commands to start a Pub/Sub to Pub/Sub Dataflow job in order to transfer the messages from the your dead-letter topic back to the primary GCP Dataflow job that was set up using Terraform:

     DM_INPUT_ID = "<copy the Data Manager Input ID here>"
     PROJECT_ID = "<copy your GCP Dataflow project ID here>"
     REGION_ID = "<copy your GCP Dataflow region here>"
   
   
   
     DATAFLOW_INPUT_TOPIC="pubsub_topic_logging_${DM_INPUT_ID}"
     DATAFLOW_DEADLETTER_SUB="pubsub_subscription_deadletter_${DM_INPUT_ID}"
   
   
     JOB_NAME=splunk-dataflow-replay-`date +"%Y%m%d-%H%M%S"`
     gcloud dataflow jobs run $JOB_NAME \
          --gcs-location= gs://dataflow-templates/latest/Cloud_PubSub_to_Cloud_PubSub \
          --worker-machine-type=n1-standard-2 \
          --max-workers=1 \
          --region=$REGION_ID \
          --parameters \
   
     inputSubscription=projects/${PROJECT_ID}/subscriptions/${DATAFLOW_DEADLETTER_SUB},\
     outputTopic=projects/${PROJECT_ID}/topics/${DATAFLOW_INPUT_TOPIC}