Troubleshoot the Splunk Add-on for Google Cloud Platform
Data not showing up
If you are upgrading from a version of the Splunk Add-on for Google Cloud Platform earlier than 1.2.0, you must run the following upgrade script in order for your data to be properly sent.
You can access internal log data for help with troubleshooting. Data collected with these source types does not appear in any dashboards.
|Data source||Source type|
|Logs from *google_cloud_pubsub_lite*.log|
|Logs from *google_cloud_pubsub_based_bucket*.log|
|Logs from *google_cloud_bucket_metadata*.log|
|Logs from *google_cloud_resource_metadata*.log|
|Logs from *google_cloud_resthandler*.log|
Configure log levels
- Click Splunk Add-on for Google Cloud Platform in your left navigation bar on Splunk Web's home page.
- Click Configuration in the app navigation bar.
- Click the Logging tab.
- Adjust the log levels for each of the Google Cloud Platform services as needed by changing the default of
INFOto one of the other available options,
- (Optional) If you are using pub/sub, restart your Splunk instance to apply changes.
These log level configurations apply only to runtime logs. Some REST endpoint logs from configuration activity log at DEBUG, and some validation logs log at ERROR. These levels cannot be configured.
Large pub/sub subscriptions
For large pub/sub subscriptions, we recommend cloning existing inputs that are ingesting the same subscriptions to increase data throughput and performance. These identical inputs can be in the same instance or in different instances.
To manage a large number of subscriptions to one Splunk instance, aggregate subscriptions belonging to the same Google Cloud Service account into one input to save resources.
Exceed Request Limit
If you see any insufficient tokens for quota group errors such as the following, then you have exceeded the Google Cloud Monitoring request limit, which is 50000 per day. You should limit your request or apply for higher quota in Google:
<"Insufficient tokens for quota group and limit DefaultGroupCLIENT_PROJECT-1d using the limit by ID 342432.">
Python version issues
If you are upgrading from versions 1.2.0 or lower, upgrade your Splunk Enterprise deployment to work with Python 3
- Plan your Splunk Enterprise upgrade to work with the Python 3 migration.
- Clean up all pycache files from your add-on's directory location. For example,
Billing data not ingesting
If you encounter the following message in your internal logs:
Billing ingestion error. You must use the Cloud BigQuery Billing input in order to ingest billing data.
You must delete your existing billing inputs, upgrade to version 3.2.0 or later of this add-on, and recreate your billing inputs using the Cloud BigQuery Billing input in order to ingest billing data.
Google has deprecated regular file export of your Cloud Billing data to CSV and JSON. To export your Cloud Billing data for analysis, use Cloud Billing export to BigQuery.
See the Configure Cloud BigQuery Billing inputs for the Splunk Add-on for Google Cloud Service topic in this manual.
Missing fields in Events/Larger events not structured in Pub/Sub input
If you get a larger size of events and face an issue with some missing fields in the Events or Events not structured correctly then perform the steps below:
- Click on Settings
- Click on Source types under the DATA section
- Unclick Show only popular checkbox if it is already checked
- Search google:gcp:pubsub:message sourcetype
- Click on Edit, it will open the Edit Source Type: google:gcp:pubsub:message dialogue box
- Click on Advanced
- Increase the TRUNCATE value based on Event size and click on the Save button
- Perform 4 to 7 steps for google:gcp:pubsub:audit:change sourcetype
Configure Resource Metadata inputs for Splunk Add-on for Google Cloud Platform
Performance reference for the Splunk Add-on for Google Cloud Platform
This documentation applies to the following versions of Splunk® Supported Add-ons: released