Enable private connectivity

Splunk Cloud administrators can enable the optional private connectivity feature for Splunk Cloud deployments in regulated environments.

Prerequisites

1. Review the scope and requirements in About private connectivity.
2. Ensure you meet the following ACS API requirements:
   • You must have the sc_admin role.
   • You must have Splunk Cloud Platform version 8.0.2007 or higher.
   • Your deployment must have one or more separate search heads or a search head cluster. ACS is not supported on single instance deployments.
3. Set up the ACS API and authentication token.
4. Obtain the AWS account ID(s) that you want to establish a PrivateLink connection with.

Enable private connectivity

The following section shows you how to enable private connectivity in the Splunk Cloud Platform using the ACS API and AWS documentation. Once you complete these steps, all traffic from your forwarders and HEC endpoints that is coming from your established AWS PrivateLink VPC will flow over private endpoints. Public endpoints for data ingestion will continue to be accessible along with your newly established service endpoints for private connectivity. To restrict the public endpoints further, configure the appropriate security group rules in your VPC.

Federal Risk and Authorization Management Program Impact Level 2 (FedRAMP IL2) customers need to open a Splunk Cloud support case instead of performing steps 1 and 2 in this process.

1. Confirm eligibility

To confirm your eligibility for private connectivity, send an HTTP GET request to the following endpoint:

GET /{stack}/adminconfig/v2/private-connectivity/eligibility

For example, to check if TestStack is eligible, send the following request:

curl --location --request GET 'https://admin.splunk.com/TestStack/adminconfig/v2/private-connectivity/eligibility' \
--header 'Authorization: Bearer abcdefgh123456....' 

The request returns a success or failure result based on whether the Splunk Cloud Deployment is in a dedicated VPC and on the Victoria experience.

If your Splunk Cloud deployment is eligible, the request returns:

{
"eligible": true
}

If your Splunk Cloud deployment is not eligible, the request returns:

{
"eligible": false
}

2. Enable private connectivity

Once you confirm that your Splunk Cloud deployment is eligible for private connectivity, send an HTTP POST request to the following endpoint:

POST /{stack}/adminconfig/v2/private-connectivity/endpoints

This request requires you to specify the AWS account ID that you want to associate with the private connectivity endpoint. For example, to enable private connectivity for TestStack with AWS account ID 112233445566:

curl --location --request POST 'https://admin.splunk.com/TestStack/adminconfig/v2/private-connectivity/endpoints' \
-H "Content-Type: application/json" --header 'Authorization: Bearer abcdefgh123456...' 
--data-raw '{
"customerAccountIds" : ["112233445566"]
}'

You can add multiple account IDs by using a comma to separate the values. For example: ["112233445566, 778899101011"].

The API creates a new endpoint and adds the AWS account ID(s) to the endpoints. The request returns the AWS account ID(s) that were added to the endpoints:

{
"customerAccountIds" : ["112233445566"]
}

2.a Retrieve endpoint name

To confirm that your private connectivity request was enabled and to retrieve the VPC endpoint ID, send an HTTP GET request to the following endpoint:

GET /{stack}/adminconfig/v2/private-connectivity/endpoints

For example, to retrieve the VPC endpoint ID for TestStack:

curl --location --request GET 'https://admin.splunk.com/TestStack/adminconfig/v2/private-connectivity/endpoints' \
--header 'Authorization: Bearer abcdefgh123456...' 

When private connectivity has been successfully enabled, the request returns the AWS account ID(s), the name of the VPC endpoint, a message confirming enablement, and an available status. For example:

{
    "customerAccountIds": [
        "123412341234",
        "123412345678"
    ],
    "endpoint": "com.amazonaws.vpce.us-east-1.vpce-svc-038b7ef7b99ca5a7b",
    "message": "Private Connectivity is enabled. Please refer to https://docs.splunk.com/Documentation/SplunkCloud/9.0.2208/Security/Privateconnectivity for more information.",
    "status": "available"
}

It can take several minutes for the private connectivity set up and initialization process to complete on AWS before your private connectivity endpoints become available.

If private connectivity has not been enabled, the request output shows the status of the request and a reason for that status. For example:

Private connectivity is not enabled:

{
"status": "unavailable",
"reason": "uninitialized"
}

Private connectivity initialization request is still in progress:

{
"status": "unavailable",
"reason": "initialization is in progress"
}

Private connectivity request failed:

{
"status": "unavailable",
"reason": "initialization failed"
}

3. Download universal forwarder credentials for private connectivity

Once you have enabled Splunk Cloud private connectivity, follow the instructions in Deploy the universal forwarder to configure your universal forwarders to connect with AWS PrivateLink. Download the private connectivity package for each universal forwarder that you want to use to send data to private endpoints.

There may be a delay after enabling private connectivity before the universal forwarder credentials appear in the portal.

To confirm your forwarder is configured for private connectivity, check the server value in outputs.conf.

• Non-private connectivity: inputs1.<stack-name>.splunkcloud.com, inputs2.<stack-name>.splunkcloud.com, ...
• Private connectivity: inputs1.pvt.<stack-name>.splunkcloud.com, inputs2.pvt.<stack-name>.splunkcloud.com, ...

4. Create VPC endpoint and enable PrivateLink

Now that you have configured your universal forwarders for AWS PrivateLink, follow the instructions in Access an AWS service using an interface VPC endpoint - Amazon Virtual Private Cloud to create an interface VPC endpoint for each VPC you want to connect to PrivateLink with the following caveats:

• In Other endpoint services, add the VPC endpoint name provided by Splunk in Step 2.a in the Service Name field.
• Set IP address type to IPv4.
• For security groups, allow port 443 and 9997 as inbound.
• In endpoint service, under the details section, take note of the first DNS record entry with the vpce-xxx.vpce-svc-xxx prefix (in other words, the first entry that does not include the availability zones). You will use this to configure your private DNS zone. For example: com.amazonaws.vpce.us-east-1.vpce-svc-10022f155a5dd8b21

After the VPC endpoint is created, the status in the console should change from "Pending" to "Available". It may take several minutes before the status gets updated.

5. Configure DNS name

To resolve Splunk forwarder traffic to the Splunk VPC from your endpoint services, you must configure the appropriate DNS mapping for each VPC that you want to connect to AWS PrivateLink. To do so, follow the instructions in Routing traffic to an Amazon Virtual Private Cloud interface endpoint by using your domain name - Amazon Route 53 with the following caveats:

• Create a private hosted zone with one of the following domain names:
  • Splunk Cloud deployments in standard AWS regions: pvt.<stack-name>.splunkcloud.com
  • Splunk Cloud deployments in GovCloud regions: pvt.<stack_name>.splunkcloudgc.com
• Create a CNAME record type for ' * ' and point it to the DNS record you copied in the previous section (for example: vpce-xxx.vpce-svc-xxx).

If your forwarders do not reside in your AWS VPC (for example, if you have extended them with DirectConnect or VPN tunneling), then you will need to complete your DNS mapping where your forwarder resides.

(Optional) Add additional AWS account IDs

If you need to add additional AWS account IDs to the PrivateLink VPC endpoint, send an HTTP PATCH request to the following endpoint:

PATCH /{stack}/adminconfig/v2/private-connectivity/endpoints

For example, to add 123456789101 to the PrivateLink VPC endpoint for TestStack:

curl --location --request PATCH 'https://admin.splunk.com/{stack}/adminconfig/v2/private-connectivity/endpoints'
--header 'Authorization: Bearer abcdefgh123456...'
--data-raw '{
"customerAccountIds" : ["123456789101"]
}'

The request returns the AWS account ID(s) that were added to the endpoints:

{
    "patchedCustomerAccountIds": [
        "123412345678"
    ]
}

Troubleshooting

To verify that the AWS PrivateLink endpoint is actually being used:

1. Resolve the Splunk environment DNS from the instance running in your VPC. The domain should resolve to a private IP address in your VPC.
2. If the DNS resolves to a public IP address, re-check your DNS and VPC configurations. The following configurations must be set:
   • DNS region (EndpointRegion) and VPC ID (Vpcid) must match the corresponding instance settings.
   • Both enableDnsHostnames and enableDnsSupport must be set to true to ensure the VPC supports privately hosted zones.
3. If the DNS resolves correctly, but your Splunk forwarders cannot connect to the endpoint on port 443, confirm incoming traffic on port 443 and 9997 is permitted by the security group settings associated with your PrivateLink endpoint.

You can also enable VPC flow logs for the network interfaces of your Splunk Cloud deployment or the network interfaces associated with PrivateLink. If you do, you can also check the IP addresses in these logs to verify that your instance is communicating with a private endpoint. If you see a REJECT entry in these logs, the traffic is likely being blocked by one of your security group settings.

Best Practices

Use the following best practices if you send data to your Splunk Cloud Platform with:

HEC or serverless endpoints

• AWS Lambda function must be bound to your VPC.
• Use the suggested HEC / AWS Lambda URLs:
  • Private endpoint: http-inputs.pvt.<stackname>.splunkcloud.com
  • Public endpoint: http-inputs.<stackname>.splunkcloud.com
• In the VPC interface, allow traffic from the security group associated with the AWS Lambda.

AWS Direct Connect or similar technologies

• Set up an additional DNS configuration to send data from on-premise targets using PrivateLink.
• Configure AWS Route 53 or similar inbound resolver to resolve inputs hostnames for your on-premise forwarder.

Routing traffic from different AWS regions

The interface VPC endpoint (PrivateLink) must be created in the VPC that is located in the same region as your Splunk Cloud deployment. However, if you want to route traffic through private endpoints from a VPC located in a different region, you can leverage AWS capabilities like inter-region VPC peering. For more details, see AWS documentation on inter-region access to endpoint services.

DNS override must be present in every VPC that has forwarders residing in it. To accomplish this, you can create a private Route 53 hosted zone in each VPC, or you can associate all the VPCs with a single private hosted zone.