Enable private connectivity

Splunk Cloud Platform administrators can turn on the optional private connectivity feature for Splunk Cloud Platform deployments.

Prerequisites

1. Review the scope and requirements in About private connectivity.
2. Ensure you meet the following ACS API requirements:
   • You must have the sc_admin role.
   • You must have Splunk Cloud Platform version 8.0.2007 or higher.
   • Your deployment must have one or more separate search heads or a search head cluster. ACS is not supported on single instance deployments.
3. Set up the ACS API and authentication token.
4. Obtain the AWS account ID(s) that you want to establish a PrivateLink connection with.

Turn on private connectivity

The following section shows you how to turn on private connectivity in the Splunk Cloud Platform using the ACS API and AWS documentation. Once you complete these steps, all traffic from your forwarders and HEC endpoints that is coming from your established AWS PrivateLink VPC will flow over private endpoints. You can also allow search tier traffic to flow over AWS PrivateLink. The default public endpoints for data ingestion and search will continue to be accessible along with your newly established service endpoints for private connectivity. To restrict the public endpoints further, configure the appropriate security group rules in your VPC.

Federal Risk and Authorization Management Program Impact Level 2 (FedRAMP Moderate) customers need to open a Splunk Cloud support case instead of performing steps 1 and 2 in this process.

1. Confirm eligibility

To confirm your eligibility for private connectivity, send an HTTP GET request to the following endpoint:

GET /{stack}/adminconfig/v2/private-connectivity/eligibility

For example, to check if TestStack is eligible, send the following request:

curl --location --request GET 'https://admin.splunk.com/TestStack/adminconfig/v2/private-connectivity/eligibility' \
--header 'Authorization: Bearer abcdefgh123456....' 

The request returns a success or failure result based on whether the Splunk Cloud Platform deployment meets the required eligibility criteria.

If your Splunk Cloud Platform deployment is eligible, the request returns:

{
"eligible": true
}

If your Splunk Cloud Platform deployment is not eligible, the request returns:

{
"eligible": false
}

2. Enable private connectivity

Once you confirm that your Splunk Cloud Platform deployment is eligible for private connectivity, send an HTTP POST request to the following endpoint:

POST /{stack}/adminconfig/v2/private-connectivity/endpoints

This request requires you to specify the AWS account ID and feature that you want to associate with the private connectivity endpoint. For example, to enable private connectivity for both ingest and search for TestStack with AWS account ID 112233445566:

curl --location --request POST 'https://admin.splunk.com/TestStack/adminconfig/v2/private-connectivity/endpoints'  \
-H "Content-Type: application/json" --header 'Authorization: Bearer abcdefgh123456...' \
--data-raw '{
"customerAccountIds" : ["112233445566"],
"feature" : ["ingest", "search"]
}'

Specify multiple account IDs using a comma to separate the values. For example: ["112233445566, 778899101011"].

Features "ingest" and "search" are supported. Specify multiple features using a comma to separate the values. For example: ["ingest", "search"]. An empty feature list, [] enables both features.

The API creates a new endpoint and adds the AWS account ID(s) to the endpoints. The request returns the AWS account ID(s) that were added to the endpoints:

{
"customerAccountIds" : ["112233445566"],
"feature" : ["ingest", "search"]
}

2.a Retrieve endpoint name

To confirm that your private connectivity request was enabled and to retrieve the VPC endpoint ID, send an HTTP GET request to the following endpoint:

GET /{stack}/adminconfig/v2/private-connectivity/endpoints

Query parameter:
"feature" (string, optional): Specify the feature to be retrieved. Valid inputs include "ingest", "search", or "all". If this parameter is not passed, the query will return information for all features.

For example, to retrieve the VPC endpoint ID for TestStack for all features:

curl --location --request GET 'https://admin.splunk.com/TestStack/adminconfig/v2/private-connectivity/endpoints?feature=all' \
--header 'Authorization: Bearer abcdefgh123456...' 

When private connectivity for ingest and search have been successfully enabled, the request returns the AWS account ID(s), the name of the VPC endpoint, a message confirming enablement, and an available status. For example:

{
    "endpoints":[
        {
        "customerAccountIds": [
            "112233445566",
            "778899101011"
        ],
        "endpoint": "com.amazonaws.vpce.us-east-1.vpce-svc-00f4e0098fce35fb9",
        "feature":"ingest",
        "message": "Private Connectivity for ingest is enabled. Please refer to 
        https://docs.splunk.com/Documentation/SplunkCloud/latest/Security/Privateconnectivity 
        for more information.",
        "status": "available"
        },
        {
        "customerAccountIds": [
            "112233445566",
            "778899101011"
        ],
        "endpoint": "com.amazonaws.vpce.us-east-1.vpce-svc-0d0f607d0a67e50f9",
        "feature":"search",
        "message": "Private Connectivity for search is enabled. Please refer to 
        https://docs.splunk.com/Documentation/SplunkCloud/latest/Security/Privateconnectivity 
        for more information.",
        "privateSearch-DNSRecords":[
            "sh1.pvt.TestStack.splunkcloud.com",
            "shc1.pvt.TestStack.splunkcloud.com",
            "pvt.TestStack.splunkcloud.com"
        ],
        "status": "available"
}

For search: Note the privateSearch-DNSRecords returned. These search tier DNS record will be used to configure DNS routing within your AWS VPC later.

It can take several minutes for the private connectivity set up and initialization process to complete on AWS before your private connectivity endpoints become available.

If private connectivity has not been enabled, the request output shows the status of the request and a reason for that status. For example:

Private connectivity for ingest is not enabled:

{
    "endpoints":[
        {
        "feature":"ingest",
        "status": "unavailable",
        "reason": "uninitialized"
        }
    ]
}

Private connectivity for ingest initialization request is still in progress:

{
    "endpoints":[
        {
        "feature":"ingest",
        "status": "unavailable",
        "reason": "initialization is in progress"
        }
    ]
}

Private connectivity for ingest request failed:

{
    "endpoints":[
        {
        "feature":"ingest",
        "status": "unavailable",
        "reason": "initialization failed"
        }
    ]
}

3. (For Data Ingest only) Download universal forwarder credentials for private connectivity

Once you have enabled Splunk Cloud Platform private connectivity, follow the instructions in Deploy the universal forwarder to configure your universal forwarders to connect with AWS PrivateLink. Download the private connectivity package for each universal forwarder that you want to use to send data to private endpoints.

There may be a delay after enabling private connectivity before the universal forwarder credentials appear in the portal.

To confirm your forwarder is configured for private connectivity, check the server value in outputs.conf.

• Non-private connectivity: inputs1.<stack-name>.splunkcloud.com, inputs2.<stack-name>.splunkcloud.com, ...
• Private connectivity: inputs1.pvt.<stack-name>.splunkcloud.com, inputs2.pvt.<stack-name>.splunkcloud.com, ...

4. Create VPC endpoint and enable PrivateLink

Now that you have configured your universal forwarders for AWS PrivateLink, follow the instructions in Access an AWS service using an interface VPC endpoint - Amazon Virtual Private Cloud to create an interface VPC endpoint for each VPC you want to connect to PrivateLink with the following caveats:

For ingest

• In Other endpoint services, add the VPC endpoint name provided by Splunk in Step 2.a under "feature : Ingest" in the Service Name field.
• Set IP address type to IPv4.
• For security groups, allow port 443 and 9997 as inbound.
• In endpoint service, under the details section, take note of the first DNS record entry with the vpce-xxx.vpce-svc-xxx prefix (in other words, the first entry that does not include the availability zones). You will use this to configure your private DNS zone. For example: com.amazonaws.vpce.us-east-1.vpce-svc-10022f155a5dd8b21

After the VPC endpoint is created, the status in the console should change from "Pending" to "Available". It may take several minutes before the status gets updated.

For search

Set up a new endpoint service to receive search results.

• In Other endpoint services, add the VPC endpoint name provided by Splunk in Step 2.a under "feature : Search" in the Service Name field.
• Set IP address type to IPv4.
• For security groups, allow port 80, 443 and 8089 as inbound.
• In endpoint service, under the details section, take note of the first DNS record entry with the vpce-xxx.vpce-svc-xxx prefix (in other words, the first entry that does not include the availability zones). You will use this to configure your private DNS zone. For example: com.amazonaws.vpce.us-east-1.vpce-svc-10026f133a3cc1d94

5. Configure DNS name

To resolve Splunk forwarder and/or Search traffic to the Splunk VPC from your endpoint services, you must configure appropriate DNS mappings for each VPC that you want to connect to AWS PrivateLink. To do so, follow the instructions in Routing traffic to an Amazon Virtual Private Cloud interface endpoint by using your domain name - Amazon Route 53 with the following caveats:

• Create a private hosted zone with one of the following domain names:
  • Splunk Cloud Platform deployments in standard AWS regions: pvt.<stack-name>.splunkcloud.com
  • Splunk Cloud Platform deployments in GovCloud regions: pvt.<stack_name>.splunkcloudgc.com

For ingest:

• Create a CNAME record type for ' * ' and point it to the DNS record you copied in the previous section (for example: vpce-xxx.vpce-svc-xxx).

If your forwarders do not reside in your AWS VPC (for example, if you have extended them with DirectConnect or VPN tunneling), then you will need to complete your DNS mapping where your forwarder resides.

For search:

• Make a note of all the DNS records associated with your search tier from Step 2a under the "Feature: Search" section.
• Create a CNAME record type for each DNS record obtained in the previous step and point them to the DNS record you copied in the previous section for Search (for example: vpce-xxx.vpce-svc-xxx).

Your Splunk Cloud Platform deployment is now configured for private connectivity.

Optional configuration steps

(Optional) Add additional AWS account IDs

If you need to add additional AWS account IDs to the PrivateLink VPC endpoint, send an HTTP PATCH request to the following endpoint:

PATCH /{stack}/adminconfig/v2/private-connectivity/endpoints

For example, to add 123456789101 to the PrivateLink VPC endpoint for TestStack for feature ingest:

curl --location --request PATCH 'https://admin.splunk.com/{stack}/adminconfig/v2/private-connectivity/endpoints' \
-H "Content-Type: application/json" --header 'Authorization: Bearer abcdefgh123456...' \
--data-raw '{
"customerAccountIds" : ["123456789101"],
"feature" : ["ingest", "search"]
}'

The request returns the AWS account ID(s) that were added to the endpoints:

{
    "feature": [
        "ingest", "search"
    ],
    "patchedCustomerAccountIds": [
        "123456789101"
    ]
}

(Optional) SAML configuration

Private connectivity--for search or data ingest--creates a private route (using AWS PrivateLink) to search or ingest your data in addition to the publicly-accessible route. Stacks configured for search using private connectivity can use SAML in one of two ways: mixed-mode or PrivateLink-only.

Mixed-mode SAML

Private search-enabled stacks can use SAML in a bimodal mode. Because most SAML IdPs only support a single redirect, this configuration requires that both Splunk and the SAML IdP have been updated to redirect to private search tier DNS entries. Clients will then rely on DNS resolution to resolve either the internet-facing IPs or the VPC endpoint IPs as applicable, respectively. When mixed-mode is enabled, a Splunk-managed private zone will be created that redirects to the regular, non-private, zone and clients accessing the search tier using PrivateLink will rely on the customer-managed private zone resolving to their VPC endpoint for private search.

Internet
Internet users must resolve against name servers that resolve against the Splunk-managed private zone.

DNS resolution flow

1. Client browses <search_tier>.<stack-name>.<domain>.
2. Search tier's SAML configuration sends auth to IdP.
3. Upon successful auth, IdP redirects to <search_tier>.pvt.<stack-name>.<domain>.
4. Client resolves <search_tier>.pvt.<stack-name>.<domain> in Splunk-managed private DNS zone, which are a CNAME(s) resolving to <search_tier>.<stack-name>.<domain>, the Splunk-managed primary DNS zone.
5. Client logs in with appropriate authorization using SAML over the Internet.
6. The client will see the URL change, which is expected, given the configuration. Despite the <search_tier>.pvt.<stack-name>.<domain> in the URL, traffic will traverse the Internet.

PrivateLink
PrivateLink users must resolve against name servers that resolve the customer-managed private zone.

DNS resolution flow

1. Client browses <search_tier>.<stack-name>.<domain>.
2. Search tier's SAML configuration sends auth to IdP. VPC or AWS Direct Connect connection must allow traffic to reach the SAML IdP.
3. Upon successful auth, IdP redirects to <search_tier>.pvt.<stack-name>.<domain>.
4. Client resolves address <search_tier>.pvt.<stack-name>.<domain> in customer-managed private DNS zone, which ultimately resolves to the VPC endpoint IP(s) from the newly created endpoint.
5. Client logs in with appropriate authorization using SAML over the AWS PrivateLink.

PrivateLink-only SAML

To configure SAML for use in a stack where Private Search is enabled and mixed-mode is set to false, ensure that Splunk and the SAML IdP have been updated to redirect to the private search tier CNAME(s) created in the steps above. DNS must resolve appropriately to the newly created VPC endpoint for the SAML handshake to complete properly.

Troubleshooting

To verify that the AWS PrivateLink endpoint is actually being used:

1. Resolve the Splunk environment DNS from the instance running in your VPC. The domain should resolve to a private IP address in your VPC.
2. If the DNS resolves to a public IP address, re-check your DNS and VPC configurations. The following configurations must be set:
   • DNS region (EndpointRegion) and VPC ID (Vpcid) must match the corresponding instance settings.
   • Both enableDnsHostnames and enableDnsSupport must be set to true to ensure the VPC supports privately hosted zones.
3. If the DNS resolves correctly, but your Splunk forwarders cannot connect to the endpoint on port 443, confirm incoming traffic on port 443 and 9997 is permitted by the security group settings associated with your PrivateLink endpoint.

You can also enable VPC flow logs for the network interfaces of your Splunk Cloud Platform deployment or the network interfaces associated with PrivateLink. If you do, you can also check the IP addresses in these logs to verify that your instance is communicating with a private endpoint. If you see a REJECT entry in these logs, the traffic is likely being blocked by one of your security group settings.

Best practices

Data Ingest

Use the following best practices if you send data to your Splunk Cloud Platform with:

HEC or serverless endpoints

• AWS Lambda function must be bound to your VPC.
• Use the suggested HEC / AWS Lambda URLs:
  • Private endpoint: http-inputs.pvt.<stackname>.splunkcloud.com
  • Public endpoint: http-inputs.<stackname>.splunkcloud.com
• In the VPC interface, allow traffic from the security group associated with the AWS Lambda.

AWS Direct Connect or similar technologies

• Set up an additional DNS configuration to send data from on-premise targets via AWS VPC using AWS PrivateLink.
• Configure AWS Route 53 or similar inbound resolver to resolve inputs hostnames for your on-premise forwarder.

Routing traffic from different AWS regions

The interface VPC endpoint (PrivateLink) must be created in the VPC that is located in the same region as your Splunk Cloud Platform deployment. However, if you want to route traffic through private endpoints from a VPC located in a different region, you can leverage AWS capabilities like inter-region VPC peering. For more details, see AWS documentation on inter-region access to endpoint services.

DNS override must be present in every VPC that has forwarders residing in it. To accomplish this, you can create a private Route 53 hosted zone in each VPC, or you can associate all the VPCs with a single private hosted zone.