Customer managed centralized SOC architectures
Many Splunk customers operate in complex organizational environments, consisting of multiple business units, agencies, and/or even companies. This can lead to multiple Splunk deployments, caused by different funding lines, regulatory compliance, data sovereignty considerations, and other reasons. Due to security / IT initiatives, customers may have a need to centralize some or all of the machine data across the various organizational units.
The following Applied Architectures are options for customers who currently have multiple Splunk deployments and need to bridge those deployments together to support initiatives like a Centralized Security Operations Center (SOC).
Please note that each of these deployment options offer various benefits and limitations and should be fully understood before implementing. If customers have questions regarding these architectures, please contact your Splunk account team for further assistance
Who is this document for?
Customers with business requirements to consolidate data between multiple organizational components. Specific personas this document is targeted for include:
Role | Description |
---|---|
Enterprise Architects | Responsible for architecting Splunk software deployments to meet enterprise needs |
Consultants | Responsible for providing services for architecture, design, and implementation of Splunk software deployments |
Splunk Engineers | Responsible for managing the Splunk lifecycle |
Managed Service Providers | Entities that deploy and run Splunk software as a service for customers |
Who is this document not for?
Managed Security Service Providers are not in scope in this document.
- For OnPremises or customer managed Splunk deployments refer https://www.splunk.com/en_us/pdfs/white-paper/splunk-for-managed-security-service-providers-technical-architecture.pdf
- For cloud MSSPs refer to: https://www.splunk.com/en_us/pdfs/white-paper/splunk-cloud-msp-blueprint.pdf
Definitions
Term | Definition |
---|---|
SOC | Security Operations Center |
Centralized SOC* | Resources for security operations are consolidated under a single authority and organization. SOC personnel have dedicated roles within this centralized SOC. |
Federated SOC* | A SOC, likely centralized but could also be hierarchical, shares a parent organization with one or more other SOCs, but generally operates independently. SOC teams may have some shared policies and authorities. |
Coordinating SOC* | A SOC responsible for coordinating the activities of other SOCs underneath it. Focus is primarily on situational awareness and overall incident management. Does not direct the day-to-day operations of the SOCs it coordinates. |
Hierarchical SOC* |
Similar to a Coordinating SOC structure; however the parent organization plays a more active role. The parent organization may offer SOC services to lower-level SOCs and has greater responsibility for coordinating a wider range of SOC functions. |
Organization | Describes the overarching designation of business components bundled together. This may be a corporation or federal agency. |
Component | A distinct subdivision under an organization that provides a particular function. |
Parent / Centralized | Description of a Splunk deployment that sits hierarchically higher than the other deployments within the organization. |
Child / Local | Description of a Splunk deployment that sits hierarchically lower than the parent deployment within the organization. |
Premium search head | Represents a deployment of a premium solution, such as Enterprise Security or ITSI within the Splunk deployment. |
Role Based Access Control (RBAC) vs. |
In Splunk there are two ways to segregate data by role. First is by index, the second is by role-based field filtering to protect sensitive data. However RBAC and ABAC do not extend to data models, or metrics indexes. Related information: |
Splunk Data Cloning | Utilizing universal or heavy forwarders to clone data to two or more Splunk deployments. Logic can be applied to heavy forwarders that allow filtering of specific data. This allows for the separation of Splunk deployments at the cost of data duplication. Data cloning usually results in similar, but not necessarily exact, copies of data on the receiving indexers. Related information: |
Splunk Search Peering | Utilizing the distributed search capability within Splunk to connect search heads to indexers. With search peering, the search head would connect to indexers in two or more Splunk deployments. This allows the searching of data across multiple deployments, but without role based access controls on data, the search head using this capability has access to all indexes, e.g index=* and index=_* Related information: |
Splunk Search Peering with Indexer Segmentation | Utilizing the distributed search capability within Splunk to connect search heads to indexers. With search peering, the search head would connect to indexers in two or more Splunk deployments. In child Splunk deployments who have security restrictions, a separate indexer cluster would be set up and only that indexer would be peered by the parent Splunk deployment. The separate indexer cluster receives different data that is forwarded to it that is not in the other indexer cluster. Related information: |
Splunk Federated Search | Utilizing API calls, it logically connects one Splunk deployment's search head to another Splunk deployment search head to allow for querying of data. This allows for some RBAC via indexes, but does not work with all premium solutions.
Transparent mode vs. Standard mode federated search has important differences. Most notably the syntax of SPL changes, the requirement that ALL knowledge objects be in sync on all search heads, and the access is brokered via a service account user and role that controls the indexes that the Federated search will be restricted to. |
- Reference: 11 Strategies of a World-Class Cybersecurity Operations Center, Page 54. https://www.mitre.org/sites/default/files/2022-04/11-strategies-of-a-world-class-cybersecurity-operations-center.pdf
Deployment options summary
The following is a listing of the available options to architect Splunk in an multi-deployment environment. Please refer to each section for further detail on the Splunk Validated Architecture.
SVA Option | Description |
---|---|
Centralized SOC Organization | |
Centralized SOC Organizational Model | A centralized SOC organization which uses a centralized Splunk platform to perform security operations at an organization level. |
Federated / Coordinating SOC Organization | |
Federated SOC Organizational Model Utilizing Data Cloning | A SOC organization where duties may be distributed among multiple customer groups within the organization. Organization level visibility accomplished through "data cloning" to a centralized Splunk deployment which isolates deployments from performance impacts. |
Federated SOC Organizational Model Utilizing Search Peering | A SOC organization where duties may be distributed among multiple customer groups within the organization. Organization level visibility accomplished through "search peering" to a centralized Splunk deployment. This method logically bridges all deployments together and has potential performance impacts. |
Federated SOC Organizational Model Utilizing Search Peering with Indexer Segmentation | A SOC organization where duties may be distributed among multiple customer groups within the organization. Organization level visibility accomplished through "search peering" and "indexer segmentation" to a centralized Splunk deployment. This method logically bridges designated indexers together and has potential performance impacts. |
Hierarchical SOC Organization | |
Hierarchical SOC Organization Model using Splunk Peering or Data Cloning | A SOC organization where duties may be handled at the centralized level while still being able to multiple child SOC groups. Organization level visibility accomplished at the parent level, while allowing for autonomy at the component level for those customers who have their own SOC |
Centralized SOC organization model
Deployment description
This deployment focuses on the centralization of SOC resources and assets under one centralized Splunk deployment. Typically this method is utilized for new organizations who don't have existing multiple Splunk instances or are looking to re-architect their Splunk environment to centralize log and metric data onto a single platform.
When to utilize
Considerations | Description |
---|---|
Centralization Initiative: | Customers, who do not have existing Splunk deployments and have buy-in from the various business components to offer Splunk as a service, organization wide. Customers, who have a few Splunk deployments and have buy-in from those business components to collapse their existing deployments in favor of a centralized deployment. |
Centralized SOC Team: | Agreement between all components that SOC activities will be the primary responsibility of the centralized SOC team. Customers may have access to their components' centralized SOC data. |
Desire the capability for service evolution: | The centralized Splunk deployment requires the capability to evolve beyond its original use case. This may include use cases around ITOps, DevOps, Business Analytics, IOT and more. |
When not to utilize
Considerations | Description |
---|---|
Data Security / Hardening: | Customers at the component level have a need to analyze data that cannot or should not be sent to the centralized SOC due to data security / hardening concerns. For example:
|
Component level analytics needed for non-sharable data: | Policy decisions were made, due to resource constraints and / or other reasons to ingest only security related data into the centralized Splunk deployment. Recommend customers set up their own Splunk deployment and utilize one of the architectures below to send security data to SOC. |
Centralized SOC organizational model architecture diagram
Incorporated SVAs
- Deployments:
- Single Server Deployment (S1)
- Distributed Non-Clustered Deployment (D1 / D11)
- Distributed Clustered Deployment - Single Site (C1 / C11)
- Distributed Clustered Deployment + SHC - Single Site (C3 / C13)
- Distributed Clustered Deployment - Multi-Site (M2 / M12)
- Distributed Clustered Deployment + SHC - Multi-Site (M3 / M13)
- Distributed Clustered Deployment + SHC - Multi-Site (M4 / M14)
- Data collection:
- Data Collection Architecture
Benefits
The centralized SOC model architecture promotes the centralization and sharing of resources across an entire organization. Benefits of this deployment include:
- Centralization and consolidation of Splunk resources under a single service for centralized monitoring.
- Centralization of machine data under one platform, which can promote use cases beyond security.
- Centralization of SOC resources.
Limitations
Implementation of this Splunk Validated architecture comes with some limitations that may impact overall deployment. considerations include:
- Limited flexibility:
- Additional premium solutions, add-ons, and ingestion methods need to be coordinated across the all components of the organizations
- Requires role based access control (RBAC) policies & indexes to ensure separation of data:
- To keep multiple components data separate, Splunk administrators will need to implement and enforce a RBAC policy along with proper index separation.
- Additional data separation strategies may be implemented through tagging and search restrictions.
- Requires workload policies:
- To ensure proper allocation and prevent overallocation of resources.
- Data Access and Onboarding Methodologies
- Depending on if you are collapsing multiple Splunk deployments together, there may be a need to maintain different versions of a TA, depending on customer needs.
Federated SOC organization model utilizing Splunk data cloning
Deployment description
This deployment utilizes a feature within Splunk Heavy Forwarders to clone data sources to multiple Splunk deployments. Typically, this model is utilized to roll-up data from multiple Splunk child deployments to a parent Splunk deployment. This method sends a copy of the data to a parent Splunk deployment which prevents performance and security issues with the child deployments.
When to utilize
Considerations | Description |
---|---|
Multiple Splunk deployments exist within the organization : | Organization has multiple Splunk deployments, typically at a component level. There is a need to roll up security data to a centralized SOC team for the following purposes:
|
Multiple SOC Groups: | SOC responsibilities are managed at a component level Centralized SOC provides organization level visibility. |
Data leakage / privacy concerns: | Component Splunk deployments have a requirement to control what data is sent to the centralized SOC level. |
Performance consistency: | Component Splunk deployments have concerns around potential performance impacts from increased load from centralized Splunk deployment. |
Audit Visibility: | Component Splunk deployments have stringent audit requirements for Splunk access. |
When not to utilize
Considerations | Description |
---|---|
Data duplication is undesirable: | Organization has concerns around duplicated storage and / or network traffic. |
Concerns around data availability and resiliency | If a heavy forwarder queue is saturated, this could prevent cloned data from reaching both locations without proper settings to mitigate the risk. The default behavior for data cloning in a universal forwarder assumes both destinations are always available. Work with your Splunk account team to evaluate heavy forwarder settings and/or architectures to mitigate this risk. |
Federated SOC organizational model utilizing data cloning architecture diagram
Data cloning can be done either at the universal forwarder level or at a heavy forwarder level. Please consult Splunk documentation (https://docs.splunk.com/Documentation/Forwarder/latest/Forwarder/Configureforwardingwithoutputs.conf#Configure_data_cloning_on_a_universal_forwarder_with_outputs.conf) for further information.
Incorporated SVAs
- Deployments:
- Single Server Deployment (S1)
- Distributed Non-Clustered Deployment (D1 / D11)
- Distributed Clustered Deployment - Single Site (C1 / C11)
- Distributed Clustered Deployment + SHC - Single Site (C3 / C13)
- Distributed Clustered Deployment - Multi-Site (M2 / M12)
- Distributed Clustered Deployment + SHC - Multi-Site (M3 / M13)
- Distributed Clustered Deployment + SHC - Multi-Site (M4 / M14)
- Data Collection:
- Data Collection Architecture
Benefits
When utilizing Splunk Data Cloning for a Federated SOC organization, this architecture allows for the rollup of security data to a higher level SOC, for visibility & coordination purposes. Benefits include:
- Provide centralized view of Splunk deployments:
- Bridge multiple existing and/or new Splunk deployments, to allow for a high level organization view.
- Performance stability:
- Promotes performance isolation between parent / child Splunk deployments.
- Data security:
- Provides the ability to selectively choose which data is sent to the parent level by the child deployments, preventing potential data security / authorization concerns.
- Increased audit controls:
- Allows for stringent audit controls and compliance around data.
- Splunk version independence
- Allows each Splunk deployment to be version independent and prevent upgrade conflicts.
- Splunk technical add-on (TA) independence between Splunk deployments.
- Allows each Splunk deployment to be TA version independent due to deployment isolation.
Limitations
Implementation of this Splunk Validated architecture comes with limitations that may impact overall deployment. Limitations include:
- Potential resource duplication:
- With the data cloning option, there may be a duplication of storage, network, and ingestion.
- Potential heavy forwarder queue backup:
- If heavy forwarders cannot reach any destination, this will cause the data to be queued up until all destinations are available. This can be mitigated with configuration changes to the heavy forwarder.
Federated SOC organization model utilizing Splunk data peering
Deployment description
This deployment model utilizes a feature within Splunk (data peering) to allow search heads to connect to multiple Splunk Indexers. This feature can virtually bridge multiple Splunk deployments together to allow searching across multiple Splunk deployments. This provides users the capability to search machine data from the location it resides at in multi-Splunk deployments.
When to utilize
Considerations | Description |
---|---|
Multiple Splunk deployments exist within the organization: | Organization has multiple Splunk deployments, typically at a component level, where there is a need to query security data from a centralized SOC team for the following purposes:
|
Multiple SOC Groups: | SOC responsibilities are managed at a component level Centralized SOC needs organization level visibility. |
Quick way to logically bridge Splunk deployments together: | Allows for the bridging of the different deployments by allowing upper level search heads to peer into child indexers. |
When not to utilize
Considerations | Description |
---|---|
Customers have data that should not be shared or viewed at the Centralized SOC level: | Peering provides all or nothing access to the data that resides on the indexers. If targeted indexers contain data that should not be searchable at the parent deployment, utilize another recommended architecture. |
Local Splunk deployments are resource constrained and cannot support an increase in queries: | Heavily utilized and / or under provisioned component level Splunk deployments can introduce performance issues with both organization and component level deployments. |
Patch coordination concerns / issues between parent / child Splunk deployments: | Coordination is required to patch logically bridged deployments (Version Releases and TAs). Local / component level Splunk deployments cannot be on the latest release until centralized deployment has upgraded. |
Federated SOC organizational model utilizing data peering architecture diagram
Incorporated SVAs
- Deployments:
- Single Server Deployment (S1)
- Distributed Non-Clustered Deployment (D1 / D11)
- Distributed Clustered Deployment - Single Site (C1 / C11)
- Distributed Clustered Deployment + SHC - Single Site (C3 / C13)
- Distributed Clustered Deployment - Multi-Site (M2 / M12)
- Distributed Clustered Deployment + SHC - Multi-Site (M3 / M13)
- Distributed Clustered Deployment + SHC - Multi-Site (M4 / M14)
- Data Collection:
- Data Collection Architecture
Benefits
When utilizing Splunk peering for a Federated SOC organization, this architecture allows for the querying of security data from the parent SOC, for visibility & coordination purposes. Benefits include:
- Easy configuration and setup:
- Requires the configuring Splunk search heads to the target indexers.
- Prevents data duplication:
- By peering the parent level SOC to child indexers, it prevents the duplication of resources, such as storage.
Limitations
Implementation of this Splunk Validated architecture comes with limitations that may impact overall deployment. Limitations include:
- Performance impacts:
- Peering provides no workload limitations to the indexers it is attached to, unless the parent implements this option.
- Performance impacts can happen due to poorly written queries executed from the parent level.
- Data Model Visibility
- Unable to utilize data models from the parent level to the child.
- Security concerns:
- RBAC is controlled at the Search Head Tier. This could cause unintended access to sensitive data at the child level if not properly set up and audited.
- Recommend customers to review data stored within the child Splunk and ensure unintended data exposure to the parent level.
- Version lock and upgrade coordination:
- With peering, all Splunk versions must be at or below the version of the parent deployment.
- Child deployments cannot upgrade to the latest releases until the parent has completed the targeted version upgrade.
- Issue Triage:
- Capability to troubleshoot issues significantly increased since child deployments cannot review performance impacts from the parent deployment unless they have access to parent deployment.
- Splunk Break/Fix Support Services:
- Due to the architecture, Splunk break/fix support services, may not be able to troubleshoot issues. This will depend on whether support services have access to all levels of the deployments.
- Support issues may transcend an individual entitlement/deployment, which may require paid services to troubleshoot and remediate.
- Recommend having a Splunk-provided Assigned Expert to ensure success in troubleshooting and remediating complexities associated with this architecture.
Federated SOC organization model utilizing Splunk peering with indexer segmentation (data separation)
Deployment description
This deployment uses Indexer segmentation which allows for the routing of data (security, shared, etc.) to a specific indexer within the child deployment to stage the data.
Once the data is separated, the centralized SOC deployment can utilize peering to connect to only those indexers that contain the data to be shared. At the local deployment level, peering can be utilized to see all the indexers giving customers access to all the data.
When to utilize
Considerations | Description |
---|---|
Multiple Splunk deployments exist within the organization: | Organization has multiple Splunk deployments, typically at a component level, where there is a need to query security data from a centralized SOC team for the following purposes:
|
Multiple SOC Groups: | SOC responsibilities are managed at a component level Centralized SOC needs organization level visibility. |
Quick way to logically bridge Splunk deployments together: | Allows for the bridging of the different deployments by allowing upper level search heads to peer into child indexers. |
Data security / hardening concerns: | By segmenting which data goes to each indexer, customers can create a staging area for the data that will be peered by the parent deployment. |
When not to utilize
Considerations | Description |
---|---|
Customers have data that should not be shared or viewed at the Centralized SOC level: | Peering provides all or nothing access to the data that resides on the indexers. If targeted indexers contain data that should not be searchable at the parent deployment, utilize another recommended architecture. |
Local Splunk deployments are resource constrained and cannot support an increase in queries: | Child Splunk deployments are underprovisioned which will potentially cause performance issues at both the organization and component level deployments. |
Patch coordination concerns / issues between parent / child Splunk deployments: | Coordination is required to patch logically bridge deployments. Local / component level Splunk deployments cannot be on the latest release until centralized deployment has upgraded. |
Complexity and support concerns: | Creating a staging indexer area will increase complexity and support of the Splunk instance.
|
Federated SOC organizational model utilizing data peering with indexer segmentation architecture diagram
Incorporated SVAs
- Deployments:
- Single Server Deployment (S1)
- Distributed Non-Clustered Deployment (D1 / D11)
- Distributed Clustered Deployment - Single Site (C1 / C11)
- Distributed Clustered Deployment + SHC - Single Site (C3 / C13)
- Distributed Clustered Deployment - Multi-Site (M2 / M12)
- Distributed Clustered Deployment + SHC - Multi-Site (M3 / M13)
- Distributed Clustered Deployment + SHC - Multi-Site (M4 / M14)
- Data Collection:
- Data Collection Architecture
Benefits
When utilizing Splunk peering for a Federated SOC organization, this architecture allows for the querying of security data from the parent SOC, for visibility & coordination purposes. Benefits include:
- Easy configuration and setup:
- Requires the pointing of Splunk search heads to the target indexers.
- Prevents data duplication:
- By peering the parent level SOC to child indexers, it prevents the duplication of resources, such as storage.
- Security / Hardening:
- Reduces data security / hardening concerns by providing a staging area on designated indexer(s) for data that can be shared at the parent level.
- If clustering is utilized, recommend that customers set up two distinct clusters to prevent data security issues due to replication.
Limitations
Implementation of this Splunk Validated Architecture comes with some limitations that may impact overall deployment. considerations include:
- Indexer proliferation:
- Increases the number of indexers needed to support the child deployment.
- This increases further if the indexer clustering option is utilized. Indexer clustering replicates data across indexers. If a staging area is utilized with indexer clustering, customers would need to set up multiple distinct clusters to prevent data security to the staging area.
- Complexity of deployment:
- Requires routing of data to proper indexers (staging or non-staging)
- Performance impacts:
- Peering provides no workload limitations to the indexers it is attached to, unless the parent level implements this option.
- Performance impacts can happen due to poorly written queries executed from the parent level.
- Security concerns:
- RBAC is controlled at the Search Head Tier. This could cause unintended access to sensitive data at the child level if not properly set up and audited.
- Recommend customers to review data stored within the child Splunk and ensure unintended data exposure to the parent level.
- Version lock and upgrade coordination:
- With peering, all Splunk versions must be at or below the version of the parent deployment.
- Child deployments cannot upgrade to the latest releases until the parent has completed the targeted version upgrade.
- Issue Triage:
- Capability to troubleshoot issues significantly increased since child deployments cannot easily review performance impacts from the parent deployment.
- Splunk Break/Fix Support Services:
- Due to the architecture, Splunk break/fix support services, may not be able to accurately troubleshoot issues. This will depend on whether support services have access to all levels of the deployments.
- Support issues may transcend an individual entitlement/deployment, which may require paid services to troubleshoot and remediate.
- Splunk recommends having a Splunk-provided Assigned Expert to ensure success in troubleshooting and remediating complexities associated with this architecture.
Hierarchical SOC organization model using Splunk peering or data cloning
Deployment Description
This deployment utilizes a combination of the "Centralized SOC" and "Federated SOC" SVA options to provide a hybrid offering. This deployment allows flexibility to customers who have existing Splunk deployments that want autonomy, while servicing other customers, who cannot deploy their own Splunk software.
When to utilize
Considerations | Description |
---|---|
A need to support existing Splunk instances a component level, within the organization AND |
Organization has multiple Splunk deployments, typically at a component level, where there is a need to query security data from a centralized SOC team for the following purposes:
Organization may also provide SOC management to components who are directly connected to the parent level Splunk deployment. |
Multiple SOC Group Levels: | SOC responsibilities are managed at a component level. Centralized SOC receives data from component deployments for organization level visibility. |
Quick way to logically bridge Splunk deployments together, while supporting other components. | Allows for the bridging of the different deployments by allowing upper level search heads to have visibility into the component levels. |
When not to utilize
Considerations | Description |
---|---|
Complexity and support concerns: | By utilizing multiple strategies, there is an increase in the overall deployment complexity. Any performance, security, auditing issues may require more effort to support. |
Inherited limitations from utilized hybrid methods | Inherits limitations from the SVA method chosen to execute within this hybrid approach. |
Hierarchical SOC organizational model using Splunk peering or data cloning architecture diagram
Incorporated SVAs
- Deployments:
- Single Server Deployment (S1)
- Distributed Non-Clustered Deployment (D1 / D11)
- Distributed Clustered Deployment - Single Site (C1 / C11)
- Distributed Clustered Deployment + SHC - Single Site (C3 / C13)
- Distributed Clustered Deployment - Multi-Site (M2 / M12)
- Distributed Clustered Deployment + SHC - Multi-Site (M3 / M13)
- Distributed Clustered Deployment + SHC - Multi-Site (M4 / M14)
- Data Collection:
- Data Collection Architecture
Benefits
When utilizing the following SVA, this architecture provides customers with multiple options to support both existing and new Splunk needs. Benefits include:
- Flexibility in supporting a wide range of customer needs internally.
- Allows for component level Splunk deployments to keep autonomy while providing data centrally for organizational visibility.
- Allows for the implementation of Splunk for those components who do not have the resources to deploy their own Splunk deployment.
Limitations
Implementation of this Splunk Validated architecture comes with some limitations that may impact overall deployment. Limitations include:
- Inheritance of limitations from other SVA models
- For this architecture setup, it will rely on either data cloning or data peering. See previous SVA's associated with Federated SOC to understand the limitations that come with these deployments.
- Support Complexity:
- Depending on the size and number of Splunk deployments, this implementation can increase support costs.
Splunk Cloud Platform IL5 | Splunk Validated Architecture for Splunk AI and ML |
This documentation applies to the following versions of Splunk® Validated Architectures: current
Feedback submitted, thanks!