Docs » Authentication and Security » Scenario: Wei maintains a secure organization with many teams and users using Splunk Observability Cloud

Scenario: Wei maintains a secure organization with many teams and users using Splunk Observability Cloud 🔗

Buttercup Games, a fictitious game company, recently refactored its e-commerce site to go cloud native. The site uses microservices for the application architecture and containers for the underlying infrastructure.

In this scenario, you learn how Wei, a site reliability engineering (SRE) manager, performs the following tasks using Splunk Observability Cloud to keep their organization secure and minimize unintentional changes.

Enhance team security with access restrictions and tokens 🔗

Kai, the on-call SRE, informs Wei that the uptime for the Buttercup Games site has dropped significantly in the past hour and that customers haven’t been able to access the website. However, Wei hasn’t received any alerts related to the drop in system uptime.

  1. Wei logs in to Splunk Observability Cloud on their laptop to investigate. They go to the alert list to check the alert configurations and notice that the relevant uptime detector has been muted.

  2. Wei goes on to check the detector history in the Detector Info tab and sees that the most recent edit was made by someone not on their SRE team. Wei realizes the person might have accidentally muted the detector when they were viewing the detector.

  3. During their investigation, Wei also notices that write permission for the uptime detector is default, which means anyone in the organization can make changes to it.

Before editing write permissions for the detector, Wei uses team management in Splunk Observability Cloud to make the process more convenient.

  1. To avoid having to add the same list of users to permission lists for all dashboards and detectors, and to account for future team expansion, Wei first creates a team in Splunk Observability Cloud that consists of only SREs in charge of monitoring uptime alerts.

  2. Wei wants to prevent other users from joining their team without approval, so they turn on access restrictions for teams in their organization.

  3. Since the SREs need to create and delete detectors, Wei generates a token for their team to be able to manage detectors using the API. Specifically, Wei chooses only the API authorization scope for the token to preserve security.

Learn more 🔗

Limit who can edit a detector 🔗

  1. Wei navigates back to the muted uptime detector to unmute it.

  2. Now that Wei has a secure team configuration, they are ready to add only the team they created to the detector permission list.

From now on, only Wei and their teammates can make further changes to uptime detectors.

Learn more 🔗

For more information on setting up detector permissions, see View and manage permissions for detectors.

Prevent people on other teams from viewing and changing your dashboards 🔗

  1. Wei knows that users can link detectors to charts in Infrastructure Monitoring, so they select Detector actions > Info to see if the uptime detector is linked to any charts.

  2. Wei sees that the detector is linked to a chart in the error budget dashboard, which belongs to their team’s dashboard group.

  3. To make sure that the chart won’t be accidentally changed like the detector, Wei navigates to the dashboard group and edits the permissions, so that only their team can make changes to the dashboard group, but the rest of the organization can still view it.

  4. Wei applies the same permission settings to other detectors and dashboard groups owned by their team.

Learn more 🔗

Summary 🔗

Using team access restrictions, limited tokens, and customized permissions for detectors, dashboard groups, and dashboards, Wei was able to reinforce security for their team as well as their organization in Splunk Observability Cloud. Wei’s use of these features also prevents accidental changes from happening in the future.

This page was last updated on Jul 17, 2023.