Manage your PostgreSQL database with data retention strategies

Use data retention strategies to schedule and cleanup records in your PostgreSQL database.

Use configure_db_maintenance to set parameters for retaining or removing database entries in the PostgreSQL database.

• A set of options is called a strategy.
• Models are collected into groups.
• Strategies are applied to groups.

Strategy

The set of configurable parameters that define when a record should be deleted, either automatically or when the db_maintenance tool runs.

Model

Any PostgreSQL database record or Django object is called a model. Models have characteristics that define what sort of information the model represents.

Group name	Description
audit	Audit logs. See Enable and download audit trail logs in .
automation_run	Automation run data including Playbook Run, Action Run, App Run, Custom Function Run, and playbook run statistics.
container	Containers and related records like Artifacts and Container Attachments. See About .
indicator	Indicators or Indicators of Compromise. See About .
playbook_run_log	Debug and error logs from historical playbook runs.

To use the configure_db_maintenance.py tool, follow these steps:

1. SSH to your instance.
   
   SSH <username>@<SOAR_hostname>
2. Use the following tool to manage data deletion.
   
   phenv configure_db_maintenance
3. Add your desired argument to the data retention tool command line to list, add, edit or schedule data retention actions.

On clustered systems, the configure_db_maintenance.py tool can be run from any node, but only the leader node runs the data retention strategy.

Data retention tool arguments

Add the --help argument to your command to get information on the data retention tool arguments;

phenv configure_db_maintenance --help

Optional arguments

Use these optional arguments to manage your data retention strategy.

Argument	Description
-h, --help	Show this help message and exit.
-v {0,1,2,3}, --verbosity {0,1,2,3}	Set the verbosity for the command's output. 0 for minimal output 1 for normal output 2 for verbose output 3 for very verbose output
--no-color	Do not use color in the command output
--skip-checks	Do not perform system checks.
Subcommand	Description
list	List all existing data retention strategies and their schedules.
add	Add a new data retention strategy, or update a data retention strategy by updating all of that strategy's settings. This command has options. See the table for add later in this topic.
edit	Selectively update an existing data retention strategy. This command has options. See the table for edit later in this topic.
schedule	Add or update a scheduled job to service all data retention strategies. This command has options. See the table for schedule later in this topic.

configure_db_maintenance subcommands

Use configure_db_maintenance to list, add, edit or schedule data retention strategies.

List

Get a list of all data retention strategies by using the list subcommand.

phenv configure_db_maintenance list

Sample output

Enabled data retention strategies will be serviced on the following cron schedule: Every hour UTC (cron string: "0 * * * *")
Data retention strategies:
 - Group: playbook_run_log, Age to Keep: 180 days, Raw Delete: True, Enabled: True, ID: 4, Description: [SOAR System Policy] Delete stale debug and error logs from historical playbook runs

In the sample output, a data retention strategy has been scheduled to run every hour, for the playbook_run_log group, setting the Age to Keep to a value of 180 days.

Add

Apply data retention strategies to a model by using the add subcommand.

phenv configure_db_maintenance add --help

Guidelines for adding a data retention strategy:

• If you do not specify an audit_category or a container label, the created strategy will apply to all audit categories or container labels, if the strategy is applicable to containers or audit categories.
• You cannot create a generic container data retention strategy (one that is for all labels) that will delete items before a label-specific strategy.
• You cannot create a generic audit data retention strategy (one that is for all audit categories) that will delete items before a category specific strategy

Argument	Required?	Description
--help, -h	Optional	Displays help for this command, then exits.
--target-group <TARGET GROUP> -t <TARGET GROUP>	Required	Name of the target group for the data retention strategy. audit automation_run container indicator playbook_run_log
--container-label <CONTAINER LABEL>	Optional	Specific label to target if this strategy targets the container group. You can omit this argument if you want the data retention strategy to apply to all labels. Because container labels are customizable, labels can vary by deployment. Use phenv configure_db_maintenance add --help and check the --container-label output for the most updated list for your deployment. Check to make sure you're applying your data retention strategy to the correct label for your deployment.
--audit-category <AUDIT CATEGORY>	Optional	Specific audit category to target if this strategy targets the audit_category group. You can omit this argument if you want the data retention strategy to apply to all audit categories. Audit categories action_run forwarder_group aggregation_rule indicator_cef_filter app license app_run note approval ph_user artifact playbook asset report automation_proxy report_run automation_rule role automation_rule_playbook_association role_permission container severity container_attachment splunk_idp_role_mapping container_comment splunk_role container_pin system_settings container_status tenant custom_function workbook_phase decided_list workbook_phase_template es_pairing workbook_task evidence workbook_task_template feature_flag workbook_template Audit categories may change from release to release. Your deployment may have more, fewer, or different categories than listed here. Use phenv configure_db_maintenance add --help and check the --audit-category output for the most updated list for your deployment.
--age-to-keep-time-unit <TIME UNIT>	Required	The time unit for measuring max-age-to-keep. minutes hours days months years
--max-age-to-keep <MAX AGE TO KEEP>	Required	The maximum number of time units of database records to keep when applying the data retention strategy. You must set a value for --age-to-keep-time-unit <TIME UNIT>.
--description <DESCRIPTION>	Optional	A description of the data retention strategy.
--disabled	Optional	Disable the data retention strategy.

If you add a data retention strategy for a model that already has one, the new strategy replaces the existing strategy.

Edit

Modify data retention strategies by using the edit subcommand.

phenv configure_db_maintenance edit --help

Argument	Required?	Description
--help, -h	Optional	Displays help for this command, then exits.
--target-group <TARGET GROUP> -t <TARGET GROUP>	Required	Name of the target group for the data retention strategy. audit automation_run container indicator playbook_run_log Use additional arguments when: your data retention strategy targets a specific label in the container group, you must also use the --container-label argument. If you want your data retention strategy to apply to all containers, do not use the --container-label argument. your data retention strategy targets specific audit items in the audit group, you must also use the --audit-category argument. If you want If you want your data retention strategy to apply to all audit items, do not use the --audit-category argument.
--container-label <CONTAINER LABEL>	Optional	Specific label to target if this strategy targets the container group. You can omit this argument if you want the data retention strategy to apply to all labels. Because container labels are customizable, labels can vary by deployment. Use phenv configure_db_maintenance edit --help and check the --container-label output for the most updated list for your deployment. Check to make sure you're applying your data retention strategy to the correct label for your deployment.
--audit-category <AUDIT CATEGORY>	Optional	Specific audit category to target if this strategy targets the audit_category group. You can omit this argument if you want the data retention strategy to apply to all audit categories. Audit categories action_run forwarder_group aggregation_rule indicator_cef_filter app license app_run note approval ph_user artifact playbook asset report automation_proxy report_run automation_rule role automation_rule_playbook_association role_permission container severity container_attachment splunk_idp_role_mapping container_comment splunk_role container_pin system_settings container_status tenant custom_function workbook_phase decided_list workbook_phase_template es_pairing workbook_task evidence workbook_task_template feature_flag workbook_template Audit categories may change from release to release. Your deployment may have more, fewer, or different categories than listed here. Use phenv configure_db_maintenance edit --help and check the --audit-category output for the most updated list for your deployment.
--delete	Optional	Permanently delete the data retention strategy.
--enable	Optional	Set the selected data retention strategy to active.
--disable	Optional	Deactivate the data retention strategy.

Schedule

Set the schedule for data retention strategies by using the schedule subcommand.

phenv configure_db_maintenance schedule --help

Argument	Required?	Description
--help, -h	Optional	Displays help for this command, then exits.
--cron-schedule <CRON_SCHEDULE>	Required	How often to run the scheduled data retention strategy. Must be a cron schedule expression.

Examples

Each example is for a specific task where you may want to list, add, edit, or schedule a data retention policy.

Example: Get a list of all data existing data retention strategies

Get a list of all data retention strategies by using the list subcommand.

phenv configure_db_maintenance list

Example: Add or update a strategy to delete indicator records

You may want to delete indicator records to improve database query performance, or to reduce the amount of disk space use by Splunk SOAR's PostgreSQL database.

Delete indicator records after three months.

phenv configure_db_maintenance add --target-group indicator --age-to-keep-time-unit months --max-age-to-keep 3

Example: Add or update a strategy to delete audit records

You may want to delete all audit records after your company's records keeping requirements no longer need them.

Add or update a data retention strategy to delete all audit records from any category after one year:

phenv configure_db_maintenance add --target-group audit --age-to-keep-time-unit years --max-age-to-keep 1

Example: Add or update a strategy to delete records for the app_run audit category

This example demonstrates deleting records from a single audit category after 90 days.

phenv configure_db_maintenance add --target-group audit --audit-category app_run --age-to-keep-time-unit days --max-age-to-keep 90

Example: Turn off an existing data retention strategy

This example demonstrates tuning off a single data retention strategy that applies to all containers.

phenv configure_db_maintenance edit --disable --target-group container

Example: Delete an existing data retention strategy

This example demonstrates how to delete a data retention strategy. This strategy applies to all containers with the events label.

phenv configure_db_maintenance edit --delete --target-group container --container-label events

Example: Change the schedule for a data retention strategy

This example changes the schedule to run data retention strategies to every two hours, on the hour.

phenv configure_db_maintenance schedule --cron-schedule "0 */2 * * *"