Convert playbooks or custom functions from Python 2 to Python 3

Python 2 reached end of life status on January 1, 2020. The platform uses Python 3.

Splunk SOAR or Splunk Phantom playbooks written in Python 2.7 should be converted to Python 3 for use with Splunk SOAR, using an on-premises deployment of Splunk SOAR or Splunk Phantom. Users who do not have an on-premises deployment of Splunk SOAR or Splunk Phantom can download and install the free Community edition. See https://www.splunk.com/en_us/software/splunk-security-orchestration-and-automation.html.

Many Splunk SOAR or Splunk Phantom deployments have significant numbers of playbooks and custom functions which were developed in Python 2 that need to be updated to Python 3. Splunk SOAR or Splunk Phantom 4.10 and later releases contains two phenv commands to help you convert your custom functions and playbooks from Python 2 to Python 3.

• customfunctions_to_py3
• playbooks_to_py3

Best Practices for converting Splunk SOAR or Splunk Phantom playbooks and custom functions from Python 2 to Python 3

Converting custom functions or playbooks to Python 3 requires some planning and careful work on both ends of the conversion process. Here are several best practices for doing the conversions.

• Create a full backup of your Splunk SOAR or Splunk Phantom deployment. You will need this backup in order to restore your git repositories and their contents in the event that there is a problem during the conversion.
• Test each custom function and playbook before converting them. You must make sure the custom function or playbook is working correctly before proceeding. Converting incomplete, inoperable, or broken custom functions and playbooks can have unforeseen consequences.
• You should perform a dry run of each conversion using the --dry_run and --verbosity 3 options of the customfunctions_to_py3 and playbooks_to_py3 phenv commands. Pay close attention to the output from these commands. They provide helpful guidance on potential pitfalls during the conversion process.
• Test each playbook or custom function after converting them before using the converted playbooks in production environments.
• Convert custom functions separately from converting playbooks.
• Playbooks and custom functions that have been converted to Python 3 will be new copies, with the --suffix appended to their names. These new copies need to be be linked to the relevant custom functions, playbooks, and any sub-playbooks after conversion by a Splunk SOAR or Splunk Phantom administrator. Edit the relevant playbooks in the Visual Playbook Editor to update any blocks that reference an incorrect name.

Code conversion is a best-effort practice for all coding languages. Splunk SOAR code that is automatically generated in Python2 by the Visual Playbook Editor follows strict coding standards and is usually converted without error. Custom code does not necessarily follow strict coding standards and the conversion tool, as a best-effort tool, might not be able to automatically convert custom code to Python 3. If you have issues with converting custom code, see the troubleshooting section later in this article.

customfunctions_to_py3 options and examples

These are the options available when using the customfunctions_to_py3 tool.

Argument	Required?	Description
repo/ repo2/playbook_name ...	Yes	A list of repositories or custom_functions to convert to Python 3. If repositories are listed, the tool will convert every custom_function in the given repositories. Only provide the repository's name, not the full path to the repository.
repo	Yes	Output repository for converted custom_functions.
-h, --help	No	Show the help message and exit the tool.
-d, --dry_run	No	Use this option to see which custom_functions would be converted with the tool, without doing the conversion.
-s <SUFFIX>, --suffix <SUFFIX>	No	Specify a suffix to be added to the names of converted custom_functions as part of the conversion process. If no suffix is specified the default is "_py3".
-v {0,1,2,3}, --verbosity {0,1,2,3}	No	Verbosity level: * 0 = minimal output * 1 = normal output * 2 = verbose output * 3 = very verbose output If you set --verbosity 3 you may see debug messages like this: Unsupported dict value of type: <class '_ast.NameConstant'>. Ignoring children and returning type. These messages can be safely ignored. They are only informational messages from the code parser, and do not indicate a problem with the custom function.

Command examples

• Convert a single custom function from the repository production and output it to the repository local:

  phenv customfunctions_to_py3 production/my_example_custom_function local

  Example output:

  Converted custom_function "my_example_custom_function"
   
  Successfully converted 1 custom_function

• Convert all custom functions in the repository production and output to the repository local.

  phenv customfunctions_to_py3 production local

  Example output:

  Converted custom_function "my_example_custom_function_01"
  Converted custom_function "my_example_custom_function_02"
   
  Successfully converted 2 custom_functions

playbooks_to_py3 options and examples

These are the options available when using the playbooks_to_py3 tool.

Argument	Required?	Description
repo/ repo2/playbook_name ...	Yes	A list of repositories or custom_functions to convert to Python 3. If repositories are listed, the tool will convert every playbook in the given repositories. Only provide the repository's name, not the full path to the repository.
repo	Yes	Output repository for converted playbooks.
-h, --help	No	Show the help message and exit the tool.
-d, --dry_run	No	Use this option to see which playbooks would be converted with the tool, without doing the conversion.
-s <SUFFIX>, --suffix <SUFFIX>	No	Specify a suffix to be added to the names of converted playbooks as part of the conversion process. If no suffix is specified the default is "_py3".
-v {0,1,2,3}, --verbosity {0,1,2,3}	No	Verbosity level: * 0 = minimal output * 1 = normal output * 2 = verbose output * 3 = very verbose output If you set --verbosity 3 you may see debug messages like this: Unsupported dict value of type: <class '_ast.NameConstant'>. Ignoring children and returning type. These messages can be safely ignored. They are only informational messages from the code parser, and do not indicate a problem with the playbook.

Command examples:

• Convert the playbook example_find_attacks in repository production to Python3 and save it to the repository local. This will make a copy in the repository local called "example_find_attacks_py3".

  phenv playbooks_to_py3 production/example_find_attacks local

  If your playbook's name includes spaces, you need to enclose the name of the playbook in quotation marks on the command line.

  phenv playbooks_to_py3 production/"example find attacks" local

  Example output:

  Converted playbook "example_find_attacks"
   
  Successfully converted 1 playbook

• Convert multiple playbooks in the repository "qatest" to Python 3 and save them to the repository local:

   phenv playbooks_to_py3 qatest/2_0_pb qatest/100_geolocates  local

  Example output:

  Converted playbook "2_0_pb"
  Converted playbook "100_geolocates"
   
  Successfully converted 2 playbooks 

• Use a custom suffix "_my_suffix" to create a Python 3 copy of playbook called audit_test_my_suffix:

  phenv playbooks_to_py3 qatest/audit_test local -s _my_suffix

  Example output:

  Converted playbook "audit_test"
   
  Successfully converted 1 playbook

Troubleshooting for converting Splunk SOAR or Splunk Phantom playbooks and custom functions from Python 2 to Python 3

If you run into issues when converting a playbook or custom function from Python 2 to Python 3, refer to the following tips:

• Resave the playbook.
  The conversion relies on how the playbook editor internally represents the playbook in JSON. When the playbook editor saves a playbook, it will update the format.
• Write code in the playbook block.
  If your new custom function validation fails, it might be validated as a legacy custom function if there is no code written in the playbook block. You might see errors such as: ValueError: not enough values to unpack (expected 2, got 1) or AssertionError: Excepted two sections after splitting CF block.
• Resolve issues with custom code
  Custom code is code you created and was not generated automatically by the Splunk SOAR Visual Playbook Editor. If you encounter conversion issues with custom code, you might need to try various methods to resolve conversion failures. Some potential resolution methods are described here.
  • Analyze the particular error provided by the conversion tool and attempt to resolve it within the code block itself. For example, you might comment out offending code, if possible.
  • Remove the custom code block from the original playbook, convert the playbook, and then re-write the custom code in Python3 in the new playbook.