Convert playbooks or custom functions from Python 2 to Python 3

Python 2 reached end of life status on January 1, 2020. The Splunk Phantom platform switched to Python 3 for in the 4.9 release and with the 4.10 release new playbooks and custom functions default to Python 3.

Python 2 is supported for playbooks and custom functions. Splunk Phantom 4.10 maintains a runtime environment for both Python 2 and Python 3 playbooks and custom functions. Because Python 2 is "end of life," you should consider converting your playbooks and custom functions to Python 3.

Many deployments have significant numbers of playbooks and custom functions which were developed in Python 2 that need to be updated to Python 3. Splunk Phantom 4.10 and later contains two phenv commands to help you convert your custom functions and playbooks from Python 2 to Python 3.

• customfunctions_to_py3
• playbooks_to_py3

Best Practices for converting Splunk Phantom playbooks and custom functions from Python 2 to Python 3

Converting custom functions or playbooks to Python 3 requires some planning and careful work on both ends of the conversion process. Here are several best practices for doing the conversions.

• Create a full backup of your Splunk Phantom deployment. You will need this backup in order to restore your git repositories and their contents in the event that there is a problem during the conversion.
• Test each custom function and playbook before converting them. You must make sure the custom function or playbook is working correctly before proceeding. Converting incomplete, inoperable, or broken custom functions and playbooks can have unforeseen consequences.
• You should perform a dry run of each conversion using the --dry_run and --verbosity 3 options of the customfunctions_to_py3 and playbooks_to_py3 phenv commands. Pay close attention to the output from these commands. They provide helpful guidance on potential pitfalls during the conversion process.
• Test each playbook or custom function after converting them before using the converted playbooks in production environments.
• Convert custom functions separately from converting playbooks.
• Playbooks and custom functions that have been converted to Python 3 will be new copies, with the --suffix appended to their names. These new copies need to be be linked to the relevant custom functions, playbooks, and any sub-playbooks after conversion by a Splunk Phantom administrator. Edit the relevant playbooks in the Visual Playbook Editor to update any blocks that reference an incorrect name.

customfunctions_to_py3 options and examples

These are the options available when using the customfunctions_to_py3 tool.

Argument	Required?	Description
repo/ repo2/playbook_name ...	Yes	A list of repositories or custom_functions to convert to Python 3. If repositories are listed, the tool will convert every custom_function in the given repositories. Only provide the repository's name, not the full path to the repository.
repo	Yes	Output repository for converted custom_functions.
-h, --help	No	Show the help message and exit the tool.
-d, --dry_run	No	Use this option to see which custom_functions would be converted with the tool, without doing the conversion.
-s <SUFFIX>, --suffix <SUFFIX>	No	Specify a suffix to be added to the names of converted custom_functions as part of the conversion process. If no suffix is specified the default is "_py3".
-v {0,1,2,3}, --verbosity {0,1,2,3}	No	Verbosity level: * 0 = minimal output * 1 = normal output * 2 = verbose output * 3 = very verbose output If you set --verbosity 3 you may see debug messages like this: Unsupported dict value of type: <class '_ast.NameConstant'>. Ignoring children and returning type. These messages can be safely ignored. They are only informational messages from the code parser, and do not indicate a problem with the custom function.

Command examples

• Convert a single custom function from the repository production and output it to the repository local:

  phenv customfunctions_to_py3 production/my_example_custom_function local

  Example output:

  Converted custom_function "my_example_custom_function"
   
  Successfully converted 1 custom_function

• Convert all custom functions in the repository production and output to the repository local.

  phenv customfunctions_to_py3 production local

  Example output:

  Converted custom_function "my_example_custom_function_01"
  Converted custom_function "my_example_custom_function_02"
   
  Successfully converted 2 custom_functions

playbooks_to_py3 options and examples

These are the options available when using the playbooks_to_py3 tool.

Argument	Required?	Description
repo/ repo2/playbook_name ...	Yes	A list of repositories or custom_functions to convert to Python 3. If repositories are listed, the tool will convert every playbook in the given repositories. Only provide the repository's name, not the full path to the repository.
repo	Yes	Output repository for converted playbooks.
-h, --help	No	Show the help message and exit the tool.
-d, --dry_run	No	Use this option to see which playbooks would be converted with the tool, without doing the conversion.
-s <SUFFIX>, --suffix <SUFFIX>	No	Specify a suffix to be added to the names of converted playbooks as part of the conversion process. If no suffix is specified the default is "_py3".
-v {0,1,2,3}, --verbosity {0,1,2,3}	No	Verbosity level: * 0 = minimal output * 1 = normal output * 2 = verbose output * 3 = very verbose output If you set --verbosity 3 you may see debug messages like this: Unsupported dict value of type: <class '_ast.NameConstant'>. Ignoring children and returning type. These messages can be safely ignored. They are only informational messages from the code parser, and do not indicate a problem with the playbook.

Command examples:

• Convert the playbook example_find_attacks in repository production to Python3 and save it to the repository local. This will make a copy in the repository local called "example_find_attacks_py3".

  phenv playbooks_to_py3 production/example_find_attacks local

  Example output:

  Converted playbook "example_find_attacks"
   
  Successfully converted 1 playbook

• Convert multiple playbooks in the repository "qatest" to Python 3 and save them to the repository local:

   phenv playbooks_to_py3 qatest/2_0_pb qatest/100_geolocates  local

  Example output:

  Converted playbook "2_0_pb"
  Converted playbook "100_geolocates"
   
  Successfully converted 2 playbooks 

• Use a custom suffix "_my_suffix" to create a Python 3 copy of playbook called audit_test_my_suffix:

  phenv playbooks_to_py3 qatest/audit_test local -s _my_suffix

  Example output:

  Converted playbook "audit_test"
   
  Successfully converted 1 playbook

If a playbook fails to convert

Some older Python 2 playbooks may fail to convert to Python 3. When this happens, take the following actions.

1. Open the Python 2 playbook in the Visual Playbook Editor in Splunk Phantom 4.10.4 or later.
2. Click the Edit Playbook button.
3. Click the Save button.
4. Run the playbooks_to_py3 utility again.

Troubleshooting for converting Splunk Phantom playbooks and custom functions from Python 2 to Python 3

If you run into issues when converting a playbook or custom function from Python 2 to Python 3, refer to the following tips:

• Resave the playbook.
  The conversion relies on how the playbook editor internally represents the playbook in JSON. When the playbook editor saves a playbook, it will update the format.
• Write code in the playbook block.
  If your new custom function validation fails, it might be validated as a legacy custom function if there is no code written in the playbook block. You might see errors such as: ValueError: not enough values to unpack (expected 2, got 1) or AssertionError: Excepted two sections after splitting CF block.