Convert RE2 regular expressions to PCRE2 regular expressions

Convert your RE2 regular expressions to Perl Compatible Regular Expressions 2 (PCRE2).

Starting on March 5, 2025, all new pipelines will use PCRE2 syntax by default, with no option to use RE2. All existing pipelines can continue using RE2, but migration to PCRE2 is recommended. Pipelines with the following regular expression functions need to be migrated:

• Like
• Replace
• Rex
• Rename
• Match
• Mvfind
• Searchmatch

Starting on June 5, 2025, RE2 support ends completely. All pipelines (new and existing) must use PCRE2 syntax.

Steps to convert RE2 regular expressions to PCRE2 regular expressions

1. Log into the Splunk Cloud Platform or Splunk Enterprise deployment used for routing test data.
2. Verify your Edge Processor node's CPU/Memory status, in order to make sure that your node has the bandwidth to route test data. Log into your Data Management deployment, and then navigate to Edge Processors, and then Manage Instances, and then See CPU / Memory.
3. In your deployment, create two new indexes for testing. For example, name the indexes re2pipelines and pcre2pipelines.
   To learn more about creating an index, see the Create an index and generate sample events topic in the Splunk Developer Guide.
4. Log into your Data Management console.
5. Refresh your system connections so that the Splunk software knows that the index exists. This can be done by selecting the Settings icon () and then selecting System connections.
6. Open a pipeline that contains regular expressions.
7. In the SPL2 statement of your pipeline, copy and paste /* RE2 copy */ directly above the $pipeline = | from $source line in your statement.
8. In the SPL2 statement of your pipeline, directly below the $pipeline = | from $source line, paste the following text:

   | eval initial_event_hash = md5(_raw)
   

9. Before the | into $destination line of your SPL2 pipeline statement, paste the following SPL2 template text:

   | thru [
      | eval final_event_hash = md5(tostring(tojson()))
      | eval index="re2pipelines"
      | eval sourcetype="regexmigration"
      | into $destination2
   ]
   

10. Navigate to the Actions section of the pipeline builder, select Send data to $destination2, and then Select a destination…", and select the Splunk platform indexer that you used to create the indexes in step 1.
11. Click the Save pipeline button, and deploy your pipeline.

    Do not check the permanently convert to PCRE2 upon save checkbox.

12. After saving and deploying your updated pipeline, perform the following changes to your SPL2 statement:
    1. Change the /* RE2 copy */ line to /* PCRE2 copy */.
    2. Change the following lines from:

       | thru [
          | eval final_event_hash = md5(tostring(tojson()))
          | eval index="re2pipelines"
          | eval sourcetype="regexmigration"
          | into $destination2
       ]
       | into $destination;
       

       
       to:
       

       | eval final_event_hash = md5(tostring(tojson()))
       | eval index="pcre2pipelines"
       | eval sourcetype="regexmigration"
       | into $destination;
       

13. In the Actions section of the pipeline builder, select Send data to $destination, and then Select a destination…, and select the same Splunk indexer that you used to create the indexes in step 1.
14. Click the Save pipeline dropdown and Save as pipeline…. Name the copy PCRE2 Copy of <pipeline of interest's name> with a description, if desired, and click Save.
    By default, the copy and any new pipelines will operate with PCRE2 regexes at runtime. Any existing pipelines (for example, your pipeline of interest), will operate with RE2 regexes at runtime.
15. Deploy the PCRE2 Copy of <pipeline of interest's name> to the same Edge Processor that is operating the original pipeline (the same as in step 6). If necessary, perform the following best practices:
    • Scale out your Edge Processor deployment before doing this if your deployment's I/O is approaching limitations.
    • Scale up your Edge Processor deployment before doing this if your CPU usage is approaching a maximum.
16. On the search head linked to the test indexer where your test indexes were created, use the Splunk software's search function to query both indexes at the same time.

    (index="re2pipelines" OR index="pcrepipelines") | eventstats count as init_count by initial_event_hash | where init_count=2 | stats count as pair_count by final_event_hash, initial_event_hash | table initial_event_hash, final_event_hash, pair_count | where pair_count=1

17. final_event_hash is the hash generated after all your business logic. initial_event_hash is generated before all your business logic. In an ideal case, all final_event_hash values should emit two copies. This means your RE2-pipeline and your PCRE2-pipeline will execute identical transformations on your data. If there are events that don't emit 2 copies, it means that the PCRE2-based pipeline emitted different results than your RE2-based pipeline. You can examine the raw event details in search to investigate for differences, by replacing the initial_event_hash value in the query below with your initial_event_hash values that only emit a count of 1 in the previous query:

    (index = "re2pipelines" OR index = "pcre2pipelines") | where initial_event_hash = 123456 | table * | transpose

18. Review your regular expressions to verify that they conform to syntax and use case. For more information, see the About Splunk regular expressions topic in the SPL2 Search Manual.
19. Once the data is identical in both locations, unapply, and delete the PCRE Copy of <pipeline of interest's name> pipeline.
20. Open the original pipeline, and perform the following tasks:
    1. Remove the logic that was created in steps 7 and 8.
    2. Click Save Pipeline and check the convert to PCRE2 upon save checkbox. This will permanently switch regex runtimes with no option to revert.
    3. Click Save.
21. Repeat for other pipelines that are still on the RE2 regex runtime. This can be done by navigating to the Pipelines page, and then navigating to each pipeline that contains the RE2 icon ().
22. If desired, delete all test data from the test indexes index=pcre2pipelines, and index=re2pipelines. To learn more, see the Remove indexes and indexed data topic in the Managing Indexers and Clusters of Indexers section of the Splunk Enterprise manual.
23. In three months, all new and existing pipelines will run on PCRE2 regex runtime by default, with no option to revert.