Enrich custom content using the ShowcaseInfo.json file

The ShowcaseInfo.json is the most important file in Splunk Security Essentials (SSE).

Architecture

Almost every SSE dashboard calls the REST endpoint /services/SSEShowcaseInfo or searches Splunk for the | sseanalytics command, which also calls the REST endpoint but on the back end. For more information, see Use the schemas in Splunk Security Essentials.

Endpoint

web.conf:

[expose:SSEShowcaseInfo]
methods = GET,POST
pattern = SSEShowcaseInfo

restmap.conf:

[script:sseshowcaseinfo]
match                 = /SSEShowcaseInfo
script                = generateShowcaseInfo.py
scripttype            = persist
handler               = generateShowcaseInfo.ShowcaseInfo
requireAuthentication = true
output_modes          = json
passPayload           = true
passHttpHeaders       = true
passHttpCookies       = true

Call process

When you call the generateShowcaseInfo.py script, the call gathers configuration parameters from the URL and then follows this process to access the ShowcaseInfo.json file, add custom content, and enrich the content with files and lookups.

1. Initialize the service object used for API calls, pulling in the URL for splunkd in case the KV store API fails and must perform a GET from the API directly.
2. Check the contents of the essentials_updates.conf file to determine which apps are activated and which are deactivated.
3. Check for MITRE ATT&CK updates. If more than one day has passed since the last check, the MITRE updates if an update is available by stashing the update in the KV store, then the pullJSON file, which allows the KV store to store large JSON files, renders the updated information.
4. Pull simple information with no modifications: bookmark, local_search_mappings, and data_source_check.
5. Pull information with slight modification: custom_content is cleaned as described in the Partner Integration topic. The data_inventory_products fields are inserted into an array of scores per DSC so they can combine at the product level, where there might be multiple DSCs. The DSC-to-productId match is created.
6. Pull in core files: ShowcaseInfo.json and each search builder file. If naming conflicts arise, SSE silently overrides.
7. Pull in supporting files: data_inventory.json as well as MITRE ATT&CK and MITRE Pre-ATT&CK, which as not grabbed directly because the pullJSON call always delivers some MITRE content.
8. Pull in custom content from the ShowcaseInfo.json file as if it comes from the JSON file itself.
9. Parse MITRE content into relevant objects, which relate to each other through relationship objects and relationship_type uses. All these objects have an ID, which maps to the source_ref and target_ref fields: x-mitre-tactic as tactics, attack-pattern as techniques, and intrusion-set as groups.
10. Clear out invalid characters.
11. Enrich and process.
12. Enrich with searches, looking for matches between information in the search builder JSON file and the ShowcaseInfo.json file. If a match appears, it's added as ShowcaseInfo['summaries']['my_summary_id']['examples'][NUM]['showcase'] = my_search_builder_obj.
13. Create local search mappings, checking each piece of content against the list of savedsearch.conf mappings in local_search_mappings and then setting search_title to match. A known limitation is that only one search can occur for each piece of SSE content. If you have multiple searches that are similar to the object you need to create custom content in SSE to complete mapping.
14. Enrich data availability, looking up each piece of content rendered in the Data Availability matrix to add the fields data_available and data_available_numeric.
15. Enrich MITRE variables already parsed with this information: the tactic and technique name, a technique description, the MITRE matrix it came from, such as MITRE ATT&CK or MITRE Pre-ATT&CK, the threat groups, and search keywords.
16. Clean up missing fields. This process can involve creating empty strings and changing strings to "None."
17. Exclude deactivated channels by iterating through all the content in the ShowcaseInfo.json file while checking for channels in the exclusions list (channel_exclusion[…]). If some content need to be excluded, it's pulled. You can override this part of the process by including ignoreChannelExclusion=true in your request.
18. Export content to the sse_content_exported file to map savedsearch.conf names to the metadata in SSE. When you run ShowcaseInfo, it checks whether the content is correct in the configuration and fixes incorrect content. You can see the fields in that export in the fields = […] line. The fields are mirrored in the collections.conf and transforms.conf files.
19. Optionally minify the ShowcaseInfo.json file by adding fields=mini to the URL. Doing this strips all fields not specified in mini_fields, drastically reducing the amount of data transferred.
20. Return data in JSON format.

Error handling

To see errors from the ShowcaseInfo.json file, look for elements added into the SSE output, including debug and throwError. The debug object contains a variety of debugging information, in particular, timing checks and exceptions. throwError denotes a critical error.

From the Security Contents page, the ShowcaseInfo.json file response appears in the window object so you can open the JavaScript console and run this command:

console.log("throwError Status", ShowcaseInfo.throwError);
console.log("debug Status", ShowcaseInfo.debug);

This is an example of the debug status:

[
    "Stage -5 Time Check:9.05990600586e-06",
    "Stage -4 Time Check:8.20159912109e-05",
    "Stage -3 Time Check:9.20295715332e-05",
    "Stage -4 Time Check:0.000141143798828",
    {
        "localecheck": ""
    },
    "Not going cached! Prepare for a long ride.",
    "Stage -1 Time Check:0.000144004821777",
    {
        "channel_exclusion": {
            "mitrepreattack": false,
            "custom": false,
            "Splunk_App_for_Enterprise_Security": false,
            "mitreattack": false,
            "Enterprise_Security_Content_Update": false,
            "Splunk_Security_Essentials": false,
            "Splunk_User_Behavior_Analytics": false
        },
        "override": false,
        "msg": "Final Channel Exclusion"
    },
    "Stage 0 Time Check:0.106211185455",
    "Stage 1 Time Check:1.20219802856",
    "Stage 2 Time Check:1.27426409721",
    {
        "store": "bookmark",
        "message": "I got a kvstore request"
    },
    "Stage 3 Time Check:1.29301118851",
    {
        "store": "local_search_mappings",
        "message": "I got a kvstore request"
    },
    "... abridged ...",
    "Stage 25 Time Check:1.62445807457"
]

Access through JavaScript

From a Splunk platform dashboard in the browser, run this command to get the resulting object:

require(['json!' + $C['SPLUNKD_PATH'] + '/services/SSEShowcaseInfo?bust=' + Math.random()], function(showcase){
    window.debug_showcase = showcase; 
    console.log("Got Showcase", showcase); 
})

Access through Splunk search

To see the output of all summaries, use the | sseanalytics command. That command breaks out key fields separately and converts inline-multivalue pipe-separated fields into native Splunk platform multi-value fields. You can also request full JSON output, identical to what appears when performing a REST call:

| sseanalytics include_json=true | search channel=ButtercupLabs | table id summaries *