Migrate event hubs data from the Splunk Add-on for Microsoft Cloud Services to Data Manager

You can migrate event hubs from the Splunk Add-on for Microsoft Cloud Services (MSCS) to Data Manager using the Migrate from Splunk Add-on for Microsoft Cloud Services version 5.4 or higher option in Data Manager. Migration allows you to automate and accelerate the data ingestion process.

The migration process consists of two parts:

• In the MSCS, export event hubs to a JSON file template. It ensures appropriate formatting of data so that it can be ingested to Data Manager.
• In Data Manager, migrate data inputs. It includes ingesting complex event hub data and reviewing created data inputs. Data inputs are created from modular inputs. Migration does not affect modular inputs in the MSCS.

Migrating the same event hub data twice may corrupt data.

To learn how to manage modular inputs in the MSCS to reduce the risk of migrating the same event hub data twice, see Validate data flow.

Azure Event Hubs Connector - automating ingestion

Azure Event Hubs connector is a key component in automating data ingestion in Data Manager. It automatically detects partitions and matches them with data collectors.

Azure Event Hubs collector provides the following capabilities:

Capability	Description
Automatic load balancing across partitions	The connector uses the Azure Event Hubs Go SDK libraries to balance load across partitions.
Flexible authentication	The connector performs the following tasks: Supports secure authentication methods and simplifies rotation of credentials, that is a periodical replacement of credentials with the new ones to increase security. Maps event attributes, like a body and properties, directly to log records for compatibility and correct formatting.

Prerequisites to exporting event hub data from the Splunk Add-on for Microsoft Cloud Services (MSCS)

To be able to use the export feature, you have upgraded the MSCS to version 5.4.0 or higher. Download the latest version of the Splunk Add-on for Microsoft Cloud Services from Splunkbase.

Export event hub data from the Splunk Add-on for Microsoft Cloud Services (MSCS) to a JSON file

In the MSCS, export event hubs by taking the following steps:

1. Navigate to Configuration > Export tab.
2. In the Input status column, set the status of the event hubs that you want to export to Inactive. Deactivating these inputs prevents data loss.
3. In the Health status column, check that the status of event hubs that you want to export is Ready.

   Configure all modular inputs associated with an event hub under the same Inputs Data Manager (IDM). Otherwise, the MSCS does not include them.

   To learn how to work with IDM, see Work with Inputs Data Manager in the Splunk Cloud Platform Admin Manual.

4. In the Ready for export column, select one or more event hubs with the Ready health status to export.
   After selecting the event hubs, on the Inputs tab, in the Name column, the Migration in progress status appears next to the event hubs selected for export. You can't activate inputs with this status until they are selected for export.
5. Select the Export button to download a JSON file that contains the point-in-time state of the event hubs.

Migrate event hubs from the JSON file to Data Manager

In Data Manager, migrate event hubs from the exported JSON file by following these steps:

1. Navigate to the Azure Data Onboarding page by taking the following steps:
   1. On the Data Management page, select the New Data Input button.
   2. On the Choose Cloud Data Platform page, select Microsoft Azure as a source to onboard and select Next.
   3. On the Azure Data Onboarding page, select Azure Event Hub.
2. Under Select your ingestion method, select Migrate from Splunk Add-on for Microsoft Cloud Services version 5.4 or higher and select Next.
3. On the Prerequisites for Onboarding Azure Data page, select Next.
4. On the Connecting Splunk Add-on for Microsoft Cloud Services to Data Manager page, take these steps:
   1. Upload the JSON file that you exported from the Splunk Add-on for Microsoft Cloud Services.
   2. In the Client Details table, enter the Client Secret to authenticate. Confirm your entry by selecting the checkmark icon.
   3. After authenticating successfully, select Next.
5. On the Migrate Azure Event Hub Data Information page, take these steps:
   1. Review the list of inputs identified by Data Manager.
      

      If needed, edit data inputs by selecting Edit in the Actions column. At this point, you can edit Data Input Name and Azure Client ID. After the migration completes, these settings are not available for editing.

   2. Select the Complete Migration of n Data Inputs button, where n indicates a number of data inputs.
6. On the Review Data Inputs page, review the information and check that the data inputs have the No issue green status.
7. Select the Finish setup and monitor configuration button.

On the Data Management page, you can see the information about initializing data inputs. It may take some time for the setup process to complete.

Validate data flow

In Data Manager, validate that the migrated event hub data flows correctly.

If you are sure that data flows correctly, wait for several days after completing the migration and then consider deleting modular inputs in the Splunk Add-on for Microsoft Cloud Services (MSCS). It helps you avoid migrating the same event hub data twice which may corrupt data.

To delete a modular input in the MSCS, take these steps:

1. Select Configuration > Export tab.
2. In the row containing the migrated event hub data, select the Delete icon.