Connect Pigment With Databricks

This article explains how to connect a Databricks Table with Pigment using the Delta Share solution.

As with all native Pigment integrations, you must first configure a connection on the Integration Library page. Once configured, the connection can then be used by any Application authorized as a data source.

Overview

This native integration with Databricks uses Delta Sharing (Azure, AWS, GCP). Delta Sharing is an open protocol developed by Databricks for secure data sharing with other organizations regardless of the computing platforms they use. To set up the integration, first create and configure a Share on Databricks. Then grant access to Pigment to enable data sharing.

Next, configure a connection in the Pigment integrations panel and select the Applications that can use this connection.

Step 1: Set up the Databricks Share and Recipient

First, prepare a Databricks Share containing the data you want to make available through Delta Sharing.

1. If a Share is not yet available, follow these steps to create a new Share in Databricks. Select all the data objects you need to make available for Pigment in the Add data assets step.

2. When the Share is ready for use, you will need to Grant recipient access to a Recipient which will be used by Pigment. In the Add recipient panel in Databricks, select Open as Recipient type and Token and authentication method.

4. You can copy and share the activation link generated.

5. Select the link and download the credential file.

6. Select the recipient you created and grant them access to the Share you have configured.

ℹ️ Note

To enhance security, Databricks supports restricting token usage to defined IP ranges. Specify the following IP addresses in the IP access section of the Recipient configured for Pigment.

• 35.242.251.111

• 34.145.54.113

• 34.163.209.119

• 35.202.142.12

Step 2: Set up the Connector in Pigment

⚠️ Important

You must be a Pigment Workspace Admin to configure the connector.

When you have your Databricks credentials set up, you can set up the connector in Pigment.

To do this:

1. In your Workspace, go to Integrations.

2. Select + Add on the Databricks Delta Sharing integration and fill in the information about the connection, this includes the Name, Application access, and Delta Sharing profile

   • Name. Enter the name of your connection.

   • Application access. Select the Applications to allow using this connection.

   • Delta Sharing profile. Copy and paste the contents of the Delta Sharing profile JSON file downloaded at the previous step.

ℹ️ Note

If the token expires or needs to be rotated, edit the connection and replace the expired Delta Share profile. For more information, see Edit Pigment Integrations.

Step 3: Load Databricks data into Pigment

After the Databricks Delta Sharing connection is configured, access an Application linked to this connection and open the Import Data panel for the relevant object.

For example, if you are importing data into a Transaction List, you would open the List and select Import and Download and then Import.

To do this:

1. Select Import, then select Integration.

2. Select the Databricks Delta Share connection you just configured.

ℹ️ Note

If you don’t see the connection, go to Integrations and confirm that the Application is on the Application Access list.

3. From the dropdown menus, select the Share, Schema and Table you want to load in Pigment. Pigment will automatically discover these fields based on the Recipient and Share configuration you’ve set up. If you think a Table is missing, check the Share and Recipient configuration with your Databricks Admin.

4. Select Start Import, the selected table will load in Pigment within seconds.

⚠️ Important

The Databricks Delta Sharing integration only allows loading Tables smaller than 500MB in size.