Databricks

Overview

Databricks is a powerful data analytics and lakehouse platform that enables organizations to unify data engineering, data science, and business analytics at scale. Integrate Databricks with Superblocks to build internal tools and workflows that query, analyze, and update your Databricks data, empowering your team to automate processes, gain insights, and accelerate decision-making.

Setting up Databricks

Set up allowed IP Addresses

If you're using Superblocks Cloud, add Superblocks' IPs as allowed IPs for Databricks. Learn how to configure this in Databrick's guide on Configuring IP access lists.

If using the Superblocks On-Premise, make sure your agent can connect to Databricks.

Configure authentication

To access your Databricks warehouse, you'll need to authenticate using a Databricks account. Superblocks provides several different ways to authenticate. See the Databricks documentation below for how to configure your preferred authentication method.

Method	Description
Personal access token (PAT)	Use a short or long-lived access tokens for a user or service principal.
Machine-to-machine OAuth	Configure oauth client credentials for a service principal. Superblocks will exchange the client credentials with Databricks to retrieve a short-lived OAuth token.
OAuth token federation	Use OAuth tokens issued by your identity provider when users log in to Superblocks to authenticate with Databricks using the authenticated user's permissions

Add integration

1. In the web app, navigate to the Integrations page
2. Click Add integration
3. Search for Databricks and select it from the list of available integrations
4. Name the integration
5. Fill out the integration configuration as follows:
   Personal access token based connection

   Field	Required	Descritpion
   Host		Databricks instance host name
   Port		Port to use when connecting to your warehouse
   HTTP Path		HTTP path either to a DBSQL endpoint (e.g. /sql/1.0/endpoints/1234567890abcdef) or to a DBR interactive cluster (e.g. /sql/protocolv1/o/1234567890123456/1234-123456-slid123)
   Default catalog		An optional initial catalog to use
   Default schema		An optional initial schema to use
   Access token		Databricks personal access token

   Machine-to-machine OAuth

   Field	Required	Descritpion
   Host		Databricks instance host name
   Port		Port to use when connecting to your warehouse
   HTTP Path		HTTP path either to a DBSQL endpoint (e.g. /sql/1.0/endpoints/1234567890abcdef) or to a DBR interactive cluster (e.g. /sql/protocolv1/o/1234567890123456/1234-123456-slid123)
   Default catalog		An optional initial catalog to use
   Default schema		An optional initial schema to use
   OAuth client ID		Client ID associated with your service principal
   OAuth client secret		Client secret associated with your service principal

   OAuth token federation

   Field	Required	Descritpion
   Host		Databricks instance host name
   Port		Port to use when connecting to your warehouse
   HTTP Path		HTTP path either to a DBSQL endpoint (e.g. /sql/1.0/endpoints/1234567890abcdef) or to a DBR interactive cluster (e.g. /sql/protocolv1/o/1234567890123456/1234-123456-slid123)
   Default catalog		An optional initial catalog to use
   Default schema		An optional initial schema to use
   Subject token source		Select Login identity provider to use the access token issued to Superblocks when users log in via SSO, or provide a Static token.
   Token URL		Databricks token URL. Choose a URL format based on the federation policy you previously configured. Workspace identity federation https://<databricks-workspace-host>/oidc/v1/token Account-wide token federation https://<databricks-account-host>/oidc/accounts/<account-id>/v1/token
   Service Principal UUID		Only required if using Workload identity federation. The service principal client ID to authenticate as.

   warning

   Login identity provider token source is only supported for Enterprise organizations with OIDC-based Single Sign-On configured. If you're not sure how you SSO provider is configured, contact support for assistance.

6. Optionally, add more configurations to set credentials for different environments
7. Click Test Connection to check that Superblocks can connect
8. Click Create

success

Databricks connected

Now you can use Databricks in any Application, Workflow, or Scheduled Job

Using Databricks in APIs

Once your Databricks integration is created, you can start using Databricks in Superblocks APIs.

Snowflake is a SQL-based integration. Learn more in our guide on Writing SQL in Superblocks.

info

See the Basic CRUD guide for additional information in setting up a CRUD app

Troubleshooting

If you run into issues, first see our guide on Troubleshooting Database Integrations.

Common errors

There are several common errors you may see when using Databricks. The table below includes error messages, why they happen, and how to address them.

Error message	Why it's happening & Resolution
IntegrationTimeoutError: Failed to connect to warehouse. Connection timed out after <N>ms	Reason Databricks warehouses can be configured to automatically hibernate after a period of inactivity. This timeout usually occurs when the warehouse is hibernating and did not restart within the timeout threshold of the agent. Resolution Retry the API/test connection after the warehouse has restarted, or increase the automatic hibernation time for your Databricks warehouse.
IntegrationOAuthError: OAuth2 - "On-Behalf-Of Token Exchange" could not find identity provider token	Reason You've selected Login identity provider as the subject token source when using OAuth token federation, but you are not currently logged in to Superblocks using an OIDC-based Identity Provider. Resolution Reach out to support@superblocks.com for assistance configuring SSO or migrating your SSO configuration to OIDC.
Failed to process token: TOKEN_EXPIRED	Reason The access token issued to Superblocks when you logged in, or the static token you've provided has expired. Resolution If using Logged in identity provider, log out of and back into Superblocks. If using a static token, obtain a new federated JWT from your identity provider.
Failed to process token: TOKEN_INVALID (Ensure a valid federation policy has been configured)	Reason Your Databricks account either does not have a federation policy configured, or the subject_token being sent to Databricks by Superblocks does not satisfy the policy. This can happen if the token is not a valid JWT, or has a different aud or iss than configured in the Databricks federation policy. Resolution Make sure the aud and iss configured in Databricks are the same aud and iss your IdP uses when issuing tokens to Superblocks. Make sure the Databricks federation policy points to a valid JWKS URI. By default, Databricks uses the URI provided at <issuer-url>/.well-known/openid-configuration. You may need to change this if your IdP uses a non-default authorization server or does not support a /.well-known discovery URL.

If you are encountering an error that's not listed, or the provided steps are insufficient to resolve the error, please contact us at support@superblocks.com