Run a CI/CD workflow with a Databricks Asset Bundle and GitHub Actions (2024)

  • Documentation
  • Develop on Databricks
  • Developer tools and guidance
  • What are Databricks Asset Bundles?
  • Run a CI/CD workflow with a Databricks Asset Bundle and GitHub Actions

This article describes how to run a CI/CD (continuous integration/continuous deployment) workflow in GitHub with GitHub Actions and a Databricks Asset Bundle. See What are Databricks Asset Bundles?

You can use GitHub Actions along with Databricks CLI bundle commands to automate, customize, and run your CI/CD workflows from within your GitHub repositories.

You can add GitHub Actions YAML files such as the following to your repo’s .github/workflows directory. The following example GitHub Actions YAML file validates, deploys, and runs the specified job in the bundle within a pre-production target named “qa” as defined within a bundle configuration file. This example GitHub Actions YAML file relies on the following:

  • A bundle configuration file at the root of the repository, which is explicitly declared through the GitHub Actions YAML file’s setting working-directory: . (This setting can be omitted if the bundle configuration file is already at the root of the repository.) This bundle configuration file defines a Databricks workflow named my-job and a target named qa. See Databricks Asset Bundle configurations.

  • A GitHub secret named SP_TOKEN, representing the Databricks access token for a Databricks service principal that is associated with the Databricks workspace to which this bundle is being deployed and run. See Encrypted secrets.

# This workflow validates, deploys, and runs the specified bundle# within a pre-production target named "qa".name: "QA deployment"# Ensure that only a single job or workflow using the same concurrency group# runs at a time.concurrency: 1# Trigger this workflow whenever a pull request is opened against the repo's# main branch or an existing pull request's head branch is updated.on: pull_request: types: - opened - synchronize branches: - mainjobs: # Used by the "pipeline_update" job to deploy the bundle. # Bundle validation is automatically performed as part of this deployment. # If validation fails, this workflow fails. deploy: name: "Deploy bundle" runs-on: ubuntu-latest steps: # Check out this repo, so that this workflow can access it. - uses: actions/checkout@v3 # Download the Databricks CLI. # See https://github.com/databricks/setup-cli - uses: databricks/setup-cli@main # Deploy the bundle to the "qa" target as defined # in the bundle's settings file. - run: databricks bundle deploy working-directory: . env: DATABRICKS_TOKEN: ${{ secrets.SP_TOKEN }} DATABRICKS_BUNDLE_ENV: qa # Validate, deploy, and then run the bundle. pipeline_update: name: "Run pipeline update" runs-on: ubuntu-latest # Run the "deploy" job first. needs: - deploy steps: # Check out this repo, so that this workflow can access it. - uses: actions/checkout@v3 # Use the downloaded Databricks CLI. - uses: databricks/setup-cli@main # Run the Databricks workflow named "my-job" as defined in the # bundle that was just deployed. - run: databricks bundle run my-job --refresh-all working-directory: . env: DATABRICKS_TOKEN: ${{ secrets.SP_TOKEN }} DATABRICKS_BUNDLE_ENV: qa

The following GitHub Actions YAML file can exist in the same repo as the preceding file. This file validates, deploys, and runs the specified bundle within a production target named “prod” as defined within a bundle configuration file. This example GitHub Actions YAML file relies on the following:

  • A bundle configuration file at the root of the repository, which is explicitly declared through the GitHub Actions YAML file’s setting working-directory: . (This setting can be omitted if the bundle configuration file is already at the root of the repository.). This bundle configuration file defines a Databricks workflow named my-job and a target named prod. See Databricks Asset Bundle configurations.

  • A GitHub secret named SP_TOKEN, representing the Databricks access token for a Databricks service principal that is associated with the Databricks workspace to which this bundle is being deployed and run. See Encrypted secrets.

# This workflow validates, deploys, and runs the specified bundle# within a production target named "prod".name: "Production deployment"# Ensure that only a single job or workflow using the same concurrency group# runs at a time.concurrency: 1# Trigger this workflow whenever a pull request is pushed to the repo's# main branch.on: push: branches: - mainjobs: deploy: name: "Deploy bundle" runs-on: ubuntu-latest steps: # Check out this repo, so that this workflow can access it. - uses: actions/checkout@v3 # Download the Databricks CLI. # See https://github.com/databricks/setup-cli - uses: databricks/setup-cli@main # Deploy the bundle to the "prod" target as defined # in the bundle's settings file. - run: databricks bundle deploy working-directory: . env: DATABRICKS_TOKEN: ${{ secrets.SP_TOKEN }} DATABRICKS_BUNDLE_ENV: prod # Validate, deploy, and then run the bundle. pipeline_update: name: "Run pipeline update" runs-on: ubuntu-latest # Run the "deploy" job first. needs: - deploy steps: # Check out this repo, so that this workflow can access it. - uses: actions/checkout@v3 # Use the downloaded Databricks CLI. - uses: databricks/setup-cli@main # Run the Databricks workflow named "my-job" as defined in the # bundle that was just deployed. - run: databricks bundle run my-job --refresh-all working-directory: . env: DATABRICKS_TOKEN: ${{ secrets.SP_TOKEN }} DATABRICKS_BUNDLE_ENV: prod
Run a CI/CD workflow with a Databricks Asset Bundle and GitHub Actions (2024)

FAQs

What is the difference between GitHub Actions and GitHub workflow? ›

GitHub Actions helps you automate your software development workflows from within GitHub. You can deploy workflows in the same place where you store code and collaborate on pull requests and issues. In GitHub Actions, a workflow is an automated process that you set up in your GitHub repository.

Is GitHub Actions CI CD? ›

GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test, and deployment pipeline. You can create workflows that build and test every pull request to your repository, or deploy merged pull requests to production.

How to run a workflow in GitHub? ›

Running a workflow
  1. On GitHub.com, navigate to the main page of the repository.
  2. Under your repository name, click Actions.
  3. In the left sidebar, click the name of the workflow you want to run.
  4. Above the list of workflow runs, click the Run workflow button.

Can GitHub be used to perform CI CD? ›

Compared with other automation or CI/CD tools, GitHub Actions offers native capabilities right in your GitHub flow. It also makes it easy to leverage any of the 10,000+ pre-written and tested automations and CI/CD actions in the GitHub Marketplace as well as the ability to write your own with easy-to-use YAML files.

What is the difference between Jenkins and GitHub Actions? ›

Jenkins deployments generally involve self-hosting, where the users oversee servers in their data centers. On the other hand, GitHub Actions uses a hybrid cloud strategy. It hosts its own job runners and also lets users host their own job runners. Find out more about runners who are on their own.

What are the two types of actions in GitHub Actions? ›

Types of actions. You can build Docker container, JavaScript, and composite actions. Actions require a metadata file to define the inputs, outputs and main entrypoint for your action. The metadata filename must be either action.

Is GitHub Actions better than Azure DevOps? ›

GitHub is suitable for smaller projects and open-source collaborations, whereas Azure DevOps is more suitable for large-scale enterprise development.

What triggers GitHub Actions workflow? ›

This example uses the push event, so a workflow run is triggered every time someone pushes a change to the repository or merges a pull request. This is triggered by a push to every branch; for examples of syntax that runs only on pushes to specific branches, paths, or tags, see "Workflow syntax for GitHub Actions."

Which CI CD tool is best? ›

  1. Gitlab CI/CD. GitLab CI/CD is a software development platform that uses Continuous methodologies, such as Continuous Integration, Delivery, and Deployment. ...
  2. Azure DevOps. Azure DevOps is a cloud service that automatically builds and tests your code. ...
  3. GitHub Actions. ...
  4. CircleCI. ...
  5. Jenkins. ...
  6. Travis CI. ...
  7. Bitbucket Pipeline. ...
  8. TeamCity.

How to automate CI CD pipeline? ›

Here are five main steps to build a CI/CD pipeline with GitHub actions:
  1. Step #1 - Create or Select a Repository.
  2. Step #2 - Open GitHub Actions in Your Repository.
  3. Step #3 - Make Changes to your Code to Trigger your CI/CD Pipeline.
  4. Step #4 - Look at the Workflow Visualizer.
  5. View the Workflow:
  6. Step #5 - Check live logs.
Feb 25, 2024

What is the difference between GitHub Actions and workflows? ›

Workflows: automated processes that run on your repository; workflows can have many GitHub Actions. GitHub Actions: individual tasks; they can be written using Docker, JavaScript and now also shell scrips with the new Composite Run Steps; you can write your own actions or use an action someone else created.

Why use GitHub Actions? ›

GitHub Actions simplifies the process with Node and Docker integrations and allows you to specify which version you want to use and then connect your code to a target environment and application platform.

What is required workflow GitHub Actions? ›

GitHub Actions must be enabled for a repository in the organization's settings in order for required workflows to run. Once enabled at an organization-level, required workflows will run even when GitHub Actions is disabled in the repository's settings.

How do I create a CI CD pipeline in Jenkins with GitHub? ›

Configure Jenkins Job:
  1. Create a new Jenkins job or edit an existing one.
  2. In the pipeline definition, choose "Pipeline script from SCM" or a similar option.
  3. Select "Git" (1) as the SCM type and provide your GitHub repository URL.
  4. Specify the branch or branches you want to monitor for changes.
Apr 17, 2024

How do you deploy react app using CI CD pipeline? ›

CI/CD for React. js
  1. Select your Git repository. The first step is hooking up the repo with your JavaScript code. ...
  2. Add a new delivery pipeline. Enter the pipeline's name, select the trigger mode, and define the branch from which Buddy will fetch code: ...
  3. Add actions. ...
  4. Summary.
Feb 26, 2024

How do I create a CI CD pipeline in DevOps? ›

Typically building a CI/CD pipeline consists of the following phases/stages.
  1. Code: Checked into the repository.
  2. Build: Build is triggered and deployed in a test environment.
  3. Test: Automated tests are executed.
  4. Deploy: Code is deployed to stage, and production environments.
Sep 12, 2022

How to create CI CD pipeline in Jenkins step by step? ›

Tutorial: How to Create a CI/CD Pipeline with Jenkins
  1. Step 1: Download and Install Jenkins. The first step is to download and install Jenkins on your local machine. ...
  2. Step 2: Start and Configure Jenkins. ...
  3. Step 3: Create CI/CD Pipeline in Jenkins.

References

Top Articles
Latest Posts
Recommended Articles
Article information

Author: Manual Maggio

Last Updated:

Views: 5853

Rating: 4.9 / 5 (49 voted)

Reviews: 88% of readers found this page helpful

Author information

Name: Manual Maggio

Birthday: 1998-01-20

Address: 359 Kelvin Stream, Lake Eldonview, MT 33517-1242

Phone: +577037762465

Job: Product Hospitality Supervisor

Hobby: Gardening, Web surfing, Video gaming, Amateur radio, Flag Football, Reading, Table tennis

Introduction: My name is Manual Maggio, I am a thankful, tender, adventurous, delightful, fantastic, proud, graceful person who loves writing and wants to share my knowledge and understanding with you.