Add databricks asset bundles docs

[noklam]

Description

Partially #3360

Development notes

• Move the old dbx docs to a separate page
• take over the go-to ide integration page with Databricks Asset bundle using kedro-databricks

Build: https://kedro--4265.org.readthedocs.build/en/4265/deployment/databricks/index.html

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

• Read the contributing guidelines
• Signed off each commit with a Developer Certificate of Origin (DCO)
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes
• Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

[noklam]

This is not new docs so no need to review, I move the original databricks_ide_developmenet.md page to a new file and use that file for DAB instead.

[merelcht]

Should we also update the index page https://docs.kedro.org/en/stable/deployment/databricks/index.html and mention asset bundles?

[astrojuanlu]

Also could be useful to clarify where the user can configure the cluster. For example, I have an existing cluster, but by default databricks bundle run will try to create a new one and fail with Unexpected user error while preparing the cluster for the job. Cause: PERMISSION_DENIED: You are not authorized to create clusters. Please contact your administrator.

[noklam]

@astrojuanlu I believe you are trying to run this with our internal Azure Databricks? From my understanding a job cluster is always created fresh.

Job Cluster has been designed to be unique for each run of a job. So, each run of your job would run against a new job cluster.

But it seems like it is also possible to run job on all-purpose cluster,, though it is not recommended. https://docs.databricks.com/en/jobs/compute.html#all-purpose.

Is the tricks here using the existing_cluster_id?
https://docs.databricks.com/en/dev-tools/bundles/settings.html#examples, pinging @JenspederM to see if you know something about it.

https://community.databricks.com/t5/data-engineering/databricks-job-scheduling-continuous-mode/td-p/38861#:~:text=Job%20Cluster%20has%20been%20designed,use%20a%20dedicated%20interactive%20cluster.

[astrojuanlu]

I remember having this problem the first time I tested the plugin and I forgot where did I write down the solution 😂

VS Code is showing me an error because the conf/base/databricks.yml is not actually a bundle definition file (it's a "bundle override configuration")

In fact I think I found a bug: after I tweak the config override, kedro databricks bundle will not update the resources/ files @JenspederM

[astrojuanlu]

Currently the best I could do is to specify the existing_cluster_id in the task overrides:

default:
    job_clusters:
        - job_cluster_key: default
          new_cluster:
              spark_version: ...
    tasks:
        - task_key: default
          existing_cluster_id: 1111-...  # <----

This seems to work.

[astrojuanlu]

Found a couple more dbx references

[JenspederM]

@astrojuanlu @noklam

Sorry for the late response. I see that you have resolved some of the issues by your self. You are correct that you should use existing_cluster_id if you do not have permissions to create job clusters.

Generally, the api for bundle resources follows the schema for creating a new job. See here.

[JenspederM]

@astrojuanlu To overwrite existing resources, you have to pass the --overwrite flag.

So you'd need to do kedro databricks bundle --overwrite

I could make this the default instead. I just thought that it would be safer to have it as a conscious decision.

[JenspederM]

Maybe the documentation for Kedro Databricks should state that it creates a valid asset bundle. So you can use the commands in Kedro Databricks or the Databricks cli if you want more control.

[noklam]

I have add an existing cluster id section and test it on databricks and added a few more screenshots for the UI flow.