This section describes how to create Data Engineering job clusters.
- In the left navigation pane, select Data Science & Engineering > Workflows.
- Select Create Job.
- The Create Job dialog opens.
- Enter the name for the job in Task name field (1).
- For Type (2), select Notebook.
- For Source (3), select Workspace.
- For Path (4), select Select notebook.
- In the Select notebook dialog, select either Shared or Users.
- Select Confirm.
- For Cluster (5), select New job cluster.
- The Job cluster creation page opens.
- For creating the job cluster, follow these steps:
- Rename the job cluster.
- For Policy, select Siemens_Job_ClusterPolicy.
- For Access mode, select Single user.
- In the Performance section, select the following options:
- Select a Databricks runtime version (1). Choose between Standard, ML and Light as required.
Note: The latest Databricks runtime version is selected by default.
- For Worker type (2), select the required value from the drop-down list.
- For Workers (3), type in the required value.
- For the Driver type (4), either choose Same as worker or pick from the available types.
- In the Tags section, add Tags as required by providing values in the Key and Value fields.
- In the Advanced options section, set the On-demand/spot composition as required.
If another number of workers is defined in the Performance section, this view might differ.
- Enable credential passthrough for user-level data access (2) option is deselected and cannot be edited.
- Select Confirm.
- The Job cluster creation page closes and returns to the Job creation page.
- Add Parameters as required.
- (Optional) In the Advanced options, add libraries, edit retry policies and notifications as required.
- Select Create.