This section describes how to create Data Engineering job clusters.

  1. In the left navigation pane, select Data Science & Engineering > Workflows.
  1. Select Create Job.
  • The Create Job dialog opens.
  1. Enter the name for the job in Task name field (1).
  1. For Type (2), select Notebook.
  1. For Source (3), select Workspace.
  1. For Path (4), select Select notebook.
  1. In the Select notebook dialog, select either Shared or Users.
  1. Select Confirm.
  1. For Cluster (5), select New job cluster.
  • The Job cluster creation page opens.
  1. For creating the job cluster, follow these steps:
  • Rename the job cluster.
  • For Policy, select Siemens_Job_ClusterPolicy.
  • For Access mode, select Single user.
  1. In the Performance section, select the following options:
  • Select a Databricks runtime version (1). Choose between Standard, ML and Light as required.
    Note: The latest Databricks runtime version is selected by default.
  • For Worker type (2), select the required value from the drop-down list.
  • For Workers (3), type in the required value.
  • For the Driver type (4), either choose Same as worker or pick from the available types.
  1. In the Tags section, add Tags as required by providing values in the Key and Value fields.
  1. In the Advanced options section, set the On-demand/spot composition as required.
    If another number of workers is defined in the Performance section, this view might differ.
  1. Enable credential passthrough for user-level data access (2) option is deselected and cannot be edited.
  1. Select Confirm.
  • The Job cluster creation page closes and returns to the Job creation page.
  1. Add Parameters as required.
  1. (Optional) In the Advanced options, add libraries, edit retry policies and notifications as required.
  1. Select Create.