IntermediateAzure Databricks Data Engineer project · about 14 hours

Batch data pipeline on Azure Databricks

Ingest raw files, transform them and load a reporting table on a schedule.

The real-world brief

A retailer wants daily sales files turned into clean reporting tables.

What you will build

  • Raw files in object storage
  • A transformation job
  • A scheduled pipeline
  • Quality checks

Skills you will practise

IngestionTransformationOrchestrationData quality

Tools

Azure DatabricksDelta LakeAuto LoaderDatabricks Workflows

Project tasks

The detailed steps, checklists and trainer review open in the SOTT student portal for enrolled students.

1Ingest Steps in portal
2Transform Steps in portal
3Schedule Steps in portal
4Check Steps in portal

How companies use Azure Databricks Data Engineer

Retail: One view of sales. Sales, stock and online data sat in separate systems. Pipelines load all sources into one warehouse with cleaned, modelled tables. Everyone uses the same numbers.

Logistics: Live delivery tracking. Delivery status updated once a day. Streaming events land in the data platform within seconds and feed dashboards. Customer service sees each parcel in real time.

Healthcare: Governed patient data. Analysts copied sensitive data into spreadsheets. Access controls, masking and lineage in a governed platform. Analysts get the data they need without uncontrolled copies.

Questions

Can I add the Batch data pipeline on Azure Databricks project to my resume?

Yes. Once your trainer approves it, list it under Projects with the tools you used, one line on the problem it solves and a link to your repository and demo video.

How long does this project take?

About 14 hours of hands-on work, and it fits best after week 5 of the Azure Databricks Data Engineer (DP-750) course.

Where do I get the step-by-step tasks?

Enrolled students get every task with steps, "done when" checks, the submission form and a trainer review in the SOTT student portal.