Design, build, and maintain scalable data ingestion pipelines from diverse source systems (databases, APIs, files, streaming) into the Databricks lakehouse.
Implement and optimize the medallion architecture (bronze/silver/gold layers), ensuring clear data quality and transformation logic at each stage.
Develop data transformation and cleansing logic using PySpark, Spark SQL, and Delta Lake to produce curated, analytics-ready datasets.
Build and orchestrate ETL/ELT workflows using Databricks Workflows, Delta Live Tables, and/or orchestration tools (Airflow, ADF).
Implement data quality checks, validation rules, and monitoring/alerting across pipelines to ensure trustworthy data.
Manage schema evolution, partitioning, and performance tuning for large-scale Delta tables.
Collaborate with data governance teams to apply cataloging, access controls, and lineage tracking (Unity Catalog).
Partner with BI analysts, data scientists, and AI engineers to understand downstream data requirements and ensure gold-layer tables meet consumption needs.
Document data pipelines, transformation logic, and data models for maintainability and knowledge sharing.
Troubleshoot and resolve data pipeline failures, latency issues, and data quality incidents.
REQUIRED SKILLS AND EXPERIENCE
4+ years of experience in data engineering, with at least 2 years working hands-on with Databricks.
Strong hands-on experience implementing medallion architecture (bronze, silver, gold layers) for data ingestion and transformation.
Proficiency in PySpark and Spark SQL for large-scale data processing.
Solid experience with Delta Lake, including ACID transactions, schema evolution, time travel, and optimization (Z-ordering, compaction).
Experience building both batch and streaming ingestion pipelines.
Strong SQL skills and experience with data modeling (dimensional modeling, star schema) for analytics consumption.
Experience with orchestration tools such as Databricks Workflows, Delta Live Tables, Apache Airflow, or Azure Data Factory.
Familiarity with cloud data platforms (Azure Databricks, AWS, or GCP) and associated storage services (ADLS, S3).
Understanding of data governance concepts: cataloging, access control, and data lineage (Unity Catalog or equivalent).
PREFERRED / NICE-TO-HAVE SKILLS
Experience with Python for pipeline automation and testing.
Familiarity with CI/CD for data pipelines (Databricks Asset Bundles, DevOps pipelines).
Exposure to data quality frameworks (Great Expectations, Deequ).
Experience supporting downstream BI tools (Power BI, Tableau) or AI/ML use cases.
Databricks certifications (Data Engineer Associate/Professional) a plus.
Key Requirements & Skills
4+ years of experience in data engineering, with at least 2 years hands-on with Databricks
