Sr. Data Engineer

Data Engineer – Data Platform

• Location: Bangalore

• Experience: 5+ Years

Role Overview

We are looking for a

hands-on Data Engineer

to design, develop, and optimize scalable

data platforms and data pipelines

. The ideal candidate will have strong expertise in

Python, SQL, PySpark/Spark, Databricks

, and modern data engineering practices.

Candidates should have hands-on experience with

at least cloud platform – AWS, Azure, or GCP

. Experience with

BigQuery, Lakehouse architecture, Delta Lake, Airflow, Kafka, and CDC

will be highly valued.

Key Responsibilities

• Design, develop, and maintain scalable

ETL/ELT data pipelines

using

Python, SQL, PySpark, and Apache Spark

.

• Build and optimize data processing solutions using

Databricks

and cloud-native data services.

• Develop

batch and real-time data ingestion pipelines

using technologies such as

Airflow, Kafka, CDC

, and other streaming/integration frameworks.

• Implement modern

Lakehouse and Medallion Architecture

using

Delta Lake

or similar technologies.

• Develop reliable data pipelines for structured and semi-structured data from multiple sources.
• Implement

data quality, validation, reconciliation, monitoring, logging, and error-handling

mechanisms.

• Optimize Spark jobs and data pipelines for

performance, scalability, reliability, and cloud cost efficiency

.

• Work with cloud storage, compute, databases, and data services across

AWS, Azure, or GCP

.

• Implement engineering best practices including

Git, CI/CD, unit testing, integration testing, and code reviews

.

• Troubleshoot data pipeline failures and resolve performance and data-quality issues.
• Collaborate with

Data Architects, Solution Architects, Analysts, Developers, and Business Stakeholders

to understand requirements and deliver scalable data solutions.

• Contribute to technical design, documentation, coding standards, and continuous improvement of the data platform.

Must-Have Skills

5+ years of hands-on experience in Data Engineering

.

• Strong programming skills in

Python

.

• Strong expertise in

SQL

and data manipulation.

• Hands-on experience with

PySpark / Apache Spark

.

• Strong hands-on experience with

Databricks

.

• Good understanding of

ETL/ELT concepts, data pipelines, data modeling, and distributed data processing

.

• Experience with

Lakehouse Architecture, Medallion Architecture, and Delta Lake

.

• Hands-on experience with

at least cloud platform: AWS / Azure / GCP

.

• Experience with or more cloud data/storage services such as:

• AWS: S3, Glue, EMR, Redshift

• Azure: ADLS, Data Factory, Synapse

• GCP: GCS, BigQuery, Dataflow

• Exposure to

Apache Airflow

or other workflow orchestration tools.

• Experience with

Kafka, CDC, or real-time/streaming data pipelines

.

• Good understanding of

data quality, monitoring, testing, and pipeline optimization

.

• Experience with

Git and CI/CD pipelines

.

• Strong analytical, troubleshooting, and problem-solving skills.
• Good communication and collaboration skills.

Good-to-Have Skills

Apache Iceberg / BigLake

dbt

Databricks Unity Catalog

Kafka / Structured Streaming

Change Data Capture (CDC)

tools

Infrastructure as Code (IaC)

– Terraform or similar

• Cloud security,

IAM, networking, and access management

• Cloud

cost optimization

• Data governance and metadata management
• Experience working with

data warehouses and dimensional data modeling

Preferred Technical Stack

• 5+ years of hands-on experience in Data Engineering
• Strong programming skills in Python
• Strong expertise in SQL and data manipulation
• Hands-on experience with PySpark / Apache Spark
• Strong hands-on experience with Databricks
• Good understanding of ETL/ELT concepts, data pipelines, data modeling, and distributed data processing
• Experience with Lakehouse Architecture, Medallion Architecture, and Delta Lake
• Hands-on experience with at least cloud platform: AWS / Azure / GCP
• Experience with or more cloud data/storage services such as AWS (S3, Glue, EMR, Redshift), Azure (ADLS, Data Factory, Synapse), or GCP (GCS, BigQuery, Dataflow)
• Exposure to Apache Airflow or other workflow orchestration tools
• Experience with Kafka, CDC, or real-time/streaming data pipelines
• Good understanding of data quality, monitoring, testing, and pipeline optimization
• Experience with Git and CI/CD pipelines
• Strong analytical, troubleshooting, and problem-solving skills
• Good communication and collaboration skills
• Apache Iceberg / BigLake
• dbt
• Databricks Unity Catalog
• Kafka / Structured Streaming
• Change Data Capture (CDC) tools
• Infrastructure as Code (IaC) – Terraform or similar
• Cloud security, IAM, networking, and access management
• Cloud cost optimization
• Data governance and metadata management
• Experience working with data warehouses and dimensional data modeling