Description - External

Job Description\: Senior Data Engineer (6–9 Years Experience)

Location\: Bengaluru

Employment Type\: Full-Time

Role Overview

We are seeking a Senior Data Engineer with 6–9 years of experience to design, build, and optimize scalable data platforms and pipelines using modern cloud-based data technologies. The ideal candidate will have strong expertise in Azure Databricks, PySpark, SQL, Unity Catalog, and workflow orchestration tools, along with experience in data governance, security, and performance optimization.

This role involves leading data engineering initiatives, enabling analytics and data science use cases, implementing governance frameworks, and ensuring high-performance, reliable data systems across enterprise environments.

Key Responsibilities

1. Data Architecture & Platform Engineering

- Design and implement scalable data architectures using Azure Databricks and modern lakehouse principles.

- Architect batch and streaming data pipelines for large-scale structured and unstructured datasets.

- Implement Databricks Delta Lake patterns including Medallion Architecture (Bronze, Silver, Gold layers).

- Define data modeling strategies for analytics and reporting workloads.

- Collaborate with architects and stakeholders to align platform design with business objectives.

2. Advanced Pipeline Engineering

- Design and manage Delta Live Tables (DLT) pipelines and Structured Streaming solutions for real-time analytics.

- Develop complex ETL/ELT pipelines using PySpark and SQL.

- Lead development of high-performance data workflows including advanced Spark tuning and optimization.

- Implement reusable frameworks and engineering standards for pipeline development.

- Optimize data processing workflows for performance, scalability, and cost efficiency.

3. Performance Engineering & Optimization

- Analyze system performance and identify bottlenecks across data pipelines and infrastructure layers.

- Optimize cluster configuration, resource utilization, shuffle behavior, caching, and partitioning strategies.

- Improve pipeline latency, throughput, and execution reliability.

- Monitor and optimize Spark workloads and SQL query performance.

- Lead performance tuning initiatives across large-scale data environments.

4. Governance & Security (Unity Catalog Administration)

- Act as the primary administrator for Databricks Unity Catalog ensuring fine-grained access control, centralized metadata management, and data lineage tracking.

- Architect and implement Delta Sharing protocols for secure cross-platform data exchange.

- Define and enforce data governance standards, policies, and compliance requirements.

- Develop and execute risk management strategies to ensure data integrity, privacy compliance, and system reliability.

- Implement data security controls including role-based access and audit mechanisms.

5. Workflow Automation & DevOps

- Automate workspace configuration, cluster provisioning, and deployment using Databricks CLI, Python scripting, and CI/CD pipelines.

- Design and manage workflow orchestration using Databricks Workflows, Azure Data Factory (ADF), or Apache Airflow.

- Ensure fault-tolerant execution of complex job dependencies.

- Implement monitoring, alerting, and automated recovery mechanisms.

- Support DevOps and Infrastructure-as-Code practices for data platforms.

6. Data Platform Reliability & Quality

- Implement automated data validation, monitoring, and quality checks.

- Maintain metadata, lineage, and auditability across datasets.

- Troubleshoot production incidents and perform root cause analysis.

- Ensure high availability, reliability, and maintainability of data pipelines.

7. Analytics, Data Science & Business Collaboration

- Partner with data analysts and data scientists to enable analytics and machine learning workflows.

- Support feature engineering pipelines and model data requirements.

- Execute complex Databricks SQL queries to derive actionable insights and audit pipeline performance.

- Provide optimized datasets for BI platforms such as Power BI, Tableau, or similar tools.

- Collaborate with business stakeholders to translate requirements into technical solutions.

8. Strategy, Solution Design & Leadership

- Partner with Data Architects and stakeholders to translate high-level requirements into technical blueprints.

- Participate in architecture discussions and technology decision-making.

- Mentor junior data engineers and provide technical leadership.

- Establish coding standards, documentation practices, and engineering best practices.

- Lead troubleshooting and resolution of complex production challenges.

Required Skills

- Strong expertise in Azure Databricks, PySpark, SQL, Unity Catalog, and Delta Lake architecture.

- Hands-on experience with Delta Live Tables (DLT), Structured Streaming, and Databricks Workflows.

- Experience with Azure Data Factory or Apache Airflow.

- Strong knowledge of distributed data processing and Spark performance tuning.

- Solid understanding of data modeling and data warehousing concepts.

- Experience with CI/CD and automation for data pipelines.

- Strong debugging, analytical, and problem-solving skills.

Preferred Qualifications

- Experience with streaming technologies such as Kafka or Event Hubs.

- Exposure to data warehousing platforms such as Synapse, Snowflake, or Redshift.

- Knowledge of Python beyond PySpark.

- Familiarity with BI tools such as Power BI or Tableau.

- Understanding of machine learning pipelines and MLOps practices.

- Azure or Databricks certifications are a plus.

Education Requirements

- Bachelor’s Degree in Engineering (Computer Science or Information Technology) is required.

- Candidates from other educational backgrounds may be considered if they have 100% relevant professional experience in data engineering.

Experience

- 6 to 9 years of experience in data engineering, big data development, or related roles.

- Proven experience working with enterprise cloud data platforms, preferably Azure.

Key Competencies

- Performance optimization mindset.

- Strong analytical and problem-solving ability.

- Technical leadership and mentoring capability.

- Collaboration and stakeholder communication skills.

- Ownership and accountability.

- Continuous learning and innovation orientation.