Azure Data Engineer

Role Overview We are seeking an experienced Azure Data Engineer to design, develop, and maintain scalable data solutions on the Microsoft Azure platform. You will work with cutting-edge technologies including Azure Data Factory, Databricks, and Azure Synapse to build robust ELT pipelines that process and transform large volumes of data. This role offers the opportunity to work on enterprise-level data architecture projects with a focus on data quality, performance optimization, and cloud-native best practices. Key Responsibilities - Design and develop metadata-driven ELT pipelines using Azure Data Factory, Databricks Autoloader, and Azure Synapse - Implement data transformation logic using Apache Spark, PySpark, and SQL across distributed computing environments - Build and maintain Data Lake architecture on Azure Data Lake Storage, ensuring optimal performance and cost efficiency - Manage code repositories using Git and implement CI/CD processes with Azure DevOps for automated testing and deployment - Perform data quality checks, validation, and profiling to ensure data integrity and accuracy in data pipelines Required Qualifications - 6-10 years of professional experience in data engineering, ETL development, or cloud data solutions - Advanced proficiency with Azure services including ADLS, Azure Data Factory, Azure Databricks, and Azure Synapse - Strong hands-on experience with PySpark, SQL, and Python for data transformation and manipulation - Solid understanding of Databricks Delta Lake and Lakehouse architecture concepts - Expert knowledge of Delta Live Tables and metadata-driven pipeline development - Experience with Git version control and CI/CD pipelines using Azure DevOps - Any Graduate or Postgraduate degree Why Join Us Join Tata Consultancy Services and work on transformative cloud data projects with industry-leading technologies. We offer opportunities to develop expertise in modern data architecture while contributing to solutions that impact global enterprises.

- Azure Data Factory expertise - Azure Databricks proficiency - Azure Synapse experience - ADLS and Data Lake Storage knowledge - PySpark and Python programming - SQL and data transformation skills - Databricks Delta Lake and Lakehouse concepts - Delta Live Tables experience - ELT pipeline development - Spark distributed computing - Git and version control - Azure DevOps and CI/CD - Data quality and profiling - Data validation and cleansing - ETL debugging and troubleshooting - Cloud architecture design patterns - Metadata-driven pipeline design - Dynamic logic implementation - Notebook scripting in Databricks - Azure Synapse Analytics Activity