Senior Data Engineer

Role Overview Alight Solutions is looking for a skilled Senior Data Engineer to join their Chennai team and take ownership of building and maintaining enterprise-grade data infrastructure. This is a full-time opportunity for a self-driven professional who can work independently, lead technical decisions, and mentor junior team members across hybrid data platforms spanning Hadoop and AWS cloud environments. Key Responsibilities - Design, build, and maintain high-volume ETL and ELT pipelines using Hadoop components such as HDFS, Hive, Spark, and Kafka alongside AWS services including Glue, EMR, Lambda, Step Functions, and Redshift - Develop distributed data processing solutions with PySpark and Spark SQL and implement reusable ingestion frameworks for both batch and streaming workloads - Optimize data workflows through partitioning, bucketing, compression, and efficient file formats like Parquet and ORC - Architect and support hybrid data lake environments using S3 and HDFS with consistent governance using tools such as Apache Atlas, Ranger, and AWS Lake Formation - Perform data profiling, create data flow diagrams, and carry out data modelling to meet reporting and analytics requirements - Orchestrate pipelines using Airflow, Control-M, Step Functions, or event-driven triggers - Own deployment, incident response, and post-incident reviews for production data systems, including Spark performance troubleshooting and cluster bottleneck resolution - Apply security best practices covering IAM, KMS, security groups, WAF, and secret management while ensuring regulatory compliance - Collaborate with business stakeholders, developers, QA, and product teams to deliver high-quality, analytics-ready datasets - Provide technical leadership through architecture reviews, code quality oversight, and mentoring of peers and junior engineers Required Qualifications - 5 to 8 years of hands-on experience with the Hadoop ecosystem including HDFS, Hive, Spark, YARN, and Kafka - Strong expertise in Scala, PySpark, Spark optimization, HiveQL, and distributed computing principles - Solid working knowledge of SQL in Hive and Impala environments - Good understanding of the AWS data stack including S3, Glue, EMR, Lambda, Kinesis, Redshift, and Step Functions - Proficiency in Python or Shell scripting for automation and pipeline management - Strong experience with CI/CD pipelines, GitHub, and Git version control - Expertise in ETL design, data warehousing concepts, and cloud architecture - Good understanding of data modelling approaches including star and snowflake schemas, partitioning strategies, and schema evolution - Hands-on experience with pipeline orchestration tools such as Airflow or Control-M - Familiarity with serverless patterns and containerization using Docker and ECS or EKS - Experience with monitoring, logging tools, and incident management practices - AWS certifications in Data Engineering, Solutions Architecture, or Development are an added advantage - Healthcare domain knowledge is a good-to-have Why Join Us Alight offers a comprehensive total rewards package including diverse health coverage, wellbeing programs, retirement benefits, generous leave policies, and ongoing education and training opportunities. You will be part of a globally growing organization committed to inclusion, innovation, and empowering its people to grow with purpose.

- 5 to 8 years of experience with Hadoop ecosystem including HDFS, Hive, Spark, YARN, and Kafka - Strong hands-on skills in Scala, PySpark, Spark optimization, and HiveQL - Experience with SQL in Hive and Impala - Good understanding of AWS data services including S3, Glue, EMR, Lambda, Kinesis, Redshift, and Step Functions - Proficiency in Python or Shell scripting - Strong experience with CI/CD, GitHub, and Git - Expertise in ETL, data warehousing, and cloud concepts - Knowledge of data modelling including star and snowflake schemas and schema evolution - Hands-on experience with Airflow, Control-M, or similar orchestration tools - Understanding of hybrid data lake architecture and governance tools like Atlas, Ranger, and Lake Formation - Familiarity with Docker and containerization using ECS or EKS - Knowledge of AWS networking including VPC, subnets, routing, and security groups - Strong analytical, problem-solving, and communication skills - Ability to work independently and guide junior team members

- Comprehensive health coverage options - Wellbeing and support programs - Retirement and financial security plans - Paid vacation and sick leave - Maternity, paternity, and adoption leave - Continuing education and training support - Several voluntary benefit options - Inclusive and diverse work culture - Reasonable accommodations for persons with disabilities