Data Engineer

• Total Experience Required: 7+ years

• Mode of Work: Work from Office

• Location: Hyderabad

• About the Role:

We are looking for a Data Engineer to own and extend the Databricks Lakehouse that powers every Bay6 AI product – Scout, Rebound, Guide, Pathways and Verify. You will work at the center of our data architecture, ingesting client CRM/SIS data (Slate, Banner, EAB Navigate, Stellic, Salesforce, etc.) in the context of complex and potentially fragmented tech stacks inside of Higher Education institutions. You will typically build out a medallion pipeline (bronze/silver/gold) and maintain a multi-tenant Unity Catalog isolation framework across a growing client portfolio. You will partner closely with the engineering team and the AI solutions team to ensure that every agent has clean, governed and well modeled data to work on.

This is a hands-on role in a small, fast-moving environment where you will also be writing pipeline code and making architectural decisions. You must be comfortable working with clients and also be able to articulate decisions or concerns eloquently.

Bay6 AI operates as a headless solution with no native UI. All agent output write into client CRMs. Data accuracy and traceability aren’t trivial concerns as they show up directly in a client’s production CRM. You will be of the few engineers who would deeply understand the full data path from a client’s SIS all the way to an agent decision, which means real ownership and visibility into product outcomes.

Must Have Skills

• Databricks - Spark (PySpark or Spark SQL, Delta Lake (medallion architecture), Unity Catalog (Multi-tenant)
• Python & SQL
• Experience owning multi-tenant data architecture
• Experience in Higher Education technology (CRM/SIS/Degree Audit platforms, etc.)
• Working knowledge of FERPA, GLBA, SOC 2
• Data Security and Governance
• GitHub, working with a CI/CD
• Architecture Reviews, decisions
• Evaluating emerging tools, models and frameworks

Nice to Have Skills

• Exposure to agentic AI systems or LLM-adjacent data pipelines.
• Experience with independently standing up net-new Unity Catalog structures for new tenants from scratch.

Job Responsibilities

• Design, build and maintain Databricks pipelines across bronze/silver/gold layers for multiple higher education clients unifying data across the institution, each within its own Unity Catalog.
• Create and implement strategies for moving client data from their CRM/SIS platforms via API calls or into drop zones (SFTP) for ingestion into Bay6 Databricks. Examples include Slate SFTP feeds, Banner derived warehouses, EAB Navigate 360 and Stellic, all core higher education platforms, with attention to schema drift and data quality.
• Own the thin, tenant scoped physical ingest versus a Lakehouse Federation decision for new client.
• Implement Data governance controls including Unity Catalog lineage and audit logging. Enforce PII de-identification gates and FERPA compliance across patterns.
• Be the resident expert on changing compliance requirements and plan as needed while keeping all the stakeholders informed.
• Partner with the AI Solutions team to translate agent data requirement into concrete schemas and feature tables.
• Support the of new institutional clients, each with their distinct CRM/SIS stacks and data dictionaries.
• Contribute and suggest reusability for components, especially with connectors so that patterns build for client can be reused across the portfolio.
• Work with engineering as the primary engineering/infrastructure partner on platform level Databricks and AWS decisions.

• Required Experience:

• At least 3 years of building production data pipelines, with hands-on Databricks experience (Spark, Delta Lake, Unity Catalog, etc.).
• Strong SQL and Python skills with familiarity working across batch ETL/ELT patterns.
• Experience with multi-tenant data architecture and a deep understanding of catalog isolation reasoning.
• Working knowledge of FERPA, GLBA, SOC 2 or equivalent regulated-data experience (higher education, healthcare, financial services).
• Experience with AWS (our deployment model is single-tenant private AWS per client).
• Comfortable working with a geographically dispersed team across the world.

• Educational Qualifications:

Bachelor's/Master's in Computer Science, AI, Engineering or related field.

• 7+ years total experience
• 3+ years building production data pipelines with hands-on Databricks experience (Spark, Delta Lake, Unity Catalog)
• Databricks - Spark (PySpark or Spark SQL), Delta Lake (medallion architecture), Unity Catalog (multi-tenant)
• Strong Python and SQL skills
• Experience owning multi-tenant data architecture and catalog isolation reasoning
• Experience in Higher Education technology (CRM/SIS/Degree Audit platforms)
• Working knowledge of FERPA, GLBA, SOC 2 or equivalent regulated-data experience
• Data Security and Governance experience
• GitHub, working with CI/CD
• Architecture Reviews and decisions
• Evaluating emerging tools, models and frameworks
• Experience with AWS (single-tenant private AWS per client)
• Comfortable working with a geographically dispersed team
• Bachelor's/Master's in Computer Science, AI, Engineering or related field
• Exposure to agentic AI systems or LLM-adjacent data pipelines
• Independently standing up net-new Unity Catalog structures for new tenants from scratch