Senior Data Engineer

About The Role:

Data Pipeline Development

• Design and implement end-to-end batch and streaming data pipelines using PySpark and Apache Spark.
• Build and manage data workflows on Databricks, including Delta Lake architecture and medallion (Bronze/Silver/Gold) patterns.
• Develop and maintain ETL/ELT processes to ingest, transform, and load data from diverse sources.
• Optimize Spark jobs for performance, cost-efficiency, and reliability at scale.

Cloud & Azure Services

• Manage data solutions using Azure services including Azure Data Lake Storage (ADLS Gen2), Azure Event Hubs, and Azure Key Vault.
• Ensure data security, access control, and compliance within Azure environments.
• Monitor and troubleshoot cloud infrastructure, pipelines, and data flows.

Data Modeling & SQL

• Design and implement dimensional and normalized data models for analytical and operational use cases.
• Write complex SQL queries for data transformation, validation, and reporting.
• Work with structured, semi-structured, and unstructured data across various storage formats (Parquet, Delta, JSON, CSV).

Required Qualifications:

• 4+ years of hands-on experience in data engineering roles.
• Strong expertise in Apache Spark and PySpark for large-scale data processing.
• Deep experience with Databricks platform — workspace setup, clusters, jobs, Delta Live Tables.
• Strong experience in building Azure Data Factory Pipeline.
• Proficiency with Azure cloud services for data engineering workloads.
• Advanced SQL skills including window functions, CTEs, and query optimization.
• Solid Python programming skills for data engineering, scripting, and automation.
• Experience with version control (Git) and CI/CD pipelines for data code deployments.
• Familiarity with data governance, data lineage, and metadata management.

Good to Have:

• Azure Synapse & Azure EventHub are good to have.

Qualification:

• Bachelor’s or Master’s degree in Computer Science, Engineering, or equivalent experience