Job Description

Overview:

We are looking for a skilled and experienced Senior Engineer specializing in Databricks to join our team. As a Senior Engineer, you will play a pivotal role in designing, developing, and optimizing data solutions leveraging the Databricks platform. You will collaborate closely with cross-functional teams to architect scalable and efficient data pipelines, implement advanced analytics solutions, and ensure high performance and reliability of our data infrastructure.

Responsibilities:

1.    Design and develop robust data pipelines and workflows using Databricks Delta, Apache Spark, and related technologies to support data ingestion, processing, and consumption.

2.    Architect scalable solutions for advanced analytics, machine learning, and big data processing on the Databricks platform.

3.    Collaborate with data scientists, analysts, and business stakeholders to understand requirements and translate them into technical designs and implementations.

4.    Optimize and fine-tune Databricks jobs and Spark applications for performance and efficiency.

5.    Implement best practices for data governance, security, and compliance on the Databricks platform.

6.    Troubleshoot and resolve technical issues related to Databricks infrastructure and data pipelines.

7.    Stay current with Databricks product updates, new features, and industry trends, and evaluate their applicability to enhance our solutions.

Requirements:

Bachelor’s degree in Computer Science, Engineering, or a related field; advanced degree preferred.

Proven experience (5+ years) as a Senior Engineer working extensively with Databricks for data engineering and analytics.

Strong proficiency in Databricks architecture, Apache Spark, Databricks Delta, and related technologies.

Experience with cloud platforms such as AWS, Azure, or GCP and their data services (e.g., S3, Blob Storage, BigQuery).

Solid understanding of data warehousing, ETL processes, and data modeling concepts.

Ability to collaborate effectively with cross-functional teams and communicate technical concepts to non-technical stakeholders.

Strong problem-solving skills and ability to troubleshoot complex issues independently.

Databricks Certification (preferred).