Data Engineer - DataBricks

  • Hyderabad
  • The Ksquare Group

Requirements

  • Bachelor's degree in computer science, information technology, or a related field.
  • Prior experience with data lakes and data warehouses, working knowledge with dimensional modeling is must.
  • Demonstrate advanced understanding in data engineering practices, Including ETL/ELT, data integrations, reusable data pipelines,
  • data management methods.
  • Advanced Pyspark, Python, Spark SQL, Dimensional models, optimization in Azure, data bricks skills required.
  • Ability to service various business stakeholders, internal and external partners to support data demands.
  • Health care experience is desired with a data-centric motive is a plus.
  • 8+ years of experience in data engineering with Azure data bricks ecosystem.
  • Strong knowledge of Azure data services, including ADF, Databricks, Delta Lake, delta live tables, Power BI, and Azure SQL Server
  • Experience with data processing languages such as SQL, Python, Pyspark and Spark SQL
  • Experience with agile methodologies on Azure DevOps and continuous integration/continuous delivery (CI/CD)
  • Strong problem-solving and analytical skills; excellent communication and collaboration skills
  • Self-motivated and a self-starter with strong ability to multitask projects/tasks effectively.
  • Ability to work independently and collaborate effectively in a team environment.
  • Proven experience with cloud platforms, Microsoft Azure, Databricks environment.
  • Applicants must be currently authorized to work in the United States

Responsibilities

  • Design, develop, and implement Databricks infrastructure and applications.
  • Collaborate with data scientists, analysts, and stakeholders to gather requirements and understand business needs.
  • Develop and maintain scalable and efficient Databricks projects.
  • Optimize and tune Databricks to ensure high-performance data processing.
  • Implement security measures to ensure privacy and compliance with healthcare regulations.
  • Mentor and provide technical guidance to junior engineers.
  • Stay up-to-date with the latest Databricks trends, tools, and technologies.