Senior Data Engineer

  • Pune
  • Intellias

We are seeking a Senior Data Engineer to join our growing team. We are looking for an experienced data developer comfortable designing, developing & maintaining complex data pipelines, with a focus on flexibility and developer experience. We are adopting the Flyte platform to help us take our document processing pipeline to the next level and looking for someone to help us develop and scale this out.

Project Overview :

The company is a fast growing start-up working to revolutionize the quality of commercial property risk data and how it can be leveraged. It was founded in 2018 by tech and finance entrepreneurial veterans with previous leadership at RMS and Apple. They are an internationally diverse organization headquartered in San Francisco with additional offices in New York, India and Netherlands.

The company serves many of the world's largest property brokers and their clients, representing over 500 of the world's largest and most dynamic commercial property portfolios to improve their data and better represent their risks.

Requirements:

  • Priority skillset: Python, Fivetran, Postgres, Snowflake, Flyte (data pipelines).
  • 5+ years of experience as a hands on data engineer with strong proficiency in Python, Pandas, etc.
  • 2+ years of experience working with data workflow platforms such as Apache Airflow, Slurm or Flyte etc.
  • Experience operating both internal and production data pipelines.
  • Experience collaborating with platform and machine learning engineers.
  • Experience with processing PDF and Excel files.
  • SQL / Analytics experience with Snowflake.
  • Experience working in cloud based environments, e.g. AWS, GCP, Azure.

Bonus Points:

  • Experience building data pipelines for the insurance industry.
  • Experience with Flyte.
  • Experience with AWS compute and EKS.
  • Experience with a compiled language such as Go, Java, C# etc

Responsibilities:

  • Develop, design, create, modify, and/or test document processing pipelines or systems to support our Machine Learning and Analytics capabilities.
  • Collaborate with Data Science, Product Managers and Software Engineers to enable the Product Support team to deliver compelling user-facing features.
  • Ensure quality of data through their flow, implement guardrails, health checks and alerts
  • Driving the continuous improvement of our existing codebase by participating in code reviews, refactoring legacy code, and measuring code coverage and performance

Key Outcomes you are responsible for:

  • Maintaining and improving accuracy and coverage metrics of the document processing pipeline.
  • In collaboration with product initiative owners support Monthly Active User growth through developing and supporting customer first user experiences.
  • Maintain and improve quality, performance, architecture, and security of existing functionality.
  • Bring your expertise to mentor and grow not just junior engineers but the whole squad.
  • Key Metrics & Results to measure your success:
  • Full coverage and high accuracy metrics for all input document types.
  • Pipeline performance & reliability.
  • Frequent iterative pipeline updates to address new document types and accuracy improvements.