Principal Data Engineer

Job Title - Principal Data Engineer

Job Location - Baner, Pune

About Company & Product

Our client is a pioneering Identity SecOps provider that arms identity and cybersecurity teams with end-to-end, real-time identity security posture management and threat detection. They are designed to secure identity and access in any cloud (e.g. AWS, Azure), as well as on-premises or data centres. In 2022, Gartner recognized our client as a Cool Vendor in Identity-First Security.

The product is heavily driven by data analytics as well as AI/ML (in the near future). The core idea is to analyze events & logs from enterprise systems & networks to enable the use cases we are trying to solve in the area of identity & access security.

Position Overview

As a Principal Data Engineer, you will have end-to-end ownership of the relevant software components, ensuring scalability, reliability, and optimal performance. You will actively contribute to the research, design and development of enterprise-grade, scalable data solutions that enable our organisation to extract maximum value from data assets. Collaboration with cross-functional teams, including QA, UI, DevOps, IT/Ops, and non-engineering stakeholders, is integral to this role.

Responsibilities

• Research, design, and develop components for our enterprise-level cybersecurity application.
• Design, develop, and deploy scalable and efficient data pipelines, ETL processes, and data integration solutions.
• Architect and implement data models, schemas, and database solutions to support our analytical and operational needs.
• Evaluate and select appropriate technologies, tools, and frameworks to enhance data processing, storage, and analytics capabilities.
• Drive initiatives to improve data quality, consistency, and reliability across the organization.
• Lead and contribute to the testing, validation and documentation of data solutions to ensure scalability, reliability, and maintainability.
• Identify areas of improvement, enhancements, and innovation in the software development process.
• Collaborate closely with the engineering team and non-engineering stakeholders throughout the development lifecycle.
• Thrive in a dynamic startup environment, contributing to a culture of innovation and excellence.
• Write clean, maintainable, and efficient code, adhering to best practices and coding standards.
• Participate in code reviews and provide constructive feedback to peers.
• Collaborate with the DevOps team to deploy and manage applications in production environments.
• Stay up-to-date with industry trends and advancements to ensure our technology stack remains current and competitive.

Requirements

• At least 14 years of hands-on experience in designing and developing enterprise-grade software, including 6+ years of experience in data engineering.
• Proven track record of designing and implementing large-scale data solutions.
• Strong hands-on experience with big data technologies, frameworks and distributed computing.
• Excellent problem-solving abilities, analytical thinking, and attention to detail.
• Solid proficiency in one or more programming languages such as Python, Java, or Ruby.
• Familiarity with cloud platforms (e.g., AWS, Azure, or Google Cloud) and containerization technologies (e.g., Docker, Kubernetes).
• Knowledge of RESTful API design and implementation.
• Experience with the ELK stack and NoSQL databases (e.g., MongoDB) is a significant advantage.
• Strong communication and collaboration skills.
• Ability to thrive in a fast-paced, dynamic work environment.
• Master’s or Bachelor's degree in Computer Science, Engineering, or a related field.