Observability Platform Engineer

  • Bengaluru
  • Stonex
Overview The Observability Engineer takes on the critical role of utilizing a ETL platform (extraction, transformation, and loading) to effectively manage and streamline log data flows. Their responsibilities encompass ensuring data integrity, security, and compliance. They collaborate closely with IT teams, engaging in tasks such as designing, implementing, and troubleshooting ETL platform configurations tailored to meet the distinct data processing needs of financial services organizations. The Observability Engineer plays a pivotal role in the design, construction, and ongoing maintenance of data pipelines and infrastructure. These elements are essential in facilitating data-driven decision-making throughout the organization. Your expertise in observability engineering technologies and best practices will prove invaluable in our quest to ensure data accuracy, availability, and security while also lending support to various data-driven initiatives. Responsibilities Design and deploy ETL solutions to optimize log data ingestion, transformation, and routing.Create custom pipelines and configurations tailored to the specific requirements of financial applications and systems.Develop and maintain log data transformation rules and functions to extract relevant information and enrich log events.Configure ETL platform to intelligently route log data to appropriate destinations, such as SIEM (Security Information and Event Management) systems, data lakes, or real-time monitoring tools.Implement filtering mechanisms to reduce noise and focus on critical events.Aid in the development of log metrics, including both KPI’s and KRI’s.Monitor and fine-tune ETL platform configurations to improve data processing efficiency and reduce latency.Monitor and optimize data pipelines for performance and scalability, identifying and resolving bottlenecks as needed.Implement security best practices to protect sensitive data within log streams.Ensure compliance with industry regulations, such as PCI DSS, GDPR, and SEC requirements, by managing and auditing log data appropriately.Create documentation and provide training for IT and operational teams on ETL platform best practices and usage.Collaborate with cross-functional teams, including IT, DevOps, and security, to integrate ETL platform seamlessly into the financial services infrastructure.Aid in the development of a log bill back model and proper reporting around that model.Work with vendors and support teams to resolve technical issues. Qualifications Proven experience in designing, implementing, and managing ETL solutions or with big data technologies (e.g., Hadoop, Spark) and data lake architectures.Strong understanding of log data processing, log management, and log analytics.Proficiency in scripting and programming languages (e.g., Python, JavaScript) for custom log data transformations.Knowledge of security industry standards (NIST, PCI, ISO), best practices (ITIL), regulations (SOX, FINRA), news, and other relevant cybersecurity information.Individual must be able to perform with minimal supervision of routine duties; must demonstrate ability to solve practical problems and deal with a variety of situations where only limited standardization exists.Communicate technical issues and concepts both verbally and in writing to audiences of varying technical backgrounds. Ability to help gather business requirements and align them to security solutions.Working a minimum of 3 days in the office.Willingness to work on US or London time zone hours.Expert organizational skills are a must as well as the ability to work successfully within a team.1-3+ years of overall technology experience. Prefer a diverse background including security, networking, compute, storage, and endpoint.1-3+ years of experience as a data engineer or log engineer.Education / Certification requirements: in at least one of or working towards the following:Associate Degree or Bachelor’s degree in information security, Information Assurance, Information Systems, or equivalent experience.SANS related certifications (GSEC, GCIA, GCED, GCIH etc.).Other field appropriate certifications may be considered.Physical requirements/Working conditions:Climate controlled office environment.Minimal physical requirements other than occasional light lifting of boxed materials.Dynamic, time-sensitive environment.