Data Engineer

Position Overview: We are seeking a highly skilled and motivated Data Engineer to join our dynamic team. As a Data Engineer, you will play a critical role in designing, building, and maintaining our data infrastructure and systems. You will collaborate closely with cross-functional teams to ensure the reliability, scalability, and efficiency of our data pipelines and analytics platforms.

Key Responsibilities:

• Data Pipeline Development: Design, develop, and maintain robust, scalable data pipelines to ingest, process, and transform large volumes of structured and unstructured data from various sources.
• Data Modeling and Architecture: Design and implement data models, schemas, and architectures that support the needs of our analytics, reporting, and machine learning initiatives.
• Data Integration: Integrate disparate data sources and systems, ensuring data consistency, accuracy, and integrity across the organization.
• Performance Optimization: Identify and implement optimizations to improve the performance, reliability, and efficiency of data processing and storage systems.
• Data Quality Assurance: Implement data quality checks, validation rules, and monitoring mechanisms to ensure data accuracy, completeness, and consistency.
• Tooling and Infrastructure: Evaluate, select, and implement appropriate tools, technologies, and infrastructure for data storage, processing, and analytics.
• Collaboration and Communication: Collaborate with cross-functional teams, including Data Scientists, Analysts, and Software Engineers, to understand data requirements and deliver data-driven solutions. Communicate effectively with stakeholders to present findings, share insights, and address challenges.
• Documentation and Best Practices: Document data pipelines, architectures, and processes. Establish and promote best practices for data engineering within the organization.

Qualifications:

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• Proven experience 10 years working as a Data Engineer or a similar role in a fast-paced, data-driven environment.
• Proficiency in programming languages such as Python, Java, or Scala, with a strong emphasis on data processing and manipulation.
• Experience with distributed computing frameworks such as Apache Hadoop, Spark, or Flink.
• Solid understanding of database systems, including SQL and NoSQL databases (e.g., PostgreSQL, MongoDB, Cassandra).
• Experience with cloud platforms and services, such as AWS, Google Cloud Platform, or Azure.
• Familiarity with data warehousing concepts and technologies (e.g., Snowflake, Redshift, BigQuery).
• Strong analytical and problem-solving skills, with a keen attention to detail.
• Excellent communication and collaboration skills, with the ability to work effectively in cross-functional teams.

Preferred Qualifications:

• Experience with containerization and orchestration technologies such as Docker and Kubernetes.
• Knowledge of streaming data processing technologies (e.g., Kafka, Spark Streaming).
• Experience with data visualization tools (e.g., Tableau, Power BI) and BI platforms.
• Familiarity with machine learning concepts and frameworks (e.g., TensorFlow, PyTorch).
• Certifications in relevant technologies (e.g., AWS Certified Big Data - Specialty, Google Professional Data Engineer).