Senior Azure Data Engineer - PySpark/Synapse Analytics

  • Hyderabad
  • Vervenest Technologies Private Limited
Job Description : Senior Azure Data Engineer : Strong experience in ADF(Azure data factory), Azure SQL, Synapse, Spark/Databricks - Excellent written and verbal communication, intellectual curiosity, a passion to understand and solve problems, consulting & customer service. - Structured and conceptual mindset coupled with strong quantitative and analytical problem-solving aptitude. - Exceptional interpersonal and collaboration skills within a team environment - Py Spark - Parquet files - Big data and modern data lake - Demonstrable experience in enterprise level data platforms involving implementation of end-to-end data pipelines. - Hands-on experience with Azure - Experience with column-oriented database technologies (e.g., Synapse), NoSQL database technologies (e.g., DynamoDB, Cosmos DB, etc.) and traditional database systems (e.g., SQL Server, Oracle, MySQL) - Experience in data pipelines and solutions for both streaming and batch integrations using tools/frameworks like Azure Data Factory, Azure functions and Stream analytics. - Metadata definition and management via data catalogs, service catalogs, and stewardship tools such as Open Metadata, and Azure Purview - Test, plan, creation and test programming using automated testing frameworks, data validation and quality frameworks, and data lineage frameworks. - Data modeling, querying, and optimization for relational, NoSQL, timeseries, graph databases, data warehouses and data lakes - Data processing programming using SQL, Python, and similar tools. - Logical programming in Python, Spark, PySpark, Java, Javascript, and/or Scala - Cloud-native data platform design with a focus on streaming and event-driven architectures - Participate in integrated validation and analysis sessions of components and subsystems on production servers. - Data ingest, validation, and enrichment pipeline design and implementation. - SDLC optimization across workstreams within a solution - Bachelor's degree in computer science, Engineering, or related field (