Elevate data processing capabilities as a Senior Data Engineer with a focus on Python and PySpark. Engage in designing optimized data models and processing pipelines.
We are searching for a Senior Data Engineer skilled in Python and PySpark to enhance our data architecture. This role involves building ETL/ELT solutions, optimizing data workflows, and ensuring effective real-time processing. Your contributions will optimize our analytics and data governance practices.
Key Responsibilities:
• Design and implement efficient ETL/ELT data pipelines
• Use Apache Spark for distributed data processing tasks
• Optimize Spark jobs for enhanced performance
• Collaborate on data integration from diverse sources
• Conduct data quality assessments and monitoring
Requirements:
• Proven experience programming in Python
• Deep knowledge of PySpark and SQL techniques
• Background in cloud services like AWS and Azure
• Familiarity with data governance and monitoring to...