Job Description:



We are seeking a Senior Data Engineer to support our client with data ingestion, data deduplication and data tagging for migration of a large-scale data environment into Databricks.



The ideal candidate will also bring hands-on expertise in end-to-end data pipeline management, including data ingestion from diverse sources, de-duplication of large-scale datasets, and data tagging to support downstream analytics, governance, and machine learning workflows.



Roles and Responsibilities (including but not limited to):

• Design, develop, and maintain scalable data ingestion pipelines to onboard structured, semi-structured, and unstructured data from batch and streaming sources (e.g., APIs, databases, flat files, message queues) into the Azure/Databricks environment.


• Implement de-duplication strategies across large-scale ...