DeepLight AI is a specialist AI and data consultancy with extensive experience implementing intelligent enterprise systems across multiple industries, with particular depth in financial services and banking. Our team combines deep expertise in data science, statistical modeling, AI/ML technologies, workflow automation, and systems integration with a practical understanding of complex business operations.
Responsibilities - Data Ingestion Development
- Building and implementing AWS Glue jobs for Bronze layer ingestion using defined standards and templates
- Implementing correct loading methods based on source requirements (CDC, full load, delta, snapshot)
- Designing and executing historical loading mechanisms to bring legacy data into the Lakehouse
- Performance Optimisation
- Optimising Glue job performance (DPU allocation, parallelization, partitioning) according to best practices
- Collaborating wit...