Project Overview

Our client is undertaking a strategic enterprise platform transformation focused on migrating data services and storage infrastructure from Microsoft Azure to VMware Cloud Foundation (VCF). The programme includes migration of databases, object storage, caching services, and AI-ready data platforms while ensuring zero data loss, high availability, and seamless business continuity.

The Senior Data Engineer will play a critical role in designing and executing the migration of enterprise data services from Azure-native platforms to VMware-based cloud infrastructure. The role combines data engineering, database migration, storage architecture, and AI platform enablement, working closely with infrastructure, platform, application, and AI teams.

Responsibilities

·       Lead migration of enterprise data services from Azure platforms to VMware Cloud Foundation (VCF) and VMware Data Services Manager (DSM).

·       Design and implement end-to-end data migration strategies, including dual-write, cutover, reconciliation, validation, and rollback approaches.

·       Lead migration of Cosmos DB workloads and other Azure-hosted data services to VMware DSM.

·       Deploy, configure, and optimise Redis Operator as the platform caching layer.

·       Implement and manage MinIO and Ceph-based object storage solutions.

·       Deploy and manage Qdrant or similar vector databases supporting AI and semantic search use cases.

·       Define and execute comprehensive data validation, reconciliation, and consistency verification processes.

·       Configure and maintain backup, disaster recovery, failover, and business continuity mechanisms.

·       Monitor and optimise database, storage, and data platform performance, scalability, and reliability.

·       Support ingestion and preparation of data for AI and vector database workloads.

·       Collaborate with platform engineering, infrastructure, security, and AI teams to design scalable storage and data architectures.

·       Develop automation and operational tooling for migration, monitoring, and platform management.

·       Participate in architecture reviews, sprint planning, backlog refinement, daily standups, and retrospectives within an Agile delivery model.

·       Provide technical leadership and guidance on cloud migration and data platform best practices.

·       Ensure compliance with data governance, security, and operational standards throughout migration activities.

Requirements

·       Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or a related field.

·       Strong hands-on experience with Microsoft Azure data platforms and cloud services.

·       Strong hands-on experience with VMware Cloud Foundation (VCF), VMware Data Services Manager (DSM), and VMware-based cloud platforms.

·       Experience migrating databases, storage platforms, and data services between cloud environments.

·       5+ years of experience in Data Engineering, Database Engineering, Data Platform Engineering, or Infrastructure Engineering roles.

·       Strong experience with distributed database technologies and DBaaS platforms.

·       Hands-on experience with NoSQL databases, including Cosmos DB or equivalent technologies.

·       Proven experience designing and executing large-scale data migrations with minimal downtime.

·       Strong understanding of migration patterns including dual-write, cutover, rollback, and reconciliation strategies.

·       Experience with Redis clusters and enterprise caching architectures.

·       Experience implementing and managing object storage platforms such as MinIO, Ceph, or S3-compatible solutions.

·       Experience with vector databases such as Qdrant, Pinecone, Weaviate, Milvus, or similar technologies.

·       Strong knowledge of data validation, reconciliation, and data quality frameworks.

·       Experience with backup, disaster recovery, and failover strategies for distributed systems.

·       Strong understanding of performance optimisation for large-scale data platforms.

·       Strong scripting and automation skills using Python, Bash, or similar languages.

·       Excellent analytical and problem-solving capabilities.

·       Strong communication, stakeholder management, and collaboration skills.

·       Experience working in Agile/Scrum environments.

Nice to Have

·       Experience supporting AI platforms and Retrieval-Augmented Generation (RAG) architectures.

·       Familiarity with Databricks, Spark, Delta Lake, or modern data platform technologies.

·       Experience with AI search solutions and vector search architectures.

·       Knowledge of Azure networking and cloud infrastructure architecture.

·       Knowledge of data governance, security, and compliance frameworks.

·       Experience with observability and monitoring platforms such as Prometheus, Grafana, or ELK.

·       Relevant Azure, VMware, Kubernetes, or cloud certifications.

Role Ownership in the Project

·       Owns the Azure to VMware Cloud data services migration workstream.

·       Leads migration of databases, storage services, caching platforms, and related data infrastructure.

·       Owns implementation of migration, validation, reconciliation, and disaster recovery processes.

·       Supports Phase 3 – AI Data Layer Enablement (Vector Databases and Storage Services).

·       Accountable for data integrity, migration quality, platform reliability, and operational readiness.

·       Acts as the technical lead for VMware DSM, storage services, and vector database implementation, ensuring a secure, scalable, and zero-data-loss migration.

Start Date: ASAP
Duration: PoC project for 4 months with possibilities for extension

Remote vs Onsite: Remote

HackerRank Challenge: No

US Hours Overlap Needed: Minimum availability until 6:00 PM CET. Until 7:00 PM CET preferred.

‍