Job Description
About the Role
We are seeking an experienced Senior Data Engineer to design, develop and support enterprise-scale cloud data platforms and modern data integration solutions. The successful candidate will play a key role in delivering scalable, secure and high-performing data engineering solutions that enable business intelligence, advanced analytics, Artificial Intelligence (AI) and Machine Learning initiatives.
The role requires strong expertise in cloud data platforms, ETL/ELT development, API integration, enterprise data governance, solution design, and modern data engineering technologies.
Key Responsibilities
- Design, develop and maintain enterprise-scale ETL and ELT data pipelines for structured, semi-structured and unstructured data.
- Design and implement cloud-based data integration solutions using Informatica Intelligent Data Management Cloud (IDMC), Azure Data Factory (ADF) and other enterprise integration technologies.
- Build scalable data platforms using Snowflake and Databricks.
- Develop robust integrations with enterprise applications and third-party systems using REST APIs and API integration frameworks.
- Design and implement scalable batch, near real-time and streaming data ingestion solutions.
- Develop automation scripts using Python, Shell scripting, SQL and other scripting technologies.
- Develop and maintain enterprise orchestration workflows using tools such as Apache Airflow, Azure Data Factory Pipelines, Control-M or equivalent workload automation platforms.
- Design enterprise data models supporting reporting, analytics, AI and Machine Learning workloads.
- Produce high-quality solution designs, technical specifications and architecture documentation.
- Implement enterprise Data Governance capabilities including metadata management, data lineage, data quality, data cataloguing and master data management.
- Ensure compliance with enterprise security, privacy and regulatory requirements.
- Optimise data platform performance, scalability, reliability and operational efficiency.
- Design and implement CI/CD pipelines and DevOps practices for data engineering solutions.
- Collaborate with Solution Architects, Business Analysts, Developers and business stakeholders to translate business requirements into scalable technical solutions.
- Support AI-enabled initiatives including Generative AI, predictive analytics, Retrieval-Augmented Generation (RAG) and machine learning data pipelines.
- Evaluate emerging technologies and recommend improvements to enterprise data architecture.
- Mentor junior team members and promote engineering best practices.
Essential Skills and Experience
Applicants must possess:
- Bachelor’s degree or higher in Computer Science, Information Technology, Engineering or a related discipline.
- Minimum five (5) years of professional experience in Data Engineering, Cloud Data Engineering or Enterprise Data Integration.
- Demonstrated experience designing and developing enterprise ETL and ELT solutions.
- Strong hands-on experience with:
- Informatica Intelligent Data Management Cloud (IDMC)
- Azure Data Factory (ADF)
- Snowflake
- Databricks
- Python
- Shell scripting
- Advanced SQL
- REST API integration
- Data modelling and data warehousing
- Enterprise orchestration tools such as Apache Airflow, Azure Data Factory Pipelines, Control-M or Autosys
- Experience designing cloud-native data platforms on Microsoft Azure.
- Strong understanding of enterprise data architecture principles.
- Experience implementing Data Governance, Metadata Management, Data Lineage, Data Quality and Master Data Management frameworks.
- Experience building secure, scalable and high-performance cloud data solutions.
- Experience implementing CI/CD pipelines, Git version control and DevOps practices.
- Strong analytical, problem-solving and troubleshooting skills.
- Excellent written, verbal and stakeholder communication skills.
Desirable Skills
The following experience will be highly regarded:
- Microsoft Fabric
- Azure Synapse Analytics
- Apache Spark and Spark SQL
- Delta Lake
- Kafka or other streaming technologies
- dbt
- Azure Event Hub
- Azure Functions
- Docker and Kubernetes
- Terraform or Infrastructure as Code
- Azure OpenAI Service, AWS Bedrock or Google Vertex AI
- Vector databases and Retrieval-Augmented Generation (RAG)
- AI and Machine Learning pipeline development
- Agile delivery methodologies including Scrum or SAFe
- Enterprise Solution Architecture frameworks
Personal Attributes
The successful candidate will demonstrate:
- Strong solution design and architecture capabilities.
- Ability to lead technical discussions and mentor engineering teams.
- Excellent stakeholder engagement and communication skills.
- Strong analytical and problem-solving abilities.
- Commitment to delivering high-quality, scalable and secure solutions.
- Ability to manage multiple priorities in a fast-paced environment.
- Continuous learning mindset with a strong interest in emerging cloud and AI technologies.
Are you interested in this position?
Apply by clicking on the “Apply Now” button below!
#GraphicDesignJobsOnline
#WebDesignRemoteJobs
#FreelanceGraphicDesigner
#WorkFromHomeDesignJobs
#OnlineWebDesignWork
#RemoteDesignOpportunities
#HireGraphicDesigners
#DigitalDesignCareers
# Dynamicbrand guru