Data Engineer – Python

Job Description

Key Responsibilities :

– Validate batch and streaming data pipelines for correctness, completeness, consistency, and timeliness.

– Create and maintain data quality checks (nulls, duplicates, schema drift, referential integrity).

– Verify business rules and transformations using SQL-based validations.

– Design, develop, and execute test strategies for Databricks-based data pipelines and analytics workflows.

– Establish data reconciliation and end-to-end traceability between source and downstream systems.

– Test ETL/ELT pipelines built using AWS services (Glue, Lambda, EMR, Step Functions).

– Validate transformations written in SQL and Python.

– Ensure correctness across data ingestion, enrichment, aggregation, and publishing layers.

– Test reprocessing, backfills, and historical data loads.

– Validate ETL/ELT processes built using Apache Spark (PySpark/Scala) in Databricks.

– Validate Kafka-based streaming pipelines for data integrity, ordering, and exactly-once/at-least-once semantics.

– Test producer and consumer logic, serialization formats (Avro, JSON, Protobuf).

– Validate topic configurations, partitions, offsets, retention policies, and schema changes.

– Simulate and test late arrivals, duplicate events, and consumer failures.

– Test data workflows using AWS S3, Glue, Lambda, Redshift, Athena, Kinesis, DynamoDB, or similar services.

– Validate IAM roles, permissions, and secure data access.

– Verify data lifecycle policies, encryption, and storage optimizations.

– Build and maintain automated data testing frameworks using Python.

– Develop reusable test utilities, fixtures, and synthetic datasets.

– Integrate data tests into CI/CD pipelines for pre-merge, scheduled, and post-deployment validation.

– Enable automated alerts for data quality failures.

– Validate pipeline performance for large-scale datasets.

– Test throughput, latency, and concurrency under peak workloads.

– Validate retry logic, error handling, idempotency, and recovery mechanisms.

– Perform soak, regression, and failover testing.

– Validate data pipeline metrics, logs, and alerts using CloudWatch, Prometheus, Grafana, or equivalent tools.

– Partner with teams to define data SLAs and SLOs.

– Participate in incident response, root-cause analysis, and postmortems related to data quality issues.

Required Qualifications :

– 5+ years of experience in Data Quality Engineering roles.

– Strong hands-on experience with SQL for complex data validation and analysis.

– Proficiency in Python for test automation and data validation.

– Experience testing data pipelines and ETL/ELT workflows.

– Hands-on experience with Kafka or other streaming platforms.

– Solid understanding of AWS data services (S3, Glue, Redshift, Lambda, Athena, etc.).

– Experience working with large datasets and distributed systems.

Are you interested in this position?
Apply by clicking on the “Apply Now” button below!
#GraphicDesignJobsOnline
#WebDesignRemoteJobs
#FreelanceGraphicDesigner
#WorkFromHomeDesignJobs
#OnlineWebDesignWork
#RemoteDesignOpportunities
#HireGraphicDesigners
#DigitalDesignCareers
# Dynamicbrand guru