Data Engineer – Cloud Platform

Job Description

Key Responsibilities :

– Design, develop, and maintain scalable data pipelines and ETL processes on Google Cloud Platform.

– Work with GCP services such as BigQuery, Dataflow, Dataproc, Dataplex, Data Fusion, Cloud SQL, Airflow, and Cloud Storage.

– Develop and maintain data ingestion and transformation workflows using Python and PySpark.

– Build and manage data orchestration workflows using Apache Airflow.

– Use Terraform and Tekton for infrastructure automation and CI/CD workflows.

– Implement data transfer utilities and batch data processing jobs.

– Write shell scripts to develop ad-hoc jobs for data import/export processes.

– Collaborate with cross-functional teams in an Agile development environment.

– Manage and maintain code using Git or other version control systems.

– Integrate Kafka-based streaming pipelines and API-based data integrations when required.

Required Skills :

– Strong experience with Google Cloud Platform (GCP).

– Hands-on experience with the following GCP services :

1. BigQuery

2. Dataflow

3. Dataproc

4. Data Fusion

5. Dataplex

6. Cloud SQL

7. Cloud Storage

8. Airflow

– Experience with Terraform and Tekton for infrastructure and deployment automation.

– Strong programming experience in Python and PySpark.

– Experience with PostgreSQL or other relational databases.

– Experience working with Data Transfer Utilities.

– Experience with Git or other version control systems.

– Knowledge of API development and integration.

– Strong experience writing shell scripts for automation and data processing jobs.

– Experience working in Agile frameworks.

Preferred Skills :

– Experience with Apache Kafka or Confluent Kafka.

– Exposure to Big Data technologies and distributed data processing systems.

Education :

– Bachelor’s Degree in Computer Science, Information Technology, or related field.

Are you interested in this position?

Apply by clicking on the “Apply Now” button below!

#GraphicDesignJobsOnline

#WebDesignRemoteJobs #FreelanceGraphicDesigner #WorkFromHomeDesignJobs #OnlineWebDesignWork #RemoteDesignOpportunities #HireGraphicDesigners #DigitalDesignCareers# Dynamicbrand guru