Department: Data & Analytics Product Group, FCSD Tech
Experience: 8+ years
Position Overview:
The Data & Analytics Product Group under FCSD Tech is seeking a highly dynamic, motivated, and results-oriented Data Engineer to join our Parts Pricing product line. The ideal candidate will have extensive experience in SQL development, cloud technologies (preferably GCP), and a solid understanding of key data processing tools and frameworks. This role offers an exciting opportunity to work in a fast-paced and innovative environment, interfacing with multiple business and technology stakeholders. The position will also provide exposure to futuristic technology domains, including BigQuery, GCP native ETL services, CI/CD pipelines, and Cloud Scheduler tools.
Key Responsibilities:
- Design, implement, and optimize data solutions using GCP native services for batch and real-time data processing.
- Collaborate with cross-functional teams (engineering, product, business) to deliver scalable and high-performance data pipelines.
- Leverage BigQuery, DataFlow, DataProc, DataFusion, Cloud Build, and Airflow for data engineering tasks.
- Develop and maintain CI/CD pipelines to streamline the deployment of data solutions.
- Support the migration of legacy ETL systems to cloud-native solutions.
- Integrate various data sources such as Oracle, Teradata, DB2, BigQuery, and flat files.
- Monitor and optimize the performance of data processing pipelines.
- Provide technical expertise and guidance in the analysis, design, development, and implementation of data warehousing applications.
- Debug and troubleshoot ETL jobs and resolve performance issues in parallel processing jobs.
Skills and Qualifications:
Required:
- 5+ years of experience in complex SQL development.
- 2+ years of experience with Google Cloud Platform (GCP), specifically in designing and implementing solutions at a production scale.
- Strong understanding of key GCP services, particularly related to data processing (Batch/Real-Time), leveraging Terraform, BigQuery, BQSQL, Dataflow, DataFusion, DataProc, Cloud Build, Airflow, and Pub/Sub.
- Hands-on experience with ETL tools, such as IBM Datastage, DataFlow, or similar data processing tools.
- Experience in integrating various data sources such as Oracle, Teradata, DB2, BigQuery, and flat files into a central data warehouse.
Preferred:
- Experience in designing ETL jobs using stages like Look-up, Filter, Sort, Copy, Remove Duplicates, Join, Funnel, and Aggregator.
- Proven experience in debugging ETL jobs and resolving errors and warnings associated with each job run.
- Expertise in tuning ETL jobs to optimize performance and resolving issues related to parallel jobs.
- Experience working with Autosys or Astronomer, Accurev, and Github for job scheduling and version control.
Education and Experience:
- 8+ years of experience in data engineering, with a strong background in building and maintaining data pipelines.
- Bachelor’s degree in Computer Science, Engineering, or a related field (or equivalent experience).