Data Engineer – Python, PySpark & AWS

We are looking for a skilled and experienced Data Engineer to join our team and contribute to the design, development, and optimization of scalable data solutions in an AWS cloud environment.

In this role, you will be responsible for building and maintaining robust ETL/ELT pipelines, integrating data from multiple sources, and developing efficient data-processing solutions that support business operations and data-driven decision-making.

You will work with modern cloud-based data technologies, leveraging Python, SQL, PySpark, and AWS services to build reliable data pipelines, Data Lakes, and Data Warehouses. You will collaborate with cross-functional teams to understand business requirements, implement technical solutions, and ensure data quality, performance, and reliability.

The ideal candidate is a hands-on data engineer with strong expertise in Python, SQL, PySpark, AWS Glue, S3, Athena, Lambda, and Redshift, who is comfortable working across the entire data engineering lifecycle, from requirements gathering and solution design to implementation, testing, troubleshooting, documentation, and technical handoff.

• Design, develop, test, and maintain scalable ETL/ELT pipelines to support data integration, transformation, and processing across multiple data sources.
• Build and optimize data-processing solutions using Python, SQL, and PySpark.
• Develop and maintain cloud-native data solutions using AWS services, including S3, Glue, Athena, Lambda, and Redshift.
• Design, implement, and support Data Lakes and Data Warehouses, ensuring efficient data storage, organization, accessibility, and processing.
• Develop data models and transformation workflows to support analytics, reporting, and business intelligence requirements.
• Integrate structured and semi-structured data from multiple sources, ensuring consistency, accuracy, and reliability.
• Implement data-quality checks, validation processes, and monitoring mechanisms to maintain data integrity across pipelines.
• Optimize data pipelines, SQL queries, and distributed processing workloads to improve performance, scalability, and cost efficiency.
• Collaborate with business stakeholders, analysts, and engineering teams to understand data requirements and translate them into effective technical solutions.
• Participate in technical design discussions, contributing to data architecture, integration patterns, and engineering best practices.
• Develop and maintain reusable, well-structured, and maintainable code following data engineering best practices.
• Work with Git, CI/CD pipelines, and Terraform to support version control, automated deployments, and infrastructure management.
• Troubleshoot complex data-processing issues, investigate pipeline failures, and implement reliable, long-term solutions.
• Perform testing, validation, and debugging to ensure the quality and reliability of data solutions.
• Produce technical documentation covering pipeline architecture, data transformations, implementation decisions, and operational procedures.
• Take end-to-end ownership of data engineering deliverables, from requirements analysis and development through testing, implementation, and technical handoff.

• Experience with workflow orchestration tools such as Apache Airflow or Amazon MWAA.
• Familiarity with dbt for data transformation and analytics engineering.
• Experience with cloud data platforms such as Snowflake or Databricks.
• Knowledge of streaming and event-driven data architectures using Apache Kafka, Amazon MSK, or Amazon Kinesis.
• Experience with Amazon EMR for large-scale distributed data processing.
• Familiarity with AWS Database Migration Service (DMS) and Change Data Capture (CDC) processes.
• Experience implementing data ingestion, migration, or synchronization solutions across heterogeneous data sources.
• Exposure to AI/ML or Generative AI (GenAI) projects, including data preparation, processing pipelines, or integration with machine learning workflows.
• Experience optimizing cloud data infrastructure for scalability, reliability, and cost efficiency.

Avenue Code reinforces its commitment to privacy and to all the principles guaranteed by the most accurate global data protection laws, such as GDPR, LGPD, CCPA and CPRA. The Candidate data shared with Avenue Code will be kept confidential and will not be transmitted to disinterested third parties, nor will it be used for purposes other than the application for open positions. As a Consultancy company, Avenue Code may share your information with its clients and other Companies from the AI/R Group to which Avenue Code’s consultants are allocated to perform its services.

• Solid professional experience in Data Engineering, with a strong background in developing and maintaining production data pipelines.
• Strong proficiency in Python for data processing, transformation, and automation.
• Advanced SQL skills, including complex queries, data transformations, performance optimization, and troubleshooting.
• Hands-on experience with PySpark and distributed data processing.
• Strong experience designing, developing, and maintaining ETL/ELT pipelines.
• Practical experience with AWS cloud services, particularly Amazon S3, AWS Glue, Amazon Athena, AWS Lambda, and Amazon Redshift.
• Solid understanding of Data Lake and Data Warehouse architectures, including data storage, processing, and integration patterns.
• Experience with data modeling, data transformation, and data integration across multiple sources.
• Familiarity with data-quality validation, monitoring, and pipeline troubleshooting.
• Experience with Git and collaborative development workflows.
• Familiarity with CI/CD practices and Terraform for automated deployments and infrastructure management.
• Strong understanding of data engineering principles, code quality, performance optimization, and maintainability.
• Experience working across the full data engineering lifecycle, including requirements gathering, implementation, testing, troubleshooting, documentation, and technical handoff.
• Strong analytical and problem-solving skills, with the ability to independently investigate and resolve complex technical issues.
• Excellent communication and collaboration skills.
• Ability to work independently, manage priorities, and deliver high-quality solutions in a collaborative environment.