Data Engineer in Test

We are undergoing a significant data transformation to ensure that accurate and consistent data is available precisely when and where it's needed.

Responsibilities:

• Write PySpark and SQL scripts to validate data pipelines, transformations, and integrations.
• Design and run tests for data validation, storage, and retrieval using Azure services like Data Lake, Synapse, and Data Factory, adhering to industry standards.
• Continuously enhance automated tests as new features are developed, ensuring Participate in data reconciliation and verify Data Quality frameworks to maintain data accuracy, completeness, and consistency across the platform.
• Formulate and maintain test strategies—including smoke, performance, functional, and regression testing—to ensure data processing and ETL jobs meet requirements.
• Collaborate with development teams to assess changes in data workflows and update test cases to preserve data integrity.
• Share knowledge and best practices by collaborating with business analysts and technology teams to document testing processes and findings.
• Communicate testing progress effectively with stakeholders, highlighting issues or blockers, and ensuring alignment with business objectives.
• Maintain a comprehensive understanding of the Azure Data Lake platform's data landscape to ensure thorough testing coverage.

Requirements: 

• Bachelor’s degree in computer science or related discipline
• 3-6 years of hands-on experience in Data Engineering particularly in Data Lake environments on Azure's cloud platform.
• Proficient in Azure Data Factory, Azure Synapse Analytics and Databricks for big data processing and scaled data quality checks.
• Proficiency in SQL, capable of writing and optimizing both simple and complex queries for data validation and testing purposes.
• Proficient in PySpark, with experience in data manipulation and transformation, and a demonstrated ability to write and execute test scripts for data processing and validation.
• Hands-on experience with Functional & system integration testing in big data environments, ensuring seamless data flow and accuracy across multiple systems.
• Knowledge and ability to design and execute test cases in a behaviour-driven development environment.
• Fluency in Agile methodologies, with active participation in Scrum ceremonies and a strong understanding of Agile principles.
• Familiarity with tools like Jira, including experience with X-Ray or Jira Zephyr for defect management and test case management.
• Proven experience working on high-traffic and large-scale software products, ensuring data quality, reliability, and performance under demanding conditions.
• Identify and resolve issues in complex Big Data environments, ensuring data accuracy and system efficiency.
• Initiate and manage tasks, delivering high-quality results within deadlines in a dynamic setting.
• Confidently challenge existing assumptions and processes to enhance data engineering and QA practices.
• Maintain organized testing schedules and documentation, ensuring thorough coverage of all scenarios.
• Exhibit strong interpersonal skills, working effectively in cross-functional, multicultural teams with minimal supervision.
• Comfortable in Agile and Scrum environments, proficient with tools like Jira, and adaptable to changing priorities.
• Manage multiple projects simultaneously, demonstrating strong planning and organizational skills to ensure timely task completion.

We have an amazing team of 700+ individuals working on highly innovative enterprise projects & products. Our customer base includes Fortune 5 retail and CPG companies, leading store chains, fast-growth fintech, and multiple Silicon Valley startups.

What makes Confiz stand out is our focus on processes and culture. Confiz is ISO 9001:2015 (QMS), ISO 27001:2022 (ISMS), ISO 20000-1:2018 (ITSM) and ISO 14001:2015 (EMS) Certified. We have a vibrant culture of learning via collaboration and making workplace fun.

People who work with us work with cutting-edge technologies while contributing success to the company as well as to themselves. 

To know more about Confiz Limited, visit: https://web.facebook.com/lifeatconfiz