Senior Data Engineer - Hadoop
Posted on Jul 13, 2021 by Request Technology
*We are unable to sponsor as this is a permanent full time role*
A prestigious fortune 500 company is on the search for Sr. Data Engineer - Hadoop. This company is doing a Hadoop Horton works to AWS migration. They need someone with experience with Hadoop, Python, Spark, AWS, CICD, containers, etc. They need experience with cloud infrastructure and cloud data lake hands on experience.
- Work with Product Owners to understand the desired capability, to define and prioritize work, determine artifacts, and manage workloads
- Assess the current Legacy application code, design and refactor the code using cloud native technology
- Involve in application development, prototyping, modelling and technical consulting
- Actively facilitate issue resolution and issue tracking. Identify mitigation steps and ensures risks and issues are mitigated/resolved in a timely manner.
- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
- Lead efforts in data, code, and systems analysis to ensure accuracy and completeness in requirements
- Support the Development TL and/or PM in managing projects/Agile Sprints
- Perform and lead thorough Unit testing and Integration testing, including test data creation for various test scenarios.
- 5+ years of Full time experience in software development including design, coding, testing, production implementation, production support, performance tuning, bug fixing and quality control
- 2+ years of strong cloud infrastructure experience working with one or more of the following
- Amazon Web Services (AWS) Cloud services: EC2, EMR, ECS, S3, SNS, SQS, Cloud Formation, Cloud watch, Lambda, EKS
- Hands-on experience of AWS architecture design, Data Management, Big Data(Hortonworks), and Data Warehousing
- 2+ years of experience in Cloud data lake engine and Cloud data warehousing technologies (Snowflake, Dremio, etc.)
- 2+ years of experience in big data technologies - Hortonworks (Spark, Hadoop, Hive, HDFS etc,) and MongoDB, PostGre SQL
- 3+ years of experience using Java, Python or Scala, Informatica IICS
- Experience of leading complex data applications with disparate semi structured data (JSON, XML)
- Proficient with CICD process, Agile and DevOps SLDC
- Experience of building data lake in AWS Cloud, moving Data applications to the Cloud, and developing cloud native data applications
- At least 2 years of experience with Scaled Agile, Kanban, or Scrum methodologies