Senior Big Data Engineer
Posted on Sep 20, 2019 by Request Technology
A prestigious fortune 500 company is on the search for a Sr. Big Data Engineer. This role is revolved around leveraging data and analytics to help automate and optimize analytic data processes. This person will be responsible for building a centralized data lake/hub using the Hadoop ecosystem that will be used by the reporting and operational analytics teams and the machine learning teams. This person should have experience with the following technologies: Data Lake, Hadoop, Python, Hive, Scala, Impala, Kafka, Github, Jenkins, and more.
- Responsible for design, prototyping and delivery of software solutions within the big data eco-system
- Leading projects and/or serving as analytics SME to provide new or enhanced data to the business
- Improving data governance and quality increasing the reliability of our data
- Influencing the creation of a single, trusted source for key Claims business data that can be shared across the Enterprise
- Responsible for designing and building new Big Data systems for turning data into actionable insights
- Train and mentor junior team members on Big Data/Hadoop tools and technologies
- Identifies opportunities for improvement and presents recommendations to management
- Participate in strategic planning discussions with technical and non-technical partners
- Uses, teaches, and supports a wide variety of Big Data and Analytics tools to achieve results (ie, Python, Hadoop, HIVE, Scala, Impala and others).
- Uses, teaches, and supports a wide variety of programming languages on Big Data and Analytics work (ie Java, Python, SQL, R)
- Undergraduate degree in Computer Science, Mathematics, Engineering (or related field) or equivalent experience preferred
- 5-7 years of experience preferred in a data integration, ETL and/or business intelligence/analytics related function
- Ability to work with broad parameters in complex situations
- Experience in developing, managing, and manipulating large, complex datasets
- Expert high-level coding skills such as SQL and Python and/or other Scripting languages(UNIX) required. Scala is a plus.
- Some understanding and exposure to - streaming toolsets such as Kafka, FLINK, Spark streaming a plus.
- Experience with source control solutions (ex git, GitHub, Jenkins, Artifactory) required
- 2+ years of experience with big data and the Hadoop ecosystem (HDFS, SPARK, SQOOP, Hive, Impala, Parquet) required
- Experience with Agile development methodologies and tools to iterate quickly on product changes, developing user stories and working through backlog (Continuous Integration and JIRA a plus)
- Experience with Airflow a plus
- Working knowledge of Tableau - a plus