Senior Big Data Engineer
Posted on Sep 29, 2020 by Visionaire Partners
Sr Big Data Engineer
We are seeking an experienced big data engineer to join an exciting industry and product. This role will provide opportunities to directly influence architecture and technical strategies, and evolve best practices!
The ideal fit is someone experienced with python, Spark, AWS, and multiple years working in data engineering environments.This role will contribute to several to product features in addition to analysis, ETL, data pipelines, and data modelling. We have three primary tools that we interface with daily. To store all data in a centralized location, we us Amazon S3 as our Data Lake. From there we pipe that data into Snowflake DB, and from there we use Databricks (Spark ecosystem) to do the ETL processing. Within Databricks we use Python as the primary language construct, but also SQL.
Author clean, tested source code that supports architectural patterns and project goals
Leverage professional experience evolving ingestion, storage, resolution, warehousing, ETL, and similar big data disciplines
Contribute to conventions and patterns standards, assist in PRs to ensure compliance
Collaborate with engineers and product to design solutions that deliver value while focusing on excellence in platform performance, scalability, extensibility, quality, and security
Work within Kanban methodology
Contribute with a spirit of continuous improvement
Be creative, humble, accountable, and collaborative
This is a 6 month contract to hire opportunity with full intention of converting into a full time role! The office is located in Cumming, Georgia. We are currently 100% remote with no plans to return at this time. Normal operations are 3 days a week WFH before COVID (Tue/Thur are in-office).
- 7+ years experience engineering within Big Data technologies and large data sets
- 2+ years experience as a Senior or Lead resource on a Big Data Engineering team
- Strong proficiency in Python and SQL
- Solid understanding of big data tooling such as Spark, Hive, and big data databases
- Experience with S3, Kinesis, Databricks, PySpark and/or Snowflake
- Understanding of common ML models
Must be authorized to work in the US.