Spark Architect

Pyramid Consulting Europe Ltd

Posted on Jun 5, 2024 by Pyramid Consulting Europe Ltd
Sheffield, Yorkshire, United Kingdom
IT
Immediate Start
Annual Salary
Contract/Project

Job Description:

Job Title: Spark Architect
Location: Sheffield, UK (3 days/week, onsite)
Job Type: 6+ month and possibility of extension

Looking for an enthusiastic Spark Architect with deep component understanding around Spark Data Integration (PySpark, Scripting, variable setting etc.), Spark SQL, Spark Explain plans. Also able to analyse Spark code failures through Spark Plans and make correcting recommendations; able to review PySpark and Spark SQL jobs and make performance improvement recommendations; able to understand Data Frames/Resilient Distributed Data Sets and understand any memory related problems and make corrective recommendations; and able to monitor Spark jobs using wider tools such as Grafana to see whether there are Cluster level failures. As a Spark architect, who can demonstrate deep knowledge of how Cloudera Spark is set up and how the run time libraries are used by PySpark code.

Your benefits:

As the Spark architect, you will have the opportunity to work with one of the biggest IT landscapes in the world. You can also look forward to opportunity to collaborate with senior leadership stakeholders, guiding technical team and drive overall program objectives.

Your responsibilities:

As a Spark Architect you will be working for - GDT (Global Data Technology) Team, you will be responsible for:

  • Working on an Enterprise scale Cloud infrastructure and Cloud Services in one of the Clouds (GCP).
  • Drive Data Integration upgrade to PySpark.

Mandatory Skills:

You need to have the below skills.

  • At least 12+ Years of IT Experience with Deep understanding of component understanding around Spark Data Integration (PySpark, Scripting, variable setting etc.), Spark SQL, Spark Explain plans.
  • Spark SME - Be able to analyse Spark code failures through Spark Plans and make correcting recommendations.
  • To be able to traverse and explain the architecture you have been a part of and why any particular tool/technology was used.
  • Spark SME - Be able to review PySpark and Spark SQL jobs and make performance improvement recommendations.
  • Spark - SME Be able to understand Data Frames/Resilient Distributed Data Sets and understand any memory related problems and make corrective recommendations.
  • Monitoring -Spark jobs using wider tools such as Grafana to see whether there are Cluster level failures.
  • Cloudera (CDP) Spark and how the run time libraries are used by PySpark code.
  • Prophecy - High level understanding of Low-Code No-Code prophecy set up and its use to generate PySpark code.
  • Ready to work at least three days from Sheffield (UK) office and accept changes as per customer/policies.

Good to have skills.

Ideally, you should be familiar with

  • Collaboration with multiple customer stakeholders
  • Knowledge of working with Cloud Databases
  • Excellent communication and solution presentation skills.

Reference: 2771795530

https://jobs.careeraddict.com/post/91511662

This Job Vacancy has Expired!

Pyramid Consulting Europe Ltd

Spark Architect

Pyramid Consulting Europe Ltd

Posted on Jun 5, 2024 by Pyramid Consulting Europe Ltd

Sheffield, Yorkshire, United Kingdom
IT
Immediate Start
Annual Salary
Contract/Project

Job Description:

Job Title: Spark Architect
Location: Sheffield, UK (3 days/week, onsite)
Job Type: 6+ month and possibility of extension

Looking for an enthusiastic Spark Architect with deep component understanding around Spark Data Integration (PySpark, Scripting, variable setting etc.), Spark SQL, Spark Explain plans. Also able to analyse Spark code failures through Spark Plans and make correcting recommendations; able to review PySpark and Spark SQL jobs and make performance improvement recommendations; able to understand Data Frames/Resilient Distributed Data Sets and understand any memory related problems and make corrective recommendations; and able to monitor Spark jobs using wider tools such as Grafana to see whether there are Cluster level failures. As a Spark architect, who can demonstrate deep knowledge of how Cloudera Spark is set up and how the run time libraries are used by PySpark code.

Your benefits:

As the Spark architect, you will have the opportunity to work with one of the biggest IT landscapes in the world. You can also look forward to opportunity to collaborate with senior leadership stakeholders, guiding technical team and drive overall program objectives.

Your responsibilities:

As a Spark Architect you will be working for - GDT (Global Data Technology) Team, you will be responsible for:

  • Working on an Enterprise scale Cloud infrastructure and Cloud Services in one of the Clouds (GCP).
  • Drive Data Integration upgrade to PySpark.

Mandatory Skills:

You need to have the below skills.

  • At least 12+ Years of IT Experience with Deep understanding of component understanding around Spark Data Integration (PySpark, Scripting, variable setting etc.), Spark SQL, Spark Explain plans.
  • Spark SME - Be able to analyse Spark code failures through Spark Plans and make correcting recommendations.
  • To be able to traverse and explain the architecture you have been a part of and why any particular tool/technology was used.
  • Spark SME - Be able to review PySpark and Spark SQL jobs and make performance improvement recommendations.
  • Spark - SME Be able to understand Data Frames/Resilient Distributed Data Sets and understand any memory related problems and make corrective recommendations.
  • Monitoring -Spark jobs using wider tools such as Grafana to see whether there are Cluster level failures.
  • Cloudera (CDP) Spark and how the run time libraries are used by PySpark code.
  • Prophecy - High level understanding of Low-Code No-Code prophecy set up and its use to generate PySpark code.
  • Ready to work at least three days from Sheffield (UK) office and accept changes as per customer/policies.

Good to have skills.

Ideally, you should be familiar with

  • Collaboration with multiple customer stakeholders
  • Knowledge of working with Cloud Databases
  • Excellent communication and solution presentation skills.

Reference: 2771795530

CareerAddict

Alert me to jobs like this:

Amplify your job search:

CV/résumé help

Increase interview chances with our downloads and specialist services.

CV Help

Expert career advice

Increase interview chances with our downloads and specialist services.

Visit Blog

Job compatibility

Increase interview chances with our downloads and specialist services.

Start Test

Similar Jobs

Security/Network Consultant

Sheffield, Yorkshire, United Kingdom

Community Fundraising Manager

Sheffield, Yorkshire, United Kingdom

Principal Electrical Engineer

Sheffield, Yorkshire, United Kingdom