Big Data Engineer

Recruiter:

Indsafri

Job Ref:

JHB000263/Nons

Date posted:

Wednesday, June 22, 2022

Location:

Johannesburg, South Africa

JOB SUMMARY:
Our client is looking for Big Data Engineer to join their vibrant team

JOB DESCRIPTION:

Big Data Engineer
Job Description
Big Data Engineer that will work on the collecting, storing, processing, and analysing of huge sets of data. The primary focus will be on choosing optimal solutions to use for these purposes, then maintaining, implementing, and monitoring them. You will also be responsible for integrating them with the architecture used across the company.
Responsibilities

Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities
Implementing ETL process {{if importing data from existing data sources is relevant}}
Monitoring performance and advising any necessary infrastructure changes
Defining data retention policies

Skills and Qualifications

Proficiency with Hadoop v2, MapReduce, HDFS
Good knowledge of Big Data querying tools, such as Pig, Hive, and Impala
Experience with Spark
Experience with various messaging systems, such as Kafka or RabbitMQ
Experience with Big Data ML toolkits, such as Mahout, SparkML, or H2OGood understanding of Lambda Architecture, along with its advantages and drawbacks
Experience with Cloudera/MapR/Hortonworks
Proficient understanding of distributed computing principles
Management of Hadoop cluster, with all included services
Ability to solve any ongoing issues with operating the cluster
Experience with building stream-processing systems, using solutions such as Storm or Spark-Streaming
Experience with integration of data from multiple data sources
Experience with NoSQL databases, such as HBase, Cassandra, MongoDB
Knowledge of various ETL techniques and frameworks, such as Flume

NB! This job is now closed. You can apply for other jobs by uploading your CV.

Similar jobs you might be interested in:

Security Analyst
Location: Johannesburg
Salary:
If you are you a defender of the digital realm who lives and breathes the Microsoft security stack, then this is for you.Our client, an industry leader in investment and technology, is on the hunt for a Security Analyst who thrives in the Microsoft ecosystem and is passionate about protecting cloud and on-prem environments. In this role, you will be the frontline of the company’s cloud and e...
3 days ago

Platform Engineer
Location: Johannesburg
Salary:
Our client is looking for a skilled Platform engineer to join their high-performing team. This is an exciting opportunity to work on modern cloud platforms, building scalable, resilient systems while collaborating with top-tier engineering talent.
7 days ago

Cloud Engineer
Location: Johannesburg
Salary:
We are seeking a specialised Cloud engineer to design, build, secure, and maintain the infrastructure powering our modern data platforms. Unlike a traditional Cloud engineer focused on general application hosting, this role is dedicated to the unique demands of large-scale data workload, spanning high-volume storage, high-throughput networking, and compute-optimised environments for Spark and SQL-...
7 days ago

Manager: Data Science
Location: Johannesburg
Salary:
Our client in the Telco sector is seeking a professional Manager: data Science on a contract duration of 6 to 12 months
41 days ago

Integration Developer (Javascript, APIs, SQL)
Location: Johannesburg
Salary:
63 days ago

Senior Data Engineer
Location: Johannesburg
Salary:
65 days ago

Portfolio Accountant
Location: Johannesburg
Salary: 0000
The Portfolio Accountant will be responsible for the accurate management, reporting and analysis of the company’s investment and loans portfolio.
1 day ago

Senior Salesforce Developer
Location: Johannesburg
Salary:
Our client is seeking a Senior Salesforce Developer to join their team, based fully in-office in Bedfordview. This role is suited to a highly experienced Salesforce professional who is passionate about designing scalable architecture, leading technical solutions, and delivering high-quality enterprise systems.
2 days ago

Project Manager
Location: Johannesburg
Salary:
Drive large scale technology change that truly matters. This role places you at the centre of complex digital transformation where your decisions shape systems, teams, and long term impact. If you thrive in high stakes environments and love turning strategy into reality, this opportunity delivers challenge, influence, and growth.
2 days ago

Full Stack Developer (Entry) 1482
Location: Menlyn
Salary:
Junior Software engineer – AIâEnabled Products (Python)We’re looking for a Junior Software engineer who wants to work at the intersection of software engineering, data, and AIâenabled products.This is not a pure frontend role, not a QA role, and not a research data scientist position.Instead, you’ll be a core engineer helping turn data and machineâlearning capabilities into...
6 days ago

Big Data Engineer

Similar jobs you might be interested in:

Create a free job alert for Big Data Engineer in Johannesburg