Big Data Engineer

Recruiter:

Indsafri

Job Ref:

JHB000263/Nons

Date posted:

Wednesday, June 22, 2022

Location:

Johannesburg, South Africa

JOB SUMMARY:
Our client is looking for Big Data Engineer to join their vibrant team

JOB DESCRIPTION:

Big Data Engineer
Job Description
Big Data Engineer that will work on the collecting, storing, processing, and analysing of huge sets of data. The primary focus will be on choosing optimal solutions to use for these purposes, then maintaining, implementing, and monitoring them. You will also be responsible for integrating them with the architecture used across the company.
Responsibilities

Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities
Implementing ETL process {{if importing data from existing data sources is relevant}}
Monitoring performance and advising any necessary infrastructure changes
Defining data retention policies

Skills and Qualifications

Proficiency with Hadoop v2, MapReduce, HDFS
Good knowledge of Big Data querying tools, such as Pig, Hive, and Impala
Experience with Spark
Experience with various messaging systems, such as Kafka or RabbitMQ
Experience with Big Data ML toolkits, such as Mahout, SparkML, or H2OGood understanding of Lambda Architecture, along with its advantages and drawbacks
Experience with Cloudera/MapR/Hortonworks
Proficient understanding of distributed computing principles
Management of Hadoop cluster, with all included services
Ability to solve any ongoing issues with operating the cluster
Experience with building stream-processing systems, using solutions such as Storm or Spark-Streaming
Experience with integration of data from multiple data sources
Experience with NoSQL databases, such as HBase, Cassandra, MongoDB
Knowledge of various ETL techniques and frameworks, such as Flume

NB! This job is now closed. You can apply for other jobs by uploading your CV.

Similar jobs you might be interested in:

AWS Data Engineer (Expert)
Location: Midrand
Salary:
Join our team as an AWS data engineer (Expert) and leverage your skills in building and maintaining big data Pipelines using advanced cloud platforms. We seek innovative problem-solvers with expertise in Python, SQL, and AWS services to drive our data initiatives forward. Be part of a collaborative environment where your contributions ensure accurate and secure data management.
Today

Intermediate Data Engineer - Johannesburg – up to R700k per annum
Location: Johannesburg
Salary: 700000
Intermediate data engineer - Johannesburg – up to R700k per annum
4 days ago

Platform Engineer (Mulesoft Integration)
Location: Johannesburg
Salary: Hourly
We are in search of a PLATFORM engineer with expertise in Mulesoft Integration for one of our banking industry clients, for a 12-month rolling employment contract. The incumbent will be joining the Platforms and engineering Gateway technology tribe responsible for the development, design and run of the Mulesoft API and Integration platform residing onâpremises and AWS Cloud.Apply platform e...
7 days ago

AWS Data Engineer (Expert)
Location: Centurion
Salary:
Join our client as an Expert AWS data engineer in South Africa. Design, implement, and optimize big data Pipelines using AWS services. Ensure data integrity, security, and compliance while collaborating with teams to integrate solutions. Bring your expertise in AWS technologies and data engineering to drive impactful data solutions.
13 days ago

Data Engineer (AWS and Redshift) - Johannesburg – up to R1mil
Location: Johannesburg
Salary: 1000000
data engineer (AWS and Redshift) - Johannesburg – up to R1mil
25 days ago

2050 - AWS Data Engineer
Location: Pretoria
Salary: Neg Monthly
Large company in Pretoria looking to fill this urgent role. Contracthybrid (2/3 times at the office)
Today

Enterprise Sales Manager (Network products)
Location: Johannesburg
Salary: 1000 Annually
Responsible for delivering growth in revenue from existing customers and on-boarding new customers in the region. You are the Customer champion and the face of our client in ensuring customer satisfaction and positioning our client as their partner of choice.Â Leverage your regional expertise within Enterprise. Defend existing revenue streams by minimising churn of services. Drive cross-functiona...
5 days ago

Data Scientist
Location: Johannesburg
Salary:
Are you a talented data Scientist seeking an exciting opportunity to revolutionize the banking industry? Join a dynamic team at a leading financial institution as we harness the power of data to drive innovation and enhance customer experiences!
7 days ago

Mulesoft Integration Platform Engineer
Location: Johannesburg
Salary:
Join an exciting and dynamic team of API Gateway Platform engineers, who are responsible for shaping the technical API economy for the bank.
7 days ago

Solutions Architect
Location: Midrand
Salary:
Falcorp Technologies is a black empowerment Digital Technology company in South Africa offering Technology Consulting, Application Services, Infrastructure Services, Portal and Content Management, and Systems Integration Services. Our focus is on enabling business enterprises through cutting-edge innovative IT solutions that provide strategic differentiation and operational superior...
7 days ago

Big Data Engineer

Similar jobs you might be interested in:

Create a free job alert for Big Data Engineer in Johannesburg