Senior Data Engineer with AWS | IT Recruitment
Minimum year of experience:
- At least 10-12 years Cloud architecture and reporting technology experience
- Extensive experience in implementing and monitoring solutions
- Experience in testing (manual or automated testing)
- Web and digital project experience advantageous
- Agile working experience advantageous
- Chief Expert (1460)
Minimum qualifications required:
- Relevant IT / Business Degree
- Applies advanced knowledge of area
- Managing projects / processes
- Ability to develop within a specific sought-after programming language
- Strong working knowledge with software development tools, techniques and approaches used to build application solutions
- Working knowledge with software development tools, techniques and approaches used to build application solutions
- Cloud computing technologies, Business drivers and emerging computing trends
Generic Technical / Functional skills:
- Understanding of integration between different technologies
- Coordination between development and support environments
- Assisting with the business case
- Planning and monitoring
- Eliciting requirements
- Requirements organisation
- Translating and simplifying requirements
- Requirements management and communication
- Requirements analysis
- Document requirements in appropriate format depending on methodology followed
- Assist with identification and management of risks
Specific Technical / Functional skills:
We are looking for a Senior AWS Developer Associate to join our Mobility Analytics team of cloud developers.
- Expertise in ETL optimization, designing, coding, and tuning big data processes using Apache Spark.
- Experience with building data pipelines and applications to stream and process datasets at low latencies.
- Show efficiency in handling data – tracking data lineage, ensuring data quality, and improving discoverability of data.
- Sound knowledge of distributed systems and data architecture (lambda)- design and implement batch and stream data processing pipelines, knows how to optimize the distribution, partitioning of high-level data structures.
- Experience designing and supporting large-scale distributed systems in a production environment
AWS Solid understanding of Components – VPC | IAM. Above average experience/understanding of AWS Components:
- Param Store
- Secrets Manager
- Step Function
- Code Build/Pipeline
Strong experience/understanding of:
- Python 3x
- Py Spark
- Linux / Unix
- Big Data
- Powershell / Bash
Experience working in Agile SDLC methodology.
- Working experience building data/ETL pipeline and data warehouse.
- Demonstrate expertise in data modelling SQL and NoSQL databases.
- Exceptional analytical skills analysing large and complex data sets.
- Perform thorough testing and data validation to ensure the accuracy of data transformations.
- Strong written and verbal communication skills, with precise documentation.
- Self -driven team player with ability to work independently and multi -task.
- Must be an analytical and creative thinker, and an innovative problem solver.
- Experience building data pipeline using AWS Glue or Data Pipeline, or similar platforms.
- Familiar with data store such as AWS S3, and AWS RDS or DynamoDB.
- Experience and solid understanding of various software design patterns.
- Experience preparing specifications from which programs will be written, designed, coded, tested and debugged.
- Experience working with a distributed team.
- Strong organizational skills.
- Certification: AWS Certified Developer Associate / Solutions Architect
- Bitbucket / Git
- Jira / Confluence
- Familiar with data streaming services such as Apache Kafka, Amazon Kinesis, or similar tools
- CI / CD Tool (Nexus / Jenkins).
- Problem solving capabilities
- Ability to work as part of a team
- Ability to work interdependently as well as independently and to submit deliverables on time and with excellent quality
- Excellent interpersonal and organisational skills with the ability to communicate effectively (both verbally and in writing) with both technical and non -technical colleagues / users
- Strong ability to understand and interpret business needs and requirements with an aptitude to move concepts through to proposal and successful implementation
- Strong presentation skills
- Above-board work ethics
- Flexibility to take up different tasks in the project
- Ability and willingness to coach and give training to fellow colleagues and users when required
- Willing and able to travel internationally