About the Role:
We are seeking a highly skilled and experienced Senior Big Data Engineer to join our dynamic team. The ideal candidate will have a strong background in Java, developing and scaling both stream and batch processing systems, and a solid understanding of public cloud technologies, especially GCP. This role involves working in a remote environment, requiring excellent communication skills and the ability to solve complex problems independently and creatively.
What you will be doing:
· Design and develop scalable data processing solutions, leveraging advanced Java techniques to ensure performance, reliability, and maintainability across streaming and batch workflows.
· Build reusable and reliable code for stream and batch processing systems at scale.
· Work with technologies such as Pub/Sub, Kafka , DataFlow, Flink, Hadoop, Pig, Oozie, Hive, and Spark.
· Implement automation/DevOps best practices for CI/CD, IaC, Containerization, etc.
Requirements:
- Driven and self-motivated to thoroughly clarify problem statements and develop well-structured plans to effectively solve challenges.
- Proficiency in Java with solid experience in asynchronous programming paradigms, including a strong understanding of API futures and the ability to implement asynchronous APIs for scalable solutions.
- Hands-on experience with reactive programming techniques, focusing on non-blocking data flow and real-time data handling.
- Ability to design, debug, and optimize concurrent processes within distributed systems.
- Adopt a proactive problem-solving approach by thoroughly analyzing problem statements, anticipating corner cases, and devising a comprehensive solution plan prior to coding. Begin coding with a clear algorithmic strategy to ensure efficient and effective solutions.
- Expertise in public cloud services, particularly in GCP.
- Experience in building reusable and reliable code for stream and batch processing systems at scale.
- Experience with GCP managed services and understanding of cloud-based messaging/stream processing systems (e.g., Data Proc, Cloud Composer, GCS, DataFlow).
- Experience in Infrastructure and Applied DevOps principles, including CI/CD and IaC (e.g., Terraform).
- Knowledge in containerization technologies like Docker and Kubernetes to enhance scalability and efficiency.
- Ability to tackle complex challenges and devise effective solutions, using critical thinking and innovative approaches.
- Proven ability to work effectively in a remote setting, maintaining strong written and verbal communication skills.
- Proven experience in engineering stream/batch processing systems at scale.
- Strong programming abilities in Java and Python.
- Hands-on experience with public cloud platforms, particularly GCP. Experience with other cloud technologies is a plus.
Must Have:
- Strong experience with Java-based frameworks and libraries for building scalable data processing and streaming applications.
- Strong Java 11-17 experience.
- Dataflow (Managed Apache Beam).
- Pub/Sub.
- Google Associate Cloud Engineer Certification or other Google Cloud Professional level certification.
- 4+ years of experience in customer-facing software/technology or consulting.
- 4+ years of experience with “on-premises to cloud” migrations or IT transformations.
- 4+ years of experience building and operating solutions on GCP (ideally) or AWS/Azure.
- Technical degree in Computer Science, Software Engineering, or related field.