Senior Streaming Engineer (GCP)

Rackspace United States - Remote Full-time

About the Role:
We are seeking a highly skilled and experienced Senior Big Data Engineer to join our dynamic team. The ideal candidate will have a strong background in Java, developing and scaling both stream and batch processing systems, and a solid understanding of public cloud technologies, especially GCP. This role involves working in a remote environment, requiring excellent communication skills and the ability to solve complex problems independently and creatively.
What you will be doing:
·         Design and develop scalable data processing solutions, leveraging advanced Java techniques to ensure performance, reliability, and maintainability across streaming and batch workflows.
·         Build reusable and reliable code for stream and batch processing systems at scale.
·         Work with technologies such as Pub/Sub, Kafka , DataFlow, Flink, Hadoop, Pig, Oozie, Hive, and Spark.
·         Implement automation/DevOps best practices for CI/CD, IaC, Containerization, etc.

Requirements:

  • Driven and self-motivated to thoroughly clarify problem statements and develop well-structured plans to effectively solve challenges.
  • Proficiency in Java with solid experience in asynchronous programming paradigms, including a strong understanding of API futures and the ability to implement asynchronous APIs for scalable solutions.
  • Hands-on experience with reactive programming techniques, focusing on non-blocking data flow and real-time data handling.
  • Ability to design, debug, and optimize concurrent processes within distributed systems.
  • Adopt a proactive problem-solving approach by thoroughly analyzing problem statements, anticipating corner cases, and devising a comprehensive solution plan prior to coding. Begin coding with a clear algorithmic strategy to ensure efficient and effective solutions.
  • Expertise in public cloud services, particularly in GCP.
  • Experience in building reusable and reliable code for stream and batch processing systems at scale.
  • Experience with GCP managed services and understanding of cloud-based messaging/stream processing systems (e.g., Data Proc, Cloud Composer, GCS, DataFlow).
  • Experience in Infrastructure and Applied DevOps principles, including CI/CD and IaC (e.g., Terraform).
  • Knowledge in containerization technologies like Docker and Kubernetes to enhance scalability and efficiency.
  • Ability to tackle complex challenges and devise effective solutions, using critical thinking and innovative approaches.
  • Proven ability to work effectively in a remote setting, maintaining strong written and verbal communication skills.
  • Proven experience in engineering stream/batch processing systems at scale.
  • Strong programming abilities in Java and Python.
  • Hands-on experience with public cloud platforms, particularly GCP. Experience with other cloud technologies is a plus.

Must Have:

  • Strong experience with Java-based frameworks and libraries for building scalable data processing and streaming applications.
  • Strong Java 11-17 experience.
  • Dataflow (Managed Apache Beam).
  • Pub/Sub.
  • Google Associate Cloud Engineer Certification or other Google Cloud Professional level certification.
  • 4+ years of experience in customer-facing software/technology or consulting.
  • 4+ years of experience with “on-premises to cloud” migrations or IT transformations.
  • 4+ years of experience building and operating solutions on GCP (ideally) or AWS/Azure.
  • Technical degree in Computer Science, Software Engineering, or related field.