Big Data Engineer Professional Services

Big Data Engineer - Professional Services

About BrightEdge

BrightEdge is a leading SEO and content performance marketing platform. We transform online content into tangible business results by processing massive amounts of data to provide actionable insights to our clients.

Role Overview

We are seeking a talented Big Data Engineer to join our Professional Services team. You will design, build, and maintain high-performance data pipelines that process terabytes of data, ensuring our data infrastructure can handle increasing volumes while maintaining exceptional performance standards.

Key Responsibilities

• Design and implement scalable batch processing systems using Python and big data technologies.
• Optimize database performance, focusing on slow-running queries and latency improvements.
• Utilize Python profilers and performance monitoring tools to identify bottlenecks.
• Reduce P95 and P99 latency metrics across our data platform.
• Build efficient ETL pipelines capable of handling large-scale data processing.
• Collaborate with data scientists and product teams to understand data requirements.
• Monitor and troubleshoot data pipeline issues in production.
• Implement data quality checks and validation mechanisms.
• Document data architecture and engineering processes.
• Stay current with emerging big data technologies and best practices.

Qualifications

• Required:
  • Bachelor's degree in Computer Science, Engineering, or related technical field.
  • 4+ years of experience in data engineering roles.
  • Strong Python programming skills with a focus on data processing libraries.
  • Experience with big data technologies (Spark, Hadoop, etc.).
  • Proven experience optimizing database performance (SQL or NoSQL).
  • Knowledge of data pipeline orchestration tools (Airflow, Luigi, etc.).
  • Understanding of performance optimization techniques and profiling tools.
• Preferred:
  • Master's degree in Computer Science or related field.
  • Experience with SEO data or web crawling systems.
  • Experience with Clickhouse Database.
  • Knowledge of distributed systems and microservices architecture.
  • Familiarity with container orchestration (Kubernetes, Docker).
  • Experience with real-time data processing.
  • Contributions to open-source projects.
  • Experience with machine learning operations.

Qualifications

Required

• Bachelor's degree in Computer Science, Engineering, or a related technical field.
• 4+ years of experience in data engineering roles.
• Strong Python programming skills with a focus on data processing libraries.
• Experience with big data technologies such as Spark and Hadoop.
• Proven experience optimizing database performance (SQL or NoSQL).
• Knowledge of data pipeline orchestration tools (e.g., Airflow, Luigi).
• Understanding of performance optimization techniques and profiling tools.

Preferred

• Master's degree in Computer Science or a related field.
• Experience with SEO data or web crawling systems.
• Experience with Clickhouse Database.
• Knowledge of distributed systems and microservices architecture.
• Familiarity with container orchestration (Kubernetes, Docker).
• Experience with real-time data processing.
• Contributions to open-source projects.
• Experience with machine learning operations.

Data Pipeline Development & Optimization

• Design and implement scalable batch processing systems using Python and big data technologies.
• Build efficient ETL pipelines capable of handling large-scale data processing.
• Optimize database performance, focusing on slow-running queries and latency improvements.
• Use Python profilers and performance monitoring tools to identify bottlenecks and implement solutions.
• Reduce P95 and P99 latency metrics across the data platform.

Collaboration & Problem Solving

• Collaborate with data scientists and product teams to understand data requirements and translate them into technical solutions.
• Monitor and troubleshoot data pipeline issues in production environments.
• Implement data quality checks and validation mechanisms to ensure data integrity.

Documentation & Continuous Learning

• Document data architecture and engineering processes for clear communication and maintainability.
• Stay current with emerging big data technologies and best practices to ensure the platform remains cutting-edge.

Candidates interested in the Big Data Engineer Professional Services role at BrightEdge will first submit their resume and cover letter through the online portal. A review of applications will be conducted, focusing on relevant experience with big data technologies, Python programming, and database optimization. Shortlisted candidates will then be invited for a technical phone screening to assess their skills and understanding of key concepts.

Those who successfully navigate the phone screening will proceed to a virtual interview with the hiring manager and potentially other team members. This interview will delve deeper into their experience, problem-solving abilities, and cultural fit within BrightEdge.

The final stage involves a practical assessment, where candidates may be asked to solve a data-related challenge or demonstrate their proficiency in using relevant tools and technologies. Based on the performance throughout all stages, the hiring team will make a final decision and extend an offer to the most qualified candidate.

BrightEdge is a leading platform that empowers businesses to transform their online content into tangible results. We leverage the power of massive data processing to deliver actionable insights that drive SEO and content performance marketing success. Our cutting-edge technology helps clients understand their audience, optimize their content strategy, and ultimately achieve their business goals. BrightEdge is committed to innovation and providing our clients with the tools and expertise they need to thrive in the ever-evolving digital landscape.