Data Pipeline Operations Engineer

We are seeking a detail-oriented and technically skilled Data Pipeline Operations Engineer to manage and execute our weekly scanning process. This critical role ensures the timely flow of customer data through our research, scanning, and UI ingest pipeline. The ideal candidate has a mix of programming, database, and Linux system administration skills to handle the various steps in the scanning workflow.

SixMap protects some of the world’s largest and most complex government and corporate enterprises with a continuous threat exposure management (CTEM) platform. The SixMap platform is powered by an advanced computational mapping engine that quickly discovers and continuously monitors the unique contours of the extended enterprise’s Internet-facing assets across IPv4 and IPv6. Upon providing cyber defenders this comprehensive enterprise visibility, the platform complements this awareness with contextual threat intelligence and a suite of remediation measures. The SixMap team brings deep intelligence community expertise and best practices to the defense of both U.S. Federal agencies and Fortune 500 corporations. For more information, please visit: www.sixmap.io.

Responsibilities

• Manage the weekly scanning process, ensuring customer data progresses through research, scanning, and UI ingest phases according to defined SLAs
• Prepare input files and kick off processes on the scanning cluster via Airflow
• Monitor and troubleshoot jobs, adjusting parameters like rate files as needed to optimize runtimes
• Perform data ingest into production databases using SQL and Python
• Clear data artifacts and caches in between ingest cycles
• Execute post-ingest data refresh routines
• Perform quality checks on ingested data to validate contractual obligations are met
• Identify process bottlenecks and suggest or implement improvements to the automated tooling to increase speed and reliability

Requirements

• Required Skills:
• Strong Linux command line skills
• Experience with Airflow or similar workflow orchestration tools
• Python programming proficiency
• Advanced SQL knowledge for data ingest, refresh, and validation
• Ability to diagnose and resolve issues with long-running batch processes
• Excellent attention to detail and problem-solving skills
• Good communication to coordinate with other teams
• Flexibility to handle off-hours work when needed to meet SLAs

• Preferred Additional Skills:
• Familiarity with network scanning tools and methodologies
• Experience optimizing database performance
• Scripting skills to automate routine tasks
• Understanding of common network protocols and services
• Knowledge of AWS services like EC2

SixMap is an Equal Opportunity Employer and considers applicants for employment without regard to race, color, religion, sex, orientation, national origin, age, disability, genetics, or any other basis forbidden under federal, state, or local laws. All new hires must pass a background check as a condition of employment.

Benefits

• Competitive compensation packages; including equity
• Employer paid medical, dental, vision, disability & life insurance
• 401(k) plans
• Flexible Spending Accounts (health & dependents)
• Unlimited PTO
• Remote Working Options