Web Crawler Engineer (C++/Rust, remote or relocation)

About Company/Product:

• Series A startup, headquartered in San Francisco and backed by top-tier investors
• building web-scale systems for search, provide web search and data to everyone: LLMs, Ai agents, in-house apps for enterprises, private markets
• the team runs batch jobs on 1000+ machines and develops production services operating on many terabytes of text

About the Role:
We are looking for a Web Crawler Engineer to join the core team, responsible for building and scaling a distributed crawler that could ingests 100M+ pages per day. You’ll be a key contributor designing systems that handle complex challenges like dynamic JavaScript rendering, anti-bot bypass, crawl scheduling, and prioritization to maximize coverage and data quality.

What You’ll Do:

• Build and scale a distributed web crawler to handle massive data ingestion
• Design and optimize anti-bot bypass techniques, dynamic JS rendering, and crawl scheduling algorithms
• Own domain-specific crawl prioritization, politeness logic, and infrastructure scaling
• Collaborate closely with backend and search infrastructure teams to support indexing and search use cases

We expect that you have:

• Proven experience building and scaling web crawlers
• Strong programming skills in C++ or Rust (or other high-performance languages)
• Familiarity with browser automation, headless rendering, and anti-bot technologies
• A systems optimization mindset and comfort working with complex distributed systems for both speed and scalability
• Passion for building tools that feed high-quality, diverse data into search
• Proficiency in English

Nice to Have(s):

• Experience working in early-stage or hyper-growth startups
• Familiarity with internet-scale data ingestion, page scoring, and prioritization
• Understanding of ML-powered ranking or LLM-based processing pipelines

Tech Stack: Rust, AWS, Kubernetes, Python, Puppeteer/Playwright

What we offer:

• High salary plus above-market equity
• Work format: Remote OR relocation to San Francisco
• Full relocation and visa support(STEM OPT, OPT, H1B, O1, E3)
• Unlimited vacation policy