OpenAI interview question

Design a web crawler at scale