MX interview question

How would you build a web crawler?