Lyft interview question

How to design a web crawler?