How can I get a list of failed requests or error info after the crawl completes?

[CodeDan-CN]

We need to collect new info. Specifically, we want to return the target and sub-links that failed during crawling. Is there a convenient way to retrieve this info right after the crawl finishes?

[Mantisus]

Hey @CodeDan-CN

You can use failed_request_handler to handle error cases, for example for tracking failed links.