Recent Scrapy Posts from Tech Monger :///feeds/scrapy/ 2018-11-20T07:06:50.542649Z Werkzeug Solving Scrapy User timeout caused connection failure https://techmonger.github.io/65/troubleshoot-scrapy-user-timeout 2018-11-20T07:06:50.542649Z 2018-11-19T18:00:39.563913Z Tech Monger <p class="lead">If the website you are scraping is not responding to your requests then your spider would report failure due to request timeout and throw following error.</p> Scrapy Get Redirect URLs and Request URL https://techmonger.github.io/63/scrapy-redirect-urls 2018-11-10T07:16:38.730746Z 2018-11-09T13:46:24.517791Z Tech Monger <p class="lead"> If you ever wanted to figure out all the redirect urls that scrapy spider hopped on or what is the currently requested URL by the spider then you easily get that using following <a href="#example">example code</a>. Scrapy Random Fake User Agent Example https://techmonger.github.io/62/example-scrapy-fake-user-agent 2018-10-19T19:16:03.373783Z 2018-10-19T18:30:06.053868Z Tech Monger <p class="lead"> If sites you are crawling with scrapy dont respond to your request then you should use randomly generated user agent in your request. Scrapy Fake User Agent is one of the open source and useful extension which will help you evade bot detection programs easily.