How To Use CrawlSpider From Scrapy To Click A Link With Javascript Onclick?

I want scrapy to crawl pages where going on to the next link looks like this: Next Will scrapy be able to interpret j

Solution 1:

The actual methodology will be as follows:

1. Post your request to reach the page (as you are doing)
2. Extract link to the next page from that particular response
3. Simple Request the next page if possible or use FormRequest again in applicable

All this have to be streamlined with the server response mechanism, e.g:

• You can try using dont_click = true in FormRequest.from_response
• Or you may want to handle the redirection (302) coming from the server (in which case you will have to mention in the meta that you require the handle redirect request also to be sent to callback.)

Now how to figure it all out: Use a web debugger like fiddler or you can use Firefox plugin FireBug, or simply hit F12 in IE 9; and check the requests a user actually makes on the website match the way you are crawling the webpage.