For example, I'm on yelp and search for 'burgers' in 'maine'
I have a for loop to scrape each of the 10 results and then click the link to get the next 10, until page 5 (so 50 results in total)
I run into a problem at number 11 (where page 2) would start.
I think a the source of the problem is that it would click 2 button and start scraping before it loads.
Code below:
This clicks the 'next' button on yelp
followed by 'wait until the page load':
However even with the do until ie. readystate = 4 as I pasted above --- it is actually NOT waiting.
Been looking into this for hours, this happens when I scrape google, and click the next button to go to page 2 as well. Any suggestions much appreciated.
I have a for loop to scrape each of the 10 results and then click the link to get the next 10, until page 5 (so 50 results in total)
I run into a problem at number 11 (where page 2) would start.
I think a the source of the problem is that it would click 2 button and start scraping before it loads.
Code below:
This clicks the 'next' button on yelp
Code:
ie.document.getelementsbyclassname("pagination-label responsive-hidden-small pagination-links_anchor")(0).Click
followed by 'wait until the page load':
Code:
Do Until (ie.readyState = 4 And Not ie.Busy)
DoEvents
Loop
However even with the do until ie. readystate = 4 as I pasted above --- it is actually NOT waiting.
Been looking into this for hours, this happens when I scrape google, and click the next button to go to page 2 as well. Any suggestions much appreciated.