VBA Scrape Data from Website

[quick_question]

I am trying to scrape data from a website, in 2 Phases:

Phase 1:

The website is a directory that is only displaying 6 profiles per page. Because of this, there are many pages of information.

I want to go through each page (1, 2, 3, etc.) and extract (or copy) the "link address" (the information found in the green square highlighted in yellow in the below image) from each page. When each link address is copied, I would like it to be pasted into "Sheet2" cell B2, B3, B4, etc. always pasting into the next available row in column B.

Once all the profile URL's have been cataloged, move on to Phase 2...


Phase 2:

Use the link addresses that have been copied in B2:B#### (which appear as a url: https://directory....) to scrape each page for more information.

These link addresses are profile page url's.

I want to extract (copy & paste) the following information from each page to Sheet3 from left to right beginning in column A row 2.

Then Cycle through each different profile url, taken from Sheets B2, B3, etc. and extract (copy & paste) the same information in row 3, 4, 5, etc.

I'm only blocking specific information which would identify individuals name, address and phone numbers.

Scroll down for more information...

 

[diddi]

i suspect you can completely bypass the first phase by POSTing the full URL ending with 999900001 and iterating the member number. it depends on the setup of the site but you can either use something like xmlhttp or selenium. without the URL i cant help further as i need to tinker to get scrapes to work, the code does not just flow and work first time.

 

[Dan_W]

I agree with Diddi. I would also make the observation that this doesn't look straightforward, judging from the obscure class names that they've opted to use. I note from the code that there is a comment that Javascript needs to be enabled - is that why you've opted for Internet Explorer?