Scrape large amounts of data

fredrerik84

Active Member
Joined
Feb 26, 2017
Messages
383
Hi I was wondring if there is a better or faster way to scrape large amounts of data that the one im using ?

I genrally run a code like this:

Code:
Sub somesite()
Dim XMLReq As New MSXML2.XMLHTTP60
Dim HTMLDoc As Object
Dim HTMLRows As MSHTML.IHTMLElementCollection
Dim i As Long, j As Long
Dim URL As String
Dim Result As String
URL = "www.comesite.com"


With XMLReq
   .Open "GET", URL, False
   .send
   Set HTMLDoc = CreateObject("HTMLfile")
   HTMLDoc.Open
   HTMLDoc.write .responseText
   HTMLDoc.Close
End With
    
Result = XMLReq.responseText


HTMLDoc.body.innerHTML = Result
Set HTMLRows = HTMLDoc.getElementsByTagName("td")
 
j = 2
For i = 0 To HTMLRows.Length - 1
    On Error Resume Next
    Cells(j, "B").value = HTMLRows(i).innerText
    j = j + 1
Next i
Application.ScreenUpdating = True
End Sub

I find the html doc and loop trough them with an for i loop

But particularly in the weekends these sites im extracting have around 1400 rows and it does take quite some time for it to run trough it all

anyone have a suggestion to how I can reduce the load time
 

Excel Facts

Wildcard in VLOOKUP
Use =VLOOKUP("Apple*" to find apple, Apple, or applesauce

Forum statistics

Threads
1,224,818
Messages
6,181,151
Members
453,021
Latest member
Justyna P

We've detected that you are using an adblocker.

We have a great community of people providing Excel help here, but the hosting costs are enormous. You can help keep this site running by allowing ads on MrExcel.com.
Allow Ads at MrExcel

Which adblocker are you using?

Disable AdBlock

Follow these easy steps to disable AdBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the icon in the browser’s toolbar.
2)Click on the "Pause on this site" option.
Go back

Disable AdBlock Plus

Follow these easy steps to disable AdBlock Plus

1)Click on the icon in the browser’s toolbar.
2)Click on the toggle to disable it for "mrexcel.com".
Go back

Disable uBlock Origin

Follow these easy steps to disable uBlock Origin

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back

Disable uBlock

Follow these easy steps to disable uBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back
Back
Top