fredrerik84
Active Member
- Joined
- Feb 26, 2017
- Messages
- 383
Hi I was wondring if there is a better or faster way to scrape large amounts of data that the one im using ?
I genrally run a code like this:
I find the html doc and loop trough them with an for i loop
But particularly in the weekends these sites im extracting have around 1400 rows and it does take quite some time for it to run trough it all
anyone have a suggestion to how I can reduce the load time
I genrally run a code like this:
Code:
Sub somesite()
Dim XMLReq As New MSXML2.XMLHTTP60
Dim HTMLDoc As Object
Dim HTMLRows As MSHTML.IHTMLElementCollection
Dim i As Long, j As Long
Dim URL As String
Dim Result As String
URL = "www.comesite.com"
With XMLReq
.Open "GET", URL, False
.send
Set HTMLDoc = CreateObject("HTMLfile")
HTMLDoc.Open
HTMLDoc.write .responseText
HTMLDoc.Close
End With
Result = XMLReq.responseText
HTMLDoc.body.innerHTML = Result
Set HTMLRows = HTMLDoc.getElementsByTagName("td")
j = 2
For i = 0 To HTMLRows.Length - 1
On Error Resume Next
Cells(j, "B").value = HTMLRows(i).innerText
j = j + 1
Next i
Application.ScreenUpdating = True
End Sub
I find the html doc and loop trough them with an for i loop
But particularly in the weekends these sites im extracting have around 1400 rows and it does take quite some time for it to run trough it all
anyone have a suggestion to how I can reduce the load time