Scraping from Javascript rendered web page

[fredrerik84]

Hi mr excel comunity I wonder if anyone know how to handle pages that are generated from javascript.

here is my basic code web scraper code:

Sub Somesite()


Dim XMLReq As New MSXML2.XMLHTTP60
Dim HTMLDoc As New MSHTML.HTMLDocument
Dim HTMLRows As MSHTML.IHTMLElementCollection
Dim URL As String
Dim i As Long, j As Long


URL = "http://www.somesite.com"
    
    With XMLReq
        .Open "GET", URL, False
        .send
        Do While .readyState <> 4
            DoEvents
        Loop
    End With
    
    If XMLReq.Status <> 200 Then
        MsgBox "Error " & XMLReq.Status & ":  " & XMLReq.statusText
        Exit Sub
    End If
    
    HTMLDoc.body.innerHTML = XMLReq.responseText
    Set HTMLRows = HTMLDoc.getElementsByTagName("")
    
    
       j = 31
       For i = 0 To HTMLRows.Length - 1
          On Error Resume Next
          Cells(j, "Q").Value = HTMLRows(i).innerText
          j = j + 1
       Next i




End Sub

This basic code works for the sites ive testet that doesnt include javascript. But when I try to scrape from a site that uses javascript to generete its content all of this is not loaded into the html doc.

Does anyone know how to get this content using XMLReq., ?

best regards

frederik