# Web Scraping multiple pages VBA



## Jacob45678 (Oct 4, 2022)

Hi, I want to web scrape information to an excel worksheet for multiple pages. example from page 1 to 3. I tried a code found online from googling but it sends out an error. Any help is much appreciated. Thanks in advance. 

I would like to sort the information like the example shown below.









The Website








						Steam Community Market :: Showing all listings
					






					steamcommunity.com
				




The Code (VBA)

Sub TableScrapeFromURL()
'
    Dim ArrayColumn         As Long, ArrayRow   As Long
    Dim TableCell           As Object, TableRow As Object
    Dim Doc                 As Object, http     As Object
    Dim ScrapedTableArray   As Variant
'
     Set Doc = CreateObject("htmlFile")                                                 ' Late bind Microsoft HTML Object Library ... This can
'                                                                                       '       be late bound as long as 'class' is not needed
    Set http = CreateObject("msxml2.xmlhttp")                                           ' Late bind Microsoft XML
'
    ArrayColumn = 1: ArrayRow = 1                                                       ' Initialize ArrayColumn & ArrayRow to 1
'
    With http                                                                           '
        .Open "GET", "Steam Community Market :: Showing all listings", False   '   Load website to load
        .Send                                                                           '   Send website to load
        Doc.body.innerHtml = .responseText                                              '   save all data received from website
    End With
'
    With Doc.getelementsbytagname("table")(0)                                           '
        ReDim ScrapedTableArray(1 To .Rows.Length, 1 To .Rows(1).Cells.Length)          '   Resize ScrapedTableArray
'
        For Each TableRow In .Rows                                                      '   Loop through each TableRow
            For Each TableCell In TableRow.Cells                                        '       Loop through each TableCell in TableRow
                ScrapedTableArray(ArrayRow, ArrayColumn) = TableCell.innerText          '           Save value into ScrapedTableArray
                ArrayColumn = ArrayColumn + 1                                           '           Increment ArrayColumn
            Next                                                                        '       Loop back
'
            ArrayColumn = 1: ArrayRow = ArrayRow + 1                                    '       Reset ArrayColumn to 1, Increment ArrayRow
        Next                                                                            '   Loop back
    End With
'
    Range("A1").Resize(UBound(ScrapedTableArray, 1), _
            UBound(ScrapedTableArray, 2)) = ScrapedTableArray                           ' Display ScrapedTableArray to sheet
End Sub


----------



## Michal_Banasinski (Dec 19, 2022)

Hi there.

I did my standard test of website resistance in python. I tried to send 400 requests in a short time, but what's come out on 24 request server response 
with error <Response [429]>. It meant too many requests and that was tested with randomized headers - 'User-Agent. ( I try to mask that I am a robot, tries to pretend to be a browser)


{'User-Agent': 'Mozilla/5.0 (Linux; Android 12; SM-S906N Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/80.0.3987.119 Mobile Safari/537.36'}

It means you should try inputting some wait function to slow down your Makro [ I do not know how I do not know VBA only Python, but I am here to learn  ].
This website look's like quite a good protected against scraping, probably heavy tools like selenium will be necessary [ automated web browser  ].

It is common in very popular websites that they are protected but if they are popular enough probably there are already some pre-build tools to scrape them, If I were you I'd check RapidAPI to see if they have anything reasonably priced.


----------

