Download web table in multiple pages

headhair

New Member
Joined
Nov 8, 2010
Messages
15
I have to copy web table data.
The data is one table across multiple pages

The Link is:

http://www.remotedatacentre.com/co-so/cbstateall.aspx?dist=Chennai-North&dates=11/9/2010

In the website itself there is a button
"export to excel" to save the whole table to an excel file.

I want to have the whole table (All data spread
in multiple pages - that is when you click
"export to excel" button) automatically
using webquery or vba in specified xls file
daily to my computer's specified location.

Please help to write code to get all
data into excel file
 

Excel Facts

Who is Mr Spreadsheet?
Author John Walkenbach was Mr Spreadsheet until his retirement in June 2019.
Here is some code which automates the whole manual process of navigating to that web page, clicking the Export To Excel button and saving the .xls download.

Put the following code in a standard module (Module1):
Code:
'10-Nov-2010
'http://www.mrexcel.com/forum/showthread.php?t=507871

'Note - IE may block the download, displaying its Information Bar at the top of the tab, and preventing this
'program from automatically downloading the file.  To prevent this, add http://www.remotedatacentre.com
'to IE's Trusted sites (Tools - Internet Options - Security - Trusted sites - Sites).


Option Explicit


Public Const baseURL As String = "http://www.remotedatacentre.com/co-so/cbstateall.aspx?dist=Chennai-North"


Public Sub Test_Download()
    Dim URL As String
    
    'Construct complete URL from the base URL and the dates parameter (mm/d/yyyy format)
    
    'Example fixed date
    URL = baseURL & "&dates=11/9/2010"
    
    'Today's date
    'URL = baseURL & "&dates=" & Format(Date, "mm/d/yyyy")

    Download_File URL, "", ""
End Sub


Public Sub Download_File(URL As String, saveInFolder As String, saveFilename As String)

    Dim IE As Object
    
    Set IE = Get_IE_Window_LB(URL)
    If IE Is Nothing Then
        Set IE = CreateObject("InternetExplorer.Application")
    End If

    With IE
        .Visible = True
        .Navigate URL

        While .busy Or .ReadyState <> 4: DoEvents: Wend

        'Click the 'Export To Excel' button
        '< input type="submit" name="ctl00$ContentPlaceHolder1$ExportToExcel1" value="Export To Excel"
        'id="ctl00_ContentPlaceHolder1_ExportToExcel1" />
        
        .Document.all("ctl00_ContentPlaceHolder1_ExportToExcel1").Click
    End With
    
    File_Download_Click_Save
        
    Save_As_Set_Filename saveInFolder, saveFilename
    
    Save_As_Click_Save
       
    Download_complete_Click_Close
    
    Debug.Print "Finished"
    
End Sub


Private Sub File_Download_Click_Save()
    
    Dim hWnd As Long
    Dim timeout As Date
    
    Debug.Print "File_Download_Click_Save"
    
    'Find the File Download window, waiting a maximum of 30 seconds for it to appear
    
    timeout = Now + TimeValue("00:00:30")
    Do
        hWnd = FindWindow("#32770", "File Download")
        DoEvents
        Sleep 200
    Loop Until hWnd Or Now > timeout
    
    Debug.Print "   File Download window "; Hex(hWnd)
    
    If hWnd Then

        'Find the child Save button
        
        hWnd = FindWindowEx(hWnd, 0, "Button", "&Save")
        Debug.Print "   Save button "; Hex(hWnd)
    End If
    
    If hWnd Then
    
        'Click the Save button
        
        SetForegroundWindow (hWnd)
        Sleep 600  'this sleep is required and 600 miiliseconds seems to be the minimum that works
        SendMessage hWnd, BM_CLICK, 0, 0
    End If

End Sub


Private Sub Test_Save_As_Set_Filename()
    'Test setting the Save As filename.  The Save As window must be displayed before running this
    Dim theFolder As String, theFilename As String
    theFolder = "C:\temp\Excel\"
    theFilename = "test " & Format(Now, "hh_mm_ss") & ".xls"
    Save_As_Set_Filename theFolder, theFilename
End Sub


Private Sub Save_As_Set_Filename(folder As String, filename As String)

    'Populate the 'File name:' edit window in the Save As dialogue with the specified folder and/or filename.
    'If folder = "" a folder path is not prepended and therefore the default save folder is used.
    'If filename = "" the default file name (already populated) is used.
    
    'The Save As window has the following child window hierarchy:
    
    '   "Save As", #32770 Dialog
    '       "FileName2011_11_11_11_00_26", ComboBoxEx32     (default value in combobox)
    '           "", ComboBox
    '               "FileName2011_11_11_11_00_26"", Edit    (default value in combobox's edit box)
    
    Dim hWnd As Long
    Dim timeout As Date
    Dim fullFilename As String
    
    Debug.Print "Save_As_Set_Filename " & folder
    
    'Find the Save As window, waiting a maximum of 10 seconds for it to appear
    
    timeout = Now + TimeValue("00:00:10")
    Do
        hWnd = FindWindow("#32770", "Save As")
        DoEvents
        Sleep 200
    Loop Until hWnd Or Now > timeout

    If hWnd Then
    
        SetForegroundWindow (hWnd)
        
        'Find the child ComboBoxEx32 window
        
        hWnd = FindWindowEx(hWnd, 0, "ComboBoxEx32", vbNullString)
        Debug.Print "   ComboBoxEx32 "; Hex(hWnd)
    End If
    
    If hWnd Then
    
        'Find the child ComboBox window
        
        hWnd = FindWindowEx(hWnd, 0, "ComboBox", "")
        Debug.Print "   ComboBox "; Hex(hWnd)
    End If
                 
    If hWnd Then
        
        SetForegroundWindow (hWnd)

        'Find the child Edit window
        
        hWnd = FindWindowEx(hWnd, 0, "Edit", "")
        Debug.Print "   Edit "; Hex(hWnd)
    End If
    
    If hWnd Then
            
        If filename = "" Then
            'Get default filename (already populated in Edit window)
            filename = Get_Window_Text(hWnd)
        End If
       
        If folder <> "" And Right(folder, 1) <> "\" Then folder = folder & "\"  'if specified, ensure folder ends with \
        
        fullFilename = folder & filename
        Debug.Print "Full filename " & fullFilename
        
        'Populate the Edit window with the full file name
        
        Sleep 200
        SendMessageByString hWnd, WM_SETTEXT, Len(fullFilename), fullFilename
    End If
    
End Sub


Private Function Get_Window_Text(hWnd As Long) As String

    'Returns the text in the specified window
    
    Dim buffer As String
    Dim length As Long
    Dim result As Long
    
    length = SendMessage(hWnd, WM_GETTEXTLENGTH, 0, 0)
    buffer = Space(length + 1)  '+1 for the null terminator
    result = SendMessage(hWnd, WM_GETTEXT, Len(buffer), ByVal buffer)
    Debug.Print "Edit File name = " & Left(buffer, length)
    Debug.Print "     length = " & length
    
    Get_Window_Text = Left(buffer, length)
    
End Function


Private Sub Save_As_Click_Save()

    'Click the Save button in the Save As dialogue
    
    Dim hWnd As Long
    Dim timeout As Date
    
    Debug.Print "Save_As_Click_Save"

    'Find the Save As window, waiting a maximum of 10 seconds for it to appear
    
    timeout = Now + TimeValue("00:00:10")
    Do
        hWnd = FindWindow(vbNullString, "Save As")
        DoEvents
        Sleep 200
    Loop Until hWnd Or Now > timeout

    If hWnd Then
    
        SetForegroundWindow (hWnd)
            
        'Get the child Save button
        
        hWnd = FindWindowEx(hWnd, 0, "Button", "&Save")
        Debug.Print "   Save button "; hWnd
    End If
    
    If hWnd Then
        'Click the Save button
        
        SendMessage hWnd, BM_CLICK, 0, 0
    End If
        
End Sub


Private Sub Download_complete_Click_Close()
    
    Dim hWnd As Long
    Dim timeout As Date
    
    Debug.Print "Download_complete_Click_Close"
        
    'Find the Download complete window, waiting a maximum of 30 seconds for it to appear.  Timeout value is dependent on the
    'size of the download, so make it longer for bigger files
    
    timeout = Now + TimeValue("00:00:30")
    Do
        hWnd = FindWindow("#32770", "Download complete")
        DoEvents
        Sleep 200
    Loop Until hWnd Or Now > timeout

    Debug.Print "   Download complete window "; Hex(hWnd)
    
    If hWnd Then

        'Find the child Close button
        
        hWnd = FindWindowEx(hWnd, 0, "Button", "Close")
        Debug.Print "   Close button "; Hex(hWnd)
    End If
    
    If hWnd Then
    
        'Click the Close button
        
        SetForegroundWindow (hWnd)
        Sleep 600  'this sleep is required and 600 miiliseconds seems to be the minimum that works
        SendMessage hWnd, BM_CLICK, 0, 0
    End If

End Sub


Public Function Get_IE_Window_LB(sUrl As String, Optional sProtocol As String = "http") As Object

    'Look for an IE window or tab already open at the specified URL (excluding sub paths) and, if found, return that browser
    'as an InternetExplorer object.  Otherwise return Nothing

    Dim sDomain As String
    Dim Shell As Object
    Dim IE As Object
    Dim i As Variant 'Integer
    
    If Left(sUrl, Len(sProtocol)) <> sProtocol Then sUrl = sProtocol & sUrl
    sDomain = Left(sUrl, InStr(Len(sProtocol) + 1, sUrl, "/"))

    Set Shell = CreateObject("Shell.Application")
    
    i = 0
    Set Get_IE_Window_LB = Nothing
    While i < Shell.Windows.Count And Get_IE_Window_LB Is Nothing

        Set IE = Shell.Windows.Item(i)
        If Not IE Is Nothing Then
            If TypeName(IE.Document) = "HTMLDocument" Then
                If InStr(IE.LocationURL, sDomain) > 0 Then
                    Set Get_IE_Window_LB = IE
                End If
            End If
        End If
        i = i + 1
    Wend
    
End Function
and put the following code in a separate standard module (Module2):
Code:
Option Explicit

Public Declare Sub Sleep Lib "kernel32" _
    (ByVal dwMilliseconds As Long)

Public Declare Function FindWindow Lib "user32" Alias "FindWindowA" _
    (ByVal lpClassName As String, ByVal lpWindowName As String) As Long

Public Declare Function FindWindowEx Lib "user32" Alias "FindWindowExA" _
    (ByVal hWnd1 As Long, ByVal hWnd2 As Long, ByVal lpsz1 As String, ByVal lpsz2 As String) As Long

Public Declare Function SetForegroundWindow Lib "user32" _
    (ByVal hWnd As Long) As Long

Public Declare Function SendMessage Lib "user32" Alias "SendMessageA" _
    (ByVal hWnd As Long, ByVal wMsg As Long, ByVal wParam As Long, lParam As Any) As Long

Public Declare Function SendMessageByString Lib "user32" Alias "SendMessageA" _
    (ByVal hWnd As Long, ByVal wMsg As Long, ByVal wParam As Long, ByVal lParam As String) As Long

Public Const BM_CLICK = &HF5
Public Const WM_SETTEXT = &HC
Public Const WM_GETTEXT = &HD
Public Const WM_GETTEXTLENGTH = &HE
Run the Test_Download macro to test it. As noted in the comments at the top of Module1, IE may block the download. If this happens you must add http://www.remotedatacentre.com to IE's trusted sites.

Note that in the line

Download_File URL, "", ""

the 2nd argument is the folder in which the file is saved. An empty string ("") means that the file is saved in IE's default folder (the last folder it saved a downloaded file in);

and the 3rd argument is the filename to save the download as. An empty string ("") means that default filename, as given by the web site, is used.

The combination of folder and filename must not exist because my code doesn't handle the 'file already exists' window if it pops up. Fortunately, this particular web site generates a unique filename (based on date and time), so you can use its own filename and there is no danger of the 'file already exists' window appearing.
 
Last edited:
Upvote 0
Hi, I have the same request as the above, except that this time the url address is different..

Would someone be able to help?

For eg, the link is http://www.airport-data.com/manuf/Rockwell_International.html

Also, I might want to do it with more than one manuf i.e. this time round its "Rockwell_International", but next time it might be "Bell" for example..where the ones I would select would be first http://www.airport-data.com/manuf/B.html -> then http://www.airport-data.com/manuf/Bell.html for example...so there are 2 level of difference..is it possible to write something for that too??
 
Upvote 0

Forum statistics

Threads
1,223,910
Messages
6,175,316
Members
452,634
Latest member
cpostell

We've detected that you are using an adblocker.

We have a great community of people providing Excel help here, but the hosting costs are enormous. You can help keep this site running by allowing ads on MrExcel.com.
Allow Ads at MrExcel

Which adblocker are you using?

Disable AdBlock

Follow these easy steps to disable AdBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the icon in the browser’s toolbar.
2)Click on the "Pause on this site" option.
Go back

Disable AdBlock Plus

Follow these easy steps to disable AdBlock Plus

1)Click on the icon in the browser’s toolbar.
2)Click on the toggle to disable it for "mrexcel.com".
Go back

Disable uBlock Origin

Follow these easy steps to disable uBlock Origin

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back

Disable uBlock

Follow these easy steps to disable uBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back
Back
Top