Web Query Issue

DharmaandGreg

New Member
Joined
Dec 29, 2009
Messages
7
Hello and thanks for taking the time to read this request for help. I've been trying to familiarize myself with MS Excel's Web Query function (MS Excel 2007) in order to collect statistics from a baseball game.

Here is this particular page I'd like to start pulling certain data from: http://www.mmobaseball.com/stats.aspx

I can get the first page of data easily, the first 50 or so players and their stats, but I cannot get anything from the other pages (which are accessed by the 'next button') nor can I get the defensive stats of any player (accessed by clicking the button labeled "defensive stats"). When I look at the page source I see that regardless of the page I'm viewing, excel only pulls data from the original page.

How can I collect the data from these other pages? Is there more information needed to assist me? Thanks in advance
smile.gif
 
<A HREF="http://cid-ea73b3a00e16f94f.skydrive.live.com/self.aspx/Mr%20Excel%20Example/www.mmobaseball.com.zip" TARGET="_blank">Example workbook: www.mmobaseball.com.xls.zip</A>

Hi Greg. I would prefer you to download the file that I actually worked on to make sure we both have the same resources available to us. If not, paste the code below into Sheet1 of a new workbook. Set two references from the VBAIDE. One to the Microsoft Internet Controls and the other to the Microsoft HTML Library. Run the procedure "Get Data". If you download the example, just click on the button. If working, it should grab all of the offensive data. Beyond this, assuming that everything is working on your end, it will be elementary to get the regions, levels, and defensive stats. The code can also be sped up a great deal but is displaying the results as it goes just to make sure it works. Let me know...

Code:
Option Explicit

Private Declare Sub Sleep Lib "kernel32" (ByVal dwMilliseconds As Long)

Private WithEvents IE As WebBrowser
Private ThisAction As String
Private GetObj As Object
Private PrevTdValue As String

Sub GetData()
    Dim OSRng As Range, DSRng As Range, Doc As HTMLDocument
    
    On Error Resume Next
        Application.DisplayAlerts = False
        Sheets("Offensive Stats").Delete
        Application.DisplayAlerts = True
    On Error GoTo 0
    
    With Sheets.Add()
        .Cells(1) = "Loading page..."
        .Cells(2, 1) = "Please wait..."
        .Name = "Offensive Stats"
        Set OSRng = .Range("A2:V2")
    End With
'    Set DSRng = Sheets.Add().Range("A2:V2")
    
    CreateInstance
    Set Doc = QueryGetDocument("DocumentComplete", "http://www.mmobaseball.com/stats.aspx")
    If Not Doc Is Nothing Then GetOffensiveTableData Doc, OSRng
    
    Do Until Doc.getElementById("lbtnNextOffense") Is Nothing
        Application.Goto OSRng(1)
        OSRng(1) = "Loading next page..."
        Doc.getElementById("lbtnNextOffense").Click
        Set Doc = QueryGetDocument("Table_onafterupdate", , Doc)
        If Not Doc Is Nothing Then GetOffensiveTableData Doc, OSRng
    Loop
    
    With OSRng.Parent
        .Range("A1:V1") = Array("Player", "POS", "TEAM", "AB", "R", "H", "2B", "3B", "HR", "AVG", "RBI", "SLG", "OBP", "TB", "BB", "SO", "SB", "CS", "SAC", "HBP", "IBB", "GDP", "TPA", "XBH")
        .Range("A1:V1").Font.Bold = True
        .Columns("A:V").EntireColumn.AutoFit
        Application.Goto .Cells(1), True
    End With
    
    Set GetObj = Nothing
    IE.Quit
    Set IE = Nothing
    PrevTdValue = vbNullString
    MsgBox "All done"
End Sub

Private Sub GetOffensiveTableData(Doc As HTMLDocument, DestRng As Range)
    Dim TR As HTMLTableRow, x As Integer
    
    For Each TR In Doc.getElementById("maintable").Children(0).tBodies(0).Rows
        If x > 0 Then
            Application.Goto DestRng(1)
            DestRng(1) = TR.Cells(0).Children(0).innerText
            For x = 1 To 21
                DestRng(x + 1) = TR.Cells(x).innerText
            Next x
            
            Set DestRng = DestRng.Offset(1)
        Else
            PrevTdValue = Doc.getElementById("maintable").Children(0).tBodies(0).Rows(1).Cells(0).innerText
        End If
        x = x + 1
    Next
End Sub

Private Sub CreateInstance(Optional Visible As Boolean = False)
    Set IE = New InternetExplorer
    IE.Visible = Visible
End Sub

Private Function QueryGetDocument(Action As String, Optional URL As String = "Event", Optional Doc As HTMLDocument) As HTMLDocument
    Dim TD As Object
    
    ThisAction = Action
    Set GetObj = Nothing
    TimedOut 30
    If URL <> "Event" Then IE.navigate URL
    
    Do
        Sleep 100
        DoEvents
        If Action = "Table_onafterupdate" Then
            On Error Resume Next
            Set TD = Doc.getElementById("maintable").Children(0).tBodies(0).Rows(1).Cells(0)
            If Not TD Is Nothing And Err.Number = 0 Then
                If TD.innerText <> "" And TD.innerText <> PrevTdValue And Err.Number = 0 Then
                    Set QueryGetDocument = Doc
                    Exit Function
                End If
            End If
            On Error GoTo 0
            Err.Clear
        End If
    Loop Until Not GetObj Is Nothing Or TimedOut
    
    If TimedOut Then
        'timed out
        Exit Function
    End If
    
    Set QueryGetDocument = GetObj
    
End Function

Private Sub IE_DocumentComplete(ByVal pDisp As Object, URL As Variant)
    If ThisAction = "DocumentComplete" Then Set GetObj = IE.document
End Sub

Private Function TimedOut(Optional Seconds As Integer) As Boolean
    Static TimeOutTime As Date
    
    If Seconds > 0 Then
        TimeOutTime = DateAdd("s", Seconds, Now)
    Else
        TimedOut = (Now > TimeOutTime)
    End If
End Function
 
Upvote 0

Excel Facts

Show numbers in thousands?
Use a custom number format of #,##0,K. Each comma after the final 0 will divide the displayed number by another thousand
I downloaded and tested the linked file and it works wonderfully. And as you said, it is very simple to adjust it to do the same thing for the other data I want. Thanks again for your assistance and patience, Tom.
 
Upvote 0
There is an error Greg. I got the number of fields mixed up. Because of this, it is only importing 22 of 24 fields for the offense. I'll send another file along shortly. The code is a bit different between the offense and defense and this file will reflect that.
 
Upvote 0
You said you needed each level for each region? Are you going to do this navigation manually?
 
Upvote 0
Ok. I'll get a file to you shortly. I have a slow internet connection (mobile) so I will not be able to test this entire thing. It should download everything. BRB.
 
Upvote 0
<A HREF="http://cid-ea73b3a00e16f94f.skydrive.live.com/self.aspx/Mr%20Excel%20Example/www.mmobaseball.com^_2.zip" TARGET="_blank">www.mmobaseball.com_2.xls.zip</A>

I did not forget about you. I cannot believe how much time I spent on this. You owe me a beer.
 
Upvote 0
Wow, this is very cool Tom. It'll save me a ton of time and works well. I'm extremely thankful you took the time to do this on my behalf, that I owe you a beer is an understatement.
 
Upvote 0
Hey Tom amazing work with this excel sheet. I wanted to make a similar web query to extract data from my site. How can i edit the code on this sheet and import data from any other site..... Please help
 
Upvote 0

Forum statistics

Threads
1,223,896
Messages
6,175,262
Members
452,627
Latest member
KitkatToby

We've detected that you are using an adblocker.

We have a great community of people providing Excel help here, but the hosting costs are enormous. You can help keep this site running by allowing ads on MrExcel.com.
Allow Ads at MrExcel

Which adblocker are you using?

Disable AdBlock

Follow these easy steps to disable AdBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the icon in the browser’s toolbar.
2)Click on the "Pause on this site" option.
Go back

Disable AdBlock Plus

Follow these easy steps to disable AdBlock Plus

1)Click on the icon in the browser’s toolbar.
2)Click on the toggle to disable it for "mrexcel.com".
Go back

Disable uBlock Origin

Follow these easy steps to disable uBlock Origin

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back

Disable uBlock

Follow these easy steps to disable uBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back
Back
Top