VBA Scraping from dynamic web page

[Bruzio]

Hi everybody. I'm trying to download soccer results from Football results 21 august 2020, I can't as I'd like.
My question was posted on a italian forum, but we were unable to resolve.
I trust in your knowledge.


What do i need?

6 columns:

LEAGUE - START - HOME - AWAY - SCORE- STATUS

What are the problems?

1) Macro doesn't download all scheduled events.
2) The content of the rows isn't correct
2) The web page has a more results button that is always visible and clickable, and i need to load all hidden pages.

I expect this, see screen:

This is my current code:

Sub fromweb()
    Dim IE As Object
    Dim Doc As Object
    Dim x As Object
    Dim i As Long
    Dim myArray As Variant
    Set IE = CreateObject("InternetExplorer.Application")
    Application.ScreenUpdating = False
    Cells.Clear
    Const myURL As String = "fctables.com/livescore/21_08_2020/"
    With IE
        .navigate myURL
        .Visible = True
        Do While .Busy: DoEvents: Loop
        Do While .readyState <> 4: DoEvents: Loop
        Application.Wait Now + TimeValue("0:00:10")
    End With
    myArray = Array("LEAGUE", "START", "HOME", "AWAY", "SCORE", "STATUS")
    With Range("A1:F1")
        .Value = myArray
    End With
    Set Doc = IE.Document
    For Each x In Doc.getElementsByClassName("league")
        i = i + 1: j = 0
        Cells(i + 1, 1) = Replace(Replace(x.innerText, "Table", ""), Chr(10), "")
        For Each y In Doc.getElementsByClassName("col-xs-8 col-sm-9 col-lg-10 truncate")
            Cells(i + 1, 1) = y.innerText
            Cells(i + 1, 2) = Doc.getElementsByClassName("date")(j).innerText
            Cells(i + 1, 3) = Doc.getElementsByClassName("home")(j).innerText
            Cells(i + 1, 4) = Doc.getElementsByClassName("away")(j).innerText
            Cells(i + 1, 5) = Doc.getElementsByClassName("score")(j).innerText
            Cells(i + 1, 6) = Doc.getElementsByClassName("status_name")(j).innerText
            j = j + 1: i = i + 1
        Next
    Next
    IE.Quit
    Set IE = Nothing
    Set Doc = Nothing
    Application.ScreenUpdating = True
End Sub

How to proceed correctly?
Thanks for the attention

 

[smozgur]

You know there is the main page and the odd pages that are opened for each match. That's where the odds data is retrieved from.

Unless the main page has no indication about the "there is no odd" situation for the matches, then the odd page will be opened anyway. So, unfortunately, the downloading time will not be different.

Hope it makes sense.

 

[Bruzio]

Yes I understand, but If gameOdd.getElementsByClassName("name").Item(0).innerText = clsOddGroupItem.items.Item(k), if item is empty I would like to skip download.

 

[smozgur]

Yes I understand, but If gameOdd.getElementsByClassName("name").Item(0).innerText = clsOddGroupItem.items.Item(k), if item is empty I would like to skip download.

To clarify better: those elements are in the odds page, which means after the individual odd page is loaded. So, the download already happens to get there. It doesn't matter you make a further comparison or not. Only if there is an indication for "no odds" on the "main" page, then you can avoid downloading the odds page.

 

[Maury1704]

Hi Bruzio Listen: As long as they agree with all the others as well
I'll try to answer this question of yours.
But to do it I would like to know:
1) If in the meantime you have already solved this question.
2) You should insert me the link of the web page in question.
3) You should kindly explain to me in a down to earth way the things you would like to Import into your excel sheet
I'll take care of the rest
Thanks and hear from us if you want!