Scrape data from a webpage but no tables show up.

[TedX]

I have tried to grab (scrape) specific data from this free website but it shows no tables, despite having tables in the sourcecode. I checked out the code on this page: Newcastle Horse Racing Odds Comparison | 2023-01-17 and it goes on forever.

At the bottom of this image, there is a Market % row, which has 16 columns. I want to hit a button and have it poll that page on any race, or any other race meeting on the day (I realise I would have to manually copy and paste the webpage url into a dialogue box), and return to me, the average number of Market %. So in the example shown in the image below, the answer would be: 122.62%

I want to wait a few minutes and hit the button again and typically the % should be lower at say 121.57% and each time I hit the button, as we get closer to race time, the % should change and in a perfect world I might see 117%. Regardless of what the % is, I want it to appear in a specific cell of AD4 on my worksheet. So in the words of whoever said it, "How hard can this possibly be?"

Here is my failed attempt at grabbing data, from this page: Newcastle Horse Racing Odds Comparison | 2023-01-17

 

[Dan_W]

It sounds like the content is being generated with Javascript. When you're accessing it. You're accessing the code as it exists at one moment in time - it's likely that this website uses Javascript to retrieve the data over a period, to be rendered it in a browser.

You'll probably have to use Selenium or some other method to scrape these such websites. Always best to use APIs, if it's an option.

 

[TedX]

It sounds like the content is being generated with Javascript. When you're accessing it. You're accessing the code as it exists at one moment in time - it's likely that this website uses Javascript to retrieve the data over a period, to be rendered it in a browser.

You'll probably have to use Selenium or some other method to scrape these such websites. Always best to use APIs, if it's an option.

Thanks Dan, I was one millisecond away from shutting down and going to sleep, it's the middle of the night in Australia. I don't know Selenium but I will tomorrow after I wake up, I'm sure there will be some tutorials about it out there, probably on YouTube. Thanks for the tip and I'll get back to you.