Power Query - Web Scraping - Timeout - How to handle or work around?

[Pak Mariman]

I've created the below function which drills down to a specific information on a particular website. If I call the function from a query for a collection of URLs, with varying Par01 between 1 and 10, I get a timeout error: "[DataSource.Error]The Web.Page function didn't finish within timeout of 100 seconds."

Using Source = Web.Page(Web.Contents(URL,[Timeout=#duration(0,0,X,0)])), where X is a couple of minutes, does not help much either; same timeout result.

If I create a query from the function, i.e. replace URL, Par01 and Par02 with specific input, the query works fine. I am guessing the website doesn't like me scraping the data much, and my data request is sent to Nirvana. I do not speak html much.

In VBA I can get around this defense by adding a pause somewhere in the code before it moves on to grab the next data from a website. How to do this in Power Query? Or is there another workaround for this timeout error?

(URL as text, Par01 as number, Par02 as number) =>
let
    Source = Web.Page(Web.Contents(URL)),
    Data8 = Source{8}[Data],
    Children0 = Data8{0}[Children],
    Children2 = Children0{2}[Children],
    Children10 = Children2{10}[Children],
    Children21 = Children10{21}[Children],
    Children1 = Children21{1}[Children],
    Children3 = Children1{3}[Children],
    Children4 = Children3{1}[Children],
    Children5 = Children4{3}[Children],
    Children6 = Children5{1}[Children],
    Children7 = Children6{1}[Children],
    Children = Children7{Par01-1}[Children],
    #"Expanded Children" = Table.ExpandTableColumn(Children, "Children", {"Text"}, {"Text.1"}),
    Output = #"Expanded Children"{Par02}
   in
    Output

 

[Pak Mariman]

*Bump*