Importing Data From Multiple URL, How do I speed up my query?

[silvanus3805]

I have a Table With URL's that I need power query to load table form each webpage and combine it as one

The problem is , my query takes long to load the data, I tried Table.buffer but it doesn't seem to speed it up any better, How do I Edit My Code to be efficient?


1st Table Named "Engine"

Column1	Column2
Get Offer Event Odds	*Special Bets*
Get Offer Event Odds	Argentina Liga Profesional Argentina
Get Offer Event Odds	Argentina Prim C Metro
Get Offer Event Odds	Argentina Prim D Metro
Get Offer Event Odds	Argentina Primera Nacional
Get Offer Event Odds	Argentina Reserve League
Get Offer Event Odds	Armenia Premier League

here is the code that extracts from the table and makes it as one but loads very slow


let
Source = Excel.CurrentWorkbook(){[Name="Engine"]}[Content],
tbl = Table.Buffer(Table.AddColumn(Source, "Tables",each Web.Page(Web.Contents(_[Column1])){0}[Data])),
#"Removed Errors" = Table.RemoveRowsWithErrors(tbl, {"Tables"}),
#"Expanded Tables" = Table.ExpandTableColumn(#"Removed Errors", "Tables", {"Kind", "Name", "Children", "Text", "Code", "Match/Offer Type", "Home", "Draw", "Away", "Kickoff", "Return to Leagues"}, {"Data.Kind", "Data.Name", "Data.Children", "Data.Text", "Data.Code", "Data.Match/Offer Type", "Data.Home", "Data.Draw", "Data.Away", "Data.Kickoff", "Data.Return to Leagues"}),
#"Removed Other Columns" = Table.SelectColumns(#"Expanded Tables",{"Column2", "Data.Match/Offer Type", "Data.Home", "Data.Draw", "Data.Away", "Data.Kickoff"}),
#"Filtered Rows" = Table.SelectRows(#"Removed Other Columns", each ([Data.Kickoff] <> null)),
#"Renamed Columns" = Table.RenameColumns(#"Filtered Rows",{{"Column2", "Custom"}}),
#"Split Column by Delimiter" = Table.SplitColumn(#"Renamed Columns", "Data.Kickoff", Splitter.SplitTextByDelimiter("/", QuoteStyle.Csv), {"Data.Kickoff.1", "Data.Kickoff.2", "Data.Kickoff.3"}),
#"Merged Columns" = Table.CombineColumns(Table.TransformColumnTypes(#"Split Column by Delimiter", {{"Data.Kickoff.2", type text}, {"Data.Kickoff.1", type text}}, "en-US"),{"Data.Kickoff.2", "Data.Kickoff.1", "Data.Kickoff.3"},Combiner.CombineTextByDelimiter("/", QuoteStyle.None),"Data.Kickoff"),
#"Replaced Value" = Table.ReplaceValue(#"Merged Columns","-"," - ",Replacer.ReplaceText,{"Data.Match/Offer Type"}),
#"Changed Type" = Table.TransformColumnTypes(#"Replaced Value",{{"Data.Kickoff", type datetime}}),
#"Filtered Rows1" = Table.SelectRows(#"Changed Type", each Date.IsInCurrentDay([Data.Kickoff])),
#"Filtered Rows2" = Table.SelectRows(#"Filtered Rows1", each ([Data.Draw] <> "-"))
in
#"Filtered Rows2"

 

[Matt Allington]

the speed is dependent on the speed the website will serve you the data. It could be throttled to prevent bots hitting the server with lots of requests, just like you are doing here.

 

[jaeiow]

I think it will load all of the data into memory, so you need a lot of memory for multiple files. But if there are any steps that use the whole table, like buffer, you will run into issues. Using multiple sources has caused slowness for me in the past. It might not be the best tool for the job.