PQ slow when refreshing, load to as pivot table

danhendo888

Board Regular
Joined
Jul 15, 2019
Messages
167
Office Version
  1. 365
Platform
  1. Windows
I have two folders, A and B, which currently have 4 Excel files in each. So 8 files total. Each file is roughly 50,000 rows, 21 columns, each cell has data (not formulas) and is between 10 to 15mb (each).
There will be more excel files added as each file represents one calendar month.

In power query, I will import folder A so that when I expand the column with the tables in editor, it automatically joins all 4 files. I'll remove columns, rename columns etc and I end up with about 15 to 20 query steps.
This is query 1

I will do the same for folder B. This is query 2. I will rename, remove columns etc. Then from this query, I will append with query 1 (so that all data from folders A and B are appended) and load to data model.
So query 2 ends up having 600,000 rows. Both the queries are in one excel file of course and the file is currently 8mb.

Is it normal for a refresh of the query 2 to be slow? (1minute 10 seconds)

And also when I load to pivot table, it is also quite slow (also 1 minute 10 seconds)

FWIW
The data in each file is not in table format.
There are two sheets in each file, I only use one of the sheets and not the other.
Before I loaded the queries into the data model, I had them loaded as a connection only and made pivot tables for testing purposes. I have since deleted those pivot tables and currently I have only one data model pivot table.
Only query 2 is loaded to the data model. Query 1 is just a connection.
 

Excel Facts

Best way to learn Power Query?
Read M is for (Data) Monkey book by Ken Puls and Miguel Escobar. It is the complete guide to Power Query.
Yes, this is normal. 70 seconds is not considered slow. You are doing ETL here, not report consumption. The whole idea is to push the load work upstream. Once loaded, the reporting should be fast.
 
Upvote 1
Solution

Forum statistics

Threads
1,224,816
Messages
6,181,138
Members
453,021
Latest member
Justyna P

We've detected that you are using an adblocker.

We have a great community of people providing Excel help here, but the hosting costs are enormous. You can help keep this site running by allowing ads on MrExcel.com.
Allow Ads at MrExcel

Which adblocker are you using?

Disable AdBlock

Follow these easy steps to disable AdBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the icon in the browser’s toolbar.
2)Click on the "Pause on this site" option.
Go back

Disable AdBlock Plus

Follow these easy steps to disable AdBlock Plus

1)Click on the icon in the browser’s toolbar.
2)Click on the toggle to disable it for "mrexcel.com".
Go back

Disable uBlock Origin

Follow these easy steps to disable uBlock Origin

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back

Disable uBlock

Follow these easy steps to disable uBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back
Back
Top