Which Data Import Method to Use in this Scenario ?

[happy1001]

Hello

I have got a few Excel Files in which I need to import the data from a few different queries from a database in SQL Server installed on my local machine. Then I intend to use this imported data inside Power Pivot Data Model.

I am not sure which particular data import method should be used, which will provide me the following flexibility -

1. Ability to change / modify the SOURCE data query which resides inside SQL Server. I will have to add or remove columns from that query based on the testing work that I am doing. The New Columns that I add to that SQL query in the future, should be able to get imported inside Excel.

2. Auto Refresh - I need to auto refresh the data that I will getting inside excel. Because the data is such that it will keep on getting modified everyday within the SQL Server Database and I need to have the LATEST Data inside Excel.

I have tried to import the data from within Power Pivot Data Model, but it does not import the latest data columns that I add within the SQL Server later on.

I am interested in using the Power Query based data import method, but I am not sure if that will help me in the above 2 requirements.

Please share any possible solutions for data import, which would satisfy the above conditions.

And if there are any links available about the BEST PRACTICES for creating such Excel and SQL Server type of Data Connections, then please share those links as well.

I am using the following software versions -
Microsoft SQL Server Management Studio version- 12.0.2000.8,
Microsoft Office Professional Plus 2016 x64
and Windows 7 x64

Thanks a lot for any help.
Regards

PS: Please do not suggest that method in which there is a risk for the Data Model to get broken because of changes in the SQL Server data source query. Because then I loose all the work that I have done within the Data Model and I have to recreate everything from start.

 

[ImkeF]

If you put Power Query between your SQL Server and your Power Pivot Model you’re basically creating sort of a logical DWH which enables you to separate your business logic that is reflected in your reports (and is modelled in Power Pivot) from your sources (your SQL server today, Oracle or csv files tomorrow…).

You just have to check for consistent output to Power Pivot and everything should be fine. Meaning: No renaming of fields that are transferred to Power Pivot already. If a column name changes in your source you simply add a renaming step in Power Query so that the original column name for the output to Power Pivot will remain.

If you edit your Power Query query properties you have several options like: Automatic refresh when opening the file or scheduled refreshes.

Prerequisite: Excel 2013 or higher. There still are technical issues in 2010 between PQ and PP.

 

[happy1001]

If you put Power Query between your SQL Server and your Power Pivot Model you’re basically creating sort of a logical DWH which enables you to separate your business logic that is reflected in your reports (and is modelled in Power Pivot) from your sources (your SQL server today, Oracle or csv files tomorrow…).

You just have to check for consistent output to Power Pivot and everything should be fine. Meaning: No renaming of fields that are transferred to Power Pivot already. If a column name changes in your source you simply add a renaming step in Power Query so that the original column name for the output to Power Pivot will remain.

If you edit your Power Query query properties you have several options like: Automatic refresh when opening the file or scheduled refreshes.

Prerequisite: Excel 2013 or higher. There still are technical issues in 2010 between PQ and PP.

Thank you so much for your reply ImkeF

If I understand it correctly, then you mean that –

1. If I keep the same column names, which have been earlier imported inside the Data Model “by doing modifications inside the Power Query” then the Data Model will not break, even if I add any NEW Columns to it, by making changes in the original SQL query from within the database ? —– If my understanding is correct, then please confirm it once.
2. If I use the power query method for data import into the Data Model, then any new columns that I add to the original SQL query from within the database, will get reflected inside the Data Model. —— I tried this approach earlier, but that did not work for some reason, maybe I should try again.

Thanks a lot for your help.

PS: we had this discussion on other place as well, but let us keep it here now.

 

[ImkeF]

Did you experience problems in the past like described here?: Breaking Power Query via Power Pivot is a thing of the pastThe Ken Puls (Excelguru) Blog
They are solved fortunately (at least for 2013 upwards).

So 1) shouldn't give you any problem now

Not sure what you mean with 2): It's basically up to you if you let new columns from your SQL DB pass through your PQ queries into your PP data model: Any new SQL columns can be retrieved by PQ. They might not show up autmatically in your existing PQ queries like you expect them if your query contains column filters somewhere. They can be defined as "Keep the following columns only" (Table.SelectColumns) or "Remove These columns" (Table.RemoveColumns). Only the latter will let new columns automatically show up, otherwise you have to include the new column in the filter expression.

 

[happy1001]

Thank you so much ImkeF. It is great that most of these common problems are solved in the new updates.


Best Regards