Pulling in new data without refreshing entire query

[jsg51483]

I'm working with powerpivot with data that flows into a table for the previous business day. It's a lot of data, though -- several million rows. My issue is, I don't want to refresh the entire query every day when 99% of the data is static. I just want to pull in the new data, and add it to the old data. Is there a way to have a data table that has both a historic, static component, and then each day just appends one days worth of data and archives it?

I have a hunch that power query may hold the answer, but I'm relatively new to power BI and am not yet familiar with the add-in, only now getting my head around powerpivot/DAX.

Thanks for any help you can provide!!

 

[geraldartman]

I'm working with powerpivot with data that flows into a table for the previous business day. It's a lot of data, though -- several million rows. My issue is, I don't want to refresh the entire query every day when 99% of the data is static. I just want to pull in the new data, and add it to the old data. Is there a way to have a data table that has both a historic, static component, and then each day just appends one days worth of data and archives it?

I have a hunch that power query may hold the answer, but I'm relatively new to power BI and am not yet familiar with the add-in, only now getting my head around powerpivot/DAX.

Thanks for any help you can provide!!

In powerquery you can filter your table on a column, for example date > or = to a constant. That would reduce the complete source to your new only list. You can then append that data to the existing parsed data. I am assuming you get a full dump of all data on any given day. PowerQuery does not alter the source file. So your archive is the daily file. The routine that parses out the daily info would essentially be the same as the routine to parse the complete file without the date filter. Excel can handle millions of rows in the model. I would try the whole thing first to see the results and time to complete issues. Redoing the whole thing always solves any sync issues from missing dates or changed source data. You may be surprised.