JackDanIce
Well-known Member
- Joined
- Feb 3, 2010
- Messages
- 9,922
- Office Version
- 365
- Platform
- Windows
Hi,
I have 3 separate csv files plus 1 table for User input, to then aggregate results and output.
The largest csv file, CSV1 has 10,0001 rows and 6 columns
The two other csv files, CSV2 and CSV3 are both 9 rows by 4 columns
I need to map CSV1 against CSV2, CSV3 (i.e. VLOOKUP to append 2 additional columns to CSV1), aggregate results and then a final calculation column.
Currently, I'm using code based on thread [URL=https://www.mrexcel.com/forum/usertag.php?do=list&action=hash&hash=1]#1 1[/URL] suggest by @Rick Rothstein as:
Above outputs a CSV file into a new sheet (the range argument is pre-defined for the new sheet), which I repeat 3 times for each CSV file.
For the line in blue, I found I couldn't output to an array and only directly to the sheet.
Ideally, I'd like to create an array of the CSV1 data and then append against CSV2 and CSV3 using dictionaries (in effect to aggregate against the Key ID value within CSV1) and then perform a few basic arithmetic calculations in code/memory and print out/hard code results.
Consideration #1
Read CSV2 and CSV3 in first (and create their own separate dictionaries) and then whilst looping over CSV1 to read in the data, apply the dictionary map for CSV2 and CSV3 respectively and create separate columns for the sheet for CSV1
With this, I should be able to perform all calculations and output into a dictionary for each loop, aggregating results into an output dictionary as I read them in.
Can anyone suggest methods or processes to make reading and then append and aggregate the data as fast as possible if consideration #1 can be improved? Please note, I'm looking for speed improvement suggestions, not specific coding directly for the spreadsheet.
TIA,
JackDanIce
I have 3 separate csv files plus 1 table for User input, to then aggregate results and output.
The largest csv file, CSV1 has 10,0001 rows and 6 columns
The two other csv files, CSV2 and CSV3 are both 9 rows by 4 columns
I need to map CSV1 against CSV2, CSV3 (i.e. VLOOKUP to append 2 additional columns to CSV1), aggregate results and then a final calculation column.
Currently, I'm using code based on thread [URL=https://www.mrexcel.com/forum/usertag.php?do=list&action=hash&hash=1]#1 1[/URL] suggest by @Rick Rothstein as:
Rich (BB code):
Private Sub Get_Data(ByRef strFilePath As String, ByRef rng As Range)
'Procedure to write data from selected data input file to range object of temporary sheet
Dim lngFileNum As Long
Dim lngRow As Long
Dim lngLooper As Long
Dim varRec As Variant
Dim strTotalFile As String
Dim strRecords() As String
Dim strFields() As String
lngFileNum = FreeFile
StatusBar_Notification "Importing data from file: " & strFilePath & " to sheet: " & rng.Parent.Name
Settings False
Open strFilePath For Binary As #lngFileNum
strTotalFile = Space(LOF(lngFileNum))
Get #lngFileNum , , strTotalFile
Close #lngFileNum
strRecords = Split(strTotalFile, vbNewLine)
On Error Resume Next
For lngLooper = LBound(strRecords) To UBound(strRecords)
strFields = Split(strRecords(lngLooper), ",")
rng.Offset(lngLooper, 0).Resize(1, UBound(strFields) + 1) = strFields
Next lngLooper
On Error GoTo 0
Settings True
Erase strRecords
Erase strFields
End Sub
For the line in blue, I found I couldn't output to an array and only directly to the sheet.
Ideally, I'd like to create an array of the CSV1 data and then append against CSV2 and CSV3 using dictionaries (in effect to aggregate against the Key ID value within CSV1) and then perform a few basic arithmetic calculations in code/memory and print out/hard code results.
Consideration #1
Read CSV2 and CSV3 in first (and create their own separate dictionaries) and then whilst looping over CSV1 to read in the data, apply the dictionary map for CSV2 and CSV3 respectively and create separate columns for the sheet for CSV1
With this, I should be able to perform all calculations and output into a dictionary for each loop, aggregating results into an output dictionary as I read them in.
Can anyone suggest methods or processes to make reading and then append and aggregate the data as fast as possible if consideration #1 can be improved? Please note, I'm looking for speed improvement suggestions, not specific coding directly for the spreadsheet.
TIA,
JackDanIce