Help!! Need macro to get data from 100000+ row CSV file

Sebastian K.

New Member
Joined
Jul 19, 2011
Messages
16
Hi

I've got a massive CSV data file, containing data loggings from a process.

I need to import specific data ranges to a worksheet.
What I want to do is average x rows before importing it to my worksheet.

My VBA skills are not any where close to the level I need to write the code for such macro.:confused:
I hope some one in here can help me.

The data looks like this:
"Analog-Real._20_BF01";"28.06.2011 14:22:59";0;1;40722599293,0787
"Analog-Real._20_BF01";"28.06.2011 14:23:04";5,115741;1;40722599351,5394
"Analog-Real._20_BF01";"28.06.2011 14:23:09";5,05787;1;40722599410,0926
"Analog-Real._20_BF01";"28.06.2011 14:23:14";5,054977;1;40722599468,5417

The reason I want to average the data is that the logging sequence is 5sec. and to avoid "flooding" my worksheet, I need to reduce it to e.g. 1 minute intervals.

It's only the data from column 2 + 3 (time stamp + process value) that is of intrest.

Best regards
Sebastian
 
Last edited:
You need to add a carriage return and a line feed between each set of data. It is chr(13) & chr(10). Or you can just use the built in constant vbCrLf.

Code:
DataPreview = DataPreview & PreviewLine & vbCrLf

I just realised that I forgot to set multi lines true ..... dooh:banghead:

Now it works with the originally code.
 
Upvote 0

Excel Facts

How to calculate loan payments in Excel?
Use the PMT function: =PMT(5%/12,60,-25000) is for a $25,000 loan, 5% annual interest, 60 month loan.
Looks good. I cleaned it up a little bit for you. This code should all go into the form code section. I simplified your file looping (this made your code a little shorter), and put the file handling into a true sub rather than a goto, return. I didn't make your whole form to test this, so no guarantees, but have a look. I also added a file exists check for some error handling. I know there could be more depending on how much someone is going to try and break this.

Code:
Option Explicit
Dim DataTotal As Single, i As Integer, j As Integer, fs As Object
Dim lnAvrg As Integer, x_dlim As String * 1
Dim LnSkip As Integer, DateSet As Boolean, FileToOpen As String
Dim SheetRef As String, SheetRow As Integer, SheetCol As Integer
 
Private Sub Cmd_ok_Click()
    lnAvrg = CInt(frm_1.txt_lnNum_1.Value)  ' Number of lines to average
    x_dlim = Left(frm_1.txt_dlim.Value, 1)          ' The delimiter character
    LnSkip = frm_1.txt_lnskip               ' Number of lines to skip at the beginning
 
    DateSet = False
    For j = 1 To 10
        If Not frm_1.Controls("Txt_" & j).Value = "" Then
            FileToOpen = frm_1.Controls("Txt_" & j).Value
            Set fs = CreateObject("Scripting.FileSystemObject")
            If fs.FileExists(FileToOpen) Then
                GetData
                SheetCol = SheetCol + 1
            Else
                MsgBox "File: '" & FileToOpen & "' not found.", vbOKOnly & vbExclamation
            End If
        End If
    Next j
End Sub
 
Sub GetData()
    Dim LineCount As Integer, LnSpool As String, SingleLine As String, x() As String
    i = FreeFile
    Open FileToOpen For Binary Access Read As #i
        If LOF(i) > 0 Then
            'This loop skips a user determin number of headers
            If LnSkip > 0 Then
                LineCount = 0
                Do
                    Line Input #1, LnSpool
                    LineCount = LineCount + 1
                Loop Until EOF(i) Or LineCount = LnSkip
            End If
            While Not EOF(i)
                DataTotal = 0
                LineCount = 0
                Do
                    Line Input #i, SingleLine
                    x = Split(SingleLine, x_dlim, , vbTextCompare)
                ' Write the date from the first line to the spreadsheet
                    If Not DateSet Then
                        ThisWorkbook.Sheets(SheetRef).Cells(SheetRow, SheetCol) = x(1)
                        SheetRow = SheetRow + 1
                        DateSet = True
                    End If
                    ' Summarize the data value for the defined lines to be averaged
                    DataTotal = DataTotal + CSng(x(2))    ' The Csng function is used to ensure the preservation of the decimal number, in this case as single
                    LineCount = LineCount + 1
                Loop Until EOF(i) Or LineCount = lnAvrg
 
                ' Write the read data to the spreadsheet
                ' using the sheet, column and row reference obtained
                ThisWorkbook.Sheets(SheetRef).Cells(SheetRow, SheetCol) = DataTotal / LineCount
                SheetRow = SheetRow + 1
            Wend
        End If
    Close #i
End Sub
 
Private Sub Cmd_ref_select_Click()
    Dim InputCell As Excel.Range
    On Error Resume Next
    frm_1.Hide
    ' Select the reference cell by using the inputbox method
    Set InputCell = _
        Application.InputBox(Prompt:="Select first input cell", _
        Title:="Cell reference", Type:=8)
    SheetRow = InputCell.Row             ' Row reference number for use when writing to the spreadsheet
    SheetCol = InputCell.Column          ' Column reference number for use when writing to the spreadsheet
    SheetRef = InputCell.Parent.Name    ' Sheet name for writing to the spreadsheet
    ' Fill the textbox with the address of the selected reference cell
    frm_1.Txt_CellRef.Text = InputCell.Parent.Name & "!" & InputCell.Address
    frm_1.Show
End Sub

I've just given it a breef try ..... but there are a few problems.

1: It freezes once the last entered file has been imported.
2: The time stamp is now missing
3: When the next file is imported the sheetrow count is not reset
which means I get 3 staggered columns instead of 3 parallel.
This one is an easy fix ;)

I'll give it som work tomorrow to se if I can get "right"

Cheers:beerchug:
 
Upvote 0
I've just given it a breef try ..... but there are a few problems.

1: It freezes once the last entered file has been imported.
2: The time stamp is now missing
3: When the next file is imported the sheetrow count is not reset
which means I get 3 staggered columns instead of 3 parallel.
This one is an easy fix ;)

I'll give it som work tomorrow to se if I can get "right"

Cheers:beerchug:

Looks like I was a bit fast on the bugs ...... It didn't freeze but it just takes a while :eek2:

I did a few changes and the code now looks like this:

Code:
Option Explicit
Dim DataTotal As Single, i As Integer, j As Integer, fs As Object
Dim LnAvrg As Integer, x_dlim As String * 1
Dim LnSkip As Integer, DateSet As Boolean, FileToOpen As String
Dim SheetRef As String, SheetRow As Integer, SheetCol As Integer, ColRef As Integer, RowRef As Integer
Dim Col_Var, x_DataVar As Integer, DataTime As String, DataAvg As Variant
Dim TxtBox, DataPreview As String
Dim File_1, File_2, File_3, File_4, File_5, File_6, File_7, File_8, File_9, File_10 As String
Private Sub Cmd_1_Click()
    File_1 = Application _
        .GetOpenFilename("Text Files (*.txt;*.csv), *.txt;*.csv")
    Txt_1.Text = File_1
End Sub
Private Sub Cmd_10_Click()
    File_10 = Application _
        .GetOpenFilename("Text Files (*.txt;*.csv), *.txt;*.csv")
    Txt_10.Text = File_10
End Sub
Private Sub Cmd_2_Click()
    File_2 = Application _
        .GetOpenFilename("Text Files (*.txt;*.csv), *.txt;*.csv")
    Txt_2.Text = File_2
    
End Sub
Private Sub Cmd_3_Click()
    File_3 = Application _
        .GetOpenFilename("Text Files (*.txt;*.csv), *.txt;*.csv")
    Txt_3.Text = File_3
    
End Sub
Private Sub Cmd_4_Click()
    File_4 = Application _
        .GetOpenFilename("Text Files (*.txt;*.csv), *.txt;*.csv")
    Txt_4.Text = File_4
End Sub
Private Sub Cmd_5_Click()
    File_5 = Application _
        .GetOpenFilename("Text Files (*.txt;*.csv), *.txt;*.csv")
    Txt_5.Text = File_5
End Sub
Private Sub Cmd_6_Click()
    File_6 = Application _
        .GetOpenFilename("Text Files (*.txt;*.csv), *.txt;*.csv")
    Txt_6.Text = File_6
End Sub
Private Sub Cmd_7_Click()
    File_7 = Application _
        .GetOpenFilename("Text Files (*.txt;*.csv), *.txt;*.csv")
    Txt_7.Text = File_7
End Sub
Private Sub Cmd_8_Click()
    File_8 = Application _
        .GetOpenFilename("Text Files (*.txt;*.csv), *.txt;*.csv")
    Txt_8.Text = File_8
End Sub
Private Sub Cmd_9_Click()
    File_9 = Application _
        .GetOpenFilename("Text Files (*.txt;*.csv), *.txt;*.csv")
    Txt_9.Text = File_9
End Sub
Private Sub Cmd_cancel_Click()
    Unload Me
End Sub
Private Sub Cmd_ok_Click()
    LnAvrg = CInt(Frm_1.Txt_LnNum_1.Value)  ' Number of lines to average
    x_dlim = Left(Frm_1.Txt_Dlim.Value, 1)  ' The delimiter character
    LnSkip = Frm_1.Txt_LnSkip               ' Number of lines to skip at the beginning
    SheetCol = ColRef
    
    DateSet = False
    For j = 1 To 10
        If Not Frm_1.Controls("Txt_" & j).Value = "" Then
            FileToOpen = Frm_1.Controls("Txt_" & j).Value
            Set fs = CreateObject("Scripting.FileSystemObject")
            If fs.FileExists(FileToOpen) Then
               If j = 1 Then
                    x_DataVar = 1
                    GetData
                    SheetCol = SheetCol + 1
                    x_DataVar = 2
                    GetData
                    SheetCol = SheetCol + 1
                Else
                    x_DataVar = 2
                    GetData
                    SheetCol = SheetCol + 1
                End If
            Else
                MsgBox "File: '" & FileToOpen & "' not found.", vbOKOnly & vbExclamation
            End If
        End If
    Next j
End Sub

Sub GetData()
    Dim LineCount As Integer, LnSpool As String, SingleLine As String, x() As String
    i = FreeFile
    SheetRow = RowRef
    Open FileToOpen For Binary Access Read As #i
        If LOF(i) > 0 Then
            'This loop skips a user determin number of headers
            If LnSkip > 0 Then
                LineCount = 0
                Do
                    Line Input #1, LnSpool
                    LineCount = LineCount + 1
                Loop Until EOF(i) Or LineCount = LnSkip
            End If
            While Not EOF(i)
                DataTotal = 0
                LineCount = 0
                Do
                    Line Input #i, SingleLine
                    x = Split(SingleLine, x_dlim, , vbTextCompare)
 
                    ' This nested If statements determin if it is time or data to be used
                    ' When it is time, only the time from the first line is used
                    ' When it is data it summarises the data value for the defined lines to be averaged
                    If x_DataVar = 1 Then
                        If LineCount = 0 Then DataTime = x(x_DataVar)  'This aquires the time stamp for the data set
                    ElseIf x_DataVar > 1 Then DataTotal = DataTotal + CSng(x(x_DataVar))    ' The Csng function is used to ensure the preservation of the decimal number, in this case as single
                    End If
                    LineCount = LineCount + 1
                Loop Until EOF(i) Or LineCount = LnAvrg
 
                ' This With statement writs the read data to the spreadsheet
                ' using the sheet, column and row reference obtained
                With ThisWorkbook.Sheets(SheetRef)
 
                    ' This If statement determins if it is time or data before writing it to the cells accordingly
                    If x_DataVar > 1 Then DataAvg = DataTotal / LineCount Else: DataAvg = DataTime
                    .Cells(SheetRow, SheetCol) = DataAvg
                    SheetRow = SheetRow + 1
                End With
             Wend
        End If
    Close #i
End Sub

Private Sub Cmd_preview_Click()
    i = FreeFile
    
    Open File_1 For Binary Access Read As #i
        If LOF(i) > 0 Then
            DataPreview = ""
            LineCount = 0
            Do
                Line Input #i, PreviewLine
                DataPreview = DataPreview & PreviewLine & Chr(13)
                LineCount = LineCount + 1
            Loop Until EOF(i) Or LineCount = 10
        End If
    Close #i
    
    Frm_2.Txt_preview.Text = DataPreview
    
    Frm_2.Show
    
End Sub
Private Sub Cmd_ref_select_Click()
    Dim InputCell As Excel.Range
    On Error Resume Next
    Frm_1.Hide
    ' Select the reference cell by using the inputbox method
    Set InputCell = _
        Application.InputBox(Prompt:="Select first input cell", _
        Title:="Cell reference", Type:=8)
    RowRef = InputCell.Row             ' Row reference number for use when writing to the spreadsheet
    ColRef = InputCell.Column          ' Column reference number for use when writing to the spreadsheet
    SheetRef = InputCell.Parent.Name   ' Sheet name for writing to the spreadsheet
    ' Fill the textbox with the address of the selected reference cell
    Frm_1.Txt_CellRef.Text = InputCell.Parent.Name & "!" & InputCell.Address
    Frm_1.Show
End Sub

Any comments are still highly appreciated.
This code will be a work in progress for quite some time as I will use it to improve my VBA skills.

Thankz
Sebastian
 
Upvote 0

Forum statistics

Threads
1,224,588
Messages
6,179,743
Members
452,940
Latest member
rootytrip

We've detected that you are using an adblocker.

We have a great community of people providing Excel help here, but the hosting costs are enormous. You can help keep this site running by allowing ads on MrExcel.com.
Allow Ads at MrExcel

Which adblocker are you using?

Disable AdBlock

Follow these easy steps to disable AdBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the icon in the browser’s toolbar.
2)Click on the "Pause on this site" option.
Go back

Disable AdBlock Plus

Follow these easy steps to disable AdBlock Plus

1)Click on the icon in the browser’s toolbar.
2)Click on the toggle to disable it for "mrexcel.com".
Go back

Disable uBlock Origin

Follow these easy steps to disable uBlock Origin

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back

Disable uBlock

Follow these easy steps to disable uBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back
Back
Top