want code import dat convert to vba array

roykana

Active Member
Joined
Mar 8, 2018
Messages
311
Office Version
  1. 2010
Platform
  1. Windows
Dear All Master,
I want the code below to be an array vba code because the actual record is 100000 so it makes it very slow


thanks
roykana

VBA Code:
Option Explicit
Sub Get_Data_From_File()
    OptimizeVBA True
    Dim FileToOpen As Variant
    Dim OpenBook As Workbook
    Dim wsSelect As Worksheet
    Dim i As Long
    Dim k As Long
    Dim L As Long
    Dim J As Long
    Dim LastRow As Long
    Dim LastColumn As Long
    Dim objTable As ListObject
    Dim objTable2 As String
    Dim startTime As Single, endTime As Single
    Dim TableFound As Boolean
    startTime = Timer
    Set wsSelect = ActiveWorkbook.Sheets("selectfile")
    With wsSelect
        .Columns("A:G").Clear
        .Range("A1").Value = "ID"
        .Range("B1").Value = "DATE & TIME"
        .Range("C1").Value = "DATE"
        .Range("D1").Value = "YEAR"
        .Range("E1").Value = "PERIOD"
        .Range("F1").Value = "CATEGORY"
        .Range("G1").Value = "NAME"
        .Range("A1:G1").HorizontalAlignment = xlCenter
        LastRow = .Range("A1").SpecialCells(xlCellTypeLastCell).Row
    End With
    FileToOpen = Application.GetOpenFilename(Title:="Browse for your File & Import Range", FileFilter:="Dat Files (*.dat*),*dat*", MultiSelect:=True)
    On Error Resume Next
    If IsArray(FileToOpen) Then
        On Error GoTo 0
        For i = LBound(FileToOpen) To UBound(FileToOpen)
            Set OpenBook = Application.Workbooks.Open(FileToOpen(i))
            OpenBook.Sheets(1).Range(Cells(1, 1), Cells(1, 2).End(xlDown)).Copy
            TableFound = TableExists(wsSelect, "TableDat")
            If Not TableFound Then
                wsSelect.Range("A2:B2").PasteSpecial xlPasteValues
                OpenBook.Close False
                wsSelect.Range("A1", "G" & wsSelect.Cells(Rows.Count, "A").End(xlUp).Row).Select
                Set objTable = wsSelect.ListObjects.Add(xlSrcRange, Selection, , xlYes)
                objTable.Name = "TableDat"
            Else
                wsSelect.Cells(Rows.Count, "A").End(xlUp).Offset(1).PasteSpecial xlPasteValues
                OpenBook.Close False
            End If
        Next
        With wsSelect
            For k = 2 To LastRow
                .Cells(k, 2).Value = WorksheetFunction.Text(.Cells(k, 2).Value, "DD/MM/YYYY HH.MM")
            Next k
            For L = 2 To LastRow
                .Cells(L, 3).Value = DateSerial(Mid(.Cells(L, 2), 7, 4), Mid(.Cells(L, 2), 4, 2), Mid(.Cells(L, 2), 1, 2))
            Next L
             For J = 2 To LastRow
                 .Cells(J, 4).Value = Format(CDate(Cells(J, 2).Text), "YYYY")
            Next J
        End With
    Else
        If FileToOpen = False Then
            On Error GoTo 0
            GoTo getout
        End If
    End If
getout:
    Application.Goto wsSelect.Range("A1")
    endTime = Timer
    Debug.Print (endTime - startTime) & " seconds have passed [VBA]"
    OptimizeVBA False
End Sub
Sub OptimizeVBA(isOn As Boolean)
    Application.Calculation = IIf(isOn, xlCalculationManual, xlCalculationAutomatic)
    Application.EnableEvents = Not (isOn)
    Application.ScreenUpdating = Not (isOn)
    ActiveSheet.DisplayPageBreaks = Not (isOn)
End Sub

Function TableExists(ws As Worksheet, sTableName As String) As Boolean
    TableExists = ws.Evaluate("ISREF(" & sTableName & ")")
End Function
 

Excel Facts

Copy formula down without changing references
If you have =SUM(F2:F49) in F50; type Alt+' in F51 to copy =SUM(F2:F49) to F51, leaving the formula in edit mode. Change SUM to COUNT.
Can you provide a link to a sample that we can work with? It will be easier to assist you if you can.

Here is a more condensed version of the code that you can test to see if it still performs properly:

VBA Code:
Option Explicit
Sub Get_Data_From_File()
'
    Dim startTime   As Single
    startTime = Timer
'
    OptimizeVBA True
'
    Dim TableFound  As Boolean
    Dim objTable    As ListObject
    Dim i           As Long, J          As Long
    Dim LastColumn  As Long, LastRow    As Long
    Dim objTable2   As String
    Dim FileToOpen  As Variant
    Dim OpenBook    As Workbook
    Dim wsSelect    As Worksheet
'
    Set wsSelect = ActiveWorkbook.Sheets("selectfile")
'
    With wsSelect
        .Columns("A:G").Clear
        .Range("A1:G1").Value = Array("ID", "DATE & TIME", "DATE", "YEAR", "PERIOD", "CATEGORY", "NAME")
        .Range("A1:G1").HorizontalAlignment = xlCenter
        LastRow = .Range("A1").SpecialCells(xlCellTypeLastCell).Row
    End With
'
    FileToOpen = Application.GetOpenFilename(Title:="Browse for your File & Import Range", FileFilter:="Dat Files (*.dat*),*dat*", MultiSelect:=True)
'
    On Error Resume Next
    If IsArray(FileToOpen) Then
    On Error GoTo 0
'
        For i = LBound(FileToOpen) To UBound(FileToOpen)
            Set OpenBook = Application.Workbooks.Open(FileToOpen(i))
            OpenBook.Sheets(1).Range(Cells(1, 1), Cells(1, 2).End(xlDown)).Copy
            TableFound = TableExists(wsSelect, "TableDat")
'
            If Not TableFound Then
                wsSelect.Range("A2:B2").PasteSpecial xlPasteValues
                OpenBook.Close False
'
                wsSelect.Range("A1", "G" & wsSelect.Cells(Rows.Count, "A").End(xlUp).Row).Select
                Set objTable = wsSelect.ListObjects.Add(xlSrcRange, Selection, , xlYes)
                objTable.Name = "TableDat"
            Else
                wsSelect.Cells(Rows.Count, "A").End(xlUp).Offset(1).PasteSpecial xlPasteValues
                OpenBook.Close False
            End If
        Next
'
        With wsSelect
            For J = 2 To LastRow
                .Cells(J, 2).Value = WorksheetFunction.Text(.Cells(J, 2).Value, "DD/MM/YYYY HH.MM")
                .Cells(J, 3).Value = DateSerial(Mid(.Cells(J, 2), 7, 4), Mid(.Cells(J, 2), 4, 2), Mid(.Cells(J, 2), 1, 2))
                .Cells(J, 4).Value = Format(CDate(Cells(J, 2).Text), "YYYY")
            Next
        End With
    End If
'
    Application.Goto wsSelect.Range("A1")
'
    OptimizeVBA False
    Debug.Print (Timer - startTime) & " seconds have passed [VBA]"
End Sub
Sub OptimizeVBA(isOn As Boolean)
    Application.Calculation = IIf(isOn, xlCalculationManual, xlCalculationAutomatic)
    Application.EnableEvents = Not (isOn)
    Application.ScreenUpdating = Not (isOn)
    ActiveSheet.DisplayPageBreaks = Not (isOn)
End Sub

Function TableExists(ws As Worksheet, sTableName As String) As Boolean
    TableExists = ws.Evaluate("ISREF(" & sTableName & ")")
End Function
 
Upvote 0
Can you provide a link to a sample that we can work with? It will be easier to assist you if you can.

Here is a more condensed version of the code that you can test to see if it still performs properly:

VBA Code:
Option Explicit
Sub Get_Data_From_File()
'
    Dim startTime   As Single
    startTime = Timer
'
    OptimizeVBA True
'
    Dim TableFound  As Boolean
    Dim objTable    As ListObject
    Dim i           As Long, J          As Long
    Dim LastColumn  As Long, LastRow    As Long
    Dim objTable2   As String
    Dim FileToOpen  As Variant
    Dim OpenBook    As Workbook
    Dim wsSelect    As Worksheet
'
    Set wsSelect = ActiveWorkbook.Sheets("selectfile")
'
    With wsSelect
        .Columns("A:G").Clear
        .Range("A1:G1").Value = Array("ID", "DATE & TIME", "DATE", "YEAR", "PERIOD", "CATEGORY", "NAME")
        .Range("A1:G1").HorizontalAlignment = xlCenter
        LastRow = .Range("A1").SpecialCells(xlCellTypeLastCell).Row
    End With
'
    FileToOpen = Application.GetOpenFilename(Title:="Browse for your File & Import Range", FileFilter:="Dat Files (*.dat*),*dat*", MultiSelect:=True)
'
    On Error Resume Next
    If IsArray(FileToOpen) Then
    On Error GoTo 0
'
        For i = LBound(FileToOpen) To UBound(FileToOpen)
            Set OpenBook = Application.Workbooks.Open(FileToOpen(i))
            OpenBook.Sheets(1).Range(Cells(1, 1), Cells(1, 2).End(xlDown)).Copy
            TableFound = TableExists(wsSelect, "TableDat")
'
            If Not TableFound Then
                wsSelect.Range("A2:B2").PasteSpecial xlPasteValues
                OpenBook.Close False
'
                wsSelect.Range("A1", "G" & wsSelect.Cells(Rows.Count, "A").End(xlUp).Row).Select
                Set objTable = wsSelect.ListObjects.Add(xlSrcRange, Selection, , xlYes)
                objTable.Name = "TableDat"
            Else
                wsSelect.Cells(Rows.Count, "A").End(xlUp).Offset(1).PasteSpecial xlPasteValues
                OpenBook.Close False
            End If
        Next
'
        With wsSelect
            For J = 2 To LastRow
                .Cells(J, 2).Value = WorksheetFunction.Text(.Cells(J, 2).Value, "DD/MM/YYYY HH.MM")
                .Cells(J, 3).Value = DateSerial(Mid(.Cells(J, 2), 7, 4), Mid(.Cells(J, 2), 4, 2), Mid(.Cells(J, 2), 1, 2))
                .Cells(J, 4).Value = Format(CDate(Cells(J, 2).Text), "YYYY")
            Next
        End With
    End If
'
    Application.Goto wsSelect.Range("A1")
'
    OptimizeVBA False
    Debug.Print (Timer - startTime) & " seconds have passed [VBA]"
End Sub
Sub OptimizeVBA(isOn As Boolean)
    Application.Calculation = IIf(isOn, xlCalculationManual, xlCalculationAutomatic)
    Application.EnableEvents = Not (isOn)
    Application.ScreenUpdating = Not (isOn)
    ActiveSheet.DisplayPageBreaks = Not (isOn)
End Sub

Function TableExists(ws As Worksheet, sTableName As String) As Boolean
    TableExists = ws.Evaluate("ISREF(" & sTableName & ")")
End Function
@johnnyL

Dear Mr. Johnnyl
Thank you for your reply.
1. if I create a new excel and use the code then the results are like below so I have to run the program code a second time in order to fill in the date & year column
2. still slow if I use to in the actual dat file
3. if I use in the original dat file there is information "there is a large amount of information on the clipboard" I should click yes according to the number of dat files I imported and I attach a screenshot below.
4. there is an inappropriate format in the actual dat file causing a 13" run time error" and I attach a screenshot below
This is the sample file you requested.

link dat file
 

Attachments

  • largeamount.JPG
    largeamount.JPG
    28.5 KB · Views: 26
  • inappropriate format.JPG
    inappropriate format.JPG
    46.2 KB · Views: 23
Upvote 0
1. if I create a new excel and use the code then the results are like below so I have to run the program code a second time in order to fill in the date & year column.
 

Attachments

  • for point one.JPG
    for point one.JPG
    48.5 KB · Views: 27
Upvote 0
this part of the macro can be replaced with ...
VBA Code:
With wsSelect
            For J = 2 To LastRow
                .Cells(J, 2).Value = WorksheetFunction.Text(.Cells(J, 2).Value, "DD/MM/YYYY HH.MM")
                .Cells(J, 3).Value = DateSerial(Mid(.Cells(J, 2), 7, 4), Mid(.Cells(J, 2), 4, 2), Mid(.Cells(J, 2), 1, 2))
                .Cells(J, 4).Value = Format(CDate(Cells(J, 2).Text), "YYYY")
            Next
        End With

i did a test with 200,000 B-cells with this macro (caution write tempory to the columns D:F).
creating the 3 arrays in memory is approx. 1.8 sec and writing them to the sheet an additional 0.9 sec.
so in total 2.7 sec

VBA Code:
Sub Speedy()
     t0 = Timer                                                 'start chrono

     With ActiveSheet.Range("B1:B" & Range("B" & Rows.Count).End(xlUp).Row)     'my range is 200,000 cells
          .Name = "test"                                        'make a defined naam
          myc = [text(test,"\'dd/mm/yyyy hh:mm")]               '1st array date&timeformat with a ' in front
          myd = [year(test)]                                    '2nd array = just the year
          myb = [trunc(test)]                                   '3rd array = dateserial

          t1 = Timer                                            'interval

          .Offset(, 2).Value = myc                              'write to D-column (later your C)
          .Offset(, 3).Value = myd                              'write to E-column (later your D)
          .Offset(, 4).Value = myb                              'write to F-column (later your original B)
     End With

     t2 = Timer                                                 'endtime

     MsgBox t1 - t0 & vbLf & t2 - t1 & vbLf & t2 - t0

End Sub
 
Upvote 0
this part of the macro can be replaced with ...
VBA Code:
With wsSelect
            For J = 2 To LastRow
                .Cells(J, 2).Value = WorksheetFunction.Text(.Cells(J, 2).Value, "DD/MM/YYYY HH.MM")
                .Cells(J, 3).Value = DateSerial(Mid(.Cells(J, 2), 7, 4), Mid(.Cells(J, 2), 4, 2), Mid(.Cells(J, 2), 1, 2))
                .Cells(J, 4).Value = Format(CDate(Cells(J, 2).Text), "YYYY")
            Next
        End With

i did a test with 200,000 B-cells with this macro (caution write tempory to the columns D:F).
creating the 3 arrays in memory is approx. 1.8 sec and writing them to the sheet an additional 0.9 sec.
so in total 2.7 sec

VBA Code:
Sub Speedy()
     t0 = Timer                                                 'start chrono

     With ActiveSheet.Range("B1:B" & Range("B" & Rows.Count).End(xlUp).Row)     'my range is 200,000 cells
          .Name = "test"                                        'make a defined naam
          myc = [text(test,"\'dd/mm/yyyy hh:mm")]               '1st array date&timeformat with a ' in front
          myd = [year(test)]                                    '2nd array = just the year
          myb = [trunc(test)]                                   '3rd array = dateserial

          t1 = Timer                                            'interval

          .Offset(, 2).Value = myc                              'write to D-column (later your C)
          .Offset(, 3).Value = myd                              'write to E-column (later your D)
          .Offset(, 4).Value = myb                              'write to F-column (later your original B)
     End With

     t2 = Timer                                                 'endtime

     MsgBox t1 - t0 & vbLf & t2 - t1 & vbLf & t2 - t0

End Sub
@BSALV
Mr. Bsalv,
Thank you for your reply.
should in column B use the format "dd/mm/yyyy hh:mm", column C should be the date format and column E should have no execution in the code. and in code you should not use activesheet code and should not use define name.
thanks
roykana
 

Attachments

  • result26012022.JPG
    result26012022.JPG
    39 KB · Views: 26
Upvote 0
@roykana how about:

VBA Code:
Option Explicit
Sub Get_Data_From_File()
'
    OptimizeVBA True
'
    Dim startTime           As Single
    Dim FileToOpen          As Variant
'
    FileToOpen = Application.GetOpenFilename(Title:="Browse for your File & Import Range", FileFilter:="Dat Files (*.dat*),*dat*", MultiSelect:=True)
'
    startTime = Timer
'
    Dim TableFound          As Boolean
    Dim objTable            As ListObject
    Dim ArraySeparatorRow   As Long
    Dim i                   As Long, J              As Long
    Dim LastRow             As Long
    Dim ListArray_A         As Object, ListArray_B  As Object, ListArray_B_Temp As Object, ListArray_C  As Object, ListArray_D  As Object
    Dim objTable2           As String
    Dim Array_A             As Variant, Array_B     As Variant, Array_A_B       As Variant
    Dim OpenBook            As Workbook
    Dim wsSelect            As Worksheet
'
    Set ListArray_A = CreateObject("System.Collections.ArrayList"): Set ListArray_B = CreateObject("System.Collections.ArrayList")
    Set ListArray_B_Temp = CreateObject("System.Collections.ArrayList")
    Set ListArray_C = CreateObject("System.Collections.ArrayList"): Set ListArray_D = CreateObject("System.Collections.ArrayList")
    Set wsSelect = ActiveWorkbook.Sheets("selectfile")
'
    With wsSelect
        .Columns("A:G").Clear
        .Range("A1:G1").Value = Array("ID", "DATE & TIME", "DATE", "YEAR", "PERIOD", "CATEGORY", "NAME")
        .Range("A1:G1").HorizontalAlignment = xlCenter
    End With
'
    LastRow = 1
'
    If IsArray(FileToOpen) Then
        For i = LBound(FileToOpen) To UBound(FileToOpen)
            Set OpenBook = Application.Workbooks.Open(FileToOpen(i))
'
            LastRow = LastRow + Range("A1").SpecialCells(xlCellTypeLastCell).Row
'
            Array_A_B = OpenBook.Sheets(1).Range(Cells(1, 1), Cells(1, 2).End(xlDown))              ' Copy A:B from dat file to Array_A_B
'
            For ArraySeparatorRow = 1 To UBound(Array_A_B, 1)
                ListArray_A.Add Array_A_B(ArraySeparatorRow, 1)
                ListArray_B_Temp.Add Array_A_B(ArraySeparatorRow, 2)
            Next
'
            OpenBook.Close False
            Erase Array_A_B
        Next
'
        For J = 2 To LastRow
            ListArray_B.Add WorksheetFunction.Text(ListArray_B_Temp(J - 2), "DD/MM/YYYY HH.MM")
            ListArray_C.Add DateSerial(Mid(ListArray_B(J - 2), 7, 4), Mid(ListArray_B(J - 2), 4, 2), Mid(ListArray_B(J - 2), 1, 2))
            ListArray_D.Add Year(ListArray_B(J - 2))
        Next
'
        Range("A2").Resize(ListArray_A.Count, 1).Value = WorksheetFunction.Transpose(ListArray_A.ToArray)
        Range("B2").Resize(ListArray_B.Count, 1).Value = WorksheetFunction.Transpose(ListArray_B.ToArray)
        Range("C2").Resize(ListArray_C.Count, 1).Value = WorksheetFunction.Transpose(ListArray_C.ToArray)
        Range("D2").Resize(ListArray_D.Count, 1).Value = WorksheetFunction.Transpose(ListArray_D.ToArray)
'
        wsSelect.Range("A1", "G" & wsSelect.Cells(Rows.Count, "A").End(xlUp).Row).Select
        Set objTable = wsSelect.ListObjects.Add(xlSrcRange, Selection, , xlYes)             ' Create Table
        objTable.Name = "TableDat"
    End If
'
    wsSelect.Columns("A:G").EntireColumn.AutoFit                                            ' AutoFit the columns for the destination sheet
'
    Application.Goto wsSelect.Range("A1")
'
    OptimizeVBA False
    Debug.Print (Timer - startTime) & " seconds have passed [VBA]"
End Sub
Sub OptimizeVBA(isOn As Boolean)
    Application.Calculation = IIf(isOn, xlCalculationManual, xlCalculationAutomatic)
    Application.EnableEvents = Not (isOn)
    Application.ScreenUpdating = Not (isOn)
    ActiveSheet.DisplayPageBreaks = Not (isOn)
End Sub

Function TableExists(ws As Worksheet, sTableName As String) As Boolean
    TableExists = ws.Evaluate("ISREF(" & sTableName & ")")
End Function

It uses Lists and arrays.
 
Upvote 0
1st method with a defined name that is deleted afterwards, 2nd method with formulas and replace the formula with its value
500,000 cells , method 1 = 3.0 sec, method 2 = 3.5 sec
Don't do both, choose one !
VBA Code:
Sub Speedy()
     t0 = Timer                                                 'start chrono

     With ActiveWorkbook.Sheets("selectfile").Range("B2:B" & Range("B" & Rows.Count).End(xlUp).Row)     'my range is 200,000 cells
          'method defined name
          .Name = "test"                                        'make a defined naam
          .NumberFormat = "dd/mm/yyyy hh:mm"                    'nothing is changed, just the format
          myd = [year(test)]                                    '2nd array = just the year
          myc = [trunc(test)]                                   '3rd array = dateserial = integer part
          .Offset(, 1).Value = myc                              'write to D-column (later your C)
          .Offset(, 1).NumberFormat = "dd/mm/yyyy"              'nothing is changed, just the format
          .Offset(, 2).Value = myd                              'write to E-column (later your D)
          ActiveWorkbook.Names("test").Delete                   'delete the defined name

          t2 = Timer

     'method formulas
          .Offset(, 1).FormulaR1C1 = "=TRUNC(RC[-1])"           'take integer part
          .Offset(, 2).FormulaR1C1 = "=YEAR(RC[-2])"            'take year
          With .Offset(, 1).Resize(, 2)
               .Value = .Value                                  'replace formula by its value
          End With

          t3 = Timer

     End With


     MsgBox "method defined name : " & t2 - t0 & vbLf & "method formulas : " & t3 - t2

End Sub

JohnnyL suggested lists and arrays, i took just a part of the macro which made the greatest difference i suppose. How much is the execution time reduced ?

After reading JohnnyL's solution, do everything in 1 dictionary !
 
Last edited:
Upvote 0
too late to put it in previous post.
VBA Code:
Sub JohnnyL()
     'in the beginning, declare dictionary
     Set dict = CreateObject("scripting.dictionary")


[...]

     'in the loop of manipulating your files
     arr = OpenBook.Sheets(1).Range(Cells(1, 1), Cells(1, 2).End(xlDown))     ' Copy A:B from dat file to Array_A_B
     For i = 1 To UBound(arr)
          dict.Add dict.Count, Array(arr(i, 1), arr(i, 2), Int(arr(i, 2)), Year(arr(i, 2)))
     Next
    
[...]

     'at the end writing to your sheet
     arr2 = Application.Index(dict.items, 0, 0)
     ActiveWorkbook.Sheets("selectfile").Range("A2").Resize(UBound(arr2), UBound(arr2, 2)).Value = arr2

End Sub
 
Upvote 0

Forum statistics

Threads
1,223,908
Messages
6,175,304
Members
452,633
Latest member
DougMo

We've detected that you are using an adblocker.

We have a great community of people providing Excel help here, but the hosting costs are enormous. You can help keep this site running by allowing ads on MrExcel.com.
Allow Ads at MrExcel

Which adblocker are you using?

Disable AdBlock

Follow these easy steps to disable AdBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the icon in the browser’s toolbar.
2)Click on the "Pause on this site" option.
Go back

Disable AdBlock Plus

Follow these easy steps to disable AdBlock Plus

1)Click on the icon in the browser’s toolbar.
2)Click on the toggle to disable it for "mrexcel.com".
Go back

Disable uBlock Origin

Follow these easy steps to disable uBlock Origin

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back

Disable uBlock

Follow these easy steps to disable uBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back
Back
Top