Copying 1 million rows quickly

keith1

New Member
Joined
Nov 30, 2010
Messages
16
I have a database with 14 million records of numbers in. I wish to copy at 1 million a time to export into excel. There are no record id's and I dont have enough space to enter a record id in without changing the registry entry, which I do not wish to do. Is there a way of quickly copying these across.
Thanks for any help
 

Excel Facts

Copy PDF to Excel
Select data in PDF. Paste to Microsoft Word. Copy from Word and paste to Excel.
Hi Keith,

As you don't have a record ID, things are a little more complicated.
But I have a solution for you, before you go to the code, read this first.

As a first step, make a copy of your table (or entire mdb/accdb).
The copy is referred to as MotherTableX from now on.
Now make an empty copy (so only copy the structure without data) of MotherTableX, this copy is referred to as tempTableX.

In the code you'll find 5 string functions to create SQL statements:
SQL_FetchTempTableX is used to fetch all records from the tempTableX
SQL_DelFromTableX is used to empty tempTableX
SQL_TOP1Mil is used to create a queryDef to delete the Top 1 million records from MotherTableX
SQL_InsertTOP1Mil is used to insert 1 million records from MotherTableX into tempTableX
SQL_AllRecords is only for a recordcount of MotherTableX

Of course you'll have to replace the table names and field names corresponding to your tables.

Now, how it works is as follows.

From MotherTableX the first 1 million records are inserted into the table tempTableX. From tempTableX the records are passed to a new excel sheet. Then the first 1 million records from MotherTableX are deleted, tempTableX is cleared and the next 1 million are copied to the tempTable.
This will be repeated until all records are deleted from MotherTablex.

Of course you'll have to be patient if you run this code with 14 million records.
Just create a new module in Access and paste the code, don't forget to set the references needed.

Code:
Option Compare Database
Option Explicit
'Set reference to Microsoft Excel xx Object Library
'Set reference to Microsoft ActiveX Data Objects 2.x Library
'Free to use, coded by Johan Kreszner
Public Sub ToExcel()
Dim oCn As New ADODB.Connection
Dim oRs As New ADODB.Recordset
Dim qDef As QueryDef
Dim oExcelApp As New Excel.Application
Dim oExcelWB As New Excel.Workbook
Dim oExcelSht As Excel.Worksheet
Dim sWorkbookName As String
Dim sSheetName As String
Dim lMaxRecPerSheet As Long
Dim iNumbSheets As Integer
Dim iSheetCnt As Integer
Set oCn = CurrentProject.Connection
'Here comes your path and workbookname
sWorkbookName = [B][COLOR=red]"M:\CreateExcel\MyExcelBook"[/COLOR][/B]
 
'First calculate number of sheets needed, if you know the exact number of records, you don't need this section
'Instead set  iNumbSheets = Round((TheNumberOfRecords / lMaxRecPerSheet) + 0.5)
'and remark the code that opens the recordset
[COLOR=royalblue][B]lMaxRecPerSheet = 1000000[/B][/COLOR]
With oRs
    'Open Mother to find recordcount
    .Open SQL_AllRecords, oCn, adOpenStatic, adLockReadOnly
    iNumbSheets = Round((.RecordCount / lMaxRecPerSheet) + 0.5)
    .Close
 
End With
Set oRs = Nothing
'create Excel object
With oExcelApp
            'Create a new workbook
            Set oExcelWB = Workbooks.Add
            With oExcelWB
                     'Create the sheets
                     For iSheetCnt = 1 To iNumbSheets
                                Set oExcelSht = .Worksheets().Add
                                'Fill tempTableX with top 1million records from MotherTableX
                                oCn.Execute (SQL_InsertTOP1Mil)
                                            'Fetch all records from tempTableX
                                            oRs.Open SQL_FetchTempTableX, oCn, adOpenStatic, adLockReadOnly
                                            'Create a name for the new sheet
                                            sSheetName = "Sheet " & iSheetCnt & " of " & iNumbSheets
                                'Copy recordset to new sheet
                                With oExcelSht
                                        .Name = sSheetName
                                        .Cells(1, 1).CopyFromRecordset oRs 'This is where the records are passed to the sheet
                                        oRs.Close
                                        Set oRs = Nothing
                                End With
                                'Delete Top 1million records from MotherTableX
                                Set qDef = CurrentDb.CreateQueryDef("x", SQL_TOP1Mil) 'Create a query def
                                oCn.Execute ("Delete * from x") 'Use query def to delete the top 1mil from MotherTableX
                                CurrentDb.QueryDefs.Delete ("x") 'Delete the query def
                                'Delete all records from tempTablex
                                oCn.Execute (SQL_DelFromTableX)
                    Next iSheetCnt
        End With
    oExcelWB.SaveAs sWorkbookName
    'Destroy object
    .Quit
End With
 
End Sub
 
Public Function SQL_FetchTempTableX() As String
    SQL_FetchTempTableX = "Select * From tempTableX"
End Function
Public Function SQL_DelFromTableX() As String
    SQL_DelFromTableX = "Delete * From tempTableX"
End Function
Public Function SQL_TOP1Mil() As String
    SQL_TOP1Mil = "SELECT Top 1000000 * FROM MotherTableX"
End Function
Public Function SQL_InsertTOP1Mil() As String
SQL_InsertTOP1Mil = "INSERT INTO tempTableX ( [B][COLOR=red]Field1, Field2, Field3[/COLOR][/B] )" _
                  & "SELECT TOP 1000000 [COLOR=red][B]MotherTableX.Field1, MotherTableX.Field2, MotherTableX.Field3[/B][/COLOR] " _
                  & "FROM MotherTableX"
End Function
Public Function SQL_AllRecords() As String
'Replace Field 1 with a valid fieldname from your table
    SQL_AllRecords = "Select [COLOR=red][B]Field1[/B][/COLOR] from MotherTableX"
End Function
 
Last edited:
Upvote 0
Hi Kreszch68

Thankyou for taking the time to compile this code for me. I have had other commitments come up today which mean I can't try this right now, but I will try it out in the next few days and let you know how I got on with it.

Thanks again for taking the time out to help.

Keith
 
Upvote 0
Hi

Thank you for your help, I am very much a novice at using code so please excuse me if I seem a little naive

I have tried the code and get a message "User type not defined" with the line

Public Sub ToExcel() highlighted.

I have renamed the tables, is there something I have missed.

Thanks
Keith :confused:
 
Upvote 0
You're welcome.<?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" /><o:p></o:p>
On second thoughts I don't think it's a good idea to place 14mil records in one workbook in one run. This will result in a really big Excel file you probably can't open at all.<o:p></o:p>
<o:p> </o:p>
But, I just finished a module that can be of use for you also.<o:p></o:p>
<o:p> </o:p>
I’ll paste the code here, but for an explanation look at the tread, retrieve N records starting at record M, on this forum.<o:p></o:p>

The function call:
Code:
Option Compare Database
Option Explicit
'Set reference to Microsoft ActiveX Data Objects 2.x Library
Public blnHasRecords As Boolean
'Coded by Johan Kreszner
'Mio-Software Netherlands
Public Sub TriggerNfromM()
'Fetch N records from a recordset starting at record M
'Use any valid SQL string function to pass to the function
Dim lStartFromRecord As Long
Dim lNumberOfRecords As Long
Dim sSQLName As String
Dim sOrderByFieldName As String
Dim oRsResult As New ADODB.Recordset
lStartFromRecord = 50000            'The first record you want to return
lNumberOfRecords = 35               'The number of records you want counted from the startfromrecord
sSQLName = "SQL_Example()"          'Mind the () in the string, if you don't add the parenthesises the function can't evaluate the name
sOrderByFieldName = "IdTableY ASC"  'The ordering determines how the collection is returned, so be aware how this affects the results
Set oRsResult = RecordsNfromM(sSQLName, lNumberOfRecords, lStartFromRecord, sOrderByFieldName)
'Test if there is a valid result
If Not blnHasRecords Then Exit Sub 'The cursor can't go further than the last record
'From here you can pick up the recordset and do whatever you need to do with it.
'<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>
'Your code
'<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>
'Test, skip this section, it's just a test to see if you get results.
oRsResult.MoveFirst
Dim arrTest() As Variant
Dim lT As Long
arrTest() = oRsResult.GetRows()
For lT = 0 To UBound(arrTest, 2)
    Debug.Print lT + 1, arrTest(1, lT), arrTest(2, lT)
Next lT
End Sub
The function:
Code:
Public Function RecordsNfromM(ByVal sSQLName As String, _
                              ByVal lNumberOfRecords As Long, _
                              ByVal lStartFromRecord As Long, _
                              Optional ByVal sOrderByFieldName As String) As ADODB.Recordset
'Coded by Johan Kreszner
'Mio-Software Netherlands
                              
'sSQLName =  name of function that returns the string to construct the SQL-Statement
'sOrderByFieldName = name of field used to order the records, if not passed records are ordered according
'to the ordering of the SQL-statement passed to the function (original order)
'Also Desc or Asc are passed with sOrderByFieldName ("fieldname ASC")
'NOTE: Records are always first ordered by the field set in the SQL-Statement you pass
Dim oCn As New ADODB.Connection
Dim oRs As New ADODB.Recordset
Dim vBookMarks() As Variant
Dim lRecCount As Long
Set oCn = CurrentProject.Connection
With oRs
        'Debug.Print "StartOpeningRecordset" & vbTab & Now()
        .CursorLocation = adUseClient
        .Open Eval(sSQLName), oCn, adOpenStatic, adLockReadOnly
        'Debug.Print "RecordsetOpen" & vbTab & Now()
        .Sort = sOrderByFieldName
        'Debug.Print "RecordsetSorted" & vbTab & Now()
        .MoveFirst
        'Debug.Print "RecordsetToFirst" & vbTab & Now()
               'Check if user doesn't ask for a record beyond the number of total records
               If lStartFromRecord > .RecordCount Then 'From record number higher than total number of records
                       MsgBox "The number of the first record asked is beyond the total number of records", vbCritical
                       blnHasRecords = False
                       Exit Function
               'Check if the total of records asked can be retrieved, if not return until last record in recordset
               ElseIf lStartFromRecord + lNumberOfRecords > .RecordCount Then 'Can't retrieve more than max number of records
                        lNumberOfRecords = .RecordCount - lStartFromRecord
                        MsgBox "Can't retrieve more than " & lNumberOfRecords & "  records", vbExclamation
               End If
               
       .Move (lStartFromRecord)
       'Debug.Print "RecordsetToFirstAsked" & vbTab & Now()
       ReDim vBookMarks(lNumberOfRecords - 1) 'Dimension the array for the bookmarks
       'Set the bookmarks for the records
       For lRecCount = 0 To UBound(vBookMarks)
           vBookMarks(lRecCount) = .Bookmark
           .MoveNext
       Next lRecCount
       'Debug.Print "BookmarksCreated" & vbTab & Now()
       .Filter = vBookMarks() 'filter the recordset by the bookmarks
       'Debug.Print "RecordsFiltered" & vbTab & Now()
       Set RecordsNfromM = oRs
       blnHasRecords = True
       'Debug.Print "Function ready" & vbTab & Now()
End With
End Function

Here the example of a query string function:
Code:
Public Function SQL_Example()
    SQL_Example = "SELECT IdTableY, TableYName, TableYDescription FROM TableY"
End Function
 
Upvote 0

Forum statistics

Threads
1,223,627
Messages
6,173,417
Members
452,514
Latest member
cjkelly15

We've detected that you are using an adblocker.

We have a great community of people providing Excel help here, but the hosting costs are enormous. You can help keep this site running by allowing ads on MrExcel.com.
Allow Ads at MrExcel

Which adblocker are you using?

Disable AdBlock

Follow these easy steps to disable AdBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the icon in the browser’s toolbar.
2)Click on the "Pause on this site" option.
Go back

Disable AdBlock Plus

Follow these easy steps to disable AdBlock Plus

1)Click on the icon in the browser’s toolbar.
2)Click on the toggle to disable it for "mrexcel.com".
Go back

Disable uBlock Origin

Follow these easy steps to disable uBlock Origin

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back

Disable uBlock

Follow these easy steps to disable uBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back
Back
Top