Simple VBA help to download data from URL

Excel Facts

What is the last column in Excel?
Excel columns run from A to Z, AA to AZ, AAA to XFD. The last column is XFD.
Hi there zaska,

How does the filename change? I see there is a day, month and year in the filename. Part of this code assumes it's going to be using the system date. As there was no file for the 10th of December 2011, I used the link you posted (on the 7th), which worked. So let us know how this works and the following code can be refined to work for you.

Basically what this will do is download the zip file to your desktop, extract the file there, then delete the zip file. If the file is already downloaded you'll be warned and given the option to continue. Same with the extracted file, if there you'll be prompted whether you want to replace the file or not (if not nothing will happen).

Code:
Option Explicit

Private Declare Function URLDownloadToFile Lib "urlmon" Alias "URLDownloadToFileA" (ByVal pCaller As Long, _
                                                                                    ByVal szURL As String, _
                                                                                    ByVal szFileName As String, _
                                                                                    ByVal dwReserved As Long, _
                                                                                    ByVal lpfnCB As Long) As Long

Sub UnzipFile()

    'Set CLEANUP to True to delete zip file after extraction
    'Leave as False to not delete zip file after extraction
    Const CLEANUP               As Boolean = True

    'Variable declaration
    Dim oApp                    As Object
    Dim vFile                   As Variant
    Dim sFileName               As Variant
    Dim aName                   As Variant
    Dim vFolderName             As Variant
    Dim vSaveFileName           As Variant
    Dim sMonth                  As String
    Dim sYear                   As String
    Dim sDate                   As String
    Dim sExtractFileName        As String
    Dim ret                     As Long

    'Set file name variables
    sMonth = UCase(MonthName(Month(VBA.Date()), True))
    sYear = CStr(Year(VBA.Date()))
    '    sDate = CStr(Day(VBA.Date()) & sMonth & sYear)
    sDate = CStr("07" & sMonth & sYear)
    'EXAMPLE: http://www.nseindia.com/content/historical/EQUITIES/2011/DEC/cm07DEC2011bhav.csv.zip
    sFileName = "http://www.nseindia.com/content/historical/EQUITIES/" & sYear & "/" & sMonth & "/cm" & sDate & "bhav.csv.zip"
    aName = Split(sFileName, "/")
    vFolderName = CreateObject("WScript.Shell").SpecialFolders("Desktop") & "\"
    vSaveFileName = vFolderName & aName(UBound(Split(sFileName, "/")))

    'Check if the file exists locally in this location first
    If Len(Dir(CStr(vSaveFileName), vbNormal)) = 0 Then
        'Download the file first to the Desktop
        ret = URLDownloadToFile(0, sFileName, vSaveFileName, 0, 0)
        'See if the download happened correctly
        If ret = 0 Then
            'all went well
        Else
            MsgBox "There was a problem downloading the file.", vbCritical, "DOWNLOADING ERROR!"
            Exit Sub
        End If
    Else
        If MsgBox("The file is already downloaded.  Continue anyway?", vbYesNo, "CONTINUE?") <> vbYes Then Exit Sub
    End If

    'Set extraction name (same of file, less the ".zip" at the end)
    sExtractFileName = Split(vSaveFileName, "\")(UBound(Split(vSaveFileName, "\")))
    sExtractFileName = Left(sExtractFileName, Len(sExtractFileName) - 4)

    'Do the actual extraction
    Set oApp = CreateObject("Shell.Application")
    For Each vFile In oApp.Namespace(vSaveFileName).items
        If Len(Dir(CStr(vFile), vbNormal)) <> 0 Then
            If MsgBox(CStr(vFile) & " already exists.  Delete and replace?", vbYesNo, "REPLACE?") = vbYes Then
                Kill vFolderName & CStr(vFile)
                oApp.Namespace(vFolderName).CopyHere oApp.Namespace(vSaveFileName).items.Item(CStr(vFile))
            End If
        Else
            oApp.Namespace(vFolderName).CopyHere oApp.Namespace(vSaveFileName).items.Item(CStr(vFile))
        End If
        '        oApp.Namespace(vFolderName).CopyHere oApp.Namespace(vSaveFileName).items.Item(CStr(vFile))
    Next vFile

    'Clean up zip files
    If CLEANUP Then
        Kill vSaveFileName
    End If
    
    MsgBox "You can find the files here: " & vFolderName

End Sub

HTH
 
Last edited:
Upvote 0
Hello Sir,

I thank you a lot for spending your time to help me out. Well your code is really awesome and it is really helpful to me and would definitely save me some time.

Let me explain my problem further. Daily i have to download Four .zip files and One .DAT file from the following URL's


http://www.nseindia.com/content/historical/EQUITIES/2010/SEP/cm20SEP2010bhav.csv.zip


www.nseindia.com/content/historical/DERIVATIVES/2009/SEP/fo18SEP2009bhav.csv.zip

http://www.bseindia.com/bhavcopy/eq180909_csv.zip

www.bseindia.com/BSEDATA/gross/2009/SCBSEALL1809.zip

http://www.nseindia.com/archives/equities/mto/MTO_18092009.DAT

The dates in the URL's change every day for Example.

cm18SEP2010bhav.csv.zip would change to cm19SEP2010bhav.csv.zip

There will be no data for Saturdays and Sundays.

All i want to accomplish is to download the .zip and .DAT files from the above URL's to some folder and extract the .zip files. so that i could process them further.

Is it possible to use some kind of loop structure in VBA code so that i could download files between two dates.

For example i want to download all the .zip and .DAT file from 01-12-2011 TO 09-12-2011 automatically.

I could specify the From and To date in Excel sheet.

I have the following code but i don't know how to insert the remaining URL's to it.


Code:
Public Sub downloadbsedelivery()

On Error Resume Next
MkDir ("E:\Macros\BseDailyBhav")

On Error Resume Next
Kill "E:\Macros\BseDailyBhav\*.txt*"
On Error Resume Next
Kill "E:\Macros\BseDailyBhav\*.zip*"
Application.ScreenUpdating = False
Application.DisplayAlerts = False

'get start date and range===============================

Range("A1").Select

Dim daycount, daycountend, startday, startmonth, startyear As Variant
Range("B1").Value = "=C3-B3"
Range("C1").Value = Range("B3").Value
daycount = 0
daycountend = Range("B1").Value
On Error GoTo line1
While daycount <= daycountend



Range("D1").Value = Range("C1").Value + daycount
Range("E1").Value = "=TEXT(D1,""dd"")"
Range("F1").Value = "=TEXT(D1,""mm"")"
Range("G1").Value = "=TEXT(D1,""yyyy"")"


startday = Range("E1").Value
startmonth = Range("F1").Value
startyear = Range("G1").Value

'download file==========================================

    Dim WebUrlStr As String, LocalFile As String
    Dim oXMLHTTP As Object, bArray() As Byte, hfile As Integer
    Dim tempWb As Workbook, newWb As Workbook
    Dim MyRange As Range
       
    Set oXMLHTTP = CreateObject("Microsoft.XMLHTTP")
    
     'download the file from the web to the hardrive
    WebUrlStr = "http://www.bseindia.com/BSEDATA/gross/" + CStr(startyear) + "/SCBSEALL" + CStr(startday) + "" + CStr(startmonth) + ".zip"
    LocalFile = "E:\Macros\BseDailyBhav\SCBSEALL" + CStr(startday) + "" + CStr(startmonth) + ".zip"
    oXMLHTTP.Open "GET", WebUrlStr, False
    oXMLHTTP.send
    bArray = oXMLHTTP.ResponseBody
    hfile = 1
    Open LocalFile For Binary As #hfile
    Put #hfile, , bArray
    Close #hfile
        
    Set oXMLHTTP = Nothing
   
   
'unzip file================================================
   
    Dim FSO As Object
    Dim oApp As Object
    Dim Fname As Variant
    Dim FileNameFolder As Variant
    Dim DefPath As String

    Fname = "E:\Macros\BseDailyBhav\SCBSEALL" + CStr(startday) + "" + CStr(startmonth) + ".zip"
    If Fname = False Then
        'Do nothing
    Else
        'Destination folder
        DefPath = "E:\Macros\BseDailyBhav\"    '<<< Change path
        If Right(DefPath, 1) <> "\" Then
            DefPath = DefPath & "\"
        End If

        FileNameFolder = DefPath

        '        'Delete all the files in the folder DefPath first if you want
        '        On Error Resume Next
        '        Kill DefPath & "*.*"
        '        On Error GoTo 0

        'Extract the files into the Destination folder
        Set oApp = CreateObject("Shell.Application")
        oApp.Namespace(FileNameFolder).CopyHere oApp.Namespace(Fname).items

       ' MsgBox "You find the files here: " & FileNameFolder

        On Error Resume Next
        Set FSO = CreateObject("scripting.filesystemobject")
        FSO.deletefolder Environ("Temp") & "\Temporary Directory*", True
    End If

   Kill "E:\Macros\BseDailyBhav\*.Zip*"
    On Error GoTo 0
   
    daycount = daycount + 1
   
    Wend

GoTo line2

line1:
MsgBox "Download Unsucessful. Kindly check Dates entered"

line2:
Range("B1:G1").ClearContents
Application.DisplayAlerts = True
Application.ScreenUpdating = True
Range("B3").Value = "mm/dd/yyyy"
Range("C3").Value = "mm/dd/yyyy"
ActiveWorkbook.Save
End Sub
Kindly help me at your own convenience and free time.

Regards,
Zaska
 
Upvote 0
Zaska, honestly I'm a little exasperated. Not only are your requirements not what you first posted, but you had code to do the brunt of the work. I hope in the future that when you post a question you will line out all of the details, so if a solution is made and time is put into one (as I did) it doesn't have to be re-done, or the solutions are a waste of time.

That being said, what code would you like to go with? You use the xmlhttp method for downloading the zip file, my code uses an API. Both will work as long as it is a valid file name. Do you want to make use of a UserForm for this? Will this be used in a single workbook only? As far as rules on your dates, must you put in two dates? What if you want the same day, would you put the same date in both fields? What if the files(s) are already downloaded? Prompt, do nothing, or delete and re-download?
 
Upvote 0
Zaska, honestly I'm a little exasperated. Not only are your requirements not what you first posted, but you had code to do the brunt of the work. I hope in the future that when you post a question you will line out all of the details, so if a solution is made and time is put into one (as I did) it doesn't have to be re-done, or the solutions are a waste of time.

That being said, what code would you like to go with? You use the xmlhttp method for downloading the zip file, my code uses an API. Both will work as long as it is a valid file name. Do you want to make use of a UserForm for this? Will this be used in a single workbook only? As far as rules on your dates, must you put in two dates? What if you want the same day, would you put the same date in both fields? What if the files(s) are already downloaded? Prompt, do nothing, or delete and re-download?

Sir,

First of all i am sorry for not lining out all the details. I thought that i could get response from respected members only when my query is simple.

I would like to go with API. I didn't intend to use xmlhttp.I just showed you an example i have found on net.

Yes i want to use userform with a calender (if possible ) for selecting dates and years.
Yes i would like to use only a single workbook
I really don't know whether using two dates is really useful. My intention is to download previous days data if i miss them due to internet or system failure for many days. Suppose if i miss a week's data then my intention is to specify that week between two days. However if you have anyohther idea or suggestion you are welcome.
If it is a same day then system date is sufficient
If a file is already downloaded then just prompt and proceed to download the next day's data.

I am really sorry for asking too much of help. All i want to accomplish is to download the files to a folder and i don't mind which way you go with your solution.

Thank you for the kind help and feedback.

Regards,
Zaska
 
Upvote 0
Thank you for the apology. A lot of members put a lot of time and care into their posts. It is just friendly netiquette. :)

As for the API vs xmlhttp, I'm not saying you need to do one or the other. It just appeared you had that code utilized already. I'm not sure there's a huge difference, except for the fact the xmlhttp needs to create an object while the other utilizes an API. I haven't run speed tests on it, but I'm willing to suspect the API is slightly faster, even just in lieu of the object.

Give me a little bit to come up with something for you.
 
Upvote 0
Thank you for the apology. A lot of members put a lot of time and care into their posts. It is just friendly netiquette. :)

As for the API vs xmlhttp, I'm not saying you need to do one or the other. It just appeared you had that code utilized already. I'm not sure there's a huge difference, except for the fact the xmlhttp needs to create an object while the other utilizes an API. I haven't run speed tests on it, but I'm willing to suspect the API is slightly faster, even just in lieu of the object.

Give me a little bit to come up with something for you.

Sir,

Thank you for showing interest. Please take your own time. I will be eagerly waiting for your post.

If anything is sill not clear in my questionnaire..Kindly give your feedback.

Have a nice day.
 
Upvote 0
Okay, so here is the code. I'm uploading the file because of the userform, as I have named the controls for which the code works. It seemed to work for me in testing, but please try it out and let us know what you think. I believe it is what you asked for in your specs though.

Userform named "ufGetData" with the following code:
Code:
Option Explicit

Private Sub ckbOneDay_Click()
    Me.lblEnd.Enabled = Not Me.ckbOneDay.Value
    Me.cbMonthEnd.Enabled = Not Me.ckbOneDay.Value
    Me.cbDayEnd.Enabled = Not Me.ckbOneDay.Value
    Me.cbYearEnd.Enabled = Not Me.ckbOneDay.Value
    If Me.ckbOneDay.Value = False Then
        Me.cbDayEnd.Value = Me.cbDayStart.Value
        Me.cbMonthEnd.Value = Me.cbMonthStart.Value
        Me.cbYearEnd.Value = Me.cbYearStart.Value
    End If
End Sub

Private Sub UserForm_Initialize()
    Me.cbMonthStart.Clear
    Me.cbMonthEnd.Clear
    Me.cbDayStart.Clear
    Me.cbDayEnd.Clear
    Me.cbYearStart.Clear
    Me.cbYearEnd.Clear
    For i = 1 To 12
        Me.cbMonthStart.AddItem MonthName(i, False)
        Me.cbMonthEnd.AddItem MonthName(i, False)
    Next i
    For i = 1 To 31
        Me.cbDayStart.AddItem i
        Me.cbDayEnd.AddItem i
    Next i
    For i = Year(VBA.Date()) - 4 To Year(VBA.Date())
        Me.cbYearStart.AddItem i
        Me.cbYearEnd.AddItem i
    Next i
    Me.cbMonthStart.ListIndex = Month(VBA.Date()) - 1
    Me.cbDayStart.ListIndex = Day(VBA.Date()) - 1
    Me.cbYearStart.ListIndex = 4
    Me.ckbOneDay.Value = True
End Sub

Private Sub cmbCancel_Click()
    Unload Me
End Sub

Private Sub cmbGO_Click()

'Declare variables
    Dim dtStart As Date, dtEnd As Date, dtLoop As Date

    'Check for a valid start date
    If Len(Me.cbDayStart.Value) = 0 Or Len(Me.cbMonthStart.Value) = 0 Or Len(Me.cbYearStart.Value) = 0 Then
        MsgBox "Please select a start date!", vbCritical, "INCOMPLETE"
        Exit Sub
    End If

    'Check if a second date is used
    If Me.ckbOneDay.Value = False Then

    'If second date used, check for valid date
        If Len(Me.cbDayEnd.Value) = 0 Or Len(Me.cbMonthEnd.Value) = 0 Or Len(Me.cbYearEnd.Value) = 0 Then
            MsgBox "Please select an end date!", vbCritical, "INCOMPLETE"
            Exit Sub
        End If
    End If

    'Set start date variable
    dtStart = DateValue(Me.cbMonthStart.Value & " " & Me.cbDayStart.Value & ", " & Me.cbYearStart.Value)

    'Check if a second date is used
    If Me.ckbOneDay.Value = False Then

    'Set end date to start date if not used
        dtEnd = DateValue(Me.cbMonthEnd.Value & " " & Me.cbDayEnd.Value & ", " & Me.cbYearEnd.Value)

    'Check if end date is prior to start date (error trapping)
        If dtEnd < dtStart Then
            MsgBox "Your end date must be after your start date!", vbCritical, "ERROR!"
            Exit Sub
        End If
    Else

    'If no end date used, set as start date
        dtEnd = dtStart
    End If

    'Turn off some application settings to speed up code
    Call TOGGLEEVENTS(False)
    sErr = vbNullString

    'Loop through each day between dates, call routine individually
    For dtLoop = dtStart To dtEnd
        Call UnzipFile(dtLoop)
    Next dtLoop

    'Turn back on some application settings
    Call TOGGLEEVENTS(True)

    'Check out error message status from running the downloads, return a message depending on it's value
    If Len(sErr) > 0 Then
        MsgBox "There was a problem downloading the following files:" & DNL & sErr & NL & "Downloads successful.", vbExclamation, "DONE WITH ERRORS"
    Else
        MsgBox "All downloads completed successfully.", vbOKOnly, "DONE!"
    End If

    'Close the userform
    Unload Me

End Sub

Standard module named "modGetData" with the following code:
Code:
Option Explicit

Private Declare Function URLDownloadToFile Lib "urlmon" Alias "URLDownloadToFileA" (ByVal pCaller As Long, _
                                                                                    ByVal szURL As String, _
                                                                                    ByVal szFileName As String, _
                                                                                    ByVal dwReserved As Long, _
                                                                                    ByVal lpfnCB As Long) As Long

Public WB                           As Workbook
Public WS                           As Worksheet
Public i                            As Long
Public sErr                         As String

Public Const NL                     As String = vbNewLine
Public Const DNL                    As String = vbNewLine & vbNewLine

Sub LaunchUserform()
    Load ufGetData
    ufGetData.Show
End Sub

Sub UnzipFile(dtLook As Date)

'Set CLEANUP to True to delete zip file after extraction
'Leave as False to not delete zip file after extraction
    Const CLEANUP                   As Boolean = True

    'Variable declaration
    Dim oApp                        As Object
    Dim vFile                       As Variant
    Dim sFileName                   As Variant
    Dim aName                       As Variant
    Dim vFolderName                 As Variant
    Dim vSaveFileName               As Variant
    Dim sDay                        As String
    Dim sMonth                      As String
    Dim sYear                       As String
    Dim sDate                       As String
    Dim sExtractFileName            As String
    Dim ret                         As Long
    Dim iMonth                      As Long
    Dim aURLs(1 To 5, 1 To 4)       As Variant

    'Set file name variables
    sDay = CStr(Format(Day(dtLook), "00"))
    sMonth = UCase(CStr(MonthName(Month(dtLook), True)))
    iMonth = CStr(Format(Month(dtLook), "00"))
    sYear = CStr(Year(dtLook))
    sDate = Format(Day(dtLook), "00") & sMonth & sYear

    'EXAMPLES:
    'http://www.nseindia.com/content/historical/EQUITIES/2010/SEP/cm20SEP2010bhav.csv.zip
    'http://www.nseindia.com/content/historical/DERIVATIVES/2009/SEP/fo18SEP2009bhav.csv.zip
    'http://www.bseindia.com/bhavcopy/eq180909_csv.zip
    'http://www.bseindia.com/BSEDATA/gross/2009/SCBSEALL1809.zip
    'http://www.nseindia.com/archives/equities/mto/MTO_18092009.DAT

    'Loop through all 5 websites
    For i = 1 To 5
        Select Case i
        Case 1
            sFileName = "http://www.nseindia.com/content/historical/EQUITIES/" & sYear & "/" & sMonth & "/cm" & sDate & "bhav.csv.zip"
        Case 2
            sFileName = "http://www.nseindia.com/content/historical/DERIVATIVES/" & sYear & "/" & sMonth & "/fo" & sDate & "bhav.csv.zip"
        Case 3
            sFileName = "http://www.bseindia.com/bhavcopy/eq" & sDay & iMonth & sYear & "_csv.zip"
        Case 4
            sFileName = "http://www.bseindia.com/BSEDATA/gross/" & sYear & "/SCBSEALL" & sDay & iMonth & ".csv.zip"
        Case 5
            sFileName = "http://www.nseindia.com/archives/equities/mto/MTO_" & sDay & iMonth & sYear & ".DAT"
        End Select
    'sFilename
        aURLs(i, 1) = sFileName
    'aNames
        aURLs(i, 2) = Split(sFileName, "/")
    'vFolderName
        aURLs(i, 3) = CreateObject("WScript.Shell").SpecialFolders("Desktop") & "\"
    'vSaveFileName
        aURLs(i, 4) = aURLs(i, 3) & aURLs(i, 2)(UBound(Split(sFileName, "/")))

    'Check if the file exists locally in this location first
        If Len(Dir(CStr(aURLs(i, 4)), vbNormal)) = 0 Then
    'Download the file first to the Desktop
            ret = URLDownloadToFile(0, aURLs(i, 1), aURLs(i, 4), 0, 0)
    'See if the download happened correctly
            If ret = 0 Then
    'all went well
            Else
    'Something went wrong, grab the URL and pass for error messaging
    '            sErr = sErr & Format(dtLook, "mmm d, yyyy") & NL
                sErr = sErr & aURLs(i, 1) & NL
                Exit Sub
            End If
        Else
            If MsgBox("The file is already downloaded.  Continue anyway?", vbYesNo, "CONTINUE?") <> vbYes Then Exit Sub
        End If

    'Check if it is a zip file, if so, extract it
        If UCase(Right(aURLs(i, 4), 4)) = ".ZIP" Then

    'Set extraction name (same of file, less the ".zip" at the end)
            sExtractFileName = Split(aURLs(i, 4), "\")(UBound(Split(aURLs(i, 4), "\")))
            sExtractFileName = Left(sExtractFileName, Len(sExtractFileName) - 4)

    'Do the actual extraction
            Set oApp = CreateObject("Shell.Application")
            For Each vFile In oApp.Namespace(aURLs(i, 4)).items
                If Len(Dir(CStr(vFile), vbNormal)) <> 0 Then
                    If MsgBox(CStr(vFile) & " already exists.  Delete and replace?", vbYesNo, "REPLACE?") = vbYes Then
                        Kill aURLs(i, 3) & CStr(vFile)
                        oApp.Namespace(aURLs(i, 3)).CopyHere oApp.Namespace(aURLs(i, 4)).items.Item(CStr(vFile))
                    End If
                Else
                    oApp.Namespace(aURLs(i, 3)).CopyHere oApp.Namespace(aURLs(i, 4)).items.Item(CStr(vFile))
                End If
    '        oApp.Namespace(vFolderName).CopyHere oApp.Namespace(vSaveFileName).items.Item(CStr(vFile))
            Next vFile

    'Clean up zip files
            If CLEANUP Then
                Kill aURLs(i, 4)
            End If

        End If

    Next i

End Sub

Public Sub TOGGLEEVENTS(blnState As Boolean)
'Originally written by Zack Barresse
    With Application
        If Not blnState And Not ActiveWorkbook Is Nothing Then .Calculation = xlCalculationManual
        .DisplayAlerts = blnState
        .EnableEvents = blnState
        .ScreenUpdating = blnState
        If blnState Then .CutCopyMode = False
        If blnState Then .StatusBar = False
        If blnState And Not ActiveWorkbook Is Nothing Then .Calculation = xlCalculationAutomatic
    End With
End Sub

A standard button on the worksheet can call the following routine "LaunchUserform", set your dates in the userform, click the GO button. Test file attached in zip format.

HTH
 
Upvote 0

Forum statistics

Threads
1,225,039
Messages
6,182,533
Members
453,124
Latest member
reshmawils

We've detected that you are using an adblocker.

We have a great community of people providing Excel help here, but the hosting costs are enormous. You can help keep this site running by allowing ads on MrExcel.com.
Allow Ads at MrExcel

Which adblocker are you using?

Disable AdBlock

Follow these easy steps to disable AdBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the icon in the browser’s toolbar.
2)Click on the "Pause on this site" option.
Go back

Disable AdBlock Plus

Follow these easy steps to disable AdBlock Plus

1)Click on the icon in the browser’s toolbar.
2)Click on the toggle to disable it for "mrexcel.com".
Go back

Disable uBlock Origin

Follow these easy steps to disable uBlock Origin

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back

Disable uBlock

Follow these easy steps to disable uBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back
Back
Top