Data scraping with URL's that keep changing.

lecxE007

New Member
Joined
Jul 22, 2011
Messages
10
I am try to get some real estate data off this website but the URL keeps changing. I would like to put a list of parcel numbers in excel and let Excel do the data scraping. The parcel number is at the very end of the URL. Here is a example of a URL:
[TABLE="width: 108"]
<tbody>[TR]
[TD="class: xl64, width: 108"]http://beacon.schneidercorp.com/Application.aspx?AppID=83&LayerID=782&PageTypeID=4&PageID=549&Q=764486438&KeyValue=03000750
[/TD]
[/TR]
</tbody>[/TABLE]

The ten digits before &KeyValue= are always changing. The URL string before the rolling ten digits seems to stay the same. I recorded a macro. See below the macro I copied out of VB.

Sub adds()

'For i = 1 to 3
For x = 1 To 3
Worksheets("Tax liens").Select
Worksheets("Tax liens").Activate
mystr = "URL;http://beacon.schneidercorp.com/Application.aspx?AppID=83&LayerID=782&PageTypeID=4&PageID=549&Q=923706357&KeyValue=01008700.html"
'mystr As String
mystr = Cells(x, 1)
Worksheets.Add(After:=Worksheets(Worksheets.Count)).Name = x

With ActiveSheet.QueryTables.Add(Connection:= _
mystr, Destination:=Range("$D$1"))
' .CommandType = 0
.Name = _
"Application.aspx?AppID=83&LayerID=782&PageTypeID=4&PageID=549&Q=1280381814&KeyValue=01008700"
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlSpecifiedTables
.WebFormatting = xlWebFormattingNone
.WebTables = """ctlBodyPane_ctl26_grdValuation"""
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.Refresh BackgroundQuery = False
End With

Next x
'Next i
End Sub


Is there any way around this? Or could I go about this in a different way? Could I record my macro a different way?

Thanks
:confused:

Excel 2010
 

Excel Facts

Show numbers in thousands?
Use a custom number format of #,##0,K. Each comma after the final 0 will divide the displayed number by another thousand
To answer my own question the number that keeps changing isn't really needed when scraping. .com/Application.aspx?AppID=83&LayerID=782&PageTypeID=4&PageID=549&Q=106474762&KeyValue=24075675

If I delete the Q=106474762& the url still works.

Here is a podcast on data scraping that worked for me.
https://www.youtube.com/watch?v=7DRGfu-BFNc
[h=1]Learn Excel 2013 - "Scraping Web Pages with Excel": Podcast #1684 with Bill Jelen[/h]Thanks You Bill Jelen
 
Upvote 0

Forum statistics

Threads
1,223,236
Messages
6,170,917
Members
452,366
Latest member
TePunaBloke

We've detected that you are using an adblocker.

We have a great community of people providing Excel help here, but the hosting costs are enormous. You can help keep this site running by allowing ads on MrExcel.com.
Allow Ads at MrExcel

Which adblocker are you using?

Disable AdBlock

Follow these easy steps to disable AdBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the icon in the browser’s toolbar.
2)Click on the "Pause on this site" option.
Go back

Disable AdBlock Plus

Follow these easy steps to disable AdBlock Plus

1)Click on the icon in the browser’s toolbar.
2)Click on the toggle to disable it for "mrexcel.com".
Go back

Disable uBlock Origin

Follow these easy steps to disable uBlock Origin

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back

Disable uBlock

Follow these easy steps to disable uBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back
Back
Top