Selenium using VBA

[amo]

I would like to hide the driver while web scraping with Selenium, just like it is done with IE object ("IE.Visible = False"). Is there a way to do what I want?

 

[diddi]

could you work the scrape using XMLHTTP instead

 

[amo]

could you work the scrape using XMLHTTP instead

I don't understand if I use this method

 

[Dan_W]

Yes, you just need to use the PhantomJS driver rather than the ChromeDriver or FirefoxDriver, though diddi's suggestion about using XMLHTTP would be quicker.

In any event, in case you need it, here is an example:

Sub HeadlessSelenium()
    
    Dim PJSD As Selenium.PhantomJSDriver
    Dim strHTML As String

    ' Instantiate Selenium through the PhantomJS Driver
    Set PJSD = New Selenium.PhantomJSDriver
    PJSD.Start
    
    ' Navigate to the URL
    PJSD.Get "https://www.inserturlhere.com"

    ' Extract the HTML code of the website
    strHTML = PJSD.PageSource
    
    ' Print the HTML code to the Immediate Window
    Debug.Print strHTML

End Sub

 

[Dan_W]

Sorry, I lie - you can do it with ChromeDriver, I've just discovered. The code is set out on StackOverflow - link. Basically, you just need to use .AddArgument "--headless" with the driver object.

 

[diddi]

XMLHTTP is a direct manipulator if web requests, whereas IE automation and (i believe) selenium are both methods which use another program as the host, so all of the code is being double handled. XMLHTTP also allows you to scrape sites where the data does not appear in the page source and is not returned in response string. so i like using it, even though it is a bit fiddly sometimes. that said there are still some sites i have not been successful in scraping

 

[amo]

Sorry, I lie - you can do it with ChromeDriver, I've just discovered. The code is set out on StackOverflow - link. Basically, you just need to use .AddArgument "--headless" with the driver object.

I've used this method
error " element not found "

 

[Dan_W]

I've used this method
error " element not found "

Can you share your code, please? It's not easy to help you otherwise. Did you try the PhantomJSDriver? XMLHTTP?

 

[amo]

Can you share your code, please? It's not easy to help you otherwise. Did you try the PhantomJSDriver? XMLHTTP?

@Dan_W

Sub tesselenium()

    Dim ch As New Selenium.ChromeDriver
    Dim URLNAME As String
    Dim sht As Worksheet
    Dim price  As String
    Dim EndRow As Long, i As Long   'ADDED
 
    Set sht = ActiveSheet
    EndRow = sht.Cells(sht.Rows.Count, "A").End(xlUp).Row
    
    For i = 3 To EndRow
        URLNAME = Cells(i, 1).Value
    
   With ch
   
    .AddArgument "--headless"
    .Get URLNAME
    
    'This is Url "https://www.tokopedia.com/petanidaun/foliage-premium-plant-food-khusus-tanaman-hias-daun
    End With
    
    ch.Wait 1000
        
    price = ch.FindElementByClass("price").Text
    
        Sheet3.Cells(i, 4) = price
       
        
        ch.Quit
        Set ch = Nothing
        Application.StatusBar = ""
        On Error GoTo 0
    Next i
   
End Sub

 

[Dan_W]

So I haven't been able to recreate the error you referenced. Instead, I encountered a number of other errors in running the same code through Selenium which reminded me that Selenium can be pretty tempermental. Turns out that, in the intervening hour, I needed to update my ChromeDriver. All I can do is suggest that you check that your ChromeDriver matches your version of Chrome, and to try restarting your system. I did this, and then tried the following code, and it worked:

Sub HeadlessSelenium_CD()
   
    Dim CD As Selenium.ChromeDriver
    Dim strHTML As String

    ' Instantiate Selenium through the ChromeDriver
    Set CD = New Selenium.ChromeDriver
   
    ' Run Selenium in Headless mode
    CD.AddArgument "--headless"
    CD.Start
   
    ' Navigate to the URL
    CD.Get "https://www.tokopedia.com/petanidaun/foliage-premium-plant-food-khusus-tanaman-hias-daun"
   
    ' Extract the HTML code of the website
    strHTML = CD.PageSource
   
    ' Print the HTML code to the Immediate Window
    Debug.Print strHTML
   
    CD.Close
    Set CD = Nothing

End Sub

I should also add that the target website looks to be a bit tricky - when I first tried accessing the site with my updated ChromeDriver, I didn't use headless mode, and it worked OK. But then when I tried to access it under headless mode, It produced the following HTML code:

<html><head>
<title>Access Denied</title>
</head><body>
<h1>Access Denied</h1>
You don't have permission to access "http://www.tokopedia.com/petanidaun/foliage-premium-plant-food-khusus-tanaman-hias-daun" on this server.<p></p></body></html>

Let me know if you've managed to get ChromeDriver to run in headless mode.