Web scraping

[Greggey1]

Is there a way to click on a drop down window located on the website and import that data when using web scraping. Trying to pull data from multiple pages but unfortunately the URL remains the same as the home page.

 

[Matt Allington]

if the click of the button produces a new, repeatable URL, you can import it in PQ. If not, you need to use a tool like Python leveraging selenium or similar

 

[Greggey1]

Thank you. Unfortunately the URL remains the same. I’ll look into python. Thanks again

 

[Edgar_]

click on a drop down window located on the website

What sort of drop down window?

Trying to pull data from multiple pages but unfortunately the URL remains the same as the home page.

1. Open the dev tools of your browser
2. Go to the network tab
3. Sort by type
4. Look for json type
5. Inspect the URL

If the browser URL has remained the same after dropping down that window, the triggered event is using REST API to bring the data.

 

[Greggey1]

What sort of drop down window?

1. Open the dev tools of your browser
2. Go to the network tab
3. Sort by type
4. Look for json type
5. Inspect the URL

If the browser URL has remained the same after dropping down that window, the triggered event is using REST API to bring the data.

Standard drop down window where select a certain criteria and click submit to get the information. Assuming REST API will not work in a standard web query?

 

[Edgar_]

Websites have diverse mechanisms to do what they do. I recommend finding out where the data is coming from. If it's coming from REST API, that's a URL that you can input in your tool. If your tool can not do the job, there are several workarounds, like:
1. browser automation through chromium using selenium, puppeteer, etc...
2. browser automation through iexplore using vba, a browser control, etc
3. browser automation through your favorite browser using a bookmarklet, browser extension, etc

Among other approaches.