Instructor: Now that we've seen how we can get started
using Selenium WebDriver to automate our browser,
the next step is to figure out how to use Selenium to find
and locate specific HTML elements on the webpage.
Just between where we quit our driver
and where we get hold of a particular page,
I'm gonna show you how easy it is to complete
the previous task in day 47 where we tried to get hold
of the price of a particular item on amazon.com.
This was the item that we were interested in.
This was the Instant Pot, and this is the price.
So if I go ahead and take this URL
and replace this previous just amazon.com URL
with the URL that takes us to this page,
then we can start inspecting on this particular element.
Now, if we take a look at the HTML,
we can see that we've got the price showing.
And at the time of recording,
the price is split into two parts.
So you've got a class called a-price-whole
which is the number of dollars
and a class called a-price-fraction, which are the cents.
So Amazon can obviously change their website at any time.
Previously, for example, the price
used to be inside an element with the ID price block price.
So you have to look inside which element the price is in
by right-clicking inspect element
and then checking to see what the ID is called
because Amazon can change this set any time.
Now, once you've noted down
if you want a particular class or a particular ID,
take a look at the Selenium locator strategies.
You can see here in the Selenium Docs,
there's many ways that we can find different elements
on a page Using Selenium.
You can see that Selenium provides support
for these eight traditional location strategies.
In our case, we're looking for the price element
on Amazon's page by selecting on a particular class name.
Now the Selenium docs are pretty good
and include handy little code snippets
for everything you wanna do.
So for example here, you can see how to find an element
by class name, and that's exactly what we need to do.
And you can do that by using the find_element method.
Depending on whether if you're looking to find an element
by class name or by ID, we need to use a class called by.
And it has helpers such as by.class_name.
And then all we need to do is just to specify
the name of the class that we're interested in finding.
So the first thing I'll do is to import the by class
and that's just from selenium.webdriver.com.common.by.
And then import_By.
And then we can create a variable called price_dollar
to hold onto the dollar price element
from the Amazon page.
The value, we'll grab the class name of the span
from the Amazon page.
So the class is called a-price-whole
and we're gonna paste it into our code.
And it's probably easier just so you don't make any typos
to just copy the name of that class
and paste it into our code.
Next, let's do the same thing
for the second part of the price, the cents on the dollar.
So we'll create a variable called price_cents
and we'll use driver.find_element by.class_name
and then set the class name value to equal
to a-price-fraction.
Again, these things can change.
At the time of recording, it's called a-price-fraction
but be sure to check what the live situation is
when you are doing this exercise.
Now let's print out the full price as an F string.
We'll write print f, and then the price is,
and we'll insert the price dollar,
dot, and then the price_cents.
Now what's gonna happen here
if you try to run this code is we've actually found
the element by class_name, but right now,
price_dollar and price-cent are actually an HTML element.
So if we want to have the text that's inside those elements,
we have to access the text content.
And you can see here in the Selenium Docs,
to get the text content,
all we need to do is just write .text
after we found the element.
So we can add dot text to the price_dollar and price_cent.
So now we can actually get hold
of the content that's inside those HTML elements.
Now all we need to do is uncomment our driver.quit
because otherwise, we're gonna have a new instance
of Chrome running every time I hit the run button.
And now if I hit run, you'll see our browser open up briefly
to the page with the Instant Pot, and then it'll close down
and you can see the price printed right here.
Now the reason why this is so much shorter than what you did
in the last project is because we're driving a browser.
So the browser's ready, sending all of those headers,
all of the information that Amazon would expect
from a actual user, instead of using a get request
using the request package.
In only three lines,
we can do so much more than we could do with BeautifulSoup.
Essentially, if you think about it
BeautifulSoup was really good for just grabbing
and scraping pieces of data from an HTML website
but it gets stuck when that website is being rendered
using JavaScript or Angular or React
and the content that was HTML was taking time to load
or it requires certain conditions to load.
Whereas in this case,
we're effectively doing exactly the same thing
as we would as a human.
Going to this particular URL
and then looking at this element and its value
I wanted to show you a couple
of other ways that you could find elements.
You can see there's a whole bunch of ways
of finding elements and there's a lot more methods
than you have with BeautifulSoup.
One that I wanna show you that's quite useful
is find element by name.
If we go to python.org, you'll remember
from your lessons on HTML and CSS
that most forms, these input forms
will have a name attribute.
So if I go ahead and inspect on this search bar,
you can see it's got an ID, it's got a name,
it's got a type, it's got a role, it's got a class
it's got a whole bunch of attributes.
Now I'm gonna search for it by this name, which is Q.
So all I need to do here is call the find_element method.
And for our by helper class,
I specify that I want to search by name.
So the name of the element
and then I provide the name of the search bar, which is Q.
Now of course, I'll have to change this URL
to actually python.org.
And now, once we've found our element by name,
we'll call it the search bar.
And then we'll go ahead and print out this search bar.
And then we'll comment out our code
that we used for Amazon and then hit run.
When Selenium locates a particular element,
it won't actually print out the actual HTML.
It'll give it to you as a selenium element like this.
Now if you wanna tap
into its various attributes or text or tag name,
then you'll actually have to do that using a dot.
So we can say search_bar.tag_name
and it will give us that that is an input tag.
Or we could say search_bar.get_attribute
and we can specify the attribute that we want the value for.
So for example,
if I wanted to know what the placeholder value was,
then all I have to do is put get_attribute placeholder
and then we can print that out
and it will tell us that the placeholder says search.
So that is something that you'll find people commonly use
with Selenium find element by name.
And it's really useful when it comes to filling in web forms
because most forms will have elements that are organized
by name because when the form is submitted,
that name is carried along with the value of the inputs.
In addition to find element by name and class name,
you can of course find an element by id.
So for example, the submit button
next to the search bar has the ID of submit.
And once again, we need to use the find element method
but this time, we specify
that we wanna find the element by ID.
And we can add in the value of the ID, which is submit.
So the code looks like this.
Button equals driver.find_element by.id
and then we insert the value of that ID, which is submit.
And similar to before, once we've found the element,
we can get hold of various properties.
For example, we can even print out the size of the element.
Let's see what it is.
Let's go ahead and print button that we found that element
and then we use .size to see what's its height.
And if we print it out, we can see
that it has a height of 40 and a width of 46.
So Selenium is really powerful
and the ability to use their helper methods to find element
by ID name or class name allows us to pretty much reach
into any website and get hold of any element that we want.
And then by using these properties
such as .size or .text, we can access whichever parts
of the element we're interested in.
So you can see that we can do a lot more
with Selenium than we could with BeautifulSoup.
So that's find_element by ID, by name, by class name.
But there's also of course, find_element by CSS selector.
And this is probably one of the easiest ways
of narrowing down on a particular element.
Let's say that I wanted to get hold
of this link to the docks.
Let's go ahead and inspect it
and it lives inside an anchor tack.
But you can see that this anchor tag has no class,
it has no ID, has no easy way of identification.
So how do we get hold of it?
Well, we can see that it lives in a paragraph element.
And then going further up,
it's inside a div with this particular class.
So this has a class of documentation widget.
So that's pretty specific.
So if we wanted to get hold
of that anchor tag, we could just say,
well inside this div of documentation widget,
let's go ahead and find any anchor tags.
And to express that as a CSS selector
we would specify the class documentation widget.
And then inside the element with that class,
we're looking for an anchor tag.
So this would get us our documentation link.
And now if I go ahead
and print the documentation link .text, you can see
that it's actually getting hold of this particular link.
And this is done even though the actual link
doesn't have any easily identifiable name or class or ID.
Now, on a similar vein, sometimes it's extremely
hard to even find an element even by CSS selector.
So if all else fails,
one that will always work is the XPath.
The XPath is a way of locating a specific HTML element
by a path structure.
So you've seen what paths look like, right?
Users/angela/development, etc.
Well, we can also express the navigation
to a particular element, drilling down from the top
of the tree to a particular node using the XPath.
Let's say I want to get hold
of this link at the bottom, Submit Website Bug.
So let's go ahead and inspect on it.
And you can see it's not particularly unique
in terms of having an ID or a class or a name.
And even in terms of its structure.
It's sort of in this ul and then in li.
And it might not be that easy to get to it.
But here's where our XPath comes in handy.
Let's go ahead and right-click on it
and then go to copy copy XPath.
And now, we can go back
to our code and use it to tap into that element.
So we'll say driver
and then we'll say find_element by.XPath.
And then inside here, we're going to paste our XPath.
Now, notice that the XPath itself
actually has some double quotes.
So we have to change our double quotes into single quotes
so that it doesn't clash with the inner double quotes.
So like this.
Now, once we found our bug link,
then we're going to print out the link,
and I'm going to get hold of the href here.
So now I'm gonna print my bug_link.text
just to prove that this worked.
And if I hit run, you can see it opens up
our browser and you can see it's managed to get hold
of that particular link using this very specific Xpath,
which looks very much
like a file path that we've been using.
But it's essentially the way that we would navigate down
through the divs, the uls, the lis,
all the way down to that particular anchor tag.
And the best part of all is of course,
you can use this button here in Chrome to locate any element
and then that would select its location in the code
and then you can right-click
and then copy its XPath to get hold
of that particular thing.
Now I'm gonna link
to the documentation on XPath from W3 schools.
It goes into a little bit more detail
about how XPaths are constructed
but you don't need to know this in order to work with them.
As I've shown, you can simply use the Chrome developer tools
and it will help you get
to the particular item you're interested in.
Now, we've been looking at finding elements using Selenium
but notice how we've always stuck
to the singular version of that method, find_element.
Now, for every single method that I've shown you,
there's also a counterpart, which is the find_elements.
So for example, find_elements by.CSS_selector.
And of course, this is just going to find you everything
on the screen that matches your criteria
and give it to you in a list.
And there is a version of this
for every single find method that we've shown so far.
In the next lesson, I've got a quick challenge for you
to put into practice what you've learned
and to be able to locate any given element on a webpage.
So for all of that and more,
I'll see you on the next lesson.