When I was really young, I remember
a piece of homework that I had to do for Geography in school.
And the teacher asked us to basically watch the weather report every single day
on the news and note down what was the temperature,
what was the weather condition of that day.
And then we would end up with the weather condition and temperature for the
previous week. Now at the time,
it was the point in time where a lot of teachers didn't actually really know what
the internet did. So, um,
I realized that I could actually just wait until the day before I have to give
the homework in, go onto the internet,
find the weather conditions for the previous seven days
and then write it out instead of having to watch the weather forecast every
single day. So I don't know what that says about me,
but if you are somebody who enjoys taking shortcuts in life and refuse to do
work that can be done by computers, then you are in the right place.
Learning Python is definitely the way to go.
So inside Google sheets
which you can access by going to sheets.google.com,
I have created a new spreadsheet of the days of the week and the temperature of
each of those days in Celsius and also the weather condition on those days.
This is a replica of that homework from many, many moons ago,
but we're going to be working with this data to learn how we can read data files
and then analyze them and how we can do lots of things with it.
So the first thing I want you to do is the head over to the link in the course
resources which will take you to this spreadsheet.
And then once you've got it open, then go to file download
and I want you to download it in the comma separated values or CSV format.
And you should end up with a file like this.
Now I want you to rename this file so that it's just weather_data.csv
, and then I've want you to create a new PyCharm project
which you can call anything you want.
I've called it day 25. And also create your main.py.
Now I'm going to drag my weather_data.csv into my folder day-25 and click
refactor to move that file into my project folder. Now,
at this point,
PyCharm recognizes that this is a CSV file and it asks you whether if you want to
install some plugins to make it easier to view this file. Now,
at this point,
you can click cancel because we want to view the data as the raw data,
which is in this CSV format. Now,
CSVs are a very common way of representing tabula data,
so data that fits into tables like a spreadsheet. And CSV,
as you've already seen stands for comma separated values.
So that's why when you look at the data,
you can see each row here is a single set of data and each piece of data is
separated by a comma without space.
So we've seen how we can open files, read files, write
to files. As a challenge,
I want you to go ahead and open up this weather_data.csv file inside your
main.py and add each line of data into a list which we'll call data.
Pause the video and give that a go.
Okay.
All right. So we know that we're going to need to open the file.
So it's stored inside the same folder as our day-25.
So we can just use a relative file path to tap into this weather_data.csv.
And we're going to save this data as a data_file.
And then we're going to get the data by reading this data file.
So data = data_file.read.
And not only are we going to read it, we're going to use readlines
which we know will take each line in this file and turn it into an item in a
list. Now, if I go ahead and print my data,
then you can see I've got this list now where each item is a row in that list.
But as you can imagine, it would be pretty painful to work with the data,
which is all in a string format.
And they are still separated by commas.
It would take a lot of cleaning to actually be able to extract each column and
each row. So what can we do instead? Well,
there's actually a inbuilt library that helps us with CSVs because Python is a
language that's used really heavily for data processing, data analysis.
There's a lot of great tools for working with tabula data,
like our weather data. First, we're going to import the CSV library,
and then we're going to, again, open up a file,
weather_data.csv as our data file,
and then we're going to use this CSV library.
And it has a method called reader,
which takes the file in question, which has already been opened
so this is going to be all data_file, and it can read it an output
the data.
So now let's go ahead and print this data and let's see what we've got.
You can see that it's created a CSV reader
object. This object can be looped through.
So if we wanted to get each row inside this data,
we can say for row in data,
go ahead and print each row. And once you've done that,
you can see it's taken each of the rows inside our weather_data.csv,
and separated out each item into a single value.
So for example, on the Monday row,
we've got the Monday as a string,
we've got the temperature as a string and also the condition as a string.
So now it's much easier for us to work with this data.
Using what you know about Python lists
I want you to create a new list called temperatures,
and this list is going to contain all of the temperatures that is inside this
weather_data.csv, like this 12 degrees, 14, 15,
and it's going to be in the format of an integer.
I don't want to see it as a string with quotation marks around.
It should be a pure number so that we can work with it more easily.
So this is your challenge.
Pause the video and see if you can extract all of the temperatures from this
file into this new list.
Okay.
All right. So when we printed out each of the rows,
we can see that we've created several lists where each list contains an entire
row of data from our weather data CSV.
If we wanted to get the temperature,
then it's going to be the item at index one in that list. For example,
if we wanted to get the Monday temperature,
then all we have to do is to tap into each of these rows and get the item at
index 1. If I go ahead and print this,
then you can see we get all of the temperatures,
but also the label for that column.
So if we want to exclude that label,
then all we have to do is use an if statement and check if row at index one
does not equal temp, which is the name of that column label,
then we're going to tap into our list of temperatures and append this row at
index one, which is going to be a temperature number.
So now after we've done the entire for loop,
then we can print out our list of temperatures. And if you take a look,
you can see we've now got a list of all the temperatures
excluding that column title, but they are all in the format of strings.
So if we want to convert that into an integer,
then all we have to do is wrap that around an int.
So that's the goal of the challenge. Now you can of course,
separate out this line into many more lines,
but I think this should make enough sense for you at this stage. While CSV is the
inbuilt CSV reading and writing library,
notice how much faff was involved in just simply getting a single column of
data. What are we going to do if we have more data,
more complex data with way more columns, way more rows and
we want to do more interesting things with it?
This is going to be quite painful to work with.
This is the point where we want to get the help of some pandas.
Not these kinds of pandas. As cute as they are
they're not going to help us with our data analysis.
But instead, I'm talking about the Pandas library and this is a Python data
analysis library
which is super helpful and super powerful to perform data analysis on tabula
data like the one that we have. In order to work with it,
we have to import this library. But because it's not in built,
you'll need to install it into your project.
So the shortcut way of this is simply just a type import pandas
and then once you see the red line,
go ahead and hover over it and then click install package pandas,
and then you can watch the progress down here. Now, while that's installing,
I want to quickly introduce you to the documentation for this library.
It's really well documented and it's really powerful,
so it has a lot of things in the documentation. If you head over
to pandas.pydata.org and then click on documentation,
then you will be able to see all the things that you can do with it.
So there's a API reference,
there's a quick getting started guide as well as a user guide on the key
concepts of pandas. When you're using a new library of any sort, a good idea is
to take a look at their getting started guide if they have one,
because it tells you how you can install it and a number of questions that you
might have, for example,
what kind of data does pandas handle or how can I read and write tabula data.
And it's actually done really, really well. So if you have a moment,
take a quick look at this page. And once you head back to PyCharm,
your packages should have installed successfully. Now,
once we've installed our pandas,
you'll see that it's still grey because we're not using it yet.
If we want to use pandas, all we have to do is say pandas.
and in our case, we actually want to read our CSV.
So we can say read_csv
and inside this method, you can do lots and lots of things,
as you can see by all of the attribute names. But most of these are optional.
The only one that is not optional is the path that leads to the CSV file.
So if we get hold of our weather_data.csv,
then we can read that CSV using pandas.
So notice how we don't have to open the file as a data file,
or you use a CSV reader, it's one step and you're done.
So now we've got hold of our data and if I print out the data,
you can see how beautifully formatted it is.
It's being printed out as an actual table,
it's got the column headings on top of each of the columns and each of the rows
gets given an index so that we can more easily identify how many records we have
and where each one is.
If we wanted to think about that previous task that we tried to do
where we just try to get hold of a single column of data from this table,
then using pandas,
it is literally as easy as saying data and then square brackets,
and then the name of that column. So in our case, it's temp.
And now if I go ahead and print this out,
you can see it's basically already identified the column and it's printed out
all of the data in that column.
So the really smart thing that Panda's doing here is
it takes that first row to be the names of each column
and it automatically knows how to find the data
when you just specify the name of the column like this.
So three lines versus eight lines,
and we get better formatting. It's no wonder that most Python developers,
as soon as they encounter a CSV fault,
they will start using pandas to work with it
no matter how simple the task or project.
So that was a quick introduction to CSV data,
working with CSV data and how to get started using pandas. In the next lesson,
we're going to dive deeper into this library and see all of the common things that
we can do with pandas.