Hey guys this is Caleb with slopes dot com and in this video we're going to dive right into adding our
core a male model.
We're going to pass an image into it and actually use machine learning to make a prediction of what
it thinks that it's seeing in the image.
Very very cool.
In the last video we left off with this we have our camera view we have our nice labels and our views
and when we tap the screen it takes a photo of what it sees and it saves it to the image of you.
We're going to use that in just a moment to pass into our core a male model.
OK so to begin we're actually going to pull open Safari and we're going to go ahead and head to the
core M-L site from Apple.
If you don't remember it's app developers Apple dot com slash machine learning.
Now if you scroll way down to the bottom there is a section called working with models and basically
Apple have provided quite a few different machine learning models that work with core M-L the one we're
going to use is called squeeze net.
And what it does is it detects the dominant objects present in an image from a set of 1000 categories
like trees animals food vehicles people and more but with an overall footprint of 4.7 megabytes it's
really really small.
And if you actually look at some of these other core and L models you'll see they're pretty big 100
to megabytes 94 megabytes.
Five hundred fifty three megabytes.
So this one is just way more efficient and way smaller.
I mean that's just good for our app it will make the app size much smaller in the end.
So to get that model go ahead and just click download core Armel model and it'll save to your downloads
which you can see right here.
Which you can see right here and it is called squeeze net.
M-L model that of course stands for machine learning model.
Now what we need to do is open our X code project and add this file into the project so go ahead and
pull open your ex code project.
And I'm actually going to just drag this from safari and drop it right into my project.
Now of course copy items if needed create groups and make sure that the target vision app has been selected.
Click Finish and we're going to go check out what is in this M-L model.
So click on that and you'll notice here it says that it is a machine learning model.
It's of type and neural network classifier.
So now you understand how the structure is is working.
And it gives you a nice description and also references the inputs.
OK this is important to know.
So we input an image and you see right here that it is BGR 6:58 and you can't see it but that's 6:58.
That means that the input image that it can classify is 227 pixels by 227 pixels.
The outputs are what we get at the end.
OK so we get a class label probs which is the probability of each category.
And we also get the class label which is the most likely image category.
So it will tell us what it thinks it is and the probability that it thinks it's correct.
If you notice here we have squeezed net and it says that it is a swift generated source.
That means that squeeze net is accessible to us as as a model like we can use it just like we can use
any other class.
And let's click this little arrow here so I can show you that.
Check it out.
So it is a Class A swift class it's ready for us to use with squeeze net input right here squeeze net
output can scroll down.
We can actually see the whole squeeze NET class right here it's very cool and all available for us.
That way you can take a look at this more and you can kind of dive in deeper if you want to.
But for now we're just going to go ahead and we're going to add this into the resources folder and go
to camera AVC.
Let's use it.
Let's see what it can do.
So to begin we're going to go ahead and we're going to actually import two frameworks.
First we're going to import core M-L and last we're going to import vision.
OK these are both part of the core M-L package Carmel l handles just kind of your basic overarching
all of the machine learning stuff and vision specifically handles things like face recognition and object
recognition.
We already talked about that and one of the first videos but just just a quick reminder.
All right so now we get to actually use this machine learning model we've dragged in Squeeze net and
to do that go ahead and scroll down and where we created the photo data we're actually going to use
that same photo data and pass it into our core M-L model squeeze net.
So to do that go ahead and go beneath the photo data and we're going to use a do try catch block.
OK.
We need to be able to catch any errors that are thrown.
And using vision can throw errors and so we're going to set it up so that it can handle that and not
cause a crash in our application.
So go ahead and type do.
And we're going to just set up the catch block as well while we're at it.
Perfect.
And this is where we're going to handle errors in just a moment.
But inside the do block we're going to type let.
Model equals try v and core I'm L model k.
Now we're going to do a parentheses and we're going to pass in an L model.
Now if you remember squeeze net is an M L model so what we can do is we can just pass in Squeeze net
we can instantiate it and if we put a period we can pull out the male model.
If you go to look at this this class you can see that there is a variable here that is for an N L model
so we can pull that out and we can use that as a vision Cornell model.
Really really cool.
So basically we're using squeeze net but we're passing it through vision cause vision has a lot of really
cool stuff for identifying things and images.
And of course we're using try because Wii and core AML model can throw errors and you'll notice in a
while little say that we're having some errors because we're not properly handling them in our catch
block.
But we will.
So we now have a model.
This is good.
The models like our brain and now we need to make a request which is kind of like a thought case where
we're allowing our brain to have a thought.
And then it's going to do something with that thought later so after we've created our model go ahead
and type let request and this is where we're going to use a V.N. core M-L request.
Gates right there ready for you.
Now if you put a parentheses you can see that there are two options.
There's one with a model and one with the completion handler and model.
We're going to use that one because our completion handler will be where we will handle what happens
when we get a response back from this request.
OK.
Now our model of course is going to be our model.
And now this completion handler This is where we're going to call a function we will right in a moment.
So for now I'm going to go ahead and write that function but I'm not going to use it yet.
So up here in our camera class we need to write that function and we're going to put it beneath did
tap camera view.
So I'm going to go ahead and type phunk results method.
All right.
Oops.
And we're going to need to pass in a request of Vienne request.
That's a vision request and we need to also give it the capability of using the errors that could come
from our from our request so that's going to be of type error.
OK.
And there isn't always an error.
That's why we're creating it as an optional.
All right.
So go ahead and give it some curly braces and we will come back to handle
changing the label text.
OK we'll come back to that in a moment.
But right now we just need to be able to use result's method.
OK.
So our completion handler is going to be result's method.
And what we're going to do is we're going to basically just call the name of our function here because
when we create a request it automatically has a V.N. request and it automatically has an error.
So that can just be passed in by default to this function so we don't actually need to put in any of
the parameters.
That's kind of the cool thing about using this.
So we have a model that's our brain.
We have a request that's kind of like the our brain is making a thought.
And now the handler is going to basically take that thought and it's going to try to turn it into something
that we can use.
So let's go ahead and let's create a handler by typing let händler and that's going to be an image request
handler.
OK.
Now of course there are lots of options here.
As you can see CIA image CEG image C-v pixel buffer data and you are l what we're going to do is we're
actually going to go ahead and just click on the one with data and options is necessary.
We don't need to use the options.
So we'll get rid of it.
But we are going to pass in our data what data you ask our photo data OK that's the data we're going
to give it so that it can analyze and determine what's in the image so pass in photo data and go ahead
and unwrap it so that we can use it.
Now what we need to do is we basically need to use our handler to perform our request our handler is
kind of like a part of our brain that handles our ability to think.
So our handlers are going to take our thought here and it's going to try to process it.
OK.
So let's go ahead and we're going to actually need to use try as well because the handler can also throw
errors and so let's go ahead and type try handler perform and you'll see right here.
It can take an array of V.N. requests.
And so we're going to pass it an array with our request OK.
So if we try to build and run here you'll notice it will work.
But the issue is that we're not handling the errors.
If there's a problem we should know.
So to do that we're just going to use Debug.
Print like we have been doing.
And we're going to just call the error.
There's an error.
It will handle it here.
OK.
That's really easy.
So now what we're doing is we're basically saying that we have a model k we are going this is like our
brain can process the image.
We have a request that is using our brain to make a thought.
We have a handler that's going to basically take the data and compare it against our little thought.
And then it's going to perform that request where it compares them and produces some data for us that
is actually all we need to do to use machine learning.
For now we're going to actually pull out the information from our request in this function result's
method.
And that's how we're going to actually display what our M-L model is thinking.
So let's do that now go to results method and what we're going to do is we're going to create a constant
to hold the results that come in from our request.
Remember it just gets passed in straight here through our completion handler.
So get rid of this commented out words it's not code and we're going to use guard Latt to create an
instance of our results so go ahead and type guard.
Let results and that is going to be equal to request.
And if you type the period after that you can see that there is results inside of a request so very
cool.
We can get that but it needs to be we need to put it into an array because we're going to cycle through
it and we're going to check to make sure that we get the most confident result.
We don't want if it's like one tenth of a percent sure.
We don't really care about that we want something that is almost positive.
So go ahead and we're going to cast this as an array of V.N. classification observation.
And you know what you guys can dive deeper into these descriptions for what these are but basically
it is seen classification information produced by an image analysis request.
All right.
So it's basically using it to determine what is in the image that was sent in the request.
So with a guard lead at the end we need to type else.
And then inside of here it's pretty common.
Just to return.
So if it doesn't work if we if we don't get any results we're just going to return so that we don't
have a crash if we do then we have a nice constant here called results that has our results.
So that looks great.
And beneath result's we're going to go ahead and type for classification in results.
And the reason for that is because we are using V.N. classification observation and that's what we're
doing is we're we're classifying objects in an image based on criteria that the male model understands.
So for each classification let's think what we want to do here what we're going to do is basically we're
going to set it up so that if it is more than 50 percent confident then it's going to say what it thinks
it is.
If it's less than 50 percent confident it's going to say I'm not really sure what this is.
Please try again.
OK simple as that.
So to do that we're going to go ahead and type if classification.
Confidence.
Now you notice that.
Well I might have to actually just a second if classification confidence is less than 0.5.
50 percent.
And if I select this you can see that V.N. confidence is of type float.
OK.
It's a float.
It's a number that comes in.
So we have 50 percent here 0.5 50 hundredths or five tenths.
If it is less than 50 percent what we're going to do is we're going to set the text label here in our
main upper bar we're going to set that to say I'm not sure what this is.
Please try again.
So go ahead and let's type self dot identification label text and that's going to be equal to.
I'm not sure what this is.
Please try again.
All right.
And you know what if we don't have if we're not really confident we shouldn't even show this confidence
label so Gordon type self-confidence label that text and we're just going to say that to be an empty
string because there's no need to show it.
If there is not much confidence that it actually works OK and if that happens as well there's not really
a need for it to continue so we're just going to call break.
So that leaves the for loop.
OK.
What we're going to do next is what happens if we do get the proper results.
Let's say that we get something it thinks it's 98 percent sure that it just saw an umbrella.
OK so else.
Right.
That means it's greater than 50 percent if the confidence is more than 50 percent.
Go ahead and type self identification label dot text and that's going to be equal to the classification
identifier the identifier is what it thinks it saw.
So in the example I showed you at the beginning of this course when you tap it takes a photo and it
shows maybe umbrella.
And we will program it later to speak out loud to say what it thinks it is but this identifier is a
string.
It returns a name of an object or an item and that's all we need to do next.
We need to go ahead and set our confidence label to show how confident it thinks it is.
So go ahead and type self confidence label that text and that's going to be equal to a string of course
and we need to type confidence.
OK.
Just like we have here with the capital letters confidence and now we're going to use string interpolation
to pass in a value that is not a string but we're going to cast it as a string so that it can be used.
So we're going to use the classification.
Confidence.
OK.
Pretty cool.
And we're going to actually multiply that by 100 because right now if the value came in at 0.5 that
might look like 0.5 percent.
Sure.
We don't want that.
So if we were to multiply this by 100 it would move the decimal place over and it would say 50 percent.
OK.
So that's what we want because 0.5 is 50 percent.
Now the issue is that it would be fifty point zero percent.
And I don't want to have to deal with the additional numbers on the outside.
So I'm actually going to cast this as an integer and then put another parentheses on the outside so
that this number is a whole number.
51 percent.
37 percent.
If you want you can leave it at thirty seven point nine eight seven six five percent sure.
I just got kind of annoyed reading like eight numbers after the decimal place so I cast it as an ant.
That's just my personal preference.
And let's see.
Confidence we have the classification and at the very end we need to add a percentage mark.
All right.
That looks pretty good.
You know what do you guys want to go try it let's go see if it worked.
OK we're going to go ahead and build and run it and it will show up here as soon as it builds and runs
it will close.
Maybe.
Yeah there it is.
All right.
Building and installing to my phone it'll pop open.
There it is.
All right.
Should we try it if it worked.
We should tap the screen.
It should analyze the photo and it should predict what it thinks it is and return a level of confidence.
Let's try it.
Ready.
Three two one.
Bing I'm not sure what this is.
Please try again.
I'm not sure what this is.
Please try again.
Interesting.
So it does not appear to know what this is.
Let's try some other other things maybe it might know what my wallet is my dustups wallet OK not sure
what this is interesting.
So what is the problem here.
Let's see we're setting the label you know what let's go ahead let's actually print out the classification
results.
Let's print out the identifier.
This way we can see everything that it's thinking through us that we know if it's.
We can get an idea of what it's thinking.
So when I tap the screen oh it says remote control and modem.
I wonder why oh ok so I think I know what the problem is.
So basically what we're doing is we're going through.
If the confidence is less than 50 percent basically we go through we say I don't know what this is.
And we break meaning we leave our for loop.
But then if it is more than 50 percent confident it goes here and it identifies this but then it loops
through again and it hits something that it's not confident on.
And it shows I'm not sure what this is so when we hit something that it is confident on more than 50
percent we should break.
All right.
OK so we should go ahead and do that.
Let's break on both so that if it's not confident it breaks if it is not if it is totally confident
on the first one it will just go right to here and that's fine.
So let's let's build and run.
Basically we were just cutting out our most confident result because we were going through some of the
non-confidence ones first.
So now that it's built let's take a look.
When I tap on the screen to look at that remote control it's ninety nine percent confident and let's
scan something that maybe it doesn't know maybe it doesn't know a Rubik's cube.
Let's try it.
OK.
I'm not sure what this is.
Please try again that's fine because I doubt it was trained to know what a Rubik's cube is.
So that's good to know.
When I scanned the remote again remote control 97 percent confident.
Very cool.
So it looks like it's working.
This is awesome.
So we have just successfully integrated machine learning.
I kid you not it's that easy to pass in images make predictions based on a trained model we just gave
our app the ability to think how incredible is that amazing work.
In the next video we're actually going to set it up so that we can toggle the flash to turn on and off.
And then we're going to actually at the very end we're going to set up a an avi speech synthesizer to
speak the results to us thinking about maybe users who might be vision impaired who might not be able
to see very well but want to classify objects they can hear what the app is seeing.
Very very cool.
So we're going to go ahead and move on to the next video.
Amazing work with this one guys we are on our way to have a full fledged machine learning app that can
analyze and identify items in photos.
Very very cool.
Awesome work guys.
We'll see in the next video.