You guys this is Caleb with Dev slopes and in this video we're going to set up an instance of a speech
synthesizer to speak what we see in our images.
It's going to be really cool.
And when I initially thought of this idea I thought of it as a way to be kind of like an assistive technology
perhaps to those who are vision impaired.
If they if they can't see what is in front of them they could use our app to scan things and it could
speak it to them so they could hear what they're seeing so kind of like they can see with their ears
supercool.
And actually to use Avie speech synthesizer is not very difficult.
So I just want to show you how to use it and how I use that in this app so pull open that X code project
and let's go ahead and let's begin by actually creating an instance of a speech synthesizer.
So go ahead and type var speech synthesizer.
And that's going to be of type Avie speech synthesizer can instantiate it.
And that's that.
Now if you look deeply inside of this you'll notice that this comes from A-V foundation.
So we've already integrated A-V foundation to capture our photo.
But the part of A-V stands for audio and that's where our Avies speech synthesizer will come into play.
OK now what we do need to do for this to work is we need to set the speech synthesisers delegate to
be this view controller.
So go ahead and type speech synthesizer dot delegate and that equals self.
Now if we try to build and run this there is an error cannot assign value of type camera AVC to A-V
speech synthesizer delegate.
Now to fix that we're going to create an extension of A-V speech synthesizer delegate.
That's going to give us the ability to call some important delegate methods that we need.
So go ahead and type extension camera AVC.
And we're going to do Avies speech synthesizer delegate.
OK.
And the function we're going to call is called did finish utterance and we'll talk about that in a second.
So just go ahead and add that function and code to finish utterance We'll come back to that in a second.
But for now if we go back up to view.
Whereas that view did appear you can see that air went away because now we are conforming to speech
synthesizer delegate.
That's all we need.
So in order to actually speak something we need to be able to create a speech utterance to get an AB
speech utterance to actually speak.
We need to pass it a string.
So we're going to create a function that basically allows us to pass a string into an avi speech utterance
and then use our synthesizer to speak it.
So go ahead and type phunk synthesize speech speech from string and we're gonna use an internal parameter
they're called string.
Of course of type string.
And basically what we're going to do is we'll create an instance of Avies speech utterance we'll pass
it the string and then we will ask our synthesizer to speak the utterance.
So go ahead and type let speech utterance equals Avies speech utterance.
And we're going to pass a string K and that's what it's going to speak.
Now of course this is the string we're passing in.
So go ahead and type string and it can be that lowercase string from the internal parameter.
OK so then now that we have that utterance we can call our synthesizer K speech synthesizer speak.
And you'll notice it asks for an example of A-V speech utterance and we have that speech whoops speech
utterance.
OK that's what we need to pass it.
Now this function for any string we pass in it will speak it k.
That's the cool thing of Avies speech synthesizer any string we pass in it's going to speak for us.
So we need to determine when we should speak this and that is basically when we classify our results.
So when we pull through all of our classifications if it comes to something that it has no idea what
it is if it's not sure we should speak that.
OK.
Now what we're going to do is we're actually going to take this string and I'm going to go ahead and
copy it and delete it.
And we're going to call this let unknown object message.
And that's what we're going to use here.
Unknown object message.
We're going to also use that in our synthesise speech function.
So go ahead and call a synthesized speech from string and we're going to give it our unknown object
message.
Now whenever we find something that we don't know what it is it's going to say I'm not sure what this
is.
Please try again.
Very cool but we need to also create a string that's going to classify our object and give a level of
confidence.
So to do that I'm going to go ahead and I'm going to actually create some variables for this as well.
So I'm going to cut this and just create var identification and let's go ahead and do var confidence.
And we're going to actually take our confidence value here as well and let's go ahead and cut that out.
Let's put it here.
Cut that out.
Confidence.
All right cool.
So we're going to pass in the confidence that we just created.
We're going to give this the identification as a string.
And so that looks good.
OK let's go ahead and make sure those little warnings go away.
Oh OK.
They were never mutated.
Let's just create them in two constants.
Good call X code.
Always looking out for me.
Very cool.
OK so that's good.
But now we need to speak what it is that we saw so let's call synthesized speech and this string what
we're going to do is we're going to compile both of these into a nice long sentence that makes sense
that feels like natural language.
So let's call this complete sentence.
And we're going to basically say this looks like a.
And we're going to pass in the identifier or sorry the identification.
This looks like a fish.
This looks like a remote.
And I'm
and here's where we pass in our confidence.
OK.
I'm blank per cent sure.
So for an example this could be this looks like a remote control and I'm 98 percent sure.
Pretty cool.
Now we're going to synthesize that complete sentence so that it speaks that outloud and.
Yeah.
So this should work.
This should synthesize our speech.
If we go inside Avies speech synthesizer if you look inside of it you can see it has a delegate it has
all kinds of booleans to check to see whether or not it's speaking if it's paused it has some functions
to speak.
And I should have shown you that earlier but I didn't.
You can look inside here and you can see all of the cool things that A-V speech synthesizer and Avies
speech utterance can do.
You can even change the voice which is kind of cool.
There's a system default but you could turn it into a different voice.
And we can play around with that later but for now let's go ahead and build and run this let's see if
it works.
OK let's pull open quick time here.
Time using two is what I'm using to record my screen.
If you don't know you can go into quick time and you can actually choose your iPhone as an input camera
which is very cool.
So anyway let's go ahead and let's tap on the screen K so it looks like it's showing the day looks like
it's showing everything but it doesn't appear that it's speaking let's see oh you know what I bet.
Wait let me try to turn up.
Whoa.
Cause that's that's broadcasting my microphone.
Let me try to set the microphone of this recording to be my iPhone.
And then when I take a picture we should be able to hear it now.
Did you hear that.
That's what we're that's what we just said let's do it again.
All right.
And did you guys hear the camera shutter sound as well that's in there we just couldn't hear it because
I didn't have the sound patched through correctly.
All right let's let's try with something that he doesn't know.
It's so cool.
Let's see maybe I can try it with.
Let's try it with some sunglasses.
Here we go.
Let's see.
I'm not sure why this is.
Please try again.
See
oh oh see.
And one thing to remember guys is this is not an exact science.
It's not absolutely perfect because it's not going to be able to identify every object it's only going
to be able to go off of what it knows and what it understands.
But sometimes you can see what why it made the association it did.
It said this was a jeweler's loop.
And if you think about it a jeweler's loop is a circular device with a circular piece of glass and a
kind of a lens looking thing so it makes sense why it thinks certain things at time.
Let's try this again.
All right cool.
So as you can see it can identify things it's not perfect.
It has its issues but a lot of things that can identify properly which is just so cool.
We have an app that can identify it can speak to us it can do all these amazing things.
All right.
So really cool this this works we can tap it
and it speaks to us which is very cool.
But the issue is right now I can tap on the screen 10 times and it's going to speak 10 times of what
I saw and when I took photos of.
So I think that we should add a little activity spinner on this image to show that it's loading and
it's speaking.
And we should basically disable the tap gesture recognizer while it's speaking.
OK.
It's really easy to set this up.
But first we need to go ahead and add a UI activity indicator.
So go ahead and type UI activity indicator view and of course we're in mind that storyboard we're going
to drag that on and center it right there on our image view.
We're going to set it to be white and we're going to pin it in the exact center of this container which
of course.
Oh you know what.
No that is not what we want.
We want to select the image view then the activity indicator and we want to give them the same horizontal
and vertical center.
OK.
Just like that.
And you know what we're going to actually set this to be hidden from the get go.
So you can actually just do that right in Interface Builder.
We don't need to do that in code or anything but what we do need to do is we need to add an IB action
or rather an outlet so that we can modify this we can show it we can animate it et cetera et cetera.
So in Interface Builder go ahead and open up the assistant editor and if it's not already there go ahead
and click automatic and it should load the proper view controller and that looks good.
When to close those get out of my way.
Perfect.
OK.
So go ahead and right click.
Right click from the activity indicator and go ahead and drag over to your other outlets which are down
here for me and I'm just going to call this the spinner because that's basically what it is is a spinner.
Now go ahead and go back to your code file and just make sure we can get.
We have our spinner.
That's awesome.
And now what we need to do is we need to basically change so that whenever we tap the camera view it's
going to go ahead and show the spinner.
It's going to go ahead and pause out the tap gesture recognizer so that input cannot be sent in anymore.
And then it's basically going to hide and re-enable the tap gesture recognizer as soon as it's done.
And know what we also added this function did finish utterance.
That's where we can re-enable everything which is perfect.
So go ahead and actually up here at the top when we call did tap camera view here's what we're going
to do.
We're going to call self camera view is user interaction and enable set to false because when we tap
it we should disable it from showing anything really.
That is what we're going to do.
So go ahead and type that and yeah we just disable the camera view from being tapped while it is capturing
the photo sending it to our model et cetera et cetera.
So we just set it up so that the user interaction is disabled.
And now we're going to also unhide our spinner and we're going to started animating because when we
tap on the camera view that's basically when everything starts floating.
So go ahead and type self whip's self-taught spinner is hidden because remember we hit it in the interface
builder that's going to be equal to false and then we're going to go ahead and type self that spinner
don't start animating.
That's right because that's going to start it actually doing its spin animation.
OK so cool so our camera views disabled our spinner is animating.
Now we need to think about when we want to re-enable the camera view and hide the spinner.
And that is when it's done speaking.
OK.
So we already added this delegate method did finish utterance and that is used to determine when the
speech synthesizer is finished talking.
So that's super easy.
All we need to do is basically do the opposite of what we just did.
So self camera view is user interaction enabled is true.
Now it can be tapped again.
Let's go ahead and call self-caused spinnerbait is hidden equals true.
OK.
That's very cool.
That's all we need to do is to hide our spinner.
Oh you know what we should also stop it from animating
OK.
So now when it's done speaking it's going to go ahead and re-enable the camera view.
It's going to hide the spinner and stop it from animating.
Let's build and run and let's double check to make sure that works.
This is just the way that we prevent the app from having too much input which could cause crashes it
could cause memory issues and there's really no need to scan 100 things like right at once.
So let's let's tap the screen and you'll hear the camera shutters sound and when I tap on the screen
you shouldn't hear it again until it's done.
We should also see the spin or show up although I see right now that it's not actually hidden.
So we're going to need to fix that.
But let's try it.
Can tapping the screen.
Nothing's happening.
OK it's done it head when I tap it again.
Ok cool.
Very cool.
So that works.
But let's go let's go determine why it's not hidden from the get go.
Maybe I put the wrong thing in the camera view is hidden.
Oh did tap camera view is hidden is supposed to.
Oh wait no it's it's supposed to show when I tap the camera view.
Is hidden it's true.
And let's go to the main story board I did hide that right.
OK yeah it says it's hidden.
Well I guess you know what let's go let's go hide it in code.
Sometimes you can't rely on interface builder so let's go ahead and type spinner that is hidden.
And let's set that to true.
Let's build it again.
Let's make sure that the spinner is actually hidden when we first load the app there's no use in having
a spinner doing nothing.
It's kind of confusing actually.
OK good.
So it's gone.
Let's go ahead and tap the screen.
Let's tap tap tap tap tap tap tap nothing.
Cool.
So when it's done I was able to tap into it again.
OK.
Pretty confident yourself there Siri.
Anyway awesome so we just added some code to make sure that too much input is not getting in to our
app.
But guys this is incredible.
Our app is done we've just built an app using machine learning that uses a pre-trained model to identify
objects.
Show what they are on the screen and speak to us what they are it's pretty amazing if you think about
the technology that this is utilizing.
Thank you guys so much for going through this section of the course.
I hope this is what your appetite for core M-L and all the things it can do.
There are so many ways you could extend this app to make it more amazing.
You could use this as a way to tag and classify photos to make different albums you could use this as
a way to catalog some of your stuff.
I have a remote control I have some sunglasses I have this thing you can use it to catalog your stuff.
There are lots of different ways that you could extend this app to make it more amazing.
Now this is just one core M-L model squeezed net.
The one that we're using.
There are other ones.
There's one a one from one called Places 2 0 5 Google net I think and it's utilized to identify scenes
and places to tell you what kind of a senior at are you in a city or you at the beach or you in the
forest.
That's just another cool type of Corum that you can use the most amazing thing is all of this is happening
on your device.
Locally we don't even need to reach out to an API or a server.
So so cool.
Thank you guys so much great work.
And let's head over to the next section of this course.