1 00:00:05,860 --> 00:00:11,200 Have you everyone what's going on this is Caleb with slopes dot com and we're now into the good good 2 00:00:11,200 --> 00:00:11,590 stuff. 3 00:00:11,590 --> 00:00:17,650 We've set the groundwork we've laid the foundation for actually processing the data that comes in in 4 00:00:17,650 --> 00:00:22,990 this video we're going to write the function that will actually perform the request and update our labels 5 00:00:22,990 --> 00:00:29,520 with all of the information that we need to show our prediction for a particular image I'm super excited. 6 00:00:29,560 --> 00:00:31,510 So let's go ahead and write that function. 7 00:00:31,510 --> 00:00:40,000 Now we're going to go down beneath process classifications and write a function called update classifications 8 00:00:40,780 --> 00:00:42,230 for image. 9 00:00:42,520 --> 00:00:45,370 OK and that's going to be of type UI image. 10 00:00:46,030 --> 00:00:51,940 Then we're going to basically start off by updating the label to say classifying giving the user a hint 11 00:00:51,970 --> 00:00:53,350 that something is happening. 12 00:00:53,410 --> 00:00:59,220 So classification label dot text is classifying dot dot dot. 13 00:00:59,650 --> 00:01:05,770 And then after that we're basically going to perform the classification request by passing in this variable 14 00:01:06,000 --> 00:01:11,680 k which when we pass in this variable what it's going to do is it's going to then instantiate the model 15 00:01:11,680 --> 00:01:18,700 perform the request process the classifications and then return that request for us to actually process. 16 00:01:18,700 --> 00:01:25,060 So let's go ahead and create what's called a handler type flat handler and that's going to be of type 17 00:01:25,070 --> 00:01:31,720 V an image request handler and as you can see it's an object that processes one or more image analysis 18 00:01:31,720 --> 00:01:36,510 requests pertaining to a single image which is perfect for our use case. 19 00:01:36,520 --> 00:01:43,600 Now as you can see you can either pass in a CD image a CD image a CV pixel buffer or data and we're 20 00:01:43,600 --> 00:01:50,250 going to basically pass in the CIA image here with orientation K options. 21 00:01:50,240 --> 00:01:55,510 I'm not going to matter in this instance so we can get rid of options now for the CIA image we need 22 00:01:55,510 --> 00:02:00,170 to basically create an instance of CIA image and pass in our UI image. 23 00:02:00,190 --> 00:02:03,640 That's just the format that this vision framework requires. 24 00:02:03,790 --> 00:02:08,230 But we also need an orientation parameter and basically we're just going to capture the orientation 25 00:02:08,290 --> 00:02:13,330 of the original image and just say this is the way that this image should be oriented so that it knows 26 00:02:13,330 --> 00:02:18,580 how to look at the image because if we pass in a bunch of sideways pictures of a cat and then we try 27 00:02:18,580 --> 00:02:20,680 to take a normal upright picture of a cat. 28 00:02:20,680 --> 00:02:25,450 It's not going to know what to look for which is a really cool way to make sure that we're passing in 29 00:02:25,450 --> 00:02:25,900 good data. 30 00:02:25,900 --> 00:02:29,850 So go ahead and type guard let orientation. 31 00:02:29,950 --> 00:02:36,210 We're going to set that to be equal to CEG image property orientation. 32 00:02:36,580 --> 00:02:42,520 And we need to actually use the raw value here which is of Una 32 and we can get that by going into 33 00:02:42,520 --> 00:02:49,180 the UI image we pass in Taiping image orientation and then getting the raw value from that. 34 00:02:49,350 --> 00:02:50,090 Okay. 35 00:02:50,500 --> 00:02:51,670 And that's how we can get that value. 36 00:02:51,690 --> 00:02:53,380 But it's still not in. 37 00:02:53,470 --> 00:02:54,600 You went 32. 38 00:02:54,610 --> 00:03:00,430 So we're going to actually have to cast it as you went 32 and then put another parentheses there at 39 00:03:00,430 --> 00:03:02,760 the end so that it's the proper type. 40 00:03:03,220 --> 00:03:10,240 And now we're going to go ahead and do one more property inside this guard let so type Let see I image 41 00:03:10,740 --> 00:03:12,940 and that's going to be equal to see-I image. 42 00:03:13,210 --> 00:03:17,710 And as you can see there is an option to pass in a UI image here. 43 00:03:17,800 --> 00:03:18,830 Pass that in. 44 00:03:19,240 --> 00:03:23,210 And now that we have both of those We're going to go ahead and type else. 45 00:03:23,350 --> 00:03:30,340 And if that does not work we are going to basically need to display an error of some kind and we'll 46 00:03:30,340 --> 00:03:32,710 come back to that in just a second. 47 00:03:32,710 --> 00:03:37,210 At the end of that if that doesn't work basically we're just going to say return and just get out of 48 00:03:37,210 --> 00:03:41,100 this guard let's so that we don't continue to move on and make mistakes. 49 00:03:41,170 --> 00:03:48,060 So we now have an orientation property and a CIA image property so we can pass both of those in of course 50 00:03:48,070 --> 00:03:52,760 ensure that you're passing in the right CIA image and the right orientation. 51 00:03:52,990 --> 00:04:01,030 So at this point we now can try to perform our classification request and return back those pieces of 52 00:04:01,030 --> 00:04:04,150 information that we get from our core a male model. 53 00:04:04,360 --> 00:04:07,650 So to do that we're going to use a do catch block. 54 00:04:07,730 --> 00:04:08,040 OK. 55 00:04:08,050 --> 00:04:09,380 So type do. 56 00:04:09,580 --> 00:04:13,570 Then make sure to put cash down below that's where we're going to catch errors. 57 00:04:13,790 --> 00:04:20,140 And basically in this do block we're going to try handler not perform. 58 00:04:20,230 --> 00:04:26,410 OK and you can see here that we can pass in some requests and it expects an array so we're going to 59 00:04:26,410 --> 00:04:29,240 pass a single item into that array. 60 00:04:29,320 --> 00:04:33,890 And what that's going to actually be is our classification request. 61 00:04:33,910 --> 00:04:36,490 We're going to pass that in right here. 62 00:04:36,890 --> 00:04:37,540 OK. 63 00:04:38,170 --> 00:04:44,280 And what the perform function is going to do is it's going to schedule our vision requests to be performed 64 00:04:44,290 --> 00:04:47,500 and of course when we pass it in we'll create all of this. 65 00:04:47,500 --> 00:04:53,050 It will process all of the requests create these properties and then actually present the label the 66 00:04:53,050 --> 00:04:54,440 way it should be. 67 00:04:54,490 --> 00:04:59,150 Once we actually perform that request which is pretty cool and we also have some good air handling. 68 00:04:59,440 --> 00:05:04,000 So in the instance that this doesn't work we're just going to print an error and basically just say 69 00:05:04,000 --> 00:05:10,130 fails to form a form classification. 70 00:05:10,960 --> 00:05:11,800 OK. 71 00:05:11,810 --> 00:05:19,150 And then we can actually do the line breaks and tax and after that we can pass in the error the errors 72 00:05:19,160 --> 00:05:22,130 localize description just like that. 73 00:05:22,130 --> 00:05:28,310 So at this point we are now ready to try it out but I'm just going to tell you that there are several 74 00:05:28,310 --> 00:05:32,770 things that we need to actually perform on some different threads. 75 00:05:32,780 --> 00:05:40,760 And the reason for that is because this classification request eventually we'll process the classifications 76 00:05:41,000 --> 00:05:45,430 which eventually actually updates the UI. 77 00:05:45,620 --> 00:05:48,940 Anything that updates the UI needs to happen on the main thread. 78 00:05:48,950 --> 00:05:51,950 So we're going to make sure that that happens by typing dispatch. 79 00:05:51,950 --> 00:05:54,880 Q global. 80 00:05:56,060 --> 00:06:01,720 And we're going to use this little handy dandy feature here called Q OS and that means quality of service. 81 00:06:02,000 --> 00:06:06,500 And in this particular instance we're going to use user interactive. 82 00:06:06,590 --> 00:06:07,230 OK. 83 00:06:07,550 --> 00:06:14,300 And essentially what this is is for when things like the UI are being updated it's going to handle these 84 00:06:14,300 --> 00:06:20,440 requests so that the user interactive elements are as usable as possible and they move nice and smoothly. 85 00:06:20,600 --> 00:06:23,620 So we don't have any type of UI bugs or hiccups. 86 00:06:23,840 --> 00:06:29,090 We're going to run this code asynchronously and we're actually going to move this entire block into 87 00:06:29,090 --> 00:06:31,450 the dispatch queue block there. 88 00:06:31,820 --> 00:06:38,990 So at this point we are now ready to actually try our classification request however now that it is 89 00:06:38,990 --> 00:06:43,640 inside of a closure we need to make a reference to self just like so. 90 00:06:44,060 --> 00:06:48,780 And I think we are nearly there. 91 00:06:49,100 --> 00:06:56,870 However we need to write our function to display errors OK because right now we're not really doing 92 00:06:56,870 --> 00:06:59,210 a great job at displaying particular errors. 93 00:06:59,210 --> 00:07:04,810 We are here we are in other places but I'm just going to paste a function in here called display error 94 00:07:05,300 --> 00:07:09,130 and I'm just going actually call that and this will be more for like. 95 00:07:09,170 --> 00:07:13,970 If you don't have a specific error to present but you need to just say hey something went wrong please 96 00:07:13,970 --> 00:07:15,660 try again. 97 00:07:15,830 --> 00:07:17,810 That would be a really good way to do it. 98 00:07:17,870 --> 00:07:20,020 So that is the display function. 99 00:07:20,030 --> 00:07:21,370 Super simple. 100 00:07:21,410 --> 00:07:26,950 Now we need to oh you know what. 101 00:07:26,950 --> 00:07:27,270 Sorry. 102 00:07:27,280 --> 00:07:30,200 That needs to be a classification label all spelled out. 103 00:07:30,610 --> 00:07:31,070 Very cool. 104 00:07:31,070 --> 00:07:36,450 So now the only thing we need to do is to call our update classifications function. 105 00:07:36,460 --> 00:07:42,790 So go ahead and call that in our function once we have picked an image and we can just pass in the image 106 00:07:42,790 --> 00:07:43,660 right here. 107 00:07:43,660 --> 00:07:45,010 Pretty cool stuff. 108 00:07:45,010 --> 00:07:51,640 And at this point we are now ready to build and run send over our new code and test out our classifier 109 00:07:51,640 --> 00:07:53,920 to see how it works. 110 00:07:53,920 --> 00:07:56,000 So press the camera button to choose an image. 111 00:07:56,230 --> 00:07:58,060 Boom here we go. 112 00:07:58,210 --> 00:07:59,170 Camera Roll. 113 00:07:59,170 --> 00:08:01,170 Choose that classifying. 114 00:08:01,170 --> 00:08:03,010 Good good. 115 00:08:03,340 --> 00:08:06,280 And we get a crash interesting. 116 00:08:06,280 --> 00:08:07,310 Check this out. 117 00:08:07,410 --> 00:08:13,300 I label dot text must be used from main thread only. 118 00:08:13,360 --> 00:08:14,770 Very interesting isn't it. 119 00:08:14,780 --> 00:08:18,290 So that means we need to also pull this code. 120 00:08:18,400 --> 00:08:23,950 This entire classification code we need to pull this out onto the main thread so to do that we're going 121 00:08:23,950 --> 00:08:24,740 to type dispatch. 122 00:08:24,760 --> 00:08:32,920 Q Dot main dot async and we're going to pull all this good stuff and run it on a main thread. 123 00:08:33,040 --> 00:08:33,570 OK. 124 00:08:33,880 --> 00:08:40,450 So let's go ahead and close that let's build and run it one more time and we should be able to see that 125 00:08:40,450 --> 00:08:42,420 it works and it updates the label. 126 00:08:42,430 --> 00:08:46,060 Thanks to putting it on the main thread so select it. 127 00:08:46,060 --> 00:08:53,560 Boom boom and let's pick that image again classifying and for some reason what it sees here it thinks 128 00:08:53,560 --> 00:08:55,310 about birds and tigers. 129 00:08:55,320 --> 00:09:00,640 Now we could do some work to update this to where if it's not totally sure. 130 00:09:00,820 --> 00:09:06,960 It would say hey you know I don't know what that is but we only showed 10 images in our training. 131 00:09:06,970 --> 00:09:09,100 So it's not super super accurate. 132 00:09:09,100 --> 00:09:14,940 However it is accurate enough to note key differences between different animals. 133 00:09:15,040 --> 00:09:19,870 Now I don't know about you but I think we should go get some actual images of you know maybe a bird 134 00:09:19,870 --> 00:09:20,780 maybe a cat. 135 00:09:20,800 --> 00:09:23,740 So let's go ahead let's head to Google. 136 00:09:23,770 --> 00:09:26,080 Or better you know what let's go to pectorals dot com. 137 00:09:26,080 --> 00:09:31,150 They have free royalty stock images and we can just say how about cats. 138 00:09:31,180 --> 00:09:32,970 Let's download an image of a cat. 139 00:09:33,250 --> 00:09:40,360 I know that it did not see this image so let's go ahead and let's download this image and I'll hold 140 00:09:40,360 --> 00:09:42,490 down on this and we'll save it. 141 00:09:42,520 --> 00:09:48,220 So now we should be able to go into our classifier here click on our image and let's see how it does 142 00:09:48,750 --> 00:09:50,020 100 percent cat. 143 00:09:50,020 --> 00:09:50,770 That's awesome. 144 00:09:50,770 --> 00:09:52,900 So it knows for sure that it's a cat. 145 00:09:52,930 --> 00:09:54,880 Let's go find one. 146 00:09:54,910 --> 00:09:56,570 How about a lizard. 147 00:09:59,620 --> 00:10:03,490 Lizard let's find a cool image of a lizard that the model hasn't seen. 148 00:10:03,490 --> 00:10:08,390 I'm going to scroll down a bit because I downloaded quite a bit of these. 149 00:10:09,490 --> 00:10:10,340 How about this guy. 150 00:10:10,370 --> 00:10:14,670 Let's see if it understands that this is in fact a lizard. 151 00:10:15,080 --> 00:10:19,510 And here we go so let's save this to our phone boom. 152 00:10:19,730 --> 00:10:26,360 Let's go ahead and click the photo and choose our lizard classifying 100 percent lizard. 153 00:10:26,360 --> 00:10:27,410 That's awesome. 154 00:10:27,410 --> 00:10:31,370 So it looks like it's really accurate for the animals that we've trained it to be accurate on. 155 00:10:31,760 --> 00:10:32,900 Let's go find. 156 00:10:33,230 --> 00:10:35,450 Let's do a bird and let's find a bird. 157 00:10:35,450 --> 00:10:38,430 Here is a bird that I did not show the model. 158 00:10:38,750 --> 00:10:39,730 And let's save it. 159 00:10:41,360 --> 00:10:44,230 And let's see how we do animal classifier. 160 00:10:44,270 --> 00:10:48,370 Let's choose our bird here 76 percent bird. 161 00:10:48,380 --> 00:10:54,830 So it's not 100 percent accurate but it knows beyond the shadow of a doubt that this is in fact a bird 162 00:10:54,830 --> 00:10:56,850 which is super super cool. 163 00:10:56,990 --> 00:10:59,660 So amazing so it looks like our app works. 164 00:10:59,690 --> 00:11:04,580 I'm going to go ahead and build this to my iPhone really quickly to verify that the camera also works 165 00:11:04,990 --> 00:11:06,060 and I will. 166 00:11:06,110 --> 00:11:07,580 We'll be back in just a second. 167 00:11:07,580 --> 00:11:08,960 Here we go. 168 00:11:08,960 --> 00:11:09,560 All right. 169 00:11:09,560 --> 00:11:15,860 So I have the app here running on my iPhone and if I tap on the camera button you can see since I do 170 00:11:15,860 --> 00:11:19,880 have a camera I have the option to choose take photo or choose photo. 171 00:11:19,880 --> 00:11:23,240 Let's tap take photo and we get a crash. 172 00:11:23,240 --> 00:11:24,720 So why would that be. 173 00:11:24,830 --> 00:11:26,900 Let's read the error we. 174 00:11:27,000 --> 00:11:29,190 Oh we can't see the error but I'll tell you what it is. 175 00:11:29,270 --> 00:11:31,100 We did not yet tell it. 176 00:11:31,100 --> 00:11:37,100 We didn't yet give it the proper usage description so we're going to go ahead and quickly add the privacy 177 00:11:37,760 --> 00:11:46,770 camera usage description and we're going to just say we need your camera to take cute pictures of animals. 178 00:11:46,880 --> 00:11:48,870 Make sure you spell it right. 179 00:11:49,130 --> 00:11:50,020 OK. 180 00:11:50,360 --> 00:11:51,830 Make sure you spell it right. 181 00:11:51,830 --> 00:11:52,610 There we go. 182 00:11:52,850 --> 00:12:00,710 And now if we build it and run it to my iPhone we can see that we have the app running right here on 183 00:12:00,710 --> 00:12:01,250 our phone. 184 00:12:01,250 --> 00:12:08,480 It's given me an error thinks X code beta but if I tap on the photo image tap take photo it will ask 185 00:12:08,480 --> 00:12:13,610 for permission to use the camera and it will bring up this usage description that we already had here 186 00:12:13,610 --> 00:12:18,090 but it appears that this is working very very cool stuff. 187 00:12:18,140 --> 00:12:24,860 We can take a photo and we can use that photo and it will do its best to classify the image based on 188 00:12:24,860 --> 00:12:26,470 what it thinks it is. 189 00:12:26,540 --> 00:12:32,660 Or alternatively you can choose a photo like I did and it will make its own classification. 190 00:12:32,660 --> 00:12:34,510 So fantastic work. 191 00:12:34,520 --> 00:12:35,450 Congratulations. 192 00:12:35,450 --> 00:12:41,390 We have just successfully built our very own Corum lap using a model that we trained. 193 00:12:41,390 --> 00:12:46,520 Imagine if you owned maybe like a machine shop for cars and there were tons of different parts you could 194 00:12:46,520 --> 00:12:52,650 take a bunch of photos of your various car parts identify them and label them and then a new employee 195 00:12:52,650 --> 00:12:57,260 you could take your app and scan someone's part that they brought into the shop to get fixed they would 196 00:12:57,260 --> 00:13:00,020 know what it is immediately they could pull up information on it. 197 00:13:00,020 --> 00:13:03,890 There are so many use cases for this fantastic work. 198 00:13:03,890 --> 00:13:05,180 Very cool stuff. 199 00:13:05,180 --> 00:13:08,150 Thank you so much for watching this section of the Iowa 12 course.