Let's not learn about character encoding and it's an absolutely fundamental lesson before we do any 2
programming that involves processing of text it is important to know about character encoding in case 3
you're not familiar with this topic then it is painful learning it now. 4
So let's get started with character encoding. 5
Let's start with absolute basics. 6
Most people classify files into bucket. 7
It is binary and text binary would include content like images audio and video right text is about files 8
with characters. 9
However fundamentally all files are binary that a sequence of bytes where each byte is a group of eight 10
bits on the bed is either 0 or 1. 11
So text files are also binary files. 12
It is just that they're stored in a certain way in binary all files whether they are binary or text 13
would look alike to computers hardware hollower software would make a distinction for instance bytes 14
are present in text are only handled by text processing software like Notepad word pide or editors like 15
vi and Eclipse. 16
Similarly bytes representing images are only handled by image processing software like Windows photo 17
or windows paint application and computers use hexadecimal numbers to represent bytes for example characters 18
is represented using the hexadecimal Korsak 61 which is stored in binary as the byte 0 1 1 0 0 0 0 1 19
6 corresponds to the first four bits. 20
Well one for the last four bits not the same bit pattern would represent something else. 21
In an image file. 22
Also note that a single byte is represented in one of the two of the six big patterns that is stewpot 23
8 as a byte has a bits. 24
So it means we can represent as many as 256 characters using different variations of a byte but to present 25
more characters we need to use more than one byte. 26
And that's with a different encoding schemes coming to agree and we will see all of that in about now 27
since the title of the lesson is character encoding. 28
We will only look at text representations unprocessed in text is very complex due to the number of languages 29
involved and also due to the number of ways in which characters in those languages can be represented 30
Nemeroff is would be a different character encoding schemes so a given text follows some encoding scheme 31
under the software like say a browser does not understand the character encoding used. 32
Then you would end up seeing familiar with characters such as this. 33
This happens usually for international characters. 34
And so every file uses some encoding scheme to represent its content and then encoding scheme simply 35
maps every character to a unique x or a similar Amber whose binary representation is just some encoding 36
scheme is basically an algorithm which maps Karakas to hexadecimal numbers and whose binary representation 37
are used. 38
And here's an example of encoding and decoding with regard to encoding character is first and coded 39
or map to hexadecimal most 61. 40
And finally the binary equal end of 61 is used for the actual storage. 41
The recording process is simply the reversal and coding process and here are some example encoding schemes 42
on each encoding scheme is basically an implementation of some characters. 43
For instance you be afraid you give 16 and you give 32 or all implementations of Unicode characters 44
it dumps encoding scheme and characters that are frequently intermixed but to speak precisely they are 45
different with regards to ASCII. 46
It can be viewed as both a character set encoding scheme initially only characters that mattered who 47
are unaccented English letters and for that 7 bit ASCII codes were used. 48
Here is an example in ASCII hex cards. 49
What we want to fight we are used to the present uppercase alphabet agency. 50
But since we also how many other languages soon other 8 bit extensions of ASCII came up. 51
I also added five line is a standard ascii based extension. 52
In fact I also did 5:9 had 15 different variations within different regions in the world. 53
In these extensions the fust one bungee cords were all less identical and would correspond to the 7 54
bit US-ASCII the remaining 128 CTS is where we see the differences based on the region. 55
No Asian languages have thousands of letters on 8 bits wasn't sufficient for them. 56
So this operator do us another encoding scheme called double byte characteristics are DDC as in short 57
Sagitta all these variations when documents were mailed from one country to another then they are not 58
getting decoded properly. 59
For instance a hexadecimal code of a particular character in one country would correspond to a completely 60
different character in some other country. 61
Here we can see an example of that the word or as you may in one country get to despair differently 62
in a different country in this because the second country is Israel. 63
Here Hicks court for Oxenford e corresponds to the Hebrew character Jamail on Israeli Beezus. 64
So there were decoding issues and the advent of internet simply made it worse and it became a huge problem 65
on to this Unicode characters that was invented which aims to cover all languages in the world. 66
Let's look at some quick facts about Unicode encoding. 67
It is Mandarin by Unicode Consortium and it is backward compatible with Southern US-ASCII. 68
So one can be it characters are seen no initial assumption was that 16 bits which would represent 65000 69
536 characters would suffice to cover all languages and these characters together as a group are referred 70
to as basic multi-lingual bin or VMP foreshock as a result UCSD which was a fixed length and going skiing 71
with 16 bits was created when Mr. Fix lent it means that it used exactly 16 bits to represent any character 72
. 73
But soon it was realized that there were more characters. 74
These characters were especially important for Asian markets. 75
These would also include things like Smiley graphics symbols which we see in various social networking 76
applications as a result other encoding schemes like usurious for a new age of 16 were created to accommodate 77
new characters but useless for was not a word as it required 4 bytes for every character. 78
And so it would be a lot of this based on memory. 79
According to Wikipedia as of today Unicode covers are on one conduct characters from on 129 scripts 80
. 81
So Unicode is really universal. 82
Now as we just discussed Unicode characters that has two parts. 83
One is BMB that is the basic multi-lingual plane. 84
Well second is referred to as supplementary characters and let's just call it as non-being. 85
So BNP how. 86
Sixty five thousand five thirty six characters why while B and B probably have it on the five key characters 87
. 88
That is as of today and we may expect more characters to be added to non-being be said and the hexadecimal 89
code that represents each character is referred to as code point on to represent a single BNP character 90
. 91
One 16 bit code point is used. 92
So that's an exciting animal called hoverboard for each supplement we got after 16 bit code points are 93
used. 94
Now this illustration also shows a range of code points for both B and B as well as non BMV. 95
And here's an example of a BMB code point for the Euro symbol. 96
You place here means Unicode. 97
I'm going DHC is the actual court which is in hexadecimal. 98
Of 16 and udy of aid are popular implementations of Unicode. 99
So let's discuss each of them briefly. 100
Let's begin with UGF 16. 101
It is a variable length encoding scheme to represent BNP characters. 102
It needs two bytes one for non BNP characters it is four bytes for the course and B and B it has a one 103
to one correspondence. 104
That is it uses the same hex scores as the code points in B and B. 105
And this applies to UGF 32 format also no FERNANDE and because UTF 16 uses unicode values put up as 106
an that you could once just we call that non BMB has to code points for each character. 107
Each Unicode value has two parts. 108
Together these two Unicode values are referred to as SoBo get peace with the Unicode value is called 109
as high Celgard. 110
Well the second one as Losar target for example here has an emoji symbol which represents the source 111
of joy and you might have seen it in your walks up keyboard or in other kinds of social networking as 112
as you can see doing it as a single character. 113
It's Unicode point is one F-16 or two and it is represented by that subtle get you know Java C-Sharp 114
on Python are some languages that internally use uty of 16 characters that is internally within the 115
memory. 116
And language like Java or visit or present a single character in Yogya of 16 that is even if the character 117
was decoded from a file that was using a different encoding but when it comes to storing in memory it 118
gets stored in uty of 16 format. 119
Not come into UTF 8 it is also available encoding scheme like Yugi of 16 and it gives the first 128 120
characters that are present the ASCII characters can be represented using a single bite. 121
And so it is backwards compatible with ASCII for all other characters. 122
It would require anywhere between Google for bytes and for non VMP characters. 123
It would be 4 bytes for sure. 124
So as it needs only one byte for plain-English got a curse it is definitely free for them Yugi of 16 125
on the other hand needed to bytes even for plain ASCII characters. 126
So there is some vestige of space when it comes to plain-English characters UTF 8 is also the most popular 127
encoding scheme on more than half the pages are encoded in UK. 128
And it was invented by Ken Thompson who also designed and implemented the Unix operating system and 129
has also invented the beat programming language which is a direct predecessor to the C programming language 130
. 131
Just know that you'll be a stands for Unicode transmission format and so it is basically an algorithmic 132
mapping from every Unicode could point to a unique byte sequence in few minutes. 133
We will also do a demo when we see how the same code point gets encoded differently using different 134
and coding schemes. 135
Norman unicorn's games like U-T of 16 cents the basic unit is two bytes they can be who is in which 136
those bytes can be stored in memory or even on disk one is quite as big endian or before shot while 137
the other as low endian or Eliphaz shock in big endian format. 138
Most significant Byard will be stored at last memory address. 139
And this is also most commonly used for Mac under default one do while in low endian most significant 140
byte would be stored at highest memory address and there is something called byte order maake or B-Boy 141
for. 142
Which is used to identify the endianness that is big endian on Indian or that it uses a special Unicode 143
character for big endian the Unicode character is F E F F will follow. 144
In the end it is all positive it is the reverse F F F G and these byte order marks are placed at the 145
beginning of a data stream and that or tell if their data stream uses big endian or low end the end 146
format. 147
And here is an example. 148
The first line shows the data stream for the string Hello second line shows the same data stream Magners 149
big endian while the next one is marked as low endian as you can see the order of MSBA is different 150
. 151
So when you add code or the code you can specify the endianness. 152
For example you can say you have 16 be on you get 16. 153
LG No one final important thing to note is that you should always decoded data stream using the same 154
encoding scheme that was used to and coded all it has to be compatible encoding scheme. 155
For instance if data stream was encoded using ASCII then you can decode it using you defeat as you'd 156
be if ADAs backward compatible with ASCII. 157
So that's about it. 158
Buteyko the resources section for some great pointers on this topic along with some additional notes 159
. 160
Altadis mentioned earlier let's not do a quick game off and coding. 161
OK in this demo we are going to end quote three different characters using different encoding schemes 162
and the characters are the character the symbol the character Eudo And finally an emoji symbol which 163
is for B-S of joy in this case. 164
So the first book out occurs in Euro RBM characters. 165
Well the last one is a non B and B got a letter and to have this you wrote in your editor in your example 166
in this because it's eclipse. 167
For that you need to specify this particular property you should make sure that it is UTF 8. 168
Only then you will be able to insert it. 169
So by default inequitably it's going to be C.P. 12:52 on Windows machine. 170
So you need to make sure that it isn't UGF and corny. 171
And so in order to apply the encoding we are using this method called print and coding the deaths on 172
all of all of this and most are are being passed as input to this method. 173
So let's look at this method. 174
So the input is a string. 175
And here we are invoking this method called Get bytes which has a single parameter which accepts the 176
name of the encoding scheme. 177
So we are encoding the input symbol with all these different encoding schemes like ASCII Aycock for 178
isolated 5:09 you are newbie of encoding schemes and it returns it. 179
And of course it underlines the byte sequence. 180
OK an R is dot string is basically used to concatenate them those byte sequences so that we can print 181
it. 182
So we are applying all these different encoding schemes that you'd be afraid you give 16 UTF 16 beat 183
. 184
So it is specific to begin here. 185
And this is low endian. 186
And here there is no Indianness. 187
So it will in Gordon bomb symbol be or m right order mark. 188
So let's just go ahead and run it and let's look at the notebook for all three of them. 189
So vastus got a curry and in ASCII it is represented using the numeric symbol 97 which has hexadecimal 190
code. 191
But it's displaying on either the byte as the byte for that which is 97 I assume also has the same Euboea 192
8 also has 97 because you'll be a four ASCII Euboea eight is backward compatible. 193
And it uses only one byte. 194
Now you do have 16 users who bytes. 195
Know that. 196
But in this case it is also including the bytes far byte order Mach 2 and to indicate the M.B.A.. 197
And in this case it is big endian because you can also see it here it is zero 6:51 and the it is 97 198
0. 199
Just the reverse. 200
So the idea of 60 needs 16 bits and half or euros Inbal ask he said 63. 201
Anders Neumann equal to 63 corresponds to the character question mark which means that it is unable 202
to uncoded. 203
And same goes with us. 204
We had as be of all you'd be able to do and quoted can be of eight needs three bytes whereas Yukiya 205
of 16 needs to base. 206
You can ignore these two bytes here and you'll be 16. 207
B needs to be NLE also need only two bytes. 208
So as you can see the böhme is indicated the start of the data stream. 209
Now let's look at the last one. 210
Last one is the emoji symbol. 211
And since it's a non BMB kind of grow it needs it needs Bucco points. 212
OK. 213
Even though it's a single Caracara it needs to code points and ASCII cannot handle it. 214
So when it looks at this first individual Foster-Carter it says I cannot do X. 215
So to answer a question mark and for the second one it is it do it is supposed to do. 216
So it prints the corresponding numeric value which is 50 unsane goes with I ISO for UTF 8. 217
It is able to encode it properly. 218
So I needs 4 bytes Judaea of 16 needs 4 bytes. 219
And once again we go we can ignore the first two. 220
Now your idea of 16 b also needs only 4 bytes. 221
And this also needs forbearance. 222
The only difference here is this is 31 5:51. 223
And here it is 9 to 6:31 0 50 and 50 0. 224
Now in the browser run the browser looks at this byte sequence. 225
It should be able it should render a single character. 226
OK. 227
The emoji symbol. 228
Now if the bot browser does not have the capability to to render non-being bmp characters then you will 229
not be able to see those symbols. 230
You'll see some other symbol. 231
So that's that's the thing. 232
And many US are supporting some of those sort are not supporting I guess so in that case you will just 233
see some other symbol. 234
So that's about it. 235
And sometimes even databases do not support non-ABS characters so they don't like to insert any data 236
which is not which has an on B and B characters. 237
You might get some kind of an exception because some databases don't support. 238
So that's about it. 239
Hope you liked this character encoding lesson and it's a very fundamental lesson and that's about it 240
. 241
Thank you. 242
And happy coding