[Tut] Python Unicode Encode Error

[xSicKxBot]

Python Unicode Encode Error

Summary: The UnicodeEncodeError generally occurs while encoding a Unicode string into a certain coding. Only a limited number of Unicode characters are mapped to strings. Thus, any character that is not-represented / mapped will cause the encoding to fail and raise UnicodeEncodeError. To avoid this error use the encode(

Code:

utf-8

) and decode(

Code:

utf-8

) functions accordingly in your code.

You might be using handling an application code that needs to deal with multilingual data or web content that has plenty of emojis and special symbols. In such situations, you will possibly come across numerous problems relating to Unicode data. But python has well-defined options to deal with Unicode characters and we shall be discussing them in this article.

What is Unicode?

Unicode is a standard that facilitates character encoding using variable bit encoding. I am sure, you must have heard of ASCII if you are into the world of computer programming. ASCII represents 128 characters while Unicode defines 221 characters. Thus, Unicode can be regarded as a superset of ASCII. If you are interested in having an in-depth look at Unicode, please follow this link. 
Click on Unicode:- U+1F40Dhttps://blog.finxter.com/wp-content/uplo...ythond.pngto find out what it represents! (Try it!!!)

What is a UnicodeEncodeError?

The best way to grasp any concept is to visualize it with an example. So let us have a look at an example of the

Code:

UnicodeEncodeError

.

Code:

u = 'é' print("Integer value for é: ", ord(u)) print("Converting the encoded value of é to Integer Equivalent: ", chr(233)) print("UNICODE Representation of é: ", u.encode('utf-8')) print("ASCII Representation of é: ", u.encode('ascii'))

Output

Code:

Integer value for é: 233 Converting the encoded value of é to Integer Equivalent: é UNICODE Representation of é: b'\xc3\xa9' Traceback (most recent call last): File "main.py", line 5, in <module> print("ASCII Representation of é: ",u.encode('ascii')) UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 0: ordinal not in range(128)

In the above code, when we tried to encode the character é to its Unicode value we got an output but while trying to convert it to the ASCII equivalent we encountered an error. The error occurred because ASCII only allows 7-bit encoding and it cannot represent characters outside the range of [0..128]. 

You now have an essence of what the

Code:

UnicodeEncodeError

looks like. Before discussing how we can avoid such errors, I feel that there is a dire need to discuss the following concepts:

Encoding and Decoding

The process of converting human-readable data into a specified format, for the secured transmission of data is known as encoding. Decoding is the opposite of encoding that is to convert the encoded information to normal text (human-readable form).

In Python, 

• encode() is an inbuilt method used for encoding. Incase no encoding is specified, UTF-8 is used as default. 
  
  
• decode() is an inbuilt method used for decoding. 
  

Example:

Code:

u = 'Πύθωνος' print("UNICODE Representation of é: ", u.encode('utf-8'))

Output:

Code:

UNICODE Representation of é: b'\xce\xa0\xcf\x8d\xce\xb8\xcf\x89\xce\xbd\xce\xbf\xcf\x82'

The following diagram should make things a little easier:


Codepoint

Unicode maps the codepoint to their respective characters. So, what do we mean by a codepoint? 

• Codepoints are numerical values or integers used to represent a character. 
  
  
• The Unicode code point for é is
  Code:

  U+00E9
  which is integer 233. When you encode a character and print it, you will generally get its hexadecimal representation as an output instead of its binary equivalent (as seen in the examples above).
  
  
• The byte sequence of a code point is different in different encoding schemes. For eg: the byte sequence for é in
  Code:

  UTF-8
  is
  Code:

  \xc3\xa9
  while in
  Code:

  UTF-16
  is \xff\xfe\xe9\x00.
  

Please have a look at the following program to get a better grip on this concept:

Code:

u = 'é' print("INTEGER value for é: ", ord(u)) print("ENCODED Representation of é in UTF-8: ", u.encode('utf-8')) print("ENCODED Representation of é in UTF-16: ", u.encode('utf-16'))

Output

Code:

INTEGER value for é: 233 ENCODED Representation of é in UTF-8: b'\xc3\xa9' ENCODED Representation of é in UTF-16: b'\xff\xfe\xe9\x00'

Now that we have an overview of Unicode and

Code:

UnicodeEncodeError

, let us discuss how we can deal with the error and avoid it in our program.

Problem: Given a string/text to be written in a text File; how to avoid the UnicodeEncodeError and write given text in the text file.

Example:

Code:

f = open('demo.txt', 'w') f.write('να έχεις μια όμορφη μέρα') f.close()

Output

Code:

Traceback (most recent call last): File "uniError.py", line 2, in <module> f.write('να έχεις μια όμορφη μέρα') File "C:\Users\Shubham-PC\AppData\Local\Programs\Python\Python38-32\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-1: character maps to <undefined>

Desired Output



Solution 1: Encode String Before Writing To File And Decode While Reading

You cannot write Unicode to a file directly. This will raise an

Code:

UnicodeEncodeError

. To avoid this you must encode the Unicode string using the

Code:

encode()

function and then write it to the file as shown in the program below:

Code:

text = u'να έχεις μια όμορφη μέρα' # write in binary mode to avoid TypeError f = open('demo.txt', 'wb') f.write(text.encode('utf8')) f.close() f = open('demo.txt', 'rb') print(f.read().decode('utf8'))

Output



Solution 2: Open File In utf-8

If you are using Python 3 or higher, all you need to do is open the file in

Code:

utf-8

, as Unicode string handling is already standardized in Python 3.

Code:

text = 'να έχεις μια όμορφη μέρα' f = open('demo2.txt', 'w', encoding="utf-8") f.write(text) f.close()

Output



Solution 3: Using The Codecs Module

Another approach to deal with the

Code:

UnicodeEncodeError

is using the codecsmodule. 

Let us have a look at the following code to understand how we can use the codecs module:

Code:

import codecs f = codecs.open("demo3.txt", "w", encoding='utf-8') f.write("να έχεις μια όμορφη μέρα") f.close()

Output



Solution 4: Using Python’s unicodecsv Module

If you are dealing with Unicode data and using a

Code:

csv

file for managing your data, then the

Code:

unicodecsv

module can be really helpful. It is an extended version of Python 2’s

Code:

csv

module and helps the user to handle Unicode data without any hassle.

Since the

Code:

unicodecsv

module is not a part of Python’s standard library, you have to install it before using it. Use the following command to install this module:

Code:

$ pip install unicodecsv

Let us have a look at the following example to get a better grip on the

Code:

unicodecsv

module:

Code:

import unicodecsv as csv with open('example.csv', 'wb') as f: writer = csv.writer(f, encoding='utf-8') writer.writerow(('English', 'Japanese')) writer.writerow((u'Hello', u'こんにちは'))

Output



Conclusion

In this article, we discussed some of the important concepts regarding Unicode character and then went on to learn about the UnicodeEncodeError and finally discussed the methods that we can use to avoid it. I hope by the end of this article you can handle Unicode characters in your python code with ease. 

Please subscribe andstay tuned for more interesting articles!

Where to Go From Here?

Enough theory, let’s get some practice!

To become successful in coding, you need to get out there and solve real problems for real people. That’s how you can become a six-figure earner easily. And that’s how you polish the skills you really need in practice. After all, what’s the use of learning theory that nobody ever needs?

Practice projects is how you sharpen your saw in coding!

Do you want to become a code master by focusing on practical code projects that actually earn you money and solve problems for people?

Then become a Python freelance developer! It’s the best way of approaching the task of improving your Python skills—even if you are a complete beginner.

Join my free webinar “How to Build Your High-Income Skill Python” and watch how I grew my coding business online and how you can, too—from the comfort of your own home.

Join the free webinar now!