Problem with encoding in mysql when storing 'non-printable' characters?

Question

MySQL

Good afternoon,

There was a task to save a large amount of data in the database for further quick search in this array, the difficulty is that the text can be in different languages ​​(hieroglyphs, Arabic characters, etc.).

How I am currently trying to implement this:
if I receive a string consisting only of 'printable' characters (which I check with the isprintable() method), then it is immediately added to my dictionary and then stored in the database
; if I receive a string that contains 'non-printable' characters then I first translate it into utf-8 and then try to save it in the database.

Example:
if some_string.text.isprintable():
data['name'] = str_filter(actual_name.text)
else:
data['name'] = actual_name.text.encode('utf-8'


Questions:
1. There are characters that are defined as printable and after 'successful' saving in the database, I simply do not find them there, or rather I find four question marks "????" in their place. here is one like this: ""
2. For example, I managed to save the text in the database in some kind of encoding, for further search using LIKE, should I just specify the hieroglyph codes?

I apologize in advance if the question is not quite technically accurate.

Answer 1

The base is probably not in UTF-8, if so, then convert the base. Use the comparison utf8mb4_unicode_ci , then it will be possible to work correctly with multibyte characters.
Also check that the scripts are in UTF-8.
PS
What's the difference between utf8_general_ci and ...