What string encoding to use on linux?

Question

linux

I do authorization by login and password in the openvpn server, there is such a script to check the login and password:

verify.py

#!/usr/bin/python3
# coding=utf-8

import sys
import os
import requests
import config

if config.show_time:
  import time
  
  start_time = time.perf_counter()
  

SUCCESS_AUTH = 0
ERROR_AUTH = 1

username = os.environ.get("username")
password = os.environ.get("password")
ip = os.environ.get("untrusted_ip")
port = os.environ.get("untrusted_port")

print(config.verify_log.format("Authorization attempt for username: " + username + " ip: " + ip + " port: " + port))

data = {
  "username": username,
  "password": password,
  "ip": ip,
  "port": port,
}

try:
  r = requests.post(url=config.verify_url, data=data)
except Exception as e:
  print(config.verify_error_log.format("Failed to complete server request: {0}".format(e)))
  sys.exit(ERROR_AUTH)


if r.status_code != 200:
  print(config.verify_error_log.format("Error status server code " + str(r.status_code) + " and should be 200"))
  sys.exit(ERROR_AUTH)
  
  
if config.show_time:
  stop_time = time.perf_counter()
  diff_time = str(config.toFixed((stop_time - start_time), 2))
  print(config.verify_log.format("Request time: " + diff_time + " seconds"))
  

try:
  r_json = r.json()
except Exception:
  print(config.verify_error_log.format("Server error returned not json"))
  sys.exit(ERROR_AUTH)


if r_json["ok"] == False:
  print(config.verify_log.format("Authorisation Error. Server message: " + r_json["message"]))
  sys.exit(ERROR_AUTH)
else:
  print(config.verify_log.format("Successful authorization username: " + username + " ip: " + ip + " port: " + port))
  sys.exit(SUCCESS_AUTH)

If Cyrillic is passed in the login or password parameters, then the request will not be sent to the server, there will be an error in this line r = requests.post(url=config.verify_url, data=data).
"'utf-8' codec can't encode characters in position 30-31: surrogates not allowed"
error

traceback

Dec 12 14:37:02 shadowvpn openvpn[4349]: Traceback (most recent call last):
Dec 12 14:37:02 shadowvpn openvpn[4349]:   File "/etc/openvpn/scripts/verify.py", line 34, in <module>
Dec 12 14:37:02 shadowvpn openvpn[4349]:     r = requests.post(url=config.verify_url, data=data)
Dec 12 14:37:02 shadowvpn openvpn[4349]:   File "/usr/lib/python3/dist-packages/requests/api.py", line 112, in post
Dec 12 14:37:02 shadowvpn openvpn[4349]:     return request('post', url, data=data, json=json, **kwargs)
Dec 12 14:37:02 shadowvpn openvpn[4349]:   File "/usr/lib/python3/dist-packages/requests/api.py", line 58, in request
Dec 12 14:37:02 shadowvpn openvpn[4349]:     return session.request(method=method, url=url, **kwargs)
Dec 12 14:37:02 shadowvpn openvpn[4349]:   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 506, in request
Dec 12 14:37:02 shadowvpn openvpn[4349]:     prep = self.prepare_request(req)
Dec 12 14:37:02 shadowvpn openvpn[4349]:   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 449, in prepare_request
Dec 12 14:37:02 shadowvpn openvpn[4349]:     hooks=merge_hooks(request.hooks, self.hooks),
Dec 12 14:37:02 shadowvpn openvpn[4349]:   File "/usr/lib/python3/dist-packages/requests/models.py", line 308, in prepare
Dec 12 14:37:02 shadowvpn openvpn[4349]:     self.prepare_body(data, files, json)
Dec 12 14:37:02 shadowvpn openvpn[4349]:   File "/usr/lib/python3/dist-packages/requests/models.py", line 499, in prepare_body
Dec 12 14:37:02 shadowvpn openvpn[4349]:     body = self._encode_params(data)
Dec 12 14:37:02 shadowvpn openvpn[4349]:   File "/usr/lib/python3/dist-packages/requests/models.py", line 104, in _encode_params
Dec 12 14:37:02 shadowvpn openvpn[4349]:     v.encode('utf-8') if isinstance(v, str) else v))
Dec 12 14:37:02 shadowvpn openvpn[4349]: UnicodeEncodeError: 'utf-8' codec can't encode characters in position 30-31: surrogates not allowed

Answer 1

any, because There are different sites, but I also met with 2 encodings at the same time
as encode / decode, play around, you need to explicitly specify the encoding in the request, I won’t say more precisely, it was a long time ago ...