Django 1.11: Issues with utf8 writing to file on windows
I am trying to write some text into a file with data from django objects (CharField) and some formatting. The problem is with accentuated characters (é in the example below).
On Linux I have no problems. However, on windows 7, I am getting extremely confusing behaviours.
This (open call without encoding):
from usermod.models import User
user = User.objects.get(pk=134)
with open('test.txt', 'w') as fout:
fout.write(user.birth_place + ',')
fout.write('Séoul')
fout.close()
produces:
SxE9l,Séoul
while this (open call with encoding):
from usermod.models import User
user = User.objects.get(pk=134)
with open('test.txt', 'w', encoding='utf8') as fout:
fout.write(user.birth_place + ',')
fout.write('Séoul')
fout.close()
produces:
Séoul,Séoul
Of course, the expected result is:
Séoul,Séoul
and this is what I get on Linux with the same database.
So the weird thing is that each of those options gets different parts right and wrong. In one case the ORM retrieved value is wrong. In the other it is the string object created in the source code that is wrong. I can't find a way to get both right.
All the files are encoded in UTF8 (as reported by notepad++). The python version is 3.5.4 The MySql database has all encodings in UTF8 and an SQL query on the user table shows the expected accents. This code is executed via the django shell (python manage.py shell) with the command:
exec(open('scripts/test.py').read()) where test.py contains the code shown above.
Any idea?
http://mysql.rjweb.org/doc.php/charcoll#python
In particular, 1st or 2nd line in source code (to have literals in the code utf8-encoded):
# -*- coding: utf-8 -*-