8) reading and writing to files in Python

2018-06-23 16:05:35

I'm having some brain failure in understanding reading and writing text to a file (Python 2.4).

# The string, which has an a-acute in it.
ss = u'Capitxe1n'
ss8 = ss.encode('utf8')
repr(ss), repr(ss8)

("u'Capitxe1n'", "'Capitxc3xa1n'")

print ss, ss8
print >> open('f1','w'), ss8

>>> file('f1').read()
'Capitxc3xa1nn'

So I type in Capitxc3xa1n into my favorite editor, in file f2.

Then:

>>> open('f1').read()
'Capitxc3xa1nn'
>>> open('f2').read()
'Capitxc3xa1nn'
>>> open('f1').read().decode('utf8')
u'Capitxe1nn'
>>> open('f2').read().decode('utf8')
u'Capitxc3xa1nn'

What am I not understanding here? Clearly there is some vital bit of magic (or good sense) that I'm missing. What does one type into text files to get proper conversions?

What I'm truly failing to grok here, is what the point of the UTF-8 representation is, if you can't actually get Python to recognize it, when it comes from outside. Maybe I should just JSON dump the string, and use that instead, since that has an asciiable representation! More to the point, is there an ASCII representation of this Unicode object that Python will recognize and decode, when coming in from a file? If so, how do I get it?

>>> print simplejson.dumps(ss)
'"Capitu00e1n"'
>>> print >> file('f3','w'), simplejson.dumps(ss)
>>> simplejson.load(open('f3'))
u'Capitxe1n'

In the notation

u'Capitxe1nn'

the "xe1" represents just one byte. "x" tells you that "e1" is in hexadecimal. When you write

Capitxc3xa1n

into your file you have "xc3" in it. Those are 4 bytes and in your code you read them all. You can see this when you display them:

>>> open('f2').read()
'Capitxc3xa1nn'

You can see that the backslash is escaped by a backslash. So you have four bytes in your string: "", "x", "c" and "3".

Edit:

As others pointed out in their answers you should just enter the characters in the editor and your editor should then handle the conversion to UTF-8 and save it.

If you actually have a string in this format you can use the string_escape codec to decode it into a normal string:

In [15]: print 'Capitxc3xa1nn'.decode('string_escape')
Capitán

The result is a string that is encoded in UTF-8 where the accented character is represented by the two bytes that were written xc3xa1 in the original string. If you want to have a unicode string you have to decode again with UTF-8.

To your edit: you don't have UTF-8 in your file. To actually see how it would look like:

s = u'Capitxe1nn'
sutf8 = s.encode('UTF-8')
open('utf-8.out', 'w').write(sutf8)

Compare the content of the file utf-8.out to the content of the file you saved with your editor.

Rather than mess with the encode and decode methods I find it easier to specify the encoding when opening the file. The io module (added in Python 2.6) provides an io.open function, which has an encoding parameter.

Use the open method from the io module.

>>>import io
>>>f = io.open("test", mode="r", encoding="utf-8")

Then after calling f's read() function, an encoded Unicode object is returned.

>>>f.read()
u'Capitxe1lnn'

Note that in Python 3, the io.read function is an alias for the built-in read function. The built-in read function only supports the encoding argument in Python 3, not Python 2.

Edit: Previously this answer recommended the codecs module. The codecs module can cause problems when mixing read() and readline() , so this answer now recommends the io module instead.

Use the open method from the codecs module.

>>>import codecs
>>>f = codecs.open("test", "r", "utf-8")

Then after calling f's read() function, an encoded Unicode object is returned.

>>>f.read()
u'Capitxe1lnn'

If you know the encoding of a file, using the codecs package is going to be much less confusing.

See http://docs.python.org/library/codecs.html#codecs.open

So, I've found a solution for what I'm looking for, which is:

print open('f2').read().decode('string-escape').decode("utf-8")

There are some unusual codecs that are useful here. This particular reading allows one to take UTF-8 representations from within Python, copy them into an ASCII file, and have them be read in to Unicode. Under the "string-escape" decode, the slashes won't be doubled.

This allows for the sort of round trip that I was imagining.

链接地址: http://www.djcxy.com/p/66352.html

上一篇: UTF有什么区别？

下一篇: 8）用Python读写文件