Re: Unicode BOM marks
From: Martin v. Löwis (martin_at_v.loewis.de)
Date: 03/07/05
- Next message: QMartin_v=2E_L=F6wis=22?=: "Re: Python 2.4 / WinXP / distutils error (cookbook example)"
- Previous message: pythonnewbie: "pyconfig.h"
- In reply to: Francis Girard: "Unicode BOM marks"
- Next in thread: Francis Girard: "Re: Unicode BOM marks"
- Reply: Francis Girard: "Re: Unicode BOM marks"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Date: Mon, 07 Mar 2005 21:54:04 +0100
Francis Girard wrote:
> If I understand well, into the UTF-8 unicode binary representation, some
> systems add at the beginning of the file a BOM mark (Windows?), some don't.
> (Linux?). Therefore, the exact same text encoded in the same UTF-8 will
> result in two different binary files, and of a slightly different length.
> Right ?
Mostly correct. I would prefer if people referred to the thing not as
"BOM" but as "UTF-8 signature", atleast in the context of UTF-8, as
UTF-8 has no byte-order issues that a "byte order mark" would deal with.
(it is correct to call it "BOM" in the context of UTF-16 or UTF-32).
Also, "some systems" is inadequate. It is not so much the operating
system that decides to add or leave out the UTF-8 signature, but much
more the application writing the file. Any high-quality tool will accept
the file with or without signature, whether it is a tool on Windows
or a tool on Unix.
I personally would write my applications so that they put the signature
into files that cannot be concatenated meaningfully (since the
signature simplifies encoding auto-detection) and leave out the
signature from files which can be concatenated (as concatenating the
files will put the signature in the middle of a file).
> I guess that this leading BOM mark are special marking bytes that can't be, in
> no way, decoded as valid text.
> Right ?
Wrong. The BOM mark decodes as U+FEFF:
>>> codecs.BOM_UTF8.decode("utf-8")
u'\ufeff'
This is what makes it a byte order mark: in UTF-16, you can tell the
byte order by checking whether it is FEFF or FFFE. The character U+FFFE
is an invalid character, which cannot be decoded as valid text
(although the Python codec will decode it as invalid text).
> I also guess that this leading BOM mark is silently ignored by any unicode
> aware file stream reader to which we already indicated that the file follows
> the UTF-8 encoding standard.
> Right ?
No. It should eventually be ignored by the application, but whether the
stream reader special-cases it or not is depends on application needs.
> If so, is it the case with the python codecs decoder ?
No; the Python UTF-8 codec is unaware of the UTF-8 signature. It reports
it to the application when it finds it, and it will never generate the
signature on its own. So processing the UTF-8 signature is left to the
application in Python.
> In python documentation, I see theseconstants. The documentation is not clear
> to which encoding these constants apply. Here's my understanding :
>
> BOM : UTF-8 only or UTF-8 and UTF-32 ?
UTF-16.
> BOM_BE : UTF-8 only or UTF-8 and UTF-32 ?
> BOM_LE : UTF-8 only or UTF-8 and UTF-32 ?
UTF-16
> Why should I need these constants if codecs decoder can handle them without my
> help, only specifying the encoding ?
Well, because the codecs don't. It might be useful to add a
"utf-8-signature" codec some day, which generates the signature on
encoding, and removes it on decoding.
Regards,
Martin
- Next message: QMartin_v=2E_L=F6wis=22?=: "Re: Python 2.4 / WinXP / distutils error (cookbook example)"
- Previous message: pythonnewbie: "pyconfig.h"
- In reply to: Francis Girard: "Unicode BOM marks"
- Next in thread: Francis Girard: "Re: Unicode BOM marks"
- Reply: Francis Girard: "Re: Unicode BOM marks"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Relevant Pages
|
