Re: byte count unicode string

willie schrieb:
Thank you for your patience and for educating me.
(Though I still have a long way to go before enlightenment)
I thought Python might have a small weakness in
lacking an efficient way to get the number of bytes
in a "UTF-8 encoded Python string object" (proper?),
but I've been disabused of that notion.

Well, to get to the enlightenment, you have to understand
that Unicode and UTF-8 are *not* synonyms.

A Python Unicode string is an abstract sequence of
characters. It does have an in-memory representation,
but that is irrelevant and depends on what microprocessor
you use. A byte string is a sequence of quantities with
8 bits each (called bytes).

For each of them, the notion of "length" exists: For
a Unicode string, it's the number of characters; for
a byte string, the number of bytes.

UTF-8 is a character encoding; it is only meaningful
to say that byte strings have an encoding (where
"UTF-8", "cp1252", "iso-2022-jp" are really very
similar). For a character encoding, "what is the
number of bytes?" is a meaningful question. For
a Unicode string, this question is not meaningful:
you have to specify the encoding first.

Now, there is no len(unicode_string, encoding) function:
len takes a single argument. To specify both the string
and the encoding, you have to write
len(unicode_string.encode(encoding)). This, as a
side effect, actually computes the encoding.

While it would be possible to answer the question
"how many bytes has Unicode string S in encoding E?"
without actually encoding the string, doing so would
require codecs to implement their algorithm twice:
once to count the number of bytes, and once to
actually perform the encoding. Since this operation
is not that frequent, it was chosen not to put the
burden of implementing the algorithm twice (actually,
doing so was never even considered).

HTH,
Martin
.

Relevant Pages

  • Further changes to source encodings (Was: PEP 263 status check)
    ... " PEP 263 status check", ... character in an 8-bit string literal. ... It appears that you are now saying that Python ... UTF-8 is such an encoding. ...
    (comp.lang.python)
  • Re: Python and decimal character entities over 128.
    ... the string "doesn't" may be coded as ... Python hates decimal entities beyond 128 so it chokes unless you do ... Web sites may use whatever encoding they please. ... Decode ignoring those character that cannot be decoded. ...
    (comp.lang.python)
  • Re: Unicode drives me crazy...
    ... every string on some level). ... Python needs to know what encoding is used. ... The decode instruction converts s into a unicode string - where Python ...
    (comp.lang.python)
  • Re: python3 urlopen(...).read() returns bytes
    ... That is, readshould return a string, as it did in py2.5. ... I can't agree that it's an obvious bug (in Python 3). ... It returns bytes because it doesn't know what encoding to use. ...
    (comp.lang.python)
  • Re: Why asci-only symbols?
    ... >> Perhaps string equivalence in keys will be treated like numeric equivalence? ... I know typewill be and in itself contain no encoding information now, ... >and a Unicode string, the system default encoding ...
    (comp.lang.python)