Re: what does "serialization" mean?

RobertMaas_at_YahooGroups.Com
Date: 07/07/04 

Date: Wed, 07 Jul 2004 09:55:29 -0700

> From: Corey Murtagh <emonk@slingshot.no.uce>
> Both headers are of course UTF-8 compatible... specifically, neither
> uses any multi-byte character sequences or multi-character
> representations.

You might be missing the point. UTF-8 means that each unit is 8 bits
(one byte), and specifics about what these 8 bits represent. UTF-16
means that each unit is 16 bits (two bytes) etc. If you look at just
the printed representation of some text, which appears to be the subset
of characters common to ASCII UTF-8 and UTF-16, you can't tell how many
bytes were used to represent each character you see. You'd have to look
at the binary dump of the data, or "know" a priori how many bytes each
character uses.

I made the assumption that because I could read the text via Google
Groups and lynx, the data as posted was using one byte per character.
But if NUL bytes are simply invisible, don't show up via lynx, then
it's possible the poster deliberately put NUL bytes between each pair
of adjacent visible characters and we weren't the wiser. I could ask
Google Groups to show me the original format of the message, and
download that to a file, and inspect it with some program that shows
NUL bytes explicitly, but if Google Groups deleted NULs that were in
the original message, then I wouldn't prove anything.

> Of course one could argue that both headers are also ASCII compliant
> in that they use only characters that exist in the 0..127 range.
> While this is also true, it should be noted that UTF-8 is basically a
> superset of ASCII, and that the UTF-8 characters in the range 0..127
> are /largely/ congruent with the ASCII characters in that range. Not
> completely, but largely.

Well let's say there's a large-majority subset of that 1..127 range
where ASCII and UTF-8 are congruent, and a few odd characters within
that range where ASCII and UTF-8 disagree as to meaning. But the two
lines we were talking about were among that large-majority subset where
ASCII and UTF-8 are congruent, providing that the data was consuming
one byte per character! If that data was actually using two bytes per
character, low-order the same as UTF-8 or ASCII and high-order
all-bits-zero, then it wasn't compatible with UTF-8 or ASCII at all, it
was only compatible with UTF-16. And if it actually used four bytes per
character, low-order compatible with UTF-8 or ASCII and the other three
all-bits-zero, then it was actually UTF-32!! When a program is reading
in a data file from some unknown source, it's really important to know
whether to input one byte or two bytes or four bytes per character! So
my first question remains: What input mode is used initially to read
the XML-version-characterset header line? If the congruent intersection
of ASCII and UTF-8 is used, then my second question remains: at what
point does the data switch from that single-byte-per-character format
to the declared UTF-16 (two bytes per code unit which is usually a full
Unicode character) or UTF-32 (four bytes per code unit which is always
a full Unicode character)?

I've created a plain-text Web page that actually contains that XML
declaration in three formats, one in UTF-8 format as posted here, one
in UTF-16 format, and one in backwards UTF-16 format.
  http://www.rawbw.com/~rem/utf.txt
I set up a link to that file, and the NULs don't show up on screen in
lynx, but then made a link to that file and used lynx to download
through that link, and the NUL characters were there in the resultant
downloaded file, so I know our HTTP server retains them. So can anybody
tell me which of the three formats in that file is correct for that
UTF-16 declaration?

Relevant Pages

  • Re: GAS-style syntax issue...
    ... but, alas, the issue becomes a little more hairy than a few simple parser ... I guess it is an issue right up there with making the assembler UTF-8 ... (UTF-16 just wastes too much memory IMO, ... majority of text is ASCII... ...
    (alt.lang.asm)
  • Re: Follow-up to multiple groups
    ... which UTF-8 would be 'alternative' rather than standard. ... to a basic UTF-8 instead of basic ascii still ... UTF-8 has the advantage over legacy character ... The debate is around my being in favor of conveniently anglicizing such as ...
    (news.software.readers)
  • RE: Login scripts?
    ... iso-8859-1 is also a part of UTF-8. ... Begining with ASCII (the real defined one, not others' redefinitions, ... codes (character number 255 in ISO-8859-1 is the same as character ... only when you refer to higher numbers, such as 256, that you need to use ...
    (Fedora)
  • Re: Unicode
    ... That's why I said read up on unicode! ... The short version is that UTF-16 is basically wasteful. ... range) where UTF-8 does not. ... other character sets are needed (really? ...
    (comp.lang.ruby)
  • Re: Follow-up to multiple groups
    ... some languages for which that would be unsatisfactory, for which UTF-8 ... basic UTF-8 instead of basic ascii still leads to ... UTF-8 has the advantage over legacy character ... sets in that much less translation between different character sets is ...
    (news.software.readers)