Re: Defacto standard string library

On Jan 1, 1:30 pm, Keith Thompson <ks...@xxxxxxx> wrote:
Phil Carmody <thefatphil_demun...@xxxxxxxxxxx> writes:
"christian.bau" <christian....@xxxxxxxxxxxxxxxxxx> writes:
On Dec 29, 4:59 pm, Jon Harrop <j...@xxxxxxxxxxxxxxxxx> wrote:
I'd like to manipulate strings from C code that may contain null characters
and may need unicode support. Is there a defacto standard string library
for C (particularly under Linux) that satisfies these needs?

You could use a method that is common in Java implementations: Use
Unicode, encoded in UTF8 format, except that a zero byte is
represented as a two byte sequence 0xc0 0x80 (which is against UTF8
rules, so this is not UTF8). Then append a zero byte at the end to
make it a C string. Standard C string functions will be fine with this
format.

Some background, in case not everyone knows this.  Unicode is a
character set that includes characters from multiple languages; as a
result, it cannot be encoded using a single byte per character, unless
a byte is at least (I think) 17 bits.  UTF8 is an encoding for Unicode
using 8-bit bytes; some Unicode characters are encoded in a single
byte, others require multiple bytes.  (It's likely I've gotten some
details of the terminology wrong.)

Concatenating looks like it will fail horribly.

Why?  As far as I can see, it would work just fine.  Do you have an
example where it would fail?

Escape character sequences.

                                                The length
of the (semantically) empty string is no longer zero.

Why?  The empty string would still be represented as a sequence of a
single byte with the value 0 (the terminating null character);
strlen() would return 0 for this string.

                                                      The
standard C str* functions seem to not have much more use
than the standard C mem* functions, to be honest.

Again, why?  You'd just have to remember that strlen() gives you the
length *in bytes*, not the length in Unicode characters.  This is just
what you want if you want to know, for example, how much memory to
allocate for a given string.  (You could have other, somewhat more
expensive functions for things like the logical length in characters.)

Sometimes you want to know things like "What is the display length?".
You may also want to create a table to store the answer in a
database. The database notion of length will be in Unicode
characters.

I can't imagine any standard 8-bit character function working properly
even on 8-bit unicode.
You won't get the lenght in units, you won't be able to reliably
locate a character or substring, concatenation will be unreliable,
etc.
They don't even have the right interface (e.g. You can't search for
the chinese character for boat {8 mouths in a box} with strchr).

If you want to do unicode, I think it is a terrible mistake to try to
manipulate it as ordinary C strings {an 8 bit Unicode character can
comprise up to 5 8-bit characters IIRC}.

The free open source IBM library will do it without any difficulty and
is fully debugged. Considering the tens of thousands of lines of code
involved, I guess that simplified efforts to reproduce the correct
behavior will end in tragedy.

It is true that there are some things that could be done using the
standard C string library. But if you have gone to the bother of
using Unicode character sets, then the right behavior will be that
provided by the IBM library and the C functions will only answer some
of the questions some of the time.

If the needs are super-simple (e.g. the interface is used only for
transport), then maybe all that is needed are the mem*() functions.
But if we ever want to *look* at the data in any useful way, then the
C library does not measure up.

I think it would be good to incorporate ICU into the standard C
library for the next standard. There is no alternative with
equivalent functionality that I know of.
.

Relevant Pages

  • Transponder Protocol Open Standard rec.pets-2005a
    ... But anybody can write an open standard, ... transponders readable by the standard is not intended to be identical ... as 20 character readings under this standard unless the reader designer ... If an excerpt string shows four or fewer transitions ...
    (rec.pets)
  • Re: Determining if a string is Unicode
    ... there's nothing magic about Unicode. ... where each character occupies 2 bytes, as opposed to a Single-Byte Character ... You could load up a string with rubbish, ... > INF file like so: ...
    (microsoft.public.vb.general.discussion)
  • Re: Determining if a string is Unicode
    ... bytes per character, and MULTI-byte occupies one!!?? ... there's nothing magic about Unicode. ... You could load up a string with rubbish, ... if I read in the INF file from a 9x based computer the string does ...
    (microsoft.public.vb.general.discussion)
  • Re: Arabic or Chinese characters in a URL link give error copying
    ... the active ANSI character set, ... Arabic/Chinese then the associated "wide" Unicode characters will have been ... Function ContainsWideChars(ByRef inString As String) As Boolean ... Dim iCh As Integer ...
    (microsoft.public.vb.general.discussion)
  • Re: Arabic characters gives ASCII code 63
    ... The only problem is that you are looking at the ASCII/ANSI values i.e. assuming that each character is represented as a number between 0 and 255. ... This is hidden from the developer - the length of a 5 character string is still 5 but it's still 10 bytes. ... all you need to do is get the unicode value for each character rather than the ANSI number. ... Dim CellValue As String ...
    (microsoft.public.excel.programming)