Re: The Register interview Nigel Brown

Hi Allen,

string. WideString is a non-reference counted BSTR and as such the
performance isn't quite as good as string. The problem is that we used
the "cool" name for the BSTR...

While we're on the topic: Have you considered implementing a native UTF-8
encoded string type with automatic conversion routines for assignment
to/from WideString, AnsiString, etc?

The reason I ask this is because UTF-8 is much more storage space efficient
than UTF-16 (the format employed by the Windows/Uniscribe API) with European
languages. UTF-16 is more efficient in storing CJK (Chinese, Japanese and
Korean) language text, but that's about the only area where it has an edge
over UTF-8 IMO. In UTF-16 you still cannot represent all characters with a
single 16-bit encoding - only characters in the BMP (basic multilingual
plane), which does not include all Chinese characters. Of course, you'll
never escape combining characters even if you overboard and use UTF-32.

An opportunity recently opened up for me to provide software to the Chinese
market, and I am hurriedly implementing Unicode support in my custom
components. The implementation I decided on is to use UTF-8 throughout and
to only convert to UTF-16 when I need to call the Uniscribe API to render
text. The conversion from UTF-8 to UTF-16 is quite fast, and insignificant
relative to the time the Uniscribe API takes to render the text. (The other
issue that affected my decision was the fact that my database backend, SQL
Server 2000, only supports UCS-2 and not UTF-16.)

The other nice things about using UTF-8 is that I can use the existing
reference counted long string type for storage, and I can also continue
using most of my existing library of fast string routines.

If this request is not too outrageous, should I QC it?

Regards,
Pierre


.

Relevant Pages

  • Re: Fast UTF-8 strlen function
    ... >> Is there a fast UTF-8 string length function floating around? ... Length in bytes, or length in characters? ... For UTF-8, the main basic "change" you have to make to your string routines ... then I could individually look up the characters in my UNICODE ...
    (alt.lang.asm)
  • Re: Writing Japanese or Chinese strings in a text file
    ... characters on the screen. ... start of the file that flags the data as UTF-8. ... VB uses Unicode internally, for 'String' data in memory. ... So they are right in the excel file. ...
    (microsoft.public.vb.general.discussion)
  • Re: CString and UTF-8
    ... installing a machine that isn't a standard locale and see how CString ... as multibyte characters, there are two chinese characters in there. ... ASCII string with UTF-8 embedded into it, delimited with quotes, the ...
    (microsoft.public.vc.mfc)
  • Re: Unicode string libraries
    ... it comes to sequences of characters? ... I know that Perl uses UTF-8 as its internal string representation. ... Ruby just didn't do Unicode. ...
    (comp.programming)
  • Re: Unicode question
    ... You're missing the existence and continued use of UTF-8 and UTF-16. ... I want to use an extended string that's good enough to cover UCS-4 ... sometimes multiple characters must be treated as one unit. ...
    (borland.public.delphi.non-technical)