The classical C strings, with its C library support standardized by ISO C and POSIX, can be used in internationalized programs with some precautions. The problem with this API is that many of the C library functions for strings don’t work correctly on strings in locale encodings, leading to bugs that only people in some cultures of the world will experience.
The first problem with the C library API is the support of multibyte
locales. According to the locale encoding, in general, every character
is represented by one or more bytes (up to 4 bytes in practice — but
MB_LEN_MAX instead of the number 4 in the code).
When every character is represented by only 1 byte, we speak of an
“unibyte locale”, otherwise of a “multibyte locale”. It is important
to realize that the majority of Unix installations nowadays use UTF-8
or GB18030 as locale encoding; therefore, the majority of users are
using multibyte locales.
The important fact to remember is:
A ‘char’ is a byte, not a character.
As a consequence:
<ctype.h>API is useless in this context; it does not work in multibyte locales.
strlenfunction does not return the number of characters in a string. Nor does it return the number of screen columns occupied by a string after it is output. It merely returns the number of bytes occupied by a string.
strncpy, can have the effect of truncating it in the middle of a multibyte character. Such a string will, when output, have a garbled character at its end, often represented by a hollow box.
strrchrdo not work with multibyte strings if the locale encoding is GB18030 and the character to be searched is a digit.
strstrdoes not work with multibyte strings if the locale encoding is different from UTF-8.
strspncannot work correctly in multibyte locales: they assume the second argument is a list of single-byte characters. Even in this simple case, they do not work with multibyte strings if the locale encoding is GB18030 and one of the characters to be searched is a digit.
strtok_rdo not work with multibyte strings unless all of the delimiter characters are ASCII characters < 0x30.
strcasestrfunctions do not work with multibyte strings.
The workarounds can be found in GNU gnulib https://www.gnu.org/software/gnulib/.
mbswidththat can be used instead of
strlenwhen the number of characters or the number of screen columns of a string is requested.
mbsrrchrthat are like
strrchr, but work in multibyte locales.
strstr, but works in multibyte locales.
mbsspnthat are like
strspn, but work in multibyte locales.
mbstok_rthat are like
strtok_rbut work in multibyte locales.
mbscasestrthat are like
strcasestr, but work in multibyte locales. Still, the function
ulc_casecmpis preferable to these functions; see below.
The second problem with the C library API is that it has some assumptions built-in that are not valid in some languages:
The correct way to deal with this problem is
This is implemented in this library, through the functions declared in
<unicase.h>, see Case mappings