---

1.5 ‘char *’ strings

The classical C strings, with its C library support standardized by ISO C and POSIX, can be used in internationalized programs with some precautions. The problem with this API is that many of the C library functions for strings don’t work correctly on strings in locale encodings, leading to bugs that only people in some cultures of the world will experience.

The first problem with the C library API is the support of multibyte locales. According to the locale encoding, in general, every character is represented by one or more bytes (up to 4 bytes in practice — but use MB_LEN_MAX instead of the number 4 in the code). When every character is represented by only 1 byte, we speak of an “unibyte locale”, otherwise of a “multibyte locale”. It is important to realize that the majority of Unix installations nowadays use UTF-8 or GB18030 as locale encoding; therefore, the majority of users are using multibyte locales.

The important fact to remember is:

A ‘char’ is a byte, not a character.

As a consequence:

• The <ctype.h> API is useless in this context; it does not work in multibyte locales.
• The strlen function does not return the number of characters in a string. Nor does it return the number of screen columns occupied by a string after it is output. It merely returns the number of bytes occupied by a string.
• Truncating a string, for example, with strncpy, can have the effect of truncating it in the middle of a multibyte character. Such a string will, when output, have a garbled character at its end, often represented by a hollow box.
• strchr and strrchr do not work with multibyte strings if the locale encoding is GB18030 and the character to be searched is a digit.
• strstr does not work with multibyte strings if the locale encoding is different from UTF-8.
• strcspn, strpbrk, strspn cannot work correctly in multibyte locales: they assume the second argument is a list of single-byte characters. Even in this simple case, they do not work with multibyte strings if the locale encoding is GB18030 and one of the characters to be searched is a digit.
• strsep and strtok_r do not work with multibyte strings unless all of the delimiter characters are ASCII characters < 0x30.
• The strcasecmp, strncasecmp, and strcasestr functions do not work with multibyte strings.

The workarounds can be found in GNU gnulib https://www.gnu.org/software/gnulib/.

• gnulib has modules ‘mbchar’, ‘mbiter’, ‘mbuiter’ that represent multibyte characters and allow to iterate across a multibyte string with the same ease as through a unibyte string.
• gnulib has functions mbslen and mbswidth that can be used instead of strlen when the number of characters or the number of screen columns of a string is requested.
• gnulib has functions mbschr and mbsrrchr that are like strchr and strrchr, but work in multibyte locales.
• gnulib has a function mbsstr that is like strstr, but works in multibyte locales.
• gnulib has functions mbscspn, mbspbrk, mbsspn that are like strcspn, strpbrk, strspn, but work in multibyte locales.
• gnulib has functions mbssep and mbstok_r that are like strsep and strtok_r but work in multibyte locales.
• gnulib has functions mbscasecmp, mbsncasecmp, mbspcasecmp, and mbscasestr that are like strcasecmp, strncasecmp, and strcasestr, but work in multibyte locales. Still, the function ulc_casecmp is preferable to these functions; see below.

The second problem with the C library API is that it has some assumptions built-in that are not valid in some languages:

• It assumes that there are only two forms of every character: uppercase and lowercase. This is not true for Croatian, where the character LETTER DZ WITH CARON comes in three forms: LATIN CAPITAL LETTER DZ WITH CARON (DZ), LATIN CAPITAL LETTER D WITH SMALL LETTER Z WITH CARON (Dz), LATIN SMALL LETTER DZ WITH CARON (dz).
• It assumes that uppercasing of 1 character leads to 1 character. This is not true for German, where the LATIN SMALL LETTER SHARP S, when uppercased, becomes ‘SS’.
• It assumes that there is 1:1 mapping between uppercase and lowercase forms. This is not true for the Greek sigma: GREEK CAPITAL LETTER SIGMA is the uppercase of both GREEK SMALL LETTER SIGMA and GREEK SMALL LETTER FINAL SIGMA.
• It assumes that the upper/lowercase mappings are position independent. This is not true for the Greek sigma and the Lithuanian i.

The correct way to deal with this problem is

1. to provide functions for titlecasing, as well as for upper- and lowercasing,
2. to view case transformations as functions that operates on strings, rather than on characters.

This is implemented in this library, through the functions declared in <unicase.h>, see Case mappings <unicase.h>.