Problem

You want to translate case when in a different locale, or you want to make \w match letters with diacritics, such as JosÙ or dÙjÐ vu.

For example, let's say you're given half a gigabyte of text written in German and told to index it. You want to extract words (with \w+) and convert them to lower-case (with lc or \L), but the normal versions of \w and lc neither match the German words nor change the case of accented letters.

Solution

Perl's regular-expression and text-manipulation routines have hooks to POSIX locale setting. If you use the use locale pragma, accented characters are taken care of  - assuming a reasonable LC_CTYPE specification and system support for the same.

use locale;

Discussion

By default, \w+ and case-mapping functions operate on upper- and lowercase letters, digits, and underscores. This works only for the simplest of English words, failing even on many common imports. The use locale directive lets you redefine what a "word character" means.

In Example 6.10 you can see the difference in output between having selected the English ("en") locale and the German ("de") one.

#!/usr/bin/perl -w
# localeg - demonstrate locale effects

use locale;
use POSIX 'locale_h';

$name = "andreas k\xF6nig";
@locale{qw(German English)} = qw(de_DE.ISO_8859-1 us-ascii);

setlocale(LC_CTYPE, $locale{English})
  or die "Invalid locale $locale{English}";
@english_names = ();
while ($name =~ /\b(\w+)\b/g) {
        push(@english_names, ucfirst($1));
}
setlocale(LC_CTYPE, $locale{German})
  or die "Invalid locale $locale{German}";
@german_names = ();
while ($name =~ /\b(\w+)\b/g) {
        push(@german_names, ucfirst($1));
}

print "English names: @english_names\n";
print "German names:  @german_names\n";
English names: Andreas K Nig
German names:  Andreas Kænig

This approach relies on POSIX locale support, which your system may or may not provide. Even if your system does claim to provide POSIX locale support, the standard does not specify the locale names. As you can tell, portability of this approach is not assured.

See Also

The treatment of \b, \w, and \s in perlre (1) and in the "Regular expression bestiary" section of Chapter 2 of Programming Perl; the treatment of locales in Perl in perllocale (1); your system's locale (3) manpage; we discuss locales in greater depth in Recipe 6.2; the "Perl and the POSIX locale" section of Chapter 7 of Mastering Regular Expressions