Chapter 5: Pattern Matching

5.3. Metacharacters and Metasymbols

Now that we've admired all the fancy cages, we can go back to looking at the critters in the cages, those funny-looking symbols you put inside the patterns. By now you'll have cottoned to the fact that these symbols aren't regular Perl code like function calls or arithmetic operators. Regular expressions are their own little language nestled inside of Perl. (There's a bit of the jungle in all of us.)

For all their power and expressivity, patterns in Perl recognize the same 12 traditional metacharacters (the Dirty Dozen, as it were) found in many other regular expression packages:

\ | ( ) [ { ^ $ * + ? .

Some simple metacharacters stand by themselves, like . and ^ and $. They don't directly affect anything around them. Some metacharacters work like prefix operators, governing what follows them, like \. Others work like postfix operators, governing what immediately precedes them, like *, +, and ?. One metacharacter, |, acts like an infix operator, standing between the operands it governs. There are even bracketing metacharacters that work like circumfix operators, governing something contained inside them, like (...) and [...]. Parentheses are particularly important, because they specify the bounds of | on the inside, and of *, +, and ? on the outside.

If you learn only one of the twelve metacharacters, choose the backslash. (Er . . . and the parentheses.) That's because backslash disables the others. When a backslash precedes a nonalphanumeric character in a Perl pattern, it always makes that next character a literal. If you need to match one of the twelve metacharacters in a pattern literally, you write them with a backslash in front. Thus, \. matches a real dot, \$ a real dollar sign, \\ a real backslash, and so on. This is known as "escaping" the metacharacter, or "quoting it", or sometimes just "backslashing" it. (Of course, you already know that backslash is used to suppress variable interpolation in double-quoted strings.)

Although a backslash turns a metacharacter into a literal character, its effect upon a following alphanumeric character goes the other direction. It takes something that was regular and makes it special. That is, together they make a metasymbol. An alphabetical list of these metasymbols can be found below in Table 5-7.

5.3.1. Metasymbol Tables

In the following tables, the Atomic column says "yes" if the given metasymbol is quantifiable (if it can match something with width, more or less). Also, we've used "..." to represent "something else". Please see the later discussion to find out what "..." means, if it is not clear from the one-line gloss in the table.)

Table 5-4 shows the basic traditional metasymbols. The first four of these are the structural metasymbols we mentioned earlier, while the last three are simple metacharacters. The . metacharacter is an example of an atom because it matches something with width (the width of a character, in this case); ^ and $ are examples of assertions, because they match something of zero width, and because they are only evaluated to see if they're true or not.

Table 5.4. General Regex Metacharacters

The quantifiers, which are further described in their own section, indicate how many times the preceding atom (that is, single character or grouping) should match. These are listed in Table 5-5.

Table 5.5. Regex Quantifiers

A minimal quantifier tries to match as few characters as possible within its allowed range. A maximal quantifier tries to match as many characters as possible within its allowed range. For instance, .+ is guaranteed to match at least one character of the string, but will match all of them given the opportunity. The opportunities are discussed later in "The Little Engine That /Could(n't)?/".

You'll note that quantifiers may never be quantified.

We wanted to provide an extensible syntax for new kinds of metasymbols. Given that we only had a dozen metacharacters to work with, we chose a formerly illegal regex sequence to use for arbitrary syntactic extensions. These metasymbols are all of the form (?KEY...); that is, a (balanced) parenthesis followed by a question mark, followed by a KEY and the rest of the subpattern. The KEY character indicates which particular regex extension it is. See Table 5-6 for a list of these. Most of them behave structurally since they're based on parentheses, but they also have additional meanings. Again, only atoms may be quantified because they represent something that's really there (potentially).

Table 5.6. Extended Regex Sequences

And finally, Table 5-7 shows all of your favorite alphanumeric metasymbols. (Symbols that are processed by the variable interpolation pass are marked with a dash in the Atomic column, since the Engine never even sees them.)

Table 5.7. Alphanumeric Regex Metasymbols

The braces are optional on \p and \P if the property name is one character. The braces are optional on \x if the hexadecimal number is two digits or less. The braces are never optional on \N.

Only metasymbols with "Match the..." or "Match any..." descriptions may be used within character classes (square brackets). That is, character classes are limited to containing specific sets of characters, so within them you may only use metasymbols that describe other specific sets of characters, or that describe specific individual characters. Of course, these metasymbols may also be used outside character classes, along with all the other nonclassificatory metasymbols. Note however that \b is two entirely different beasties: it's a backspace character inside the character class, but a word boundary assertion outside.

There is some amount of overlap between the characters that a pattern can match and the characters an ordinary double-quoted string can interpolate. Since regexes undergo two passes, it is sometimes ambiguous which pass should process a given character. When there is ambiguity, the variable interpolation pass defers the interpretation of such characters to the regular expression parser.

But the variable interpolation pass can only defer to the regex parser when it knows it is parsing a regex. You can specify regular expressions as ordinary double-quoted strings, but then you must follow normal double-quote rules. Any of the previous metasymbols that happen to map to actual characters will still work, even though they're not being deferred to the regex parser. But you can't use any of the other metasymbols in ordinary double quotes (or in any similar constructs such as `...`, qq(...), qx(...), or the equivalent here documents). If you want your string to be parsed as a regular expression without doing any matching, you should be using the qr// (quote regex) operator.

Note that the case and metaquote translation escapes (\U and friends) must be processed during the variable interpolation pass because the purpose of those metasymbols is to influence how variables are interpolated. If you suppress variable interpolation with single quotes, you don't get the translation escapes either. Neither variables nor translation escapes (\U, etc.) are expanded in any single quoted string, nor in single-quoted m'...' or qr'...' operators. Even when you do interpolation, these translation escapes are ignored if they show up as the result of variable interpolation, since by then it's too late to influence variable interpolation.

Although the transliteration operator doesn't take regular expressions, any metasymbol we've discussed that matches a single specific character also works in a tr/// operation. The rest do not (except for backslash, which continues to work in the backward way it always works.)

5.3.2. Specific Characters

As mentioned before, everything that's not special in a pattern matches itself. That means an /a/ matches an "a", an /=/ matches an "=", and so on. Some characters, though, aren't very easy to type in from the keyboard or, even if you manage that, don't show up on a printout; control characters are notorious for this. In a regular expression, Perl recognizes the following double-quotish character aliases:

Just as in double-quoted strings, Perl also honors the following four metasymbols in patterns:

\cX

A named control character, like \cC for Control-C, \cZ for Control-Z, \c[ for ESC, and \c? for DEL.

\NNN

A character specified using its two- or three-digit octal code. The leading 0 is optional, except for values less than 010 (8 decimal) since (unlike in double-quoted strings) the single-digit versions are always considered to be backreferences to captured strings within a pattern. Multiple digits are interpreted as the nth backreference if you've captured at least n substrings earlier in the pattern (where n is considered as a decimal number). Otherwise, they are interpreted as a character specified in octal.

\x{LONGHEX}

\xHEX

A character number specified as one or two hex digits ([0-9a-fA-F]), as in \x1B. The one-digit form is usable only if the character following it is not a hex digit. If braces are used, you may use as many digits as you'd like, which may result in a Unicode character. For example, \x{262f} matches a Unicode YIN YANG.

\N{NAME}

A named character, such \N{GREEK SMALL LETTER EPSILON}, \N{greek:epsilon}, or \N{epsilon}. This requires the use charnames pragma described in Chapter 31, "Pragmatic Modules", which also determines which flavors of those names you may use (":long", ":full", ":short" respectively, corresponding to the three styles just shown).

A list of all Unicode character names can be found in your closest Unicode standards document, or in PATH_TO_PERLLIB/unicode/Names.txt.

5.3.3. Wildcard Metasymbols

Three special metasymbols serve as generic wildcards, each of them matching "any" character (for certain values of "any"). These are the dot ("."), \C, and \X. None of these may be used in a character class. You can't use the dot there because it would match (nearly) any character in existence, so it's something of a universal character class in its own right. If you're going to include or exclude everything, there's not much point in having a character class. The special wildcards \C and \X have special structural meanings that don't map well to the notion of choosing a single Unicode character, which is the level at which character classes work.

The dot metacharacter matches any one character other than a newline. (And with the /s modifier, it matches that, too.) Like any of the dozen special characters in a pattern, to match a dot literally, you must escape it with a backslash. For example, this checks whether a filename ends with a dot followed by a one-character extension:

if ($pathname =~ /\.(.)\z/s) {
    print "Ends in $1\n";
}

The dot metacharacter is most often used with a quantifier. A .* matches a maximal number of characters, while a .*? matches a minimal number of characters. But it's also sometimes used without a quantifier for its width: /(..):(..):(..)/ matches three colon-separated fields, each of which is two characters long.

If you use a dot in a pattern compiled under the lexically scoped use utf8 pragma, then it will match any Unicode character. (You're not supposed to need a use utf8 for that, but accidents will happen. The pragma may not be necessary by the time you read this.)

use utf8;
use charnames qw/:full/;
$BWV[887] = "G\N{MUSIC SHARP SIGN} minor";
($note, $black, $mode) = $BWV[887] =~ /^([A-G])(.)\s+(\S+)/;
print "That's lookin' sharp!\n" if $black eq chr(9839);

The \X metasymbol matches a character in a more extended sense. It really matches a string of one or more Unicode characters known as a "combining character sequence". Such a sequence consists of a base character followed by any "mark" characters (diacritical markings like cedillas or diereses) that combine with that base character to form one logical unit. \X is exactly equivalent to (?:\PM\pM*). This allows it to match one logical character, even when that really comprises several separate characters. The length of the match in /\X/ would exceed one character if it matched any combining characters. (And that's character length, which has little to do with byte length).

If you are using Unicode and really want to get at a single byte instead of a single character, you can use the \C metasymbol. This will always match one byte (specifically, one C language char type), even if this gets you out of sync with your Unicode character stream. See the appropriate warnings about doing this in Chapter 15, "Unicode".

5.2. Pattern-Matching Operators		5.4. Character Classes

Copyright © 2001 O'Reilly & Associates. All rights reserved.