Section (3) pcre2pattern
Name
PCRE2 — Perl-compatible regular expressions (revised API)
PCRE2 REGULAR EXPRESSION DETAILS
The syntax and semantics of the regular expressions that are supported by PCRE2 are described in detail below. There is a quick-reference syntax summary in the pcre2syntax(3) page. PCRE2 tries to match Perl syntax and semantics as closely as it can. PCRE2 also supports some alternative regular expression syntax (which does not conflict with the Perl syntax) in order to provide some compatibility with regular expressions in Python, .NET, and Oniguruma.
Perl_zsingle_quotesz_s regular expressions are described in its own documentation, and regular expressions in general are covered in a number of books, some of which have copious examples. Jeffrey Friedl_zsingle_quotesz_s Mastering Regular Expressions, published by O_zsingle_quotesz_Reilly, covers regular expressions in great detail. This description of PCRE2_zsingle_quotesz_s regular expressions is intended as reference material.
This document discusses the regular expression patterns
that are supported by PCRE2 when its main matching function,
pcre2_match
(), is used. PCRE2
also has an alternative matching function, pcre2_dfa_match
(), which matches using a
different algorithm that is not Perl-compatible. Some of the
features discussed below are not available when DFA matching
is used. The advantages and disadvantages of the alternative
function, and how it differs from the normal function, are
discussed in the pcre2matching(3) page.
SPECIAL START-OF-PATTERN ITEMS
A number of options that can be passed to pcre2_compile
() can also be set by special
items at the start of a pattern. These are not
Perl-compatible, but are provided to make these options
accessible to pattern writers who are not able to change the
program that processes the pattern. Any number of these items
may appear, but they must all be together right at the start
of the pattern string, and the letters must be in upper
case.
UTF support
In the 8-bit and 16-bit PCRE2 libraries, characters may be coded either as single code units, or as multiple UTF-8 or UTF-16 code units. UTF-32 can be specified for the 32-bit library, in which case it constrains the character values to valid Unicode code points. To process UTF strings, PCRE2 must be built to include Unicode support (which is the default). When using UTF strings you must either call the compiling function with one or both of the PCRE2_UTF or PCRE2_MATCH_INVALID_UTF options, or the pattern must start with the special sequence (*UTF), which is equivalent to setting the relevant PCRE2_UTF. How setting a UTF mode affects pattern matching is mentioned in several places below. There is also a summary of features in the pcre2unicode(3) page.
Some applications that allow their users to supply
patterns may wish to restrict them to non-UTF data for
security reasons. If the PCRE2_NEVER_UTF option is passed
to pcre2_compile
(), (*UTF) is
not allowed, and its appearance in a pattern causes an
error.
Unicode property support
Another special sequence that may appear at the start of a pattern is (*UCP). This has the same effect as setting the PCRE2_UCP option: it causes sequences such as d and w to use Unicode properties to determine character types, instead of recognizing only characters with codes less than 256 via a lookup table.
Some applications that allow their users to supply
patterns may wish to restrict them for security reasons. If
the PCRE2_NEVER_UCP option is passed to pcre2_compile
(), (*UCP) is not allowed,
and its appearance in a pattern causes an error.
Locking out empty string matching
Starting a pattern with (*NOTEMPTY) or (*NOTEMPTY_ATSTART) has the same effect as passing the PCRE2_NOTEMPTY or PCRE2_NOTEMPTY_ATSTART option to whichever matching function is subsequently called to match the pattern. These options lock out the matching of empty strings, either entirely, or only at the start of the subject.
Disabling auto-possessification
If a pattern starts with (*NO_AUTO_POSSESS), it has the same effect as setting the PCRE2_NO_AUTO_POSSESS option. This stops PCRE2 from making quantifiers possessive when what follows cannot match the repeated item. For example, by default a+b is treated as a++b. For more details, see the pcre2api(3) documentation.
Disabling start-up optimizations
If a pattern starts with (*NO_START_OPT), it has the same effect as setting the PCRE2_NO_START_OPTIMIZE option. This disables several optimizations for quickly reaching no match results. For more details, see the pcre2api(3) documentation.
Disabling automatic anchoring
If a pattern starts with (*NO_DOTSTAR_ANCHOR), it has the same effect as setting the PCRE2_NO_DOTSTAR_ANCHOR option. This disables optimizations that apply to patterns whose top-level branches all start with .* (match any number of arbitrary characters). For more details, see the pcre2api(3) documentation.
Disabling JIT compilation
If a pattern that starts with (*NO_JIT) is successfully
compiled, an attempt by the application to apply the JIT
optimization by calling pcre2_jit_compile
() is ignored.
Setting match resource limits
The pcre2_match
() function
contains a counter that is incremented every time it goes
round its main loop. The caller of pcre2_match
() can set a limit on this
counter, which therefore limits the amount of computing
resource used for a match. The maximum depth of nested
backtracking can also be limited; this indirectly restricts
the amount of heap memory that is used, but there is also
an explicit memory limit that can be set.
These facilities are provided to catch runaway matches
that are provoked by patterns with huge matching trees. A
common example is a pattern with nested unlimited repeats
applied to a long string that does not match. When one of
these limits is reached, pcre2_match
() gives an error return. The
limits can also be set by items at the start of the pattern
of the form
(*LIMIT_HEAP=d) (*LIMIT_MATCH=d) (*LIMIT_DEPTH=d)
where d is any number of decimal digits. However, the
value of the setting must be less than the value set (or
defaulted) by the caller of pcre2_match
() for it to have any effect.
In other words, the pattern writer can lower the limits set
by the programmer, but not raise them. If there is more
than one setting of one of these limits, the lower value is
used. The heap limit is specified in kibibytes (units of
1024 bytes).
Prior to release 10.30, LIMIT_DEPTH was called LIMIT_RECURSION. This name is still recognized for backwards compatibility.
The heap limit applies only when the pcre2_match
() or pcre2_dfa_match
() interpreters are used
for matching. It does not apply to JIT. The match limit is
used (but in a different way) when JIT is being used, or
when pcre2_dfa_match
() is
called, to limit computing resource usage by those matching
functions. The depth limit is ignored by JIT but is
relevant for DFA matching, which uses function recursion
for recursions within the pattern and for lookaround
assertions and atomic groups. In this case, the depth limit
controls the depth of such recursion.
Newline conventions
PCRE2 supports six different conventions for indicating
line breaks in strings: a single CR (carriage return)
character, a single LF (linefeed) character, the
two-character sequence CRLF, any of the three preceding,
any Unicode newline sequence, or the NUL character (binary
zero). The pcre2api(3) page has
further discussion about newlines, and shows how to set the
newline convention when calling pcre2_compile
().
It is also possible to specify a newline convention by starting a pattern string with one of the following sequences:
(*CR) carriage return (*LF) linefeed (*CRLF) carriage return, followed by linefeed (*ANYCRLF) any of the three above (*ANY) all Unicode newline sequences (*NUL) the NUL character (binary zero)
These override the default and the options given to the compiling function. For example, on a Unix system where LF is the default newline sequence, the pattern
(*CR)a.b
changes the convention to CR. That pattern matches a b because LF is no longer a newline. If more than one of these settings is present, the last one is used.
The newline convention affects where the circumflex and dollar assertions are true. It also affects the interpretation of the dot metacharacter when PCRE2_DOTALL is not set, and the behaviour of N when not followed by an opening brace. However, it does not affect what the R escape sequence matches. By default, this is any Unicode newline sequence, for Perl compatibility. However, this can be changed; see the next section and the description of R in the section entitled Newline sequences below. A change of R setting can be combined with a change of newline convention.
Specifying what R matches
It is possible to restrict R to match only CR, LF, or CRLF (instead of the complete set of Unicode line endings) by setting the option PCRE2_BSR_ANYCRLF at compile time. This effect can also be achieved by starting a pattern with (*BSR_ANYCRLF). For completeness, (*BSR_UNICODE) is also recognized, corresponding to PCRE2_BSR_UNICODE.
EBCDIC CHARACTER CODES
PCRE2 can be compiled to run in an environment that uses EBCDIC as its character code instead of ASCII or Unicode (typically a mainframe system). In the sections below, character code values are ASCII or Unicode; in an EBCDIC environment these characters may have different code values, and there are no code points greater than 255.
CHARACTERS AND METACHARACTERS
A regular expression is a pattern that is matched against a subject string from left to right. Most characters stand for themselves in a pattern, and match the corresponding characters in the subject. As a trivial example, the pattern
The quick brown fox
matches a portion of a subject string that is identical to itself. When caseless matching is specified (the PCRE2_CASELESS option), letters are matched independently of case.
The power of regular expressions comes from the ability to
include wild cards, character classes, alternatives, and
repetitions in the pattern. These are encoded in the pattern
by the use of metacharacters
,
which do not stand for themselves but instead are interpreted
in some special way.
There are two different sets of metacharacters: those that are recognized anywhere in the pattern except within square brackets, and those that are recognized within square brackets. Outside square brackets, the metacharacters are as follows:
general escape character with several uses ^ assert start of string (or line, in multiline mode) $ assert end of string (or line, in multiline mode) . match any character except newline (by default) [ start character class definition | start of alternative branch ( start group or control verb ) end group or control verb * 0 or more quantifier + 1 or more quantifier; also possessive quantifier ? 0 or 1 quantifier; also quantifier minimizer { start min/max quantifier
Part of a pattern that is in square brackets is called a character class. In a character class the only metacharacters are:
general escape character ^ negate the class, but only if the first character - indicates character range [ POSIX character class (if followed by POSIX syntax) ] terminates the character class
The following sections describe the use of each of the metacharacters.
BACKSLASH
The backslash character has several uses. Firstly, if it is followed by a character that is not a digit or a letter, it takes away any special meaning that character may have. This use of backslash as an escape character applies both inside and outside character classes.
For example, if you want to match a * character, you must write * in the pattern. This escaping action applies whether or not the following character would otherwise be interpreted as a metacharacter, so it is always safe to precede a non-alphanumeric with backslash to specify that it stands for itself. In particular, if you want to match a backslash, you write \.
In a UTF mode, only ASCII digits and letters have any special meaning after a backslash. All other characters (in particular, those whose code points are greater than 127) are treated as literals.
If a pattern is compiled with the PCRE2_EXTENDED option, most white space in the pattern (other than in a character class), and characters between a # outside a character class and the next newline, inclusive, are ignored. An escaping backslash can be used to include a white space or # character as part of the pattern.
If you want to treat all characters in a sequence as literals, you can do so by putting them between Q and E. This is different from Perl in that $ and @ are handled as literals in Q...E sequences in PCRE2, whereas in Perl, $ and @ cause variable interpolation. Also, Perl does double-quotish backslash interpolation on any backslashes between Q and E which, its documentation says, may lead to confusing results. PCRE2 treats a backslash between Q and E just like any other character. Note the following examples:
Pattern PCRE2 matches Perl matches
Qabc$xyzE abc$xyz abc followed by the contents of $xyz Qabc$xyzE abc$xyz abc$xyz QabcE$QxyzE abc$xyz abc$xyz QABE AB AB Q\E \E
The Q...E sequence is recognized both inside and outside character classes. An isolated E that is not preceded by Q is ignored. If Q is not followed by E later in the pattern, the literal interpretation continues to the end of the pattern (that is, E is assumed at the end). If the isolated Q is inside a character class, this causes an error, because the character class is not terminated by a closing square bracket.
Non-printing characters
A second use of backslash provides a way of encoding non-printing characters in patterns in a visible manner. There is no restriction on the appearance of non-printing characters in a pattern, but when a pattern is being prepared by text editing, it is often easier to use one of the following escape sequences instead of the binary character it represents. In an ASCII or Unicode environment, these escapes are as follows:
a alarm, that is, the BEL character (hex 07) cx control-x, where x is any printable ASCII character e escape (hex 1B) f form feed (hex 0C) linefeed (hex 0A) carriage return (hex 0D) (but see below) tab (hex 09) dd character with octal code 0dd ddd character with octal code ddd, or backreference o{ddd..} character with octal code ddd.. xhh character with hex code hh x{hhh..} character with hex code hhh.. N{U+hhh..} character with Unicode hex code point hhh..
By default, after x that is not followed by {, from zero to two hexadecimal digits are read (letters can be in upper or lower case). Any number of hexadecimal digits may appear between x{ and }. If a character other than a hexadecimal digit appears between x{ and }, or if there is no terminating }, an error occurs.
Characters whose code points are less than 256 can be defined by either of the two syntaxes for x or by an octal sequence. There is no difference in the way they are handled. For example, xdc is exactly the same as x{dc} or 334. However, using the braced versions does make such sequences easier to read.
Support is available for some ECMAScript (aka JavaScript) escape sequences via two compile-time options. If PCRE2_ALT_BSUX is set, the sequence x followed by { is not recognized. Only if x is followed by two hexadecimal digits is it recognized as a character escape. Otherwise it is interpreted as a literal x character. In this mode, support for code points greater than 256 is provided by u, which must be followed by four hexadecimal digits; otherwise it is interpreted as a literal u character.
PCRE2_EXTRA_ALT_BSUX has the same effect as PCRE2_ALT_BSUX and, in addition, u{hhh..} is recognized as the character specified by hexadecimal code point. There may be any number of hexadecimal digits. This syntax is from ECMAScript 6.
The N{U+hhh..} escape sequence is recognized only when PCRE2 is operating in UTF mode. Perl also uses N{name} to specify characters by Unicode name; PCRE2 does not support this. Note that when N is not followed by an opening brace (curly bracket) it has an entirely different meaning, matching any character that is not a newline.
There are some legacy applications where the escape sequence is expected to match a newline. If the PCRE2_EXTRA_ESCAPED_CR_IS_LF option is set, in a pattern is converted to so that it matches a LF (linefeed) instead of a CR (carriage return) character.
The precise effect of cx on ASCII characters is as follows: if x is a lower case letter, it is converted to upper case. Then bit 6 of the character (hex 40) is inverted. Thus cA to cZ become hex 01 to hex 1A (A is 41, Z is 5A), but c{ becomes hex 3B ({ is 7B), and c; becomes hex 7B (; is 3B). If the code unit following c has a value less than 32 or greater than 126, a compile-time error occurs.
When PCRE2 is compiled in EBCDIC mode, N{U+hhh..} is
not supported. a, e, f,
,
, and generate the
appropriate EBCDIC code values. The c escape is processed
as specified for Perl in the perlebcdic
document. The only characters
that are allowed after c are A-Z, a-z, or one of @, [, ,
], ^, _, or ?. Any other character provokes a compile-time
error. The sequence [email protected] encodes character code 0; after c
the letters (in either case) encode characters 1-26 (hex 01
to hex 1A); [, , ], ^, and _ encode characters 27-31 (hex
1B to hex 1F), and c? becomes either 255 (hex FF) or 95
(hex 5F).
Thus, apart from c?, these escapes generate the same character code values as they do in an ASCII environment, though the meanings of the values mostly differ. For example, cG always generates code value 7, which is BEL in ASCII but DEL in EBCDIC.
The sequence c? generates DEL (127, hex 7F) in an ASCII environment, but because 127 is not a control character in EBCDIC, Perl makes it generate the APC character. Unfortunately, there are several variants of EBCDIC. In most of them the APC character has the value 255 (hex FF), but in the one Perl calls POSIX-BC its value is 95 (hex 5F). If certain other characters have POSIX-BC values, PCRE2 makes c? generate 95; otherwise it generates 255.
After up to two further octal digits are read. If there are fewer than two digits, just those that are present are used. Thus the sequence x 15 specifies two binary zeros followed by a CR character (code value 13). Make sure you supply two digits after the initial zero if the pattern character that follows is itself an octal digit.
The escape o must be followed by a sequence of octal digits, enclosed in braces. An error occurs if this is not the case. This escape is a recent addition to Perl; it provides way of specifying character code points as octal numbers greater than 0777, and it also allows octal numbers and backreferences to be unambiguously specified.
For greater clarity and unambiguity, it is best to avoid following by a digit greater than zero. Instead, use o{} or x{} to specify numerical character code points, and g{} to specify backreferences. The following paragraphs describe the old, ambiguous syntax.
The handling of a backslash followed by a digit other than 0 is complicated, and Perl has changed over time, causing PCRE2 also to change.
Outside a character class, PCRE2 reads the digit and any
following digits as a decimal number. If the number is less
than 10, begins with the digit 8 or 9, or if there are at
least that many previous capture groups in the expression,
the entire sequence is taken as a backreference
. A description of how this
works is given later, following the discussion of
parenthesized groups. Otherwise, up to three octal digits
are read to form a character code.
Inside a character class, PCRE2 handles 8 and 9 as the literal characters 8 and 9, and otherwise reads up to three octal digits following the backslash, using them to generate a data character. Any subsequent digits stand for themselves. For example, outside a character class: