[Tut] Python Regex Greedy vs Non-Greedy Quantifiers

[xSicKxBot]

Python Regex Greedy vs Non-Greedy Quantifiers

Ready to earn the black belt of your regex superpower? This tutorial shows you the subtle but important difference between greedy and non-greedy regex quantifiers.

But first things first: what are “quantifiers” anyway? Great question – I’m glad you asked! So let’s dive into Python’s three main regex quantifiers.

Python Regex Quantifiers

The word “quantifier” originates from latin: it’s meaning is quantus = how much / how often.

This is precisely what a regular expression quantifier means: you tell the regex engine how often you want to match a given pattern.

If you think you don’t define any quantifier, you do it implicitly: no quantifier means to match the regular expression exactly once.

So what are the regex quantifiers in Python?

Quantifier |
Meaning

Code:

A?

|
Match regular expression

Code:

A

zero or one times

Code:

A*

|
Match regular expression

Code:

A

zero or more times

Code:

A+

|
Match regular expression

Code:

A

one or more times

Code:

A{m}

|
Match regular expression

Code:

A

exactly m times

Code:

A{m,n}

|
Match regular expression

Code:

A

between m and n times (included)

Note that in this tutorial, I assume you have at least a remote idea of what regular expressions actually are. If you haven’t, no problem, check out my detailed regex tutorial on this blog.

You see in the table that the quantifiers

Code:

?

,

Code:

*

,

Code:

+

,

Code:

{m}

, and

Code:

{m,n}

define how often you repeat the matching of regex

Code:

A

.

Let’s have a look at some examples—one for each quantifier:

Code:

>>> import re >>> re.findall('a?', 'aaaa') ['a', 'a', 'a', 'a', ''] >>> re.findall('a*', 'aaaa') ['aaaa', ''] >>> re.findall('a+', 'aaaa') ['aaaa'] >>> re.findall('a{3}', 'aaaa') ['aaa'] >>> re.findall('a{1,2}', 'aaaa') ['aa', 'aa']

In each line, you try a different quantifier on the same text

Code:

'aaaa'

. And, interestingly, each line leads to a different output:

• The zero-or-one regex
  Code:

  'a?'
  matches four times one
  Code:

  'a'
  . Note that it doesn’t match zero characters if it can avoid doing so.
  
  
• The zero-or-more regex
  Code:

  'a*'
  matches once four
  Code:

  'a'
  s and consumes them. At the end of the string, it can still match the empty string.
  
  
• The one-or-more regex
  Code:

  'a+'
  matches once four
  Code:

  'a'
  s. In contrast to the previous quantifier, it cannot match an empty string.
  
  
• The repeating regex
  Code:

  'a{3}'
  matches up to three
  Code:

  'a'
  s in a single run. It can do so only once.
  
  
• The repeating regex
  Code:

  'a{1,2}'
  matches one or two
  Code:

  'a'
  s. It tries to match as many as possible.
  

You’ve learned the basic quantifiers of Python regular expressions. Now, it’s time to explore the meaning of the term greedy. Shall we?

Python Regex Greedy Match

A greedy match means that the regex engine (the one which tries to find your pattern in the string) matches as many characters as possible.

For example, the regex

Code:

'a+'

will match as many

Code:

'a'

s as possible in your string

Code:

'aaaa'

. Although the substrings

Code:

'a'

,

Code:

'aa'

,

Code:

'aaa'

all match the regex

Code:

'a+'

, it’s not enough for the regex engine. It’s always hungry and tries to match even more.



In other words, the greedy quantifiers give you the longest match from a given position in the string.

As it turns out, all default quantifiers

Code:

?

,

Code:

*

,

Code:

+

,

Code:

{m}

, and

Code:

{m,n}

you’ve learned above are greedy: they “consume” or match as many characters as possible so that the regex pattern is still satisfied.

Here are the above examples again that all show how greedy the regex engine is:

Code:

>>> import re >>> re.findall('a?', 'aaaa') ['a', 'a', 'a', 'a', ''] >>> re.findall('a*', 'aaaa') ['aaaa', ''] >>> re.findall('a+', 'aaaa') ['aaaa'] >>> re.findall('a{3}', 'aaaa') ['aaa'] >>> re.findall('a{1,2}', 'aaaa') ['aa', 'aa']

In all cases, a shorter match would also be valid. But as the regex engine is greedy per default, those are not enough for the regex engine.

Okay, so how can we do a non-greedy match?

Python Regex Non-Greedy Match

A non-greedy match means that the regex engine matches as few characters as possible—so that it still can match the pattern in the given string.

For example, the regex

Code:

'a+?'

will match as few

Code:

'a'

s as possible in your string

Code:

'aaaa'

. Thus, it matches the first character

Code:

'a'

and is done with it. Then, it moves on to the second character (which is also a match) and so on.

In other words, the non-greedy quantifiers give you the shortest possible match from a given position in the string.

You can make the default quantifiers

Code:

?

,

Code:

*

,

Code:

+

,

Code:

{m}

, and

Code:

{m,n}

non-greedy by appending a question mark symbol

Code:

'?'

to them:

Code:

??

,

Code:

*?

,

Code:

+?

, and

Code:

{m,n}?

. they “consume” or match as few characters as possible so that the regex pattern is still satisfied.

Here are some examples that show how non-greedy matching works:

Non-Greedy Question Mark Operator (??)

Let’s start with the question mark (zero-or-one operator):

Code:

>>> import re >>> re.findall('a?', 'aaaa') ['a', 'a', 'a', 'a', ''] >>> re.findall('a??', 'aaaa') ['', 'a', '', 'a', '', 'a', '', 'a', '']

In the first instance, you use the zero-or-one regex

Code:

'a?'

. It’s greedy so it matches one

Code:

'a'

character if possible.

In the second instance, you use the non-greedy zero-or-one version

Code:

'a??'

. It matches zero

Code:

'a'

s if possible. Note that it moves from left to right so it matches the empty string and “consumes” it. Only then, it cannot match the empty string anymore so it is forced to match the first

Code:

'a'

character. But after that, it’s free to match the empty string again. This pattern of first matching the empty string and only then matching the

Code:

'a'

if it is absolutely needed repeats. That’s why this strange pattern occurs.

Non-Greedy Asterisk Operator (*?)

Let’s start with the asterisk (zero-or-more operator):

Code:

>>> re.findall('a*', 'aaaa') ['aaaa', ''] >>> re.findall('a*?', 'aaaa') ['', 'a', '', 'a', '', 'a', '', 'a', '']

First, you use the zero-or-more asterisk regex

Code:

'a*'

. It’s greedy so it matches as many

Code:

'a'

characters as it can.

Second, you use the non-greedy zero-or-one version

Code:

'a*?'

. Again, it matches zero

Code:

'a'

s if possible. Only if it has already matched zero characters at a certain position, it matches one character at that position, “consumes” it, and moves on.

Non-Greedy Plus Operator (+?)

Let’s start with the plus (one-or-more operator):

Code:

>>> re.findall('a+', 'aaaa') ['aaaa'] >>> re.findall('a+?', 'aaaa') ['a', 'a', 'a', 'a']

First, you use the one-or-more plus regex

Code:

'a+'

. It’s greedy so it matches as many

Code:

'a'

characters as it can (but at least one).

Second, you use the non-greedy one-or-more version

Code:

'a+?'

. In this case, the regex engine matches only one character

Code:

'a'

, consumes it, and moves on with the next match.

Let’s summarize what you’ve learned so far:

Greedy vs Non-Greedy Match – What’s the Difference?

Given a pattern with a quantifier (e.g. the asterisk operator) that allows the regex engine to match the pattern multiple times.

A given string may match the regex in multiple ways. For example, both substrings

Code:

'a'

and

Code:

'aaa'

are valid matches when matching the pattern

Code:

'a*'

in the string

Code:

'aaaa'

.

So the difference between the greedy and the non-greedy match is the following: The greedy match will try to match as many repetitions of the quantified pattern as possible. The non-greedy match will try to match as few repetitions of the quantified pattern as possible.

Examples Greedy vs Non-Greedy Match

Let’s consider a range of examples that help you understand the difference between greedy and non-greedy matches in Python:

Code:

>>> import re >>> re.findall('a+', 'aaaa') ['aaaa'] >>> re.findall('a+?', 'aaaa') ['a', 'a', 'a', 'a'] >>> re.findall('a*', 'aaaa') ['aaaa', ''] >>> re.findall('a*?', 'aaaa') ['', 'a', '', 'a', '', 'a', '', 'a', ''] >>> re.findall('a?', 'aaaa') ['a', 'a', 'a', 'a', ''] >>> re.findall('a??', 'aaaa') ['', 'a', '', 'a', '', 'a', '', 'a', '']

Make sure you completely understand those examples before you move on. If you don’t, please read the previous paragraphs again.

Which is Faster: Greedy vs Non-Greedy?

Considering that greedy quantifiers match a maximal and non-greedy a minimal number of patterns, is there any performance difference?

Great question!

Indeed, some benchmarks suggest that there’s a significant performance difference: the greedy quantifier is 100% slower in realistic experiments on benchmark data.

So if you optimize for speed and you don’t care about greedy or non-greedy matches—and you don’t know anything else—go for the non-greedy quantifier!

However, the truth is not as simple. For example, consider the following basic experiment that falsifies the previous hypothesis that the non-greedy version is faster:

Code:

>>> import timeit >>> timeit.timeit('import re;re.findall("a*", "aaaaaaaaaaaa")') 1.0579840000000331 >>> timeit.timeit('import re;re.findall("a*?", "aaaaaaaaaaaa")') 3.7830938000000742

I used the speed testing tool timeitthat allows to throw in some simple Python statements and check how long they run. Per default, the passed statement is executed 1,000,000 times.

You can see a notable performance difference of more than 300%! The non-greedy version is three times slower than the greedy version.

Why is that?

The reason is the re.findall() method that returns a list of matching substrings. Here’s the output both statements would produce:

Code:

>>> re.findall("a*", "aaaaaaaaaaaa") ['aaaaaaaaaaaa', ''] >>> re.findall("a*?", "aaaaaaaaaaaa") ['', 'a', '', 'a', '', 'a', '', 'a', '', 'a', '', 'a', '', 'a', '', 'a', '', 'a', '', 'a', '', 'a', '', 'a', '']

You can see that the greedy version finds one match and is done with it. The non-greedy version finds 25 matches which leads to far more processing and memory overhead.

So what happens if you use the re.search() method that returns only the first match rather than the re.findall() method that returns all matches?

Code:

>>> timeit.timeit('import re;re.search("a*", "aaaaaaaaaaaa")') 0.8420328999998219 >>> timeit.timeit('import re;re.search("a*?", "aaaaaaaaaaaa")') 0.7955709000000297

As expected, this changes things again. Both regex searches yield a single result, but the non-greedy match is much shorter: it matches the empty string

Code:

''

rather than the whole string

Code:

'aaaaaaaaaaaa'

. Of course, this is a bit faster.

However, the difference is negligible in this minimal example.

There’s More: Greedy, Docile, Lazy, Helpful, Possessive Match

In this article, I’ve classified the regex world into greedy and non-greedy quantifiers. But you can differentiate the “non-greedy” class even more!

Next, I’ll give you a short overview based on this great article of the most important terms in this regard:

• Greedy: match as many instances of the quantified pattern as you can.
  
  
• Docile: match as many instances of the quantified pattern as long as it still matches the overall pattern—if this is possible. Note that what I called “greedy” in this article is really “docile”.
  
  
• Lazy: match as few instances of the quantified pattern as needed. This is what I called “non-greedy” in this article.
  
  
• Possessive: never gives up a partial match. So the regex engine may not even find a match that actually exist—just because it’s so greedy. This is very unusual and you won’t see it a lot in practice.
  

If you want to learn more about those, I’d recommend that you read this excellent online tutorial.

Where to Go From Here

Summary: You’ve learned that the greedy quantifiers

Code:

?

,

Code:

*

, and

Code:

+

match as many repetitions of the quantified pattern as possible. The non-greedy quantifiers

Code:

??

,

Code:

*?

, and

Code:

+?

match as few repetitions of the quantified pattern as possible.

If you want to master Python and regular expressions, join my free email academy—it’s fun!