Category: Programming

Regular expressions are a strange animal. Many students find them difficult to understand – do you?

I realized that a major reason for this is simply that they don’t understand the special regex characters. To put it differently: understand the special characters and everything else in the regex space will come much easier to you.

Regular expressions are built from characters. There are two types of characters: literal characters and special characters.

Literal Characters

Let’s start with the absolute first thing you need to know with regular expressions: a regular expression (short: regex) searches for a given pattern in a given string.

What’s a pattern? In its most basic form, a pattern can be a literal character. So the literal characters 'a', 'b', and 'c' are all valid regex patterns.

For example, you can search for the regex pattern 'a' in the string 'hello world' but it won’t find a match. You can also search for the pattern 'a' in the string 'hello woman' and there is a match: the second last character in the string.

Based on the simple insight that a literal character is a valid regex pattern, you’ll find that a combination of literal characters is also a valid regex pattern. For example, the regex pattern 'an' matches the last two characters in the string 'hello woman'.

Summary: Regular expressions are built from characters. An important class of characters are the literal characters. In principle, you can use all Unicode literal characters in your regex pattern.

Special Characters

However, the power of regular expressions come from their abstraction capability. Instead of writing the character set [abcdefghijklmnopqrstuvwxyz], you’d write [a-z] or even \w. The latter is a special regex character—and pros know them by heart. In fact, regex experts seldomly match literal characters. In most cases, they use more advanced constructs or special characters for various reasons such as brevity, expressiveness, or generality.

So what are the special characters you can use in your regex patterns?

Let’s have a look at the following table that contains all special characters in Python’s re package for regular expression processing.

But these are not all characters you can use in a regular expression.

There are also meta characters for the regex engine that allow you to do much more powerful stuff.

A good example is the asterisk operator that matches “zero or more” occurrences of the preceding regex. For example, the pattern .*txt matches an arbitrary number of arbitrary characters followed by the suffix 'txt'. This pattern has two special regex meta characters: the dot . and the asterisk operator *. You’ll now learn about those meta characters:

Regex Meta Characters

Feel free to watch the short video about the most important regex meta characters:

Next, you’ll get a quick and dirty overview of the most important regex operations and how to use them in Python.

Here are the most important regex operators:

Note that I gave the above operators some more meaningful names (in bold) so that you can immediately grasp the purpose of each regex. For example, the ‘^’ operator is usually denoted as the ‘caret’ operator. Those names are not descriptive so I came up with more kindergarten-like words such as the “start-of-string” operator.

Let’s dive into some examples!

Examples

import re text = ''' Ha! let me see her: out, alas! he's cold: Her blood is settled, and her joints are stiff; Life and these lips have long been separated: Death lies on her like an untimely frost Upon the sweetest flower of all the field. ''' print(re.findall('.a!', text)) '''
Finds all occurrences of an arbitrary character that is
followed by the character sequence 'a!'.
['Ha!'] ''' print(re.findall('is.*and', text)) '''
Finds all occurrences of the word 'is',
followed by an arbitrary number of characters
and the word 'and'.
['is settled, and'] ''' print(re.findall('her:?', text)) '''
Finds all occurrences of the word 'her',
followed by zero or one occurrences of the colon ':'.
['her:', 'her', 'her'] ''' print(re.findall('her:+', text)) '''
Finds all occurrences of the word 'her',
followed by one or more occurrences of the colon ':'.
['her:'] ''' print(re.findall('^Ha.*', text)) '''
Finds all occurrences where the string starts with
the character sequence 'Ha', followed by an arbitrary
number of characters except for the new-line character. Can you figure out why Python doesn't find any?
[] ''' print(re.findall('\n$', text)) '''
Finds all occurrences where the new-line character '\n'
occurs at the end of the string.
['\n'] ''' print(re.findall('(Life|Death)', text)) '''
Finds all occurrences of either the word 'Life' or the
word 'Death'.
['Life', 'Death'] '''

In these examples, you’ve already seen the special symbol \n which denotes the new-line character in Python (and most other languages). There are many special characters, specifically designed for regular expressions.

Where to Go From Here

You’ve learned all special characters of regular expressions, as well as meta characters. This will give you a strong basis for improving your regex skills.

If you want to accelerate your skills, you need a good foundation. Check out my brand-new Python book “Python One-Liners (Amazon Link)” which boosts your skills from zero to hero—in a single line of Python code!

This tutorial explains everything you need to know about matching groups in Python’s re package for regular expressions. You may have also read the term “capture groups” which points to the same concept.

As you read through the tutorial, you can also watch the tutorial video where I explain everything in a simple way:

So let’s start with the basics:

Matching Group ()

What’s a matching group?

Like you use parentheses to structure mathematical expressions, (2 + 2) * 2 versus 2 + (2 * 2), you use parentheses to structure regular expressions. An example regex that does this is 'a(b|c)'. The whole content enclosed in the opening and closing parentheses is called matching group (or capture group). You can have multiple matching groups in a single regex. And you can even have hierarchical matching groups, for example 'a(b|(cd))'.

One big advantage of a matching group is that it captures the matched substring. You can retrieve it in other parts of the regular expression—or after analyzing the result of the whole regex matching.

Let’s have a short example for the most basic use of a matching group—to structure the regex.

Say you create regex b?(a.)* with the matching group (a.) that matches all patterns starting with zero or one occurrence of character 'b' and an arbitrary number of two-character-sequences starting with the character 'a'. Hence, the strings 'bacacaca', 'aaaa', '' (the empty string), and 'Xababababab' all match your regex.

The use of the parentheses for structuring the regular expression is intuitive and should come naturally to you because the same rules apply as for arithmetic operations. However, there’s a more advanced use of regex groups: retrieval.

You can retrieve the matched content of each matching group. So the next question naturally arises:

How to Get the First Matching Group?

There are two scenarios when you want to access the content of your matching groups:

1. Access the matching group in the regex pattern to reuse partially matched text from one group somewhere else.
2. Access the matching group after the whole match operation to analyze the matched text in your Python code.

In the first case, you simply get the first matching group with the \number special sequence. For example, to get the first matching group, you’d use the \1 special sequence. Here’s an example:

>>> import re
>>> re.search(r'(j.n) is \1','jon is jon')
<re.Match object; span=(0, 10), match='jon is jon'>

You’ll use this feature a lot because it gives you much more expression power: for example, you can search for a name in a text-based on a given pattern and then process specifically this name in the rest of the text (and not all other names that would also fit the pattern).

Note that the numbering of the groups start with \1 and not with \0—a rare exception to the rule that in programming, all numbering starts with 0.

In the second case, you want to know the contents of the first group after the whole match. How do you do that?

The answer is also simple: use the m.group(0) method on the matching object m. Here’s an example:

>>> import re
>>> m = re.search(r'(j.n)','jon is jon')
>>> m.group(1) 'jon'

The numbering works consistently with the previously introduced regex group numbering: start with identifier 1 to access the contents of the first group.

How to Get All Other Matching Groups?

Again, there are two different intentions when asking this question:

1. Access the matching group in the regex pattern to reuse partially matched text from one group somewhere else.
2. Access the matching group after the whole match operation to analyze the matched text in your Python code.

In the first case, you use the special sequence \2 to access the second matching group, \3 to access the third matching group, and \99 to access the ninety-ninth matching group.

Here’s an example:

>>> import re
>>> re.search(r'(j..) (j..)\s+\2', 'jon jim jim')
<re.Match object; span=(0, 11), match='jon jim jim'>
>>> re.search(r'(j..) (j..)\s+\2', 'jon jim jon')
>>> 

As you can see, the special sequence \2 refers to the matching contents of the second group 'jim'.

In the second case, you can simply increase the identifier too to access the other matching groups in your Python code:

>>> import re
>>> m = re.search(r'(j..) (j..)\s+\2', 'jon jim jim')
>>> m.group(0) 'jon jim jim'
>>> m.group(1) 'jon'
>>> m.group(2) 'jim'

This code also shows an interesting feature: if you use the identifier 0 as an argument to the m.group(0) method, the regex module will give you the contents of the whole match. You can think of it as the first group being the whole match.

Named Groups: (?P<name>…) and (?P=name)

Accessing the captured group using the notation \number is not always convenient and sometimes not even possible (for example if you have more than 99 groups in your regex). A major disadvantage of regular expressions is that they tend to be hard to read. It’s therefore important to know about the different tweaks to improve readability.

One such optimization is a named group. It’s really just that: a matching group that captures part of the match but with one twist: it has a name. Now, you can use this name to access the captured group at a later point in your regular expression pattern. This can improve readability of the regular expression.

import re
pattern = '(?P<quote>["\']).*(?P=quote)'
text = 'She said "hi"'
print(re.search(pattern, text))
# <re.Match object; span=(9, 13), match='"hi"'>

The code searches for substrings that are enclosed in either single or double quotes. You first match the opening quote by using the regex ["\']. You escape the single quote, \' so that the Python regex engine does not assume (wrongly) that the single quote indicates the end of the string. You then use the same group to match the closing quote of the same character (either a single or double quote).

Non-Capturing Groups (?:…)

In the previous examples, you’ve seen how to match and capture groups with the parentheses (...). You’ve learned that each match of this basic group operator is captured so that you can retrieve it later in the regex with the special commands \1, \2, …, \99 or after the match on the matched object m with the method m.group(1), m.group(2), and so on.

But what if you don’t need that? What if you just need to keep your regex pattern in order—but you don’t want to capture the contents of a matching group?

The simple solution is the non-capturing group operation (?: ... ). You can use it just like the capturing group operation ( ... ). Here’s an example:

>>>import re
>>> re.search('(?:python|java) is great', 'python is great. java is great.')
<re.Match object; span=(0, 15), match='python is great'>

The non-capturing group exists with the sole purpose to structure the regex. You cannot use its content later:

>>> m = re.search('(?:python|java) is great', 'python is great. java is great.')
>>> m.group(1)
Traceback (most recent call last): File "<pyshell#28>", line 1, in <module> m.group(1)
IndexError: no such group
>>> 

If you try to access the contents of the non-capturing group, the regex engine will throw an IndexError: no such group.

Of course, there’s a straightforward alternative to non-capturing groups. You can simply use the normal (capturing) group but don’t access its contents. Only rarely will the performance penalty of capturing a group that’s not needed have any meaningful impact on your overall application.

Positive Lookahead (?=…)

The concept of lookahead is a very powerful one and any advanced coder should know it. A friend recently told me that he had written a complicated regex that ignores the order of occurrences of two words in a given text. It’s a challenging problem and without the concept of lookahead, the resulting code will be complicated and hard to understand. However, the concept of lookahead makes this problem simple to write and read.

But first things first: how does the lookahead assertion work?

In normal regular expression processing, the regex is matched from left to right. The regex engine “consumes” partially matching substrings. The consumed substring cannot be matched by any other part of the regex.

Figure: A simple example of lookahead. The regular expression engine matches (“consumes”) the string partially. Then it checks whether the remaining pattern could be matched without actually matching it.

Think of the lookahead assertion as a non-consuming pattern match. The regex engine goes from the left to the right—searching for the pattern. At each point, it has one “current” position to check if this position is the first position of the remaining match. In other words, the regex engine tries to “consume” the next character as a (partial) match of the pattern.

The advantage of the lookahead expression is that it doesn’t consume anything. It just “looks ahead” starting from the current position whether what follows would theoretically match the lookahead pattern. If it doesn’t, the regex engine cannot move on. Next, it “backtracks”—which is just a fancy way of saying: it goes back to a previous decision and tries to match something else.

Positive Lookahead Example: How to Match Two Words in Arbitrary Order?

What if you want to search a given text for pattern A AND pattern B—but in no particular order? If both patterns appear anywhere in the string, the whole string should be returned as a match.

Now, this is a bit more complicated because any regular expression pattern is ordered from left to right. A simple solution is to use the lookahead assertion (?.*A) to check whether regex A appears anywhere in the string. (Note we assume a single line string as the .* pattern doesn’t match the newline character by default.)

Let’s first have a look at the minimal solution to check for two patterns anywhere in the string (say, patterns ‘hi’ AND ‘you’).

>>> import re
>>> pattern = '(?=.*hi)(?=.*you)'
>>> re.findall(pattern, 'hi how are yo?')
[]
>>> re.findall(pattern, 'hi how are you?')
['']

In the first example, both words do not appear. In the second example, they do.

Let’s go back to the expression (?=.*hi)(?=.*you) to match strings that contain both ‘hi’ and ‘you’. Why does it work?

The reason is that the lookahead expressions don’t consume anything. You first search for an arbitrary number of characters .*, followed by the word hi. But because the regex engine hasn’t consumed anything, it’s still in the same position at the beginning of the string. So, you can repeat the same for the word you.

Note that this method doesn’t care about the order of the two words:

>>> import re
>>> pattern = '(?=.*hi)(?=.*you)'
>>> re.findall(pattern, 'hi how are you?')
['']
>>> re.findall(pattern, 'you are how? hi!')
['']

No matter which word “hi” or “you” appears first in the text, the regex engine finds both.

You may ask: why’s the output the empty string? The reason is that the regex engine hasn’t consumed any character. It just checked the lookaheads. So the easy fix is to consume all characters as follows:

>>> import re
>>> pattern = '(?=.*hi)(?=.*you).*'
>>> re.findall(pattern, 'you fly high')
['you fly high']

Now, the whole string is a match because after checking the lookahead with ‘(?=.*hi)(?=.*you)’, you also consume the whole string ‘.*’.

Negative Lookahead (?!…)

The negative lookahead works just like the positive lookahead—only it checks that the given regex pattern does not occur going forward from a certain position.

Here’s an example:

>>> import re
>>> re.search('(?!.*hi.*)', 'hi say hi?')
<re.Match object; span=(8, 8), match=''>

The negative lookahead pattern (?!.*hi.*) ensures that, going forward in the string, there’s no occurrence of the substring 'hi'. The first position where this holds is position 8 (right after the second 'h'). Like the positive lookahead, the negative lookahead does not consume any character so the result is the empty string (which is a valid match of the pattern).

You can even combine multiple negative lookaheads like this:

>>> re.search('(?!.*hi.*)(?!\?).', 'hi say hi?')
<re.Match object; span=(8, 9), match='i'>

You search for a position where neither ‘hi’ is in the lookahead, nor does the question mark character follow immediately. This time, we consume an arbitrary character so the resulting match is the character 'i'.

Group Flags (?aiLmsux:…) and (?aiLmsux)

You can control the regex engine with the flags argument of the re.findall(), re.search(), or re.match() methods. For example, if you don’t care about capitalization of your matched substring, you can pass the re.IGNORECASE flag to the regex methods:

>>> re.findall('PYTHON', 'python is great', flags=re.IGNORECASE)
['python']

But using a global flag for the whole regex is not always optimal. What if you want to ignore the capitalization only for a certain subregex?

You can do this with the group flags: a, i, L, m, s, u, and x. Each group flag has its own meaning:

For example, if you want to switch off the differentiation of capitalization, you’ll use the i flag as follows:

>>> re.findall('(?i:PYTHON)', 'python is great')
['python']

You can also switch off the capitalization for the whole regex with the “global group flag” (?i) as follows:

>>> re.findall('(?i)PYTHON', 'python is great')
['python']

Where to Go From Here?

Summary: You’ve learned about matching groups to structure the regex and capture parts of the matching result. You can then retrieve the captured groups with the \number syntax within the regex pattern itself and with the m.group(i) syntax in the Python code at a later stage.

To learn the Python basics, check out my free Python email academy with many advanced courses—including a regex video tutorial in your INBOX.

Join 20,000+ ambitious coders for free!

Morse code was developed in telecommunications to encode the alphabetic characters in binary signals (“long” and “short”, or “1” and “0”).

Here’s the Morse code table:

This tutorial shows you how to implement a simple Morse code translator in Python.

The code is contributed by Finxter student Albrecht.

Goal: Given a string that is either Morse code or normal text. Write a function that transforms the string into the other language: Morse code should be translated to normal text. Normal text should be translated to Morse code.

Output Example: Create a function morse(txt) that takes an input string argument txt and returns its translation:

>>> morse('python') '.--. -.-- - .... --- -.'
>>> morse('.--. -.-- - .... --- -.') 'PYTHON'
>>> morse(morse('HEY')) 'HEY'

Note that Morse code doesn’t differentiate lowercase or uppercase characters. So you just use uppercase characters as default translation output.

Algorithm Idea: A simple algorithm is enough to solve the problem:

• Detect if a string is Morse code or normal text. The simple but not perfect solution is to check if the first character is either the dot symbol '.' or the minus symbol '-'. Note that you can easily extend this by checking if all characters are either the dot symbol or the minus symbol (a simple regular expression will be enough).
• Prepare a dictionary that maps all “normal text” symbols to their respective Morse code translations. Use the inverse dictionary (or create it ad-hoc) to get the inverse mapping.
• Iterate over all characters in the string and use the dictionary to translate each character separately.

Implementation: Here’s the Python implementation of the above algorithm for Morse code translation:

def morse(txt): '''Morse code encryption and decryption''' d = {'A':'.-','B':'-...','C':'-.-.','D':'-..','E':'.', 'F':'..-.','G':'--.','H':'....','I':'..','J':'.---', 'K':'-.-','L':'.-..','M':'--','N':'-.','O':'---', 'P':'.--.','Q':'--.-','R':'.-.','S':'...','T':'-', 'U':'..-','V':'...-','W':'.--','X':'-..-','Y':'-.--', 'Z':'--..', ' ':'.....'} translation = '' # Encrypt Morsecode if txt.startswith('.') or txt.startswith('−'): # Swap key/values in d: d_encrypt = dict([(v, k) for k, v in d.items()]) # Morse code is separated by empty space chars txt = txt.split(' ') for x in txt: translation += d_encrypt.get(x) # Decrypt to Morsecode: else: txt = txt.upper() for x in txt: translation += d.get(x) + ' ' return translation.strip() print(morse('python'))
# .--. -.-- - .... --- -.
print(morse('.--. -.-- - .... --- -.'))
# PYTHON
print(morse(morse('HEY')))
# HEY

Algorithmic complexity: The runtime complexity is linear in the length of the input string to be translated—one translation operation per character. Dictionary membership has constant runtime complexity. The memory overhead is also linear in the input text as all the characters have to be hold in memory.

Alternative Implementation: Albrecht also proposed a much shorter alternative:

def morse(txt): encrypt = {'A':'.-', 'B':'-...', 'C':'-.-.', 'D':'-..', 'E':'.', 'F':'..-.', 'G':'--.', 'H':'....', 'I':'..', 'J':'.---', 'K':'-.-', 'L':'.-..', 'M':'--', 'N':'-.', 'O':'---', 'P':'.--.', 'Q':'--.-', 'R':'.-.', 'S':'...', 'T':'-', 'U':'..-', 'V':'...-', 'W':'.--', 'X':'-..-', 'Y':'-.--', 'Z':'--..', ' ':'.....'} decrypt = {v: k for k, v in encrypt.items()} if '-' in txt: return ''.join(decrypt[i] for i in txt.split()) return ' '.join(encrypt[i] for i in txt.upper()) print(morse('python'))
# .--. -.-- - .... --- -.
print(morse('.--. -.-- - .... --- -.'))
# PYTHON
print(morse(morse('HEY')))
# HEY

It uses dict comprehension and generator expressions to make it much more concise.

Ready to earn the black belt of your regex superpower? This tutorial shows you the subtle but important difference between greedy and non-greedy regex quantifiers.

But first things first: what are “quantifiers” anyway? Great question – I’m glad you asked! So let’s dive into Python’s three main regex quantifiers.

Python Regex Quantifiers

The word “quantifier” originates from latin: it’s meaning is quantus = how much / how often.

This is precisely what a regular expression quantifier means: you tell the regex engine how often you want to match a given pattern.

If you think you don’t define any quantifier, you do it implicitly: no quantifier means to match the regular expression exactly once.

So what are the regex quantifiers in Python?

Note that in this tutorial, I assume you have at least a remote idea of what regular expressions actually are. If you haven’t, no problem, check out my detailed regex tutorial on this blog.

You see in the table that the quantifiers ?, *, +, {m}, and {m,n} define how often you repeat the matching of regex A.

Let’s have a look at some examples—one for each quantifier:

>>> import re
>>> re.findall('a?', 'aaaa')
['a', 'a', 'a', 'a', '']
>>> re.findall('a*', 'aaaa')
['aaaa', '']
>>> re.findall('a+', 'aaaa')
['aaaa']
>>> re.findall('a{3}', 'aaaa')
['aaa']
>>> re.findall('a{1,2}', 'aaaa')
['aa', 'aa']

In each line, you try a different quantifier on the same text 'aaaa'. And, interestingly, each line leads to a different output:

• The zero-or-one regex 'a?' matches four times one 'a'. Note that it doesn’t match zero characters if it can avoid doing so.
• The zero-or-more regex 'a*' matches once four 'a's and consumes them. At the end of the string, it can still match the empty string.
• The one-or-more regex 'a+' matches once four 'a's. In contrast to the previous quantifier, it cannot match an empty string.
• The repeating regex 'a{3}' matches up to three 'a's in a single run. It can do so only once.
• The repeating regex 'a{1,2}' matches one or two 'a's. It tries to match as many as possible.

You’ve learned the basic quantifiers of Python regular expressions. Now, it’s time to explore the meaning of the term greedy. Shall we?

Python Regex Greedy Match

A greedy match means that the regex engine (the one which tries to find your pattern in the string) matches as many characters as possible.

For example, the regex 'a+' will match as many 'a's as possible in your string 'aaaa'. Although the substrings 'a', 'aa', 'aaa' all match the regex 'a+', it’s not enough for the regex engine. It’s always hungry and tries to match even more.

In other words, the greedy quantifiers give you the longest match from a given position in the string.

As it turns out, all default quantifiers ?, *, +, {m}, and {m,n} you’ve learned above are greedy: they “consume” or match as many characters as possible so that the regex pattern is still satisfied.

Here are the above examples again that all show how greedy the regex engine is:

>>> import re
>>> re.findall('a?', 'aaaa')
['a', 'a', 'a', 'a', '']
>>> re.findall('a*', 'aaaa')
['aaaa', '']
>>> re.findall('a+', 'aaaa')
['aaaa']
>>> re.findall('a{3}', 'aaaa')
['aaa']
>>> re.findall('a{1,2}', 'aaaa')
['aa', 'aa']

In all cases, a shorter match would also be valid. But as the regex engine is greedy per default, those are not enough for the regex engine.

Okay, so how can we do a non-greedy match?

Python Regex Non-Greedy Match

A non-greedy match means that the regex engine matches as few characters as possible—so that it still can match the pattern in the given string.

For example, the regex 'a+?' will match as few 'a's as possible in your string 'aaaa'. Thus, it matches the first character 'a' and is done with it. Then, it moves on to the second character (which is also a match) and so on.

In other words, the non-greedy quantifiers give you the shortest possible match from a given position in the string.

You can make the default quantifiers ?, *, +, {m}, and {m,n} non-greedy by appending a question mark symbol '?' to them: ??, *?, +?, and {m,n}?. they “consume” or match as few characters as possible so that the regex pattern is still satisfied.

Here are some examples that show how non-greedy matching works:

Non-Greedy Question Mark Operator (??)

Let’s start with the question mark (zero-or-one operator):

>>> import re
>>> re.findall('a?', 'aaaa')
['a', 'a', 'a', 'a', '']
>>> re.findall('a??', 'aaaa')
['', 'a', '', 'a', '', 'a', '', 'a', '']

In the first instance, you use the zero-or-one regex 'a?'. It’s greedy so it matches one 'a' character if possible.

In the second instance, you use the non-greedy zero-or-one version 'a??'. It matches zero 'a's if possible. Note that it moves from left to right so it matches the empty string and “consumes” it. Only then, it cannot match the empty string anymore so it is forced to match the first 'a' character. But after that, it’s free to match the empty string again. This pattern of first matching the empty string and only then matching the 'a' if it is absolutely needed repeats. That’s why this strange pattern occurs.

Non-Greedy Asterisk Operator (*?)

Let’s start with the asterisk (zero-or-more operator):

>>> re.findall('a*', 'aaaa')
['aaaa', '']
>>> re.findall('a*?', 'aaaa')
['', 'a', '', 'a', '', 'a', '', 'a', '']

First, you use the zero-or-more asterisk regex 'a*'. It’s greedy so it matches as many 'a' characters as it can.

Second, you use the non-greedy zero-or-one version 'a*?'. Again, it matches zero 'a's if possible. Only if it has already matched zero characters at a certain position, it matches one character at that position, “consumes” it, and moves on.

Non-Greedy Plus Operator (+?)

Let’s start with the plus (one-or-more operator):

>>> re.findall('a+', 'aaaa')
['aaaa']
>>> re.findall('a+?', 'aaaa')
['a', 'a', 'a', 'a']

First, you use the one-or-more plus regex 'a+'. It’s greedy so it matches as many 'a' characters as it can (but at least one).

Second, you use the non-greedy one-or-more version 'a+?'. In this case, the regex engine matches only one character 'a', consumes it, and moves on with the next match.

Let’s summarize what you’ve learned so far:

Greedy vs Non-Greedy Match – What’s the Difference?

Given a pattern with a quantifier (e.g. the asterisk operator) that allows the regex engine to match the pattern multiple times.

A given string may match the regex in multiple ways. For example, both substrings 'a' and 'aaa' are valid matches when matching the pattern 'a*' in the string 'aaaa'.

So the difference between the greedy and the non-greedy match is the following: The greedy match will try to match as many repetitions of the quantified pattern as possible. The non-greedy match will try to match as few repetitions of the quantified pattern as possible.

Examples Greedy vs Non-Greedy Match

Let’s consider a range of examples that help you understand the difference between greedy and non-greedy matches in Python:

>>> import re
>>> re.findall('a+', 'aaaa')
['aaaa']
>>> re.findall('a+?', 'aaaa')
['a', 'a', 'a', 'a']
>>> re.findall('a*', 'aaaa')
['aaaa', '']
>>> re.findall('a*?', 'aaaa')
['', 'a', '', 'a', '', 'a', '', 'a', '']
>>> re.findall('a?', 'aaaa')
['a', 'a', 'a', 'a', '']
>>> re.findall('a??', 'aaaa')
['', 'a', '', 'a', '', 'a', '', 'a', '']

Make sure you completely understand those examples before you move on. If you don’t, please read the previous paragraphs again.

Which is Faster: Greedy vs Non-Greedy?

Considering that greedy quantifiers match a maximal and non-greedy a minimal number of patterns, is there any performance difference?

Great question!

Indeed, some benchmarks suggest that there’s a significant performance difference: the greedy quantifier is 100% slower in realistic experiments on benchmark data.

So if you optimize for speed and you don’t care about greedy or non-greedy matches—and you don’t know anything else—go for the non-greedy quantifier!

However, the truth is not as simple. For example, consider the following basic experiment that falsifies the previous hypothesis that the non-greedy version is faster:

>>> import timeit
>>> timeit.timeit('import re;re.findall("a*", "aaaaaaaaaaaa")')
1.0579840000000331
>>> timeit.timeit('import re;re.findall("a*?", "aaaaaaaaaaaa")')
3.7830938000000742

I used the speed testing tool timeit that allows to throw in some simple Python statements and check how long they run. Per default, the passed statement is executed 1,000,000 times.

You can see a notable performance difference of more than 300%! The non-greedy version is three times slower than the greedy version.

Why is that?

The reason is the re.findall() method that returns a list of matching substrings. Here’s the output both statements would produce:

>>> re.findall("a*", "aaaaaaaaaaaa")
['aaaaaaaaaaaa', '']
>>> re.findall("a*?", "aaaaaaaaaaaa")
['', 'a', '', 'a', '', 'a', '', 'a', '', 'a', '', 'a', '', 'a', '', 'a', '', 'a', '', 'a', '', 'a', '', 'a', '']

You can see that the greedy version finds one match and is done with it. The non-greedy version finds 25 matches which leads to far more processing and memory overhead.

So what happens if you use the re.search() method that returns only the first match rather than the re.findall() method that returns all matches?

>>> timeit.timeit('import re;re.search("a*", "aaaaaaaaaaaa")')
0.8420328999998219
>>> timeit.timeit('import re;re.search("a*?", "aaaaaaaaaaaa")')
0.7955709000000297

As expected, this changes things again. Both regex searches yield a single result, but the non-greedy match is much shorter: it matches the empty string '' rather than the whole string 'aaaaaaaaaaaa'. Of course, this is a bit faster.

However, the difference is negligible in this minimal example.

There’s More: Greedy, Docile, Lazy, Helpful, Possessive Match

In this article, I’ve classified the regex world into greedy and non-greedy quantifiers. But you can differentiate the “non-greedy” class even more!

Next, I’ll give you a short overview based on this great article of the most important terms in this regard:

• Greedy: match as many instances of the quantified pattern as you can.
• Docile: match as many instances of the quantified pattern as long as it still matches the overall pattern—if this is possible. Note that what I called “greedy” in this article is really “docile”.
• Lazy: match as few instances of the quantified pattern as needed. This is what I called “non-greedy” in this article.
• Possessive: never gives up a partial match. So the regex engine may not even find a match that actually exist—just because it’s so greedy. This is very unusual and you won’t see it a lot in practice.

If you want to learn more about those, I’d recommend that you read this excellent online tutorial.

Where to Go From Here

Summary: You’ve learned that the greedy quantifiers ?, *, and + match as many repetitions of the quantified pattern as possible. The non-greedy quantifiers ??, *?, and +? match as few repetitions of the quantified pattern as possible.

If you want to master Python and regular expressions, join my free email academy—it’s fun!

This tutorial makes you a master of character sets in Python. (I know, I know, it feels awesome to see your deepest desires finally come true.)

As I wrote this article, I saw a lot of different terms describing this same powerful concept such as “character class“, “character range“, or “character group“. However, the most precise term is “character set” as introduced in the official Python regex docs. So in this tutorial, I’ll use this term throughout.

Python Regex – Character Set

So, what is a character set in regular expressions?

The character set is (surprise) a set of characters: if you use a character set in a regular expression pattern, you tell the regex engine to choose one arbitrary character from the set. As you may know, a set is an unordered collection of unique elements. So each character in a character set is unique and the order doesn’t really matter (with a few minor exceptions).

Here’s an example of a character set as used in a regular expression:

>>> import re
>>> re.findall('[abcde]', 'hello world!')
['e', 'd']

You use the re.findall(pattern, string) method to match the pattern '[abcde]' in the string 'hello world!'. You can think of all characters a, b, c, d, and e as being in an OR relation: either of them would be a valid match.

The regex engine goes from the left to the right, scanning over the string ‘hello world!’ and simultaneously trying to match the (character set) pattern. Two characters from the text ‘hello world!’ are in the character set—they are valid matches and returned by the re.findall() method.

You can simplify many character sets by using the range symbol ‘-‘ that has a special meaning within square brackets: [a-z] reads “match any character from a to z”, while [0-9] reads “match any character from 0 to 9”.

Here’s the previous example, simplified:

>>> re.findall('[a-e]', 'hello world!')
['e', 'd']

You can even combine multiple character ranges in a single character set:

>>> re.findall('[a-eA-E0-4]', 'hello WORLD 42!')
['e', 'D', '4', '2']

Here, you match three ranges: lowercase characters from a to e, uppercase characters from A to E, and numbers from 0 to 4. Note that the ranges are inclusive so both start and stop symbols are included in the range.

Python Regex Negative Character Set

But what if you want to match all characters—except some? You can achieve this with a negative character set!

The negative character set works just like a character set, but with one difference: it matches all characters that are not in the character set.

Here’s an example where you match all sequences of characters that do not contain characters a, b, c, d, or e:

>>> import re
>>> re.findall('[^a-e]+', 'hello world')
['h', 'llo worl']

We use the “at-least-once quantifier +” in the example that matches at least one occurrence of the preceding regex (if you’re unsure about how it works, check out my detailed Finxter tutorial about the plus operator).

There are only two such sequences: the one-character sequence ‘h’ and the eight-character sequence ‘llo worl’. You can see that even the empty space matches the negative character set.

Summary: the negative character set matches all characters that are not enclosed in the brackets.

How to Fix “re.error: unterminated character set at position”?

Now that you know character classes, you can probably fix this error easily: it occurs if you use the opening (or closing) bracket ‘[‘ in your pattern. Maybe you want to match the character ‘[‘ in your string?

But Python assumes that you’ve just opened a character class—and you forgot to close it.

Here’s an example:

>>> re.findall('[', 'hello [world]')
Traceback (most recent call last): File "<pyshell#5>", line 1, in <module> re.findall('[', 'hello [world]') File "C:\Users\xcent\AppData\Local\Programs\Python\Python37\lib\re.py", line 223, in findall return _compile(pattern, flags).findall(string) File "C:\Users\xcent\AppData\Local\Programs\Python\Python37\lib\re.py", line 286, in _compile p = sre_compile.compile(pattern, flags) File "C:\Users\xcent\AppData\Local\Programs\Python\Python37\lib\sre_compile.py", line 764, in compile p = sre_parse.parse(p, flags) File "C:\Users\xcent\AppData\Local\Programs\Python\Python37\lib\sre_parse.py", line 930, in parse p = _parse_sub(source, pattern, flags & SRE_FLAG_VERBOSE, 0) File "C:\Users\xcent\AppData\Local\Programs\Python\Python37\lib\sre_parse.py", line 426, in _parse_sub not nested and not items)) File "C:\Users\xcent\AppData\Local\Programs\Python\Python37\lib\sre_parse.py", line 532, in _parse source.tell() - here)
re.error: unterminated character set at position 0

The error happens because you used the bracket character ‘[‘ as if it was a normal symbol.

So, how to fix it? Just escape the special bracket character ‘\[‘ with the single backslash:

>>> re.findall('\[', 'hello [world]')
['[']

This removes the “special” meaning of the bracket symbol.

Related Re Methods

There are seven important regular expression methods which you must master:

• The re.findall(pattern, string) method returns a list of string matches. Read more in our blog tutorial.
• The re.search(pattern, string) method returns a match object of the first match. Read more in our blog tutorial.
• The re.match(pattern, string) method returns a match object if the regex matches at the beginning of the string. Read more in our blog tutorial.
• The re.fullmatch(pattern, string) method returns a match object if the regex matches the whole string. Read more in our blog tutorial.
• The re.compile(pattern) method prepares the regular expression pattern—and returns a regex object which you can use multiple times in your code. Read more in our blog tutorial.
• The re.split(pattern, string) method returns a list of strings by matching all occurrences of the pattern in the string and dividing the string along those. Read more in our blog tutorial.
• The re.sub(The re.sub(pattern, repl, string, count=0, flags=0) method returns a new string where all occurrences of the pattern in the old string are replaced by repl. Read more in our blog tutorial.

These seven methods are 80% of what you need to know to get started with Python’s regular expression functionality. If you want to learn more, check out the most comprehensive Python regex tutorial in the world!

Where to Go From Here?

You’ve learned everything you need to know about the Python Regex Character Set Operator.

Summary:

If you use a character set [XYZ] in a regular expression pattern, you tell the regex engine to choose one arbitrary character from the set: X, Y, or Z.

Want to earn money while you learn Python? Average Python programmers earn more than $50 per hour. You can certainly become average, can’t you?

Join the free webinar that shows you how to become a thriving coding business owner online!

[Webinar] Become a Six-Figure Freelance Developer with Python

Join us. It’s fun!

A regular expression is a decades-old concept in computer science. Invented in the 1950s by famous mathematician Stephen Cole Kleene, the decades of evolution brought a huge variety of operations. Collecting all operations and writing up a comprehensive list would result in a very thick and unreadable book by itself.

Fortunately, you don’t have to learn all regular expressions before you can start using them in your practical code projects. Next, you’ll get a quick and dirty overview of the most important regex operations and how to use them in Python. In follow-up chapters, you’ll then study them in detail — with many practical applications and code puzzles.

Here are the most important regex operators:

• . The wild-card operator (‘dot’) matches any character in a string except the newline character ‘\n’. For example, the regex ‘…’ matches all words with three characters such as ‘abc’, ‘cat’, and ‘dog’.  
• * The zero-or-more asterisk operator matches an arbitrary number of occurrences (including zero occurrences) of the immediately preceding regex. For example, the regex ‘cat*’ matches the strings ‘ca’, ‘cat’, ‘catt’, ‘cattt’, and ‘catttttttt’. 
• ? The zero-or-one operator matches (as the name suggests) either zero or one occurrences of the immediately preceding regex. For example, the regex ‘cat?’ matches both strings ‘ca’ and ‘cat’ — but not ‘catt’, ‘cattt’, and ‘catttttttt’. 
• + The at-least-one operator matches one or more occurrences of the immediately preceding regex. For example, the regex ‘cat+’ does not match the string ‘ca’ but matches all strings with at least one trailing character ‘t’ such as ‘cat’, ‘catt’, and ‘cattt’. 
• ^ The start-of-string operator matches the beginning of a string. For example, the regex ‘^p’ would match the strings ‘python’ and ‘programming’ but not ‘lisp’ and ‘spying’ where the character ‘p’ does not occur at the start of the string.
• $ The end-of-string operator matches the end of a string. For example, the regex ‘py$’ would match the strings ‘main.py’ and ‘pypy’ but not the strings ‘python’ and ‘pypi’. 
• A|B The OR operator matches either the regex A or the regex B. Note that the intuition is quite different from the standard interpretation of the or operator that can also satisfy both conditions. For example, the regex ‘(hello)|(hi)’ matches strings ‘hello world’ and ‘hi python’. It wouldn’t make sense to try to match both of them at the same time.
• AB  The AND operator matches first the regex A and second the regex B, in this sequence. We’ve already seen it trivially in the regex ‘ca’ that matches first regex ‘c’ and second regex ‘a’. 

Note that I gave the above operators some more meaningful names (in bold) so that you can immediately grasp the purpose of each regex. For example, the ‘^’ operator is usually denoted as the ‘caret’ operator. Those names are not descriptive so I came up with more kindergarten-like words such as the “start-of-string” operator.

Let’s dive into some examples!

Examples

import re text = ''' Ha! let me see her: out, alas! he's cold: Her blood is settled, and her joints are stiff; Life and these lips have long been separated: Death lies on her like an untimely frost Upon the sweetest flower of all the field. ''' print(re.findall('.a!', text)) '''
Finds all occurrences of an arbitrary character that is
followed by the character sequence 'a!'.
['Ha!'] ''' print(re.findall('is.*and', text)) '''
Finds all occurrences of the word 'is',
followed by an arbitrary number of characters
and the word 'and'.
['is settled, and'] ''' print(re.findall('her:?', text)) '''
Finds all occurrences of the word 'her',
followed by zero or one occurrences of the colon ':'.
['her:', 'her', 'her'] ''' print(re.findall('her:+', text)) '''
Finds all occurrences of the word 'her',
followed by one or more occurrences of the colon ':'.
['her:'] ''' print(re.findall('^Ha.*', text)) '''
Finds all occurrences where the string starts with
the character sequence 'Ha', followed by an arbitrary
number of characters except for the new-line character. Can you figure out why Python doesn't find any?
[] ''' print(re.findall('\n$', text)) '''
Finds all occurrences where the new-line character '\n'
occurs at the end of the string.
['\n'] ''' print(re.findall('(Life|Death)', text)) '''
Finds all occurrences of either the word 'Life' or the
word 'Death'.
['Life', 'Death'] '''

In these examples, you’ve already seen the special symbol \n which denotes the new-line character in Python (and most other languages). There are many special characters, specifically designed for regular expressions.

Where to Go From Here?

If you want to master regular expressions once and for all, I’d recommend that you read the massive regular expression tutorial on the Finxter blog — for free!

https://blog.finxter.com/python-regex/