PYTHON / ADVANCED PYTHON
Regular expressions with re
Use re to search, capture, and rewrite text with compiled patterns, named groups, match objects, and non-greedy quantifiers.
What you will learn
- Choose between re.match, re.search, and re.fullmatch by where you need anchoring
- Pull data out of a match object with group, groupdict, and span
- Control backtracking with lazy quantifiers and negated character classes
- Rewrite text with re.sub, passing a function when the replacement is computed
Understanding Regular expressions with re
A regular expression is a small program for a backtracking scanner. The engine plants itself at position 0 of the subject string, tries to consume the pattern piece by piece, and if any piece fails it backs up and retries alternatives; if no alternative works it shifts the starting position one character right and begins again. This is why re.search can find a match anywhere while re.match only ever reports a match that begins at position 0, and why a pattern that looks obviously satisfiable can still be slow: every failure costs a walk back through the quantifiers.
Nothing in re returns plain strings except findall and sub. search, match, fullmatch, and finditer hand you a match object, which is a record of what the engine did: group(0) is the whole matched text, group(n) or group("name") is what a capturing group held at the moment the match succeeded, span() gives the half-open index range, and groupdict() collects every named group. Keeping the match object around instead of immediately calling .group() is what lets you slice the original string, report error positions, or check whether an optional group participated at all (it will be None, not an empty string).
Quantifiers like + and * are greedy: they take as much as the rest of the pattern will allow, then give characters back one at a time under backtracking pressure. Adding ? makes them lazy so they take as little as possible and grow only when forced. Most "my regex matched way too much" bugs are greed, and the usual fix is not the lazy form but a negated character class such as [^>]+, which cannot cross the delimiter at all and therefore needs no backtracking.
import re
log = "2024-03-01 12:04:11 WARN disk=91% host=web-03"
pattern = re.compile(
r"(?P<date>\d{4}-\d{2}-\d{2})\s+"
r"(?P<time>\d{2}:\d{2}:\d{2})\s+"
r"(?P<level>[A-Z]+)\s+"
r"disk=(?P<pct>\d+)%"
)
m = pattern.search(log)
print(m.group("level"), int(m.group("pct")))
print(m.span("date"))
print(m.groupdict()["time"])
print(re.findall(r"\w+=(\w+)", log))
print(re.sub(r"host=\S+", "host=REDACTED", log))Every re call is the same backtracking left-to-right scan, and what you get back is a match object describing exactly where and what the engine matched.
Worked examples
Greed, laziness, and negated classes
Shows how the same subject yields wildly different results from .+, .+?, and [^>]+.
import re
html = "<b>bold</b> and <i>italic</i>"
print(re.findall(r"<.+>", html))
print(re.findall(r"<.+?>", html))
print(re.findall(r"<[^>]+>", html))Example explained
Line 1<.+> starts at the first < and lets .+ run to the end of the string, then backtracks only far enough to find one > , which is the final one.
Line 2<.+?> makes .+? surrender at the first opportunity, so it stops at the nearest > and produces four tags.
Line 3<[^>]+> forbids > inside the match outright, so it gets the same answer without any backtracking, which is the version to prefer.
Line 4The . in all three never matches a newline, so a tag split across lines would be missed unless re.DOTALL is passed.
sub with a function, and finditer for positions
Computes each replacement from the matched text and reports where each match started.
import re
text = "cart: 2 x 4.50, 1 x 12.00, 3 x 0.99"
def double(m):
return f"{float(m.group()) * 2:.2f}"
print(re.sub(r"\d+\.\d{2}", double, text))
for m in re.finditer(r"(\d+) x (\d+\.\d{2})", text):
total = int(m.group(1)) * float(m.group(2))
print(m.start(), f"{total:.2f}")Example explained
Line 1When the second argument to re.sub is callable, re calls it once per match with the match object and splices in the returned string, so no \\1 backreference syntax is involved.
Line 2m.group() with no argument is group(0), the entire matched text, here the price alone.
Line 3finditer is lazy and yields match objects, so m.start() is available; findall would have thrown the positions away.
Line 4The last total is printed as 2.97 because 3 * 0.99 is 2.9699999999999998 in binary floating point and the format spec rounds it.
match vs fullmatch
Demonstrates that re.match anchors only at the start while re.fullmatch requires the pattern to consume the whole string.
import re
for code in ["AB-1234", "xAB-1234", "AB-1234x"]:
anchored = bool(re.match(r"AB-\d{4}", code))
whole = bool(re.fullmatch(r"AB-\d{4}", code))
print(code, anchored, whole)Example explained
Line 1re.match refuses to shift its starting position, so xAB-1234 fails even though the pattern occurs one character in.
Line 2AB-1234x passes re.match because the pattern is satisfied at position 0 and trailing text is simply ignored.
Line 3re.fullmatch is the correct tool for validation: it is equivalent to wrapping the pattern in \\A and \\Z without you editing the pattern.
Line 4Both return None rather than False on failure, which is why bool() is used to print a clean flag.
Important notes
An optional group that did not participate gives None, not "", so int(m.group(2)) on an unmatched group raises TypeError rather than ValueError.
re already caches compiled patterns internally, so re.compile is about readability and reuse of one object, not a large speed win for a handful of patterns.
Common mistakes
Writing "\d+\b" without the r prefix: \b becomes the backspace character U+0008, so the pattern silently never matches instead of raising an error.
Using re.findall to get whole matches from a pattern that contains capturing groups: findall returns only the groups, so you get a list of tuples or the wrong substring and wonder where the rest went.
Reaching for re.match to test whether a string contains something: it anchors at position 0, so any match that is not at the very start returns None.
Try it yourself
Change, predict, then run
Given settings = "user=amy;role=admin;retries=3", use re.finditer with named groups to build a dict of the key/value pairs and print it, then use re.sub to replace only the user value with "***" and print the rewritten string.
Open the Python workspaceCheck your understanding
re.findall(r"(\d+)-(\d+)", "10-20 30-40") returns [('10', '20'), ('30', '40')] instead of ['10-20', '30-40']. Why?
- When the pattern has capturing groups, findall returns the groups for each match, so each match becomes a tuple of its groups
- findall always returns tuples, each holding the matched text and its starting index
- The hyphen is a special character that makes findall split every match in two
- findall returns tuples only when the subject string contains more than one match
Show answer
findall never reports group 0 when groups are present: with one group it returns a list of strings, with two or more it returns a tuple per match. The index option is wrong because findall discards positions entirely; if you want the full match text use finditer and m.group(0), or make the groups non-capturing with (?:...).