PYTHON / PANDAS
Filtering rows with boolean conditions
Build boolean masks from column comparisons and use them to select DataFrame rows, combining conditions with &, |, ~ and isin.
What you will learn
- Create a boolean Series from a comparison and pass it to df[...] to keep rows
- Combine conditions with & | ~ and parenthesise each comparison
- Use isin for membership tests and between for inclusive range tests
- Know that NaN fails every comparison, so such rows drop out silently
Understanding Filtering rows with boolean conditions
A comparison like sensors["temp_c"] > 22 does not return True or False. It returns a whole new Series of dtype bool, one value per row, carrying the same index labels as the column it came from. Passing that Series back into sensors[...] keeps only the rows whose mask value is True. Two objects are involved: the mask, which you can print, store in a variable, and reuse, and the frame being indexed.
The mask is matched to the frame by index label, not by position, so a mask built from a column of the same frame always lines up. Because a Series holds many booleans, Python's and, or, and not cannot be used on it; they demand one truth value and pandas raises "The truth value of a Series is ambiguous" instead of guessing. The element-wise operators & (and), | (or), and ~ (not) work per row, but they bind more tightly than < and >, so each comparison needs its own parentheses or the expression is parsed as something else entirely.
Filtering returns a new DataFrame that keeps the original index labels, so the index will have gaps such as 2 and 4 rather than restarting at 0. That result is a copy, so writing to it does not change the source frame; to edit matching rows in place, put the mask on the left of .loc. Missing values are the other trap: any comparison against NaN is False, so a row with NaN in the tested column is excluded by both x > 10 and x <= 10.
import pandas as pd
sensors = pd.DataFrame({
"station": ["north", "south", "east", "west", "north"],
"temp_c": [21.5, 18.0, 25.3, 19.8, 30.1],
"humidity": [55, 80, 40, 65, 35],
})
hot = sensors["temp_c"] > 22
print(hot)
print(sensors[hot])
print(sensors[(sensors["temp_c"] > 22) & (sensors["humidity"] < 38)])A boolean condition on a column produces a per-row bool Series, and indexing a DataFrame with that Series keeps the rows marked True.
Worked examples
Membership with isin and negation with ~
Selects rows whose city is in a list of interest, then inverts the same mask to get everything else.
import pandas as pd
orders = pd.DataFrame({
"city": ["Lisbon", "Oslo", "Lima", "Oslo", "Cairo"],
"total": [42.0, 15.5, 88.0, 61.25, 30.0],
})
nordic = orders["city"].isin(["Oslo", "Helsinki"])
print(orders[nordic]["total"].sum())
print(orders.loc[~nordic, "city"].tolist())Example explained
Line 1isin tests each value against the whole list at once, replacing a chain of == comparisons joined by |.
Line 2"Helsinki" is absent from the column; isin simply marks nothing for it rather than erroring.
Line 3orders[nordic]["total"].sum() adds 15.5 and 61.25, the two rows the mask kept.
Line 4~nordic flips every element of the stored mask, so the second line needs no new comparison.
query strings and between
Shows a string-based filter where column names are bare identifiers, and an inclusive numeric range test.
import pandas as pd
df = pd.DataFrame({"name": ["a", "b", "c", "d"], "score": [55, 71, 90, 71]})
print(df.query("score >= 70 and name != 'd'"))
print(df["score"].between(60, 71).tolist())Example explained
Line 1Inside query the text is parsed by pandas, so and is legal there and score needs no quoting or df[...] prefix.
Line 2String literals inside the query need their own quotes, hence 'd' in single quotes.
Line 3Row 3 has score 71 but name 'd', so it fails the second half of the condition.
Line 4between(60, 71) is inclusive on both ends by default, which is why 71 maps to True.
Masks align by label, lists by position
Demonstrates that a boolean Series is reindexed to the target's labels while a plain list is applied positionally.
import pandas as pd
s = pd.Series([10, 20, 30], index=["a", "b", "c"])
by_label = pd.Series([True, True, False], index=["c", "b", "a"])
print(s.loc[by_label].tolist())
print(s[[True, True, False]].tolist())Example explained
Line 1by_label says c=True, b=True, a=False, so alignment selects b and c and returns them in s's own order.
Line 2The plain list has no labels, so its three values are read against positions 0, 1, 2 and pick a and b.
Line 3Same booleans, different results: this is why a mask should be derived from the frame you are filtering.
Line 4If by_label were missing a label present in s, pandas would raise an unalignable boolean Series error.
Important notes
A row with NaN in the tested column is excluded by both x > 10 and x <= 10; use ~(x > 10) or an explicit notna() check if you need those rows counted.
The filtered frame keeps the original index labels, so call reset_index(drop=True) if downstream code expects a 0-based index.
Common mistakes
Using and/or between two conditions, as in df[cond1 and cond2], which raises ValueError: The truth value of a Series is ambiguous.
Omitting parentheses: df[df["temp_c"] > 22 & df["humidity"] < 38] evaluates 22 & df["humidity"] first and fails or gives nonsense, because & binds tighter than >.
Assigning through a filter with df[mask]["col"] = 0, which writes into a temporary copy, leaves the original unchanged and triggers a chained-assignment warning.
Try it yourself
Change, predict, then run
Build a DataFrame of six products with price and stock columns, then print the rows where price is under 20 and stock is greater than zero. Print the negated mask's sum to report how many rows were rejected.
Open the Python workspaceCheck your understanding
A DataFrame has 100 rows and its price column contains 4 NaN values. df[df["price"] > 10] returns 30 rows. How many rows does df[df["price"] <= 10] return?
- 70, because the two conditions cover every row
- 66, because the 4 NaN rows fail both comparisons
- 74, because NaN is treated as smaller than 10
- 100, because the mask is reindexed back to full length
Show answer
Any comparison involving NaN yields False, so the 4 NaN rows are dropped by > 10 and by <= 10 alike, leaving 100 - 30 - 4 = 66. The tempting answer 70 assumes the two masks are exact complements, which only holds for columns with no missing values; ~(df["price"] > 10) would be the true complement and would return 70.