PYTHON / FILE HANDLING
Directory operations with os and shutil
Create, inspect, copy, move and delete directory trees with os and shutil, and know which module owns which job.
What you will learn
- Use os.makedirs(path, exist_ok=True) instead of testing os.path.exists first
- Walk a whole tree with os.walk and prune branches by slice-assigning to dirnames
- Pick os.remove for files, os.rmdir for empty dirs, shutil.rmtree for whole trees
- Copy trees with shutil.copytree and preserve timestamps with shutil.copy2
Understanding Directory operations with os and shutil
The os module gives you thin wrappers around single operating-system calls: os.mkdir creates exactly one directory, os.rmdir removes exactly one empty directory, os.listdir and os.scandir read the entries of exactly one directory. Because each function maps to one syscall, it fails loudly whenever the situation is not exactly right: a missing parent gives FileNotFoundError, an existing target gives FileExistsError, and a directory with contents gives a plain OSError carrying errno.ENOTEMPTY. shutil sits one layer above and is written in Python: copytree, rmtree and move loop over trees for you. Almost every directory task comes down to deciding which of those two layers you need.
For reading a tree, os.walk is the workhorse. It is a generator that yields one (dirpath, dirnames, filenames) tuple per directory, top-down by default, and it re-reads dirnames after your loop body runs. That is why pruning works: dirnames[:] = [...] mutates the list walk is holding, so the pruned subdirectories are never visited, while dirnames = [...] only rebinds your local name and changes nothing. Paths that come back from walk, listdir or scandir are just names or joined strings, so use os.path.join and os.path.relpath rather than string concatenation; those functions produce backslashes on Windows and forward slashes elsewhere.
Writing and deleting need more care than reading. os.makedirs(path, exist_ok=True) is preferable to "if not os.path.exists(path): os.mkdir(path)" because the check-then-act version has a gap in which another process can create the directory, while exist_ok pushes the decision down to the atomic mkdir call. shutil.rmtree deletes recursively with no confirmation and no recycle bin, and if it hits an error halfway it leaves a partly deleted tree behind. shutil.move renames when source and destination share a filesystem, but falls back to copy-then-delete across devices, so it can be slow and briefly leave two copies on disk.
import errno
import os
import shutil
import tempfile
base = tempfile.mkdtemp()
project = os.path.join(base, "project")
os.makedirs(os.path.join(project, "data", "raw"), exist_ok=True)
os.makedirs(os.path.join(project, "__pycache__"), exist_ok=True)
for path, text in [
(os.path.join(project, "notes.txt"), "first draft\n"),
(os.path.join(project, "data", "raw", "a.csv"), "id,value\n1,10\n"),
(os.path.join(project, "__pycache__", "junk.pyc"), "x"),
]:
with open(path, "w") as f:
f.write(text)
for dirpath, dirnames, filenames in os.walk(project):
dirnames[:] = sorted(d for d in dirnames if d != "__pycache__")
print(os.path.relpath(dirpath, base), sorted(filenames))
backup = shutil.copytree(project, os.path.join(base, "backup"))
print("backup holds:", sorted(os.listdir(backup)))
try:
os.rmdir(backup)
except OSError as err:
print("os.rmdir(backup) ->", type(err).__name__, errno.errorcode[err.errno])
shutil.rmtree(base)
print("base exists:", os.path.isdir(base))os provides one-directory-at-a-time primitives that fail loudly, while shutil provides recursive tree operations built on top of them.
Worked examples
Why makedirs exists
Shows the two ways os.mkdir refuses to work and how os.makedirs handles both.
import os
import shutil
import tempfile
root = tempfile.mkdtemp()
target = os.path.join(root, "logs", "2024")
os.makedirs(target, exist_ok=True)
os.makedirs(target, exist_ok=True)
print("created:", os.path.isdir(target))
try:
os.mkdir(target)
except OSError as err:
print("mkdir on existing dir:", type(err).__name__)
try:
os.mkdir(os.path.join(root, "cache", "tmp"))
except OSError as err:
print("mkdir with missing parent:", type(err).__name__)
shutil.rmtree(root)
print("cleaned:", os.path.exists(root))Example explained
Line 1os.makedirs builds both logs and logs/2024 in one call; exist_ok=True turns the repeat call into a no-op.
Line 2os.mkdir maps straight onto the mkdir syscall, so an already existing path raises FileExistsError.
Line 3os.mkdir will not invent parents either: cache does not exist, so the error is FileNotFoundError.
Line 4shutil.rmtree removes root plus everything under it, which os.rmdir could not do here.
Listing one directory with scandir
Uses os.scandir to tell files from directories and to total the sizes in a subfolder.
import os
import shutil
import tempfile
root = tempfile.mkdtemp()
os.mkdir(os.path.join(root, "images"))
os.mkdir(os.path.join(root, "text"))
with open(os.path.join(root, "README.md"), "w") as f:
f.write("hi\n")
for name, size in [("a.txt", 10), ("b.txt", 25)]:
with open(os.path.join(root, "text", name), "w") as f:
f.write("x" * size)
with os.scandir(root) as entries:
for entry in sorted(entries, key=lambda e: e.name):
print("DIR" if entry.is_dir() else "FILE", entry.name)
with os.scandir(os.path.join(root, "text")) as entries:
total = sum(entry.stat().st_size for entry in entries)
print("bytes in text/:", total)
shutil.rmtree(root)Example explained
Line 1os.scandir yields DirEntry objects that already carry the type flag from the directory read, so entry.is_dir() normally costs no extra syscall, unlike os.path.isdir(os.path.join(...)) after os.listdir.
Line 2Using scandir as a context manager releases the directory handle immediately instead of leaving it to the garbage collector.
Line 3Filesystem order is arbitrary, so sorting by entry.name is what makes this output reproducible.
Line 4entry.stat().st_size does touch the filesystem, because size is not part of the directory entry itself.
copy, copy2 and move
Demonstrates that only copy2 carries metadata across and that move leaves nothing behind.
import os
import shutil
import tempfile
root = tempfile.mkdtemp()
src = os.path.join(root, "report.txt")
with open(src, "w") as f:
f.write("numbers\n")
os.utime(src, (1_000_000_000, 1_000_000_000))
archive = os.path.join(root, "archive")
os.mkdir(archive)
plain = shutil.copy(src, archive)
exact = shutil.copy2(src, os.path.join(archive, "report_copy.txt"))
print("copy landed at:", os.path.basename(plain))
print("copy2 kept mtime:", os.path.getmtime(exact) == 1_000_000_000)
print("copy kept mtime:", os.path.getmtime(plain) == 1_000_000_000)
shutil.move(src, os.path.join(archive, "report_final.txt"))
print("source still there:", os.path.exists(src))
print("archive:", sorted(os.listdir(archive)))
shutil.rmtree(root)Example explained
Line 1When the destination of shutil.copy is a directory, the source file name is reused, and the function returns the full path it wrote.
Line 2copy2 also calls copystat, so the timestamp forced by os.utime survives; plain copy leaves the new file with the current time.
Line 3shutil.move here is a rename inside one filesystem, so the original path stops existing immediately.
Line 4The final listing proves all three destinations are separate files in the same directory.
Important notes
shutil.rmtree is not transactional; if it fails partway (a read-only file on Windows, a permission error) the tree is left half deleted, so handle failures with the onexc/onerror callback rather than assuming all-or-nothing.
Pruning in os.walk only works with the default topdown=True and only with in-place mutation of dirnames; with topdown=False the children are already visited before you see the parent.
Common mistakes
Calling os.mkdir('out/2024/raw') when out/2024 does not exist yet: os.mkdir creates one level only, so you get FileNotFoundError instead of the tree you expected.
Passing bare names from os.listdir straight to open() or os.path.isdir: the names are relative to the listed directory, not the current one, so calls silently look in the wrong place and raise FileNotFoundError.
Reaching for shutil.rmtree when os.rmdir complains about ENOTEMPTY, on a path built from user input or a wrong variable: rmtree deletes recursively with no prompt and nothing goes to a recycle bin.
Try it yourself
Change, predict, then run
In a temporary directory create sub/ and sub/deep/ containing a mix of .txt and .log files, then write a script that walks the tree, copies only the .log files into a new logs/ directory with shutil.copy2, prints the sorted listing of logs/, and finally removes the whole temporary directory with shutil.rmtree.
Open the Python workspaceCheck your understanding
You are walking a source tree with os.walk and want it to never descend into any directory named node_modules. Which line at the top of the loop body actually achieves that?
- dirnames[:] = [d for d in dirnames if d != 'node_modules']
- dirnames = [d for d in dirnames if d != 'node_modules']
- if os.path.basename(dirpath) == 'node_modules': continue
- filenames[:] = [f for f in filenames if 'node_modules' not in f]
Show answer
os.walk reads the same dirnames list object again after your loop body finishes, so an in-place slice assignment changes which subdirectories it visits. Rebinding the name (option 2) creates a new list and leaves walk's list untouched, and option 3 only skips printing that one directory: walk has already recorded its subdirectories and will still descend into them.