A few months into a collaboration with an astrophysics group, the simulation’s post-processing pipeline takes about an hour to walk 200 GB of output tables, and nobody is sure why. The data is split across tens of thousands of small text files. The script loops over them, sometimes rereading the same file multiple times. Somewhere inside is a dictionary of dictionaries that simulates a binary tree by hand using string keys, and every leaf holds a pandas dataframe. It works, slowly, and it is extremely difficult to read.
This is the everyday version of a problem that quietly shapes most scientific software. The scientists who write the code are usually excellent at their domain and only passingly familiar with the language they happen to be using. Python gets the job done because the data science library surface is forgiving and the on-ramp is short, but the gap between “I can write a script that gets the answer” and “I can write a script that runs in reasonable time and is debuggable six months from now” is wide, and almost nothing in a typical scientific training path closes it.
The result is not bad science. It is slow software that ships, ships successfully, and accumulates as institutional drag. The same pipeline gets rewritten every few years by whoever is brave enough to touch it, and the rewrite usually does not get upstreamed because the original author has moved on to the next postdoc.
Why the bottleneck shows up where it does
Three patterns recur in the tools that show up in scientific codebases, and all three trace back to the same gap in training.
First, files are read in loops, sometimes the same file several times, because the alternative (loading everything into memory at once) sounds expensive and the language does not warn you about the cost. For a 200 GB dataset, the right move is neither “load it all” nor “open and close the same file fifteen times.” It is some combination of indexed binary storage, an out-of-core data structure (a data structure designed to work on data that does not fit in RAM by reading and writing blocks to disk), and a query layer that lets you ask for a subset without walking every file. None of that is in the standard scientific Python toolbox.
Second, complex nested data structures get assembled by hand. The dictionary-of-dictionaries-with-string-keys pattern is not a design choice; it is what happens when the only data structure anyone in the room knows how to use is a dict. pandas dataframes are designed for tabular data with consistent column types, and using them as leaves in a manually-keyed tree is a sign that the modeling layer is doing too much.
Third, profilers and debuggers do not get used until somebody on the outside asks. The cProfile module, snakeviz (a browser-based visualizer for cProfile output), memory_profiler, py-spy (a sampling profiler that can attach to a running Python process without restarting it), and the standard pdb debugger are all bundled with CPython or install with one pip command. Most scientific Python codebases never touch them.
What a missing semester for science would teach
A practical curriculum for scientists who write Python would be short and would not require a computer science degree. The pieces are well-documented and well-maintained; they just do not show up in most graduate programs.
Four short modules cover most of what scientific Python users are missing:
- Profiling with cProfile (the standard-library call-graph profiler), snakeviz (its browser-based sunburst visualizer), and py-spy (a sampling profiler that attaches to a running process without restart)
- Memory model and data structures, including CPython object overhead, when to use numpy arrays or array.array instead of dicts, and out-of-core options like polars, duckdb, vaex, or sqlite-on-disk
- Debuggers and test frameworks: the built-in pdb, the breakpoint() builtin from Python 3.7, and pytest as a one-line install that gives you reusable regression checks
- When not to use pandas: hierarchical data, streaming data, very wide rows, or exploratory analysis where the schema is changing mid-flight
Profiling first, because nothing else matters until you know where the time is going. cProfile is in the standard library; snakeviz turns its output into a sunburst diagram you can click through; py-spy can attach to a running process so you do not have to instrument your script ahead of time. The first time a researcher sees a snakeviz window pop open with a clear “this one line is taking 80 percent of the runtime” callout, the reaction is usually physical surprise. That surprise is the entire point.
Memory model and data structures second, because the wrong choice early on is expensive to undo. CPython’s memory model is straightforward but unforgiving: every object has overhead, dictionaries are hash tables under the hood, and a million small dicts in a list will quietly consume gigabytes. Arrays from numpy or the standard array module, or the typed dictionaries in the typing module, exist because untyped dicts are not free. For data that does not fit in RAM, polars, duckdb, vaex, or sqlite-on-disk give you real query semantics without forcing you to redesign your whole pipeline.
Debuggers and test frameworks third, because the alternative is print statements. The built-in pdb (Python debugger) and the breakpoint() builtin dropped in Python 3.7 are good enough for most scientific scripts. For anything bigger, pytest is a one-line install and gives you a way to write a sanity check once and reuse it. The hardest sell is the testing one, because scientific code is often “I ran it once and the output looked right,” but that is exactly the case where a saved regression test pays for itself.
When not to use a dataframe fourth, and this is the topic that surprises people. Pandas is excellent for the shape of data it was designed for: rectangular, column-oriented, fits in memory, has consistent types. It is the wrong tool for hierarchical data, for streaming data, for data where one row is ten megabytes, or for the kind of exploratory analysis where the schema is changing as you go. Knowing when not to reach for it is the difference between a clean pipeline and a years-long fight.
Trade-offs
The “Missing Semester of Your CS Education” model works well for computer science undergraduates because they are willing to spend a semester on tooling and they have advisors who will grade them on it. Scientists have neither. Their job is to do science, and any time spent on tooling is time not spent on their actual research question, which their advisor, their committee, and their funding agency all care about. A curriculum that ignores that incentive structure will not be adopted.
Software Carpentry and similar efforts have been pushing on this for years with mixed results. Workshops are useful, but a one-day workshop does not stick. The pieces that do stick tend to be the ones a lab installs and maintains locally: a shared snakeviz walkthrough in the onboarding doc, a pinned set of py-spy commands for the cluster, a wiki page on how to interpret a memory profile. Those are operational changes, not curriculum changes.
There is also a real question about who teaches this. The scientists who already know it are usually the ones with the least time. The computer scientists who could teach it usually do not understand the data shape well enough to give useful examples. A workable answer is to fund embedded software engineers inside scientific labs for multi-year stints, but that is a structural funding change, not a teaching change.
Finally, the article this is based on is itself a small anecdote. One astrophysics simulation, one postdoc, one afternoon of profiling. The pattern is recognizable, and similar stories come up across labs and disciplines, but the fix is not going to land the same way for every group. A lab with a stable codebase and a five-year horizon can invest in rewriting the pipeline. A lab running a one-off simulation with a six-month deadline should probably just buy more disk and let the next group fix it.
What to take away
If you write Python for a living in a scientific context, the cheapest change is to spend an afternoon with snakeviz and py-spy on your actual script, not a toy one. Most researchers who do this once find a 5x to 50x speedup without changing the algorithm, just by removing accidental rereads and switching the wrong data structures. That afternoon is worth more than any curriculum.
If you mentor a graduate student who writes Python, the second-cheapest change is to make a profile of their script together, in person, the first time they say it is slow. The conversation that follows is the curriculum.