Scientific codebases stall when nobody on the team profiles Python

A CUNY astrophysics group spent months building a Python pipeline that took an hour to walk through 200 GB of simulation output. The grad student who wrote it had a good reason for every choice. None of those reasons survived the moment someone ran a profiler on the result. The lesson was not about their code. It was about a missing semester of training that affects every lab I have seen use Python for real work.

The setup was familiar. A small research group ran a large astrophysical simulation on a cluster. The output landed in tens of thousands of plain-text files, one set per timestep, some only a few rows wide, others tens of thousands. A grad student had written a tool to walk those files and assemble a tree of black hole merger events. The tree was the point. The tool was the bottleneck.

Two things jumped out when I sat down with the code. First, the data was spread across tens of thousands of files, with no central index. Second, the postprocessing code was rereading the same files multiple times, holding a giant dictionary-of-dictionaries where the leaves were pandas dataframes keyed by labels like root, A, A1, A2, a binary tree encoded by hand instead of by a data structure. Each traversal of the tree walked back to disk to recompute things the previous traversal already knew.

The code was difficult to understand, and not in a “well, it’s research code” way. It was difficult because the choice of representation was fighting the problem. The labels-as-keys trick is something I have seen in academic code often enough to be a pattern. Grad students are smart, focused, and rarely trained in computer science. They reach for the first thing that works and rarely revisit it.

There was no clean way to optimize the code in place. The shapes were wrong. We agreed to rewrite the tree construction and preserve the graphing code, which was elaborate but sane. That was the first non-trivial ask of credibility: I had to spend a few weeks showing up, doing small cleanups, and letting them watch me profile their simulator before they would let me touch the postprocessor. By the time I asked, they had seen me catch a meaningful speedup on the main simulator with a one-line change, and they trusted the diagnosis.

What I asked them to do before I touched the code

Three small things, in order. None of them required a CS degree.

• Convert the Jupyter notebook to a script. Notebooks hide control flow. A main.py with argparse and a --profile flag is the smallest step that lets a profiler see the whole run. Most lab code I have looked at lives in notebooks for the entire lifecycle, and every profiler I know works better on a script.
• Install snakeviz and run it on a short input. Snakeviz (a Python profiler that turns cProfile output into an interactive flame graph in the browser) is one pip install and one decorator away. The team watched the profile window pop open and immediately saw where the time was going: walking lots of small in-memory dataframes and loading them from txt.
• Trust that the tool exists. They had seen snakeviz before, in earlier meetings, when I used it on their main simulator. They had filed it under “weird thing the consultant does.” They did not realize it was a stock tool anyone with cProfile could run.

The third point is the one I keep coming back to. Scientists do not lack intelligence. They lack a map of the standard toolset. Two years ago I was in much the same place, working scientist who wrote Python, used pandas, and had no idea what a profiler was. I still consider myself a beginner at most of this.

Why this gap exists

The reason nobody teaches this stuff in physics or chemistry grad programs is structural. Most grad students take one or two required stats or methods courses, learn MATLAB or Python by osmosis, and start running their own simulations by year two. There is no required course on memory models, on the cost of repeatedly indexing into a dataframe in a loop, on when a set outperforms a list, on why reading the same 200 GB of simulation output four times is a bad idea. These are Computer Science 101 topics. They are also the topics that decide whether your pipeline takes an hour or takes four days.

There is a near-perfect template for fixing this. The MIT “Missing Semester” course (a set of unofficial lectures that cover the tools most CS programs skip, like shell, version control, editors, and debuggers) was created because a generation of CS students arrived at MIT having never used a debugger, never written a Makefile, and never used a shell beyond double-clicking icons. The same gap exists in every scientific computing group I have worked with, scaled up by an order of magnitude.

What a “Missing Semester for Scientists” would actually cover, in order of how often I have seen them cause problems:

• Profilers. cProfile, line_profiler, py-spy. The one lecture that would save the most researcher-hours across all of academia.
• The Python memory model. What a list, a set, a dict, and a numpy array actually cost. When df.iterrows() is faster than df.itertuples(). Why df.apply(lambda x: ...) is often orders of magnitude slower than a vectorized operation.
• When not to use a dataframe. Dataframes are great for analysis. They are a poor choice for hot inner loops. Most slow pandas code I have profiled was slow because the writer used a dataframe where a list of tuples or a numpy array would have done.
• I/O patterns. Why opening a file once and streaming it beats opening it 10,000 times. Why parquet (a columnar binary format) is faster than csv for the data shapes scientific code actually uses. Why the cluster filesystem does not love 30,000 small files.
• Version control beyond git push. Branches, merges, bisecting. Most lab code lives on one branch forever.

There are groups trying to teach this. Software Carpentry (a volunteer-run organization that runs short workshops on scientific computing basics) is the closest thing to a national program. Their workshops cover shell, Python, git, and testing. They are the reason several of my collaborators can bisect their own bugs.

What I would tell past me

Three things I would say to the version of me that started doing this work:

• Do not assume the team knows the standard tools. They probably saw you use snakeviz once. They did not know they could install it themselves.
• Profile before you rewrite. The CUNY team’s tree code looked unfixable. After running snakeviz, the rewriting scope shrank dramatically because the bottleneck was not where the code looked worst. It was in the I/O loop.
• Earn the rewrite trust with small wins first. A meaningful speedup on the main simulator, in a one-line change, is the cheapest credibility you will ever buy. Spend the first few weeks on those.

Trade-offs

Profilers are not free. cProfile adds 10 to 30 percent overhead on most Python code, which is fine for diagnosing but useless for benchmarking. py-spy runs in a separate process and adds almost no overhead, but it does not work on every Python build. Snakeviz is the most beginner-friendly, but it produces a flame graph (a visual where the width of each box represents how much time was spent in that function) that takes practice to read. Picking one and learning to read its output is the highest-value 90 minutes a research programmer can spend.

The “convert the notebook to a script” step has its own cost. Notebooks are popular for a reason: they keep notes, plots, and code in one place. Asking a scientist to convert their working notebook to a script before they can profile it is a real friction. The lighter-weight alternative is jupyter nbconvert --to script (a one-line conversion to a .py file), which loses the plots but keeps the code runnable. For profiling-only purposes, that is enough.

For groups that already have credibility with each other, the meta-lesson is “let one person profile and report.” For groups that do not, the meta-lesson is “first spend a week doing small cleanups until the team trusts your read of the codebase.” Both are real costs. The first costs one profiler run. The second costs weeks.

Anyone whose pipeline takes more than ten minutes and has never seen a flame graph is leaving days on the table. The toolset exists. The training gap is real, and the fix is short.