Climate science runs on data. A single weather station records temperature, pressure, humidity, and wind speed every minute. A satellite records dozens of channels across the electromagnetic spectrum, every few seconds, for decades. A climate model simulates the atmosphere on a grid that can be a hundred kilometres across, and produces terabytes of output per simulation. If you work in this field, the part of your job that eats most of your week is not the science. It is the data wrangling.
The LinuxLinks roundup of seven free climate, weather, and Earth-science data analysis tools is a decent starting point. It lists the seven tools that have stayed relevant as the field moved from text-file NetCDF (Network Common Data Form, a self-describing binary format used across atmospheric and ocean science) datasets to multi-terabyte archives on cloud object storage. What I want to add is which of those seven I have actually used on real projects, and what they cost in time and frustration.
What most newcomers should start with
If you are new to atmospheric or climate data, the two tools you will hear about first are GrADS and NCO. They are also the two that have been around the longest, which tells you something about the maturity of the field. Both work primarily from the command line. Both expect you to write a small script to load a NetCDF file, do something to it, and write the result back out.
GrADS is the easier of the two to get started with. The script language is forgiving. A common starter task is to plot a single field from a model output, and GrADS will do that in three or four lines. NCO is the operator’s toolkit; it does arithmetic on NetCDF files without loading them into memory. If you need to subtract one model run from another, or average across an ensemble (a collection of similar simulations used to estimate uncertainty in a model’s predictions), NCO does this in a single command.
The honest description of both is that they are tools from an earlier era. They are fast, they are reliable, and they do not have a graphical user interface that does anything you would want to use. For a workflow that runs on a remote server and outputs NetCDF, they are still the right choice.
When you need to look at the data
ncview and VAPOR are the two tools in the roundup that have a graphical interface. ncview is the lighter one. Open a file, get a basic plot, scrub through timesteps with the arrow keys. It does not do anything you cannot do from the command line, but it is much faster to use when you are trying to figure out what is in a file you have never seen before.
VAPOR is the heavyweight option. It does interactive 3D visualisation, which means you can rotate a volumetric render of a model output and slice through it on any axis. VAPOR is what atmospheric scientists reach for when they need to understand a 3D phenomenon like a convective system or a tropical cyclone. The catch is that VAPOR is a significant install. It pulls in Qt, OpenGL extensions, and a few other things you may not have on your machine.
A practical decision rule:
- If your question is “what does this field look like at one timestep,” ncview is enough.
- If your question is “how does this field evolve over the simulation,” ncview is still enough.
- If your question is “what does the 3D structure of this convection look like,” VAPOR is the right tool.
I have used both. ncview is the one I open first. VAPOR is the one I open when ncview tells me I am looking at something 3D that needs to be looked at properly.
When you need a programming language
MetPy and wradlib are the two Python-based tools in the roundup. Both are libraries, not standalone applications. You install them with pip and import them into your own scripts.
MetPy is for general-purpose meteorological calculations. It has functions for thermodynamics (potential temperature, equivalent potential temperature, moist static energy), for plotting on common map projections, and for reading the common station-data formats (METAR (Meteorological Aviation Report, the standard hourly weather report from airports), BUFR (Binary Universal Form for the Representation of meteorological data, the standard format for gridded observations from national weather services), and a few others). If you are writing a Python script that needs to do anything weather-related, MetPy is the first library to install.
wradlib is more specialised. It is for weather radar, which is the data type that comes from the rotating dish antennas you see at airports. wradlib handles the reflectivity-to-rainrate conversion, the Doppler velocity unfolding, and the quality control that is specific to radar data. If you do not work with radar, you do not need wradlib.
Metview is the odd one out. It is a hybrid: a graphical interface, a scripting language, and a library. It was built by the European Centre for Medium-Range Weather Forecasts (ECMWF, the European intergovernmental organisation that runs one of the two major global weather models), and it is the workhorse at most European national meteorological services.
A short description of when each one is worth installing:
– MetPy. For any Python script that needs meteorological calculations or plotting.
– wradlib. For any Python script that needs to work with weather radar data.
– Metview. For ECMWF-style bulk data processing, or when your institution already uses it.
What I actually run day-to-day
A short tour of which tool handles what in my own workflow, in the order I reach for them.
- Ad-hoc exploration of a new dataset. ncview. It is the fastest way to answer “what is in this file” without writing any code.
- Any script that reads NetCDF and does arithmetic. NCO. The operator’s toolkit. ncdiff, ncra, ncwa, nces are the commands I use most often.
- Any script that needs meteorological calculations. MetPy. The thermodynamic functions alone are worth the install.
- Radar-specific work. wradlib. There is no real substitute.
- 3D visualisation of large model output. VAPOR. Slower to set up than the others, but the only thing in the roundup that does what it does.
GrADS I keep around for legacy scripts that I do not want to rewrite. Metview I have used at ECMWF workstations, but I do not have it installed locally because the use cases overlap with MetPy + Cartopy and I have already built the workflow I need.
Trade-offs
ncview is the right choice if you need to look at a file you have never seen before. The install is light, the learning curve is flat, and it does not pretend to be anything other than a viewer.
NCO is the right choice if your work is mostly arithmetic on NetCDF files. It is fast, scriptable, and has been stable for twenty years.
MetPy is the right choice if you are writing Python and need meteorological functions. The documentation is solid, the maintainers are responsive, and the API is what you would expect from a scientific Python library.
wradlib is the right choice if you work with radar. There is no general-purpose alternative.
VAPOR is the right choice if you need 3D visualisation of large model output. The install is heavy and the learning curve is steep, but the payoff is real.
GrADS is worth keeping around for legacy workflows but not worth learning new.
Metview is the right choice if you work at an institution that uses it, but the documentation gap relative to MetPy makes it a poor choice for someone starting from scratch.
The cost of getting this wrong is mostly time. A wrong tool for a workflow will eat a week of setup and produce something uglier than it should be. A wrong tool for a one-off exploration costs you an hour of installing before you switch to the right one.
What I would tell past me
Three things, if I were starting over.
- Install ncview first. It will answer more “what is in this file” questions than any other tool, and it takes five minutes to set up.
- Learn NCO before you learn MetPy. The arithmetic operations on NetCDF files are the workhorse of any analysis pipeline. You will use NCO on every project.
- Do not install VAPOR until you actually need 3D visualisation. The setup cost is real, and you should not pay it for a tool you might use once.