Simplifying argparse usage with subcommands

One of the best things about Python is its standard library: it's frequently possible to create complex applications while requiring few (if any) external dependencies. For example, command line interfaces can be easily built with the argparse module. Despite this, there exist several alternative, third-party modules (e.g., docopt, click, and begins). These all tend to share similar motivations: while argparse is powerful, it is by inherently verbose and is therefore cumbersome to use for more complex CLIs which use advanced features such as subcommands. Nevertheless, I tend to prefer sticking with argparse in part because I am already familiar with the API and because using it means I don't need to bring in another dependency from PyPI just to add a small bit of extra functionality. The good news is that with a simple decorator and a convenience function, writing CLIs with subcommands in argparse is pretty trivial and clean.

Start by creating a parser and subparsers in cli.py:

from argparse import ArgumentParser

cli = ArgumentParser()
subparsers = cli.add_subparsers(dest="subcommand")

Note that we are storing the name of the called subcommand so that we can later print help if either no subcommand is given or if ...

more ...

Javascript for Python programmers

Unless you're just writing a simple HTTP API server, any amount of web programming in Python will likely require at least a little bit of Javascript. Like it or not (and I will try to argue in this post that you should like it for what it's good at), Javascript is really the only game in town when it comes to client-side scripting on a web page. Sure, there are a number of Python-to-Javascript transpilers out there, but using these just tends to limit the ability to use new Javascript features as they are rolled out to browsers and may limit the ability to use third-party Javascript libraries. At the very least, using one these transpilers introduces added complexity to deploying a web app1.

In this post, I will describe some things I've learned about Javascript from the perspective of someone who prefers to use Python as much as possible. This guide is mainly aimed at scientists and others who are not primarily programmers but who may find it useful to make a web app for their main work. It is assumed that the reader is at least moderately familiar with Javascript (Mozilla has a nice ...

more ...

Getting Matplotlib's colors in order

Matplotlib can be very easy to use at times, especially if you just want to make a simple "y vs. x" type of plot. But when it comes to specialized customization, it can be a bit challenging to find the proper solution. The situation is not helped by the fact that a lot of times, an obscure answer on Stack Overflow no longer works because the API changed.

One common need is to color things in the same way. For example, say you want to plot two dependent variables with widely different scales that share an independent variable. This is often represented by having two separate vertical axes which are colored to match the lines or markers of each data set. The most basic approach is to manually assign colors for the lines and axes, but if using a custom style, such as the ggplot style, we need a way to access the color cycle used if we want to remain consistent with the selected style. Here is the best way I have found which works at least in version 1.5:

colors = plt.rcParams["axes.prop_cycle"].by_key()["color"]

colors should now be a list which contains the colors defined ...

more ...

Fitting with lmfit

General-purpose fitting in Python can sometimes be a bit more challenging than one might at first suspect given the robust nature of tools like Numpy and Scipy. First we had leastsq. It works, although often requires a bit of manual tuning of initial guesses and always requires manual calculation of standard error from a covariance matrix (which isn't even one of the return values by default). Later we got curve_fit which is a bit more user friendly and even estimates and returns standard error for us by default! Alas, curve_fit is just a convenience wrapper on top of leastsq and suffers from some of the same general headaches.

These days, we have the wonderful lmfit package. Not only can lmfit make fitting more user friendly, but it also is quite a bit more robust than using scipy directly. The documentation is thorough and rigorous, but that can also mean that it can be a bit overwhelming to get started with it. Here I work through a basic example in two slightly different ways in order to demonstrate how to use it.

Generating the data

Let's assume we have data that resembles a decaying sine wave (e.g., a ...

more ...

Using Postgres as a time series database

Time series databases (TSDBs) are quite popular these days. To name a few, there are InfluxDB, Graphite, Druid, Kairos, and Prometheus. All aim to optimize data storage and querying for time-based data, which is highly relevant in a physics labs where there are multitude of "metrics" (to borrow a phrase used frequently in TSDB documentation) that naturally lend themselves to time series representation: lab (and individual device) temperatures, vacuum chamber pressures, and laser powers, just to name a few. Ideally, one could log various data to one of these databases and then use a tool like Grafana to visualize it. Sadly, more traditional relational databases like SQLite and PostgreSQL are not (currently) supported by Grafana (although this is now being addressed by a datasource plugin in development).

Nevertheless, there are quite a few reasons to favor a traditional RDBMS over a newfangled TSDB. To name a few:

  • Longevity: SQL has been around since the 1970s and became standardized in the 1980s.
  • Ubiquity: almost every server (web or otherwise) has an instance of SQL installed. If not, SQLite doesn't even require a server!
  • Community: not to suggest there aren't good communities with TSDBs, but the Postgres and SQLite communities ...
more ...