Binning Data to a New Grid¶
Often we have some y values the correspond to a particular grid of x values, and we want to resample them onto a different grid of x values. Interpolation is one way to do this, but it won't necessarily provide reasonable averages over wiggly features. There are lots of different features we might hope for in a resampling routine, but one common one is that we'd like get the same answer when integrating between two x limits, whether we're using the original or the resampled grid of quantities. One way to ensure such integrals are conserved is to calculate the cumulative distribution function (= the integral up to a particular limit) of the original arrays, interpolate onto the new grid, and differentiate; Diamond-Lowe et al. 2020 provides a literature example of this algorthim being used for exoplanet transmission spectrum observations.
For chromatic, the tools used to achieve this are the bintogrid and bintoR functions, which we demonstrate below.
from chromatic import bintogrid, bintoR, version
import numpy as np, matplotlib.pyplot as plt
plt.matplotlib.rcParams["figure.figsize"] = (8, 3)
plt.matplotlib.rcParams["figure.dpi"] = 300
version()
'0.4.12'
How do we bin some input arrays?¶
Let's create a fake input dataset with an input grid that is uniformly spaced in x.
N = 100
x = np.linspace(1, 5, N)
y = x**2
Then, let's use bintogrid to bin these input arrays onto a new grid with wider spacing. The results of this function are a dictionary that contain:
x= the center of the output gridy= the resampled value on the output gridx_edge_lower= the lower edges of the output gridx_edge_upper= the upper edges of the output gridN_unbinned/N_binned= the approximate number of input bins that contributed to each output bin
binned = bintogrid(x, y, dx=0.5)
list(binned.keys())
['x', 'x_edge_lower', 'x_edge_upper', 'y', 'N_unbinned/N_binned']
Let's compare the results on a plot. The resampled values line up very neatly with the
plt.scatter(x, y, alpha=0.5, label="input")
plt.scatter(binned["x"], binned["y"], s=100, alpha=0.5, label="output")
plt.xlabel("x")
plt.ylabel("y")
plt.legend(frameon=False);
This code should work similarly even if the input arrays are non-uniform in x, which can be a nice way to arrange a heterogeneous dataset into something easier to work with.
x = np.sort(np.random.uniform(1, 5, N))
y = x**2
binned = bintogrid(x, y, dx=0.5)
plt.scatter(x, y, alpha=0.5, label="input")
plt.scatter(binned["x"], binned["y"], s=100, alpha=0.5, label="output")
plt.xlabel("x")
plt.ylabel("y")
plt.legend(frameon=False);
The bintoR function is a wrapper around bintogrid that provides a quick way to bin onto logarithmic grid. In spectroscopy, it's common to want to work with wavelengths $\lambda$ that are spaced according to a constant value of $R = \lambda/\Delta \lambda = 1 / \Delta [\ln \lambda]$. This quantity $R$ is often called the spectral resolution.
binned = bintoR(x, y, R=10)
plt.scatter(x, y, alpha=0.5, label="input")
plt.scatter(binned["x"], binned["y"], s=100, alpha=0.5, label="output")
plt.xlabel("x")
plt.ylabel("y")
plt.legend(frameon=False);
This may seem like a weird way to define a new output grid, but its usefulness becomes apparent when we plot on logarithmic axes. It's uniform in log space!
plt.scatter(x, y, alpha=0.5, label="input")
plt.scatter(binned["x"], binned["y"], s=100, alpha=0.5, label="output")
plt.xlabel("x")
plt.ylabel("y")
plt.legend(frameon=False)
plt.xscale("log");
How do we bin with uncertainties?¶
For any real measurements, there are probably uncertainties associated with them. Both bintogrid and bintoR will try their best to propagate uncertainties by using inverse-variance weighting and its maximum likelihood estimate for the binned uncertainty.
Let's look at a similar example as before, but with some uncertainties associated with each input y value.
x = np.linspace(1, 5, N)
uncertainty = np.ones_like(x) * 5
y = np.random.normal(x**2, uncertainty)
Let's resample it to a new grid, providing the known uncertainties on the original points. Notice that the result now also includes an uncertainty key.
binned = bintogrid(x, y, uncertainty, dx=0.5)
list(binned.keys())
['x', 'x_edge_lower', 'x_edge_upper', 'y', 'uncertainty', 'N_unbinned/N_binned']
When we plot the input and output values, we can see that the typical output uncertainties are smaller than the typical input uncertainties, because we've effectively averaged together a few data points and therefore decreased the uncertainty for the new values.
kw = dict(linewidth=0, elinewidth=1, marker="o", alpha=0.5, markeredgecolor="none")
plt.errorbar(x, y, uncertainty, label="input", **kw)
plt.errorbar(
binned["x"], binned["y"], binned["uncertainty"], label="output", markersize=10, **kw
)
plt.xlabel("x")
plt.ylabel("y")
plt.legend(frameon=False);
When we bin onto a logarithmic grid with bintoR, we can see that the uncertainties typically smaller for the higher values of x, where more input points are getting averaged together to make each output point.
binned = bintoR(x, y, uncertainty, R=10)
plt.errorbar(x, y, uncertainty, label="input", **kw)
plt.errorbar(
binned["x"], binned["y"], binned["uncertainty"], label="output", markersize=10, **kw
)
plt.xlabel("x")
plt.ylabel("y")
plt.legend(frameon=False)
plt.xscale("log");
How do we customize the output grid?¶
There are a few different options you can use to specify the exact output grid you would like.
For bintogrid, the options are:
nx= the number of adjacent inputs points that should be binned together to create the output grid (for example, "bin every 3 points together")dx= the spacing for a linearly-uniform output gridnewx= a custom output grid, referring to the centers of the new binsnewx_edges= a custom output grid, referring to the edges of the new bins. The left and right edges of the bins will be, respectively,newx_edges[:-1]andnewx_edges[1:], so the size of the output array will belen(newx_edges) - 1
For bintoR, the options are:
R= the spectral resolution R=x/dx for a logarithmically-uniform output gridxlim= a two-element list indicating the min and max values of x for the new logarithmic output grid. If not supplied, this will center the first output bin on the first value ofx