Previous: Sky value misconceptions, Up: Sky value [Contents][Index]

In order to define detection thresholds on the image, or calibrate it for measurements (subtract the signal of the background sky and define errors), we need some basic measurements.
for example, the quantile threshold in NoiseChisel (`--qthresh` option), or the mean of the undetected regions (Sky) and the Sky standard deviation (Sky STD) which are the output of NoiseChisel and Statistics.
But astronomical images will contain a lot of stars and galaxies that will bias those measurements if not properly accounted for.
Quantifying where signal is present is thus a very important step in the usage of a dataset: for example, if the Sky level is over-estimated, your target object’s brightness will be under-estimated.

Let’s start by clarifying some definitions:
*Signal* is defined as the non-random source of flux in each pixel (you can think of this as the mean in a Gaussian or Poisson distribution).
In astronomical images, signal is mostly photons coming of a star or galaxy, and counted in each pixel.
*Noise* is defined as the random source of flux in each pixel (or the standard deviation of a Gaussian or Poisson distribution).
Noise is mainly due to counting errors in the detector electronics upon data collection.
*Data* is defined as the combination of signal and noise (so a noisy image of a galaxy is one *data*set).

When a dataset does not have any signal (for example, you take an image with a closed shutter, producing an image that only contains noise), the mean, median and mode of the distribution are equal within statistical errors. Signal from emitting objects, like astronomical targets, always has a positive value and will never become negative, see Figure 1 in Akhlaghi and Ichikawa (2015). Therefore, when signal is added to the data (you take an image with an open shutter pointing to a galaxy for example), the mean, median and mode of the dataset shift to the positive, creating a positively skewed distribution. The shift of the mean is the largest. The median shifts less, since it is defined after ordering all the elements/pixels (the median is the value at a quantile of 0.5), thus it is not affected by outliers. Finally, the mode’s shift to the positive is the least.

Inverting the argument above gives us a robust method to quantify the significance of signal in a dataset: when the mean and median of a distribution are approximately equal we can argue that there is no significant signal. In other words: when the quantile of the mean (\(q_{mean}\)) is around 0.5. This definition of skewness through the quantile of the mean is further introduced with a real image the tutorials, see Skewness caused by signal and its measurement.

However, in an astronomical image, some of the pixels will contain more signal than the rest, so we cannot simply check \(q_{mean}\) on the whole dataset. for example, if we only look at the patch of pixels that are placed under the central parts of the brightest stars in the field of view, \(q_{mean}\) will be very high. The signal in other parts of the image will be weaker, and in some parts it will be much smaller than the noise (for example, 1/100-th of the noise level). When the signal-to-noise ratio is very small, we can generally assume no signal (because its effectively impossible to measure it) and \(q_{mean}\) will be approximately 0.5.

To address this problem, we break the image into a grid of tiles^{172} (see Tessellation).
for example, a tile can be a square box of size \(30\times30\) pixels.
By measuring \(q_{mean}\) on each tile, we can find which tiles that contain significant signal and ignore them.
Technically, if a tile’s \(|q_{mean}-0.5|\) is larger than the value given to the `--meanmedqdiff` option, that tile will be ignored for the next steps.
You can read this option as “mean-median-quantile-difference”.

The raw dataset’s pixel distribution (in each tile) is noisy, to decrease the noise/error in estimating \(q_{mean}\), we convolve the image before tessellation (see Convolution process. Convolution decreases the range of the dataset and enhances its skewness, See Section 3.1.1 and Figure 4 in Akhlaghi and Ichikawa (2015). This enhanced skewness can be interpreted as an increase in the Signal to noise ratio of the objects buried in the noise. Therefore, to obtain an even better measure of the presence of signal in a tile, the mean and median discussed above are measured on the convolved image.

There is one final hurdle: raw astronomical datasets are commonly peppered with Cosmic rays. Images of Cosmic rays are not smoothed by the atmosphere or telescope aperture, so they have sharp boundaries. Also, since they do not occupy too many pixels, they do not affect the mode and median calculation. But their very high values can greatly bias the calculation of the mean (recall how the mean shifts the fastest in the presence of outliers), for example, see Figure 15 in Akhlaghi and Ichikawa (2015). The effect of outliers like cosmic rays on the mean and standard deviation can be removed through \(\sigma\)-clipping, see Sigma clipping for a complete explanation.

Therefore, after asserting that the mean and median are approximately equal in a tile (see Tessellation), the Sky and its STD are measured on each tile after \(\sigma\)-clipping with the `--sigmaclip` option (see Sigma clipping).
In the end, some of the tiles will pass the test and will be given a value.
Others (that had signal in them) will just be assigned a NaN (not-a-number) value.
But we need a measurement over each tile (and thus pixel).
We will therefore use interpolation to assign a value to the NaN tiles.

However, prior to interpolating over the failed tiles, another point should be considered: large and extended galaxies, or bright stars, have wings which sink into the noise very gradually. In some cases, the gradient over these wings can be on scales that is larger than the tiles (for example, the pixel value changes by \(0.1\sigma\) after 100 pixels, but the tile has a width of 30 pixels).

In such cases, the \(q_{mean}\) test will be successful, even though there is signal. Recall that \(q_{mean}\) is a measure of skewness. If we do not identify (and thus set to NaN) such outlier tiles before the interpolation, the photons of the outskirts of the objects will leak into the detection thresholds or Sky and Sky STD measurements and bias our result, see Detecting large extended targets. Therefore, the final step of “quantifying signal in a tile” is to look at this distribution of successful tiles and remove the outliers. \(\sigma\)-clipping is a good solution for removing a few outliers, but the problem with outliers of this kind is that there may be many such tiles (depending on the large/bright stars/galaxies in the image). We therefore apply the following local outlier rejection strategy.

For each tile, we find the nearest \(N_{ngb}\) tiles that had a usable value (\(N_{ngb}\) is the value given to `--interpnumngb`).
We then sort them and find the difference between the largest and second-to-smallest elements (The minimum is not used because the scatter can be large).
Let’s call this the tile’s *slope* (measured from its neighbors).
All the tiles that are on a region of flat noise will have similar slope values, but if a few tiles fall on the wings of a bright star or large galaxy, their slope will be significantly larger than the tiles with no signal.
We just have to find the smallest tile slope value that is an outlier compared to the rest, and reject all tiles with a slope larger than that.

To identify the smallest outlier, we will use the distribution of distances between sorted elements.
Let’s assume the total number tiles with a good mean-median quantile difference is \(N\).
We sort them in increasing order based on their *slope* (defined above).
So the array of tile slopes can be written as \({s[0], s[1], ..., s[N]}\), where \(s[0]~~
~~

The \(\sigma\)-clipped median and standard deviation of this distribution are then found (this \(\sigma\)-clipping is configured with `--outliersclip` which takes two values, see Sigma clipping).
If the distance between the element and its previous element is more than `--outliersigma` multiples of the \(\sigma\)-clipped standard deviation added with the \(\sigma\)-clipped median, that element is considered an outlier and all tiles larger than that value are ignored.

Formally, if we assume there are \(N\) elements.
They are first sorted.
Searching for the outlier starts on element \(N/3\) (integer division).
Let’s take \(v_i\) to be the \(i\)-th element of the sorted input (with no blank values) and \(m\) and \(\sigma\) as the \(\sigma\)-clipped median and standard deviation from the distances of the previous \(N/3-1\) elements (not including \(v_i\)).
If the value given to `--outliersigma` is displayed with \(s\), the \(i\)-th element is considered as an outlier when the condition below is true.

$${(v_i-v_{i-1})-m\over \sigma}>s$$

Since \(i\) begins from the \(N/3\)-th element in the sorted array (a quantile of \(1/3=0.33\)), the outlier has to be larger than the \(0.33\) quantile value of the dataset.

Once the outlying tiles have been successfully identified and set to NaN, we use nearest-neighbor interpolation to give a value to all tiles in the image.
We do not use parametric interpolation methods (like bicubic), because they will effectively extrapolate on the edges, creating strong artifacts.
Nearest-neighbor interpolation is very simple: for each tile, we find the \(N_{ngb}\) nearest tiles that had a good value, the tile’s value is found by estimating the median.
You can set \(N_{ngb}\) through the `--interpnumngb` option.
Once all the tiles are given a value, a smoothing step is implemented to remove the sharp value contrast that can happen on the edges of tiles.
The size of the smoothing box is set with the `--smoothwidth` option.

As mentioned above, the process above is used for any of the basic measurements (for example, identifying the quantile-based thresholds in NoiseChisel, or the Sky value in Statistics).
You can use the check-image feature of NoiseChisel or Statistics to inspect the steps and visually see each step (all the options that start with `--check`).
For example, as mentioned in the NoiseChisel optimization tutorial, when given a dataset from a new instrument (with differing noise properties), we highly recommend to use `--checkqthresh` in your first call and visually inspect how the parameters above affect the final quantile threshold (e.g., have the wings of bright sources leaked into the threshold?).
The same goes for the `--checksky` option of Statistics or NoiseChisel.

Previous: Sky value misconceptions, Up: Sky value [Contents][Index]

JavaScript license information

GNU Astronomy Utilities 0.19 manual, October 2022.