GNU Astronomy Utilities


Next: , Previous: , Up: MakeCatalog   [Contents][Index]


7.4.2 Quantifying measurement limits

No measurement on a real dataset can be perfect: you can only reach a certain level/limit of accuracy. Therefore, a meaningful (scientific) analysis requires an understanding of these limits for the dataset and your analysis tools: different datasets have different noise properties and different detection methods (one method/algorith/software that is run with a different set of parameters is considered as a different detection method) will have different abilities to detect or measure certain kinds of signal (astronomical objects) and their properties in the dataset. Hence, quantifying the detection and measurement limitations with a particular dataset and analysis tool is the most crucial/critical aspect of any high-level analysis.

Here, we’ll review some of the most general limits that are important in any astronomical data analysis and how MakeCatalog makes it easy to find them. Depending on the higher-level analysis, there are more tests that must be done, but these are relatively low-level and usually necessary in most cases. In astronomy, it is common to use the magnitude (a unit-less scale) and physical units, see Flux Brightness and magnitude. Therefore the measurements discussed here are commonly used in units of magnitudes.

Surface brightness limit (of whole dataset)

As we make more observations on one region of the sky, and add the observations into one dataset, the signal and noise both increase. However, the signal increase much faster than the noise: assuming you add \(N\) datasets with equal exposure times, the signal will increases as a multiple of \(N\), while noise increases as \(\sqrt{N}\). Thus this increases the signal-to-noise ratio. Qualitatively, fainter (per pixel) parts of the objects/signal in the image will become more visible/detectable. The noise-level is known as the dataset’s surface brightness limit.

You can think of the noise as muddy water that is completely covering a flat ground136. The signal (or astronomical objects in this analogy) will be summits/hills that start from the flat sky level (under the muddy water) and can sometimes reach outside of the muddy water. Let’s assume that in your first observation the muddy water has just been stirred and you can’t see anything through it. As you wait and make more observations/exposures, the mud settles down and the depth of the transparent water increases, making the summits visible. As the depth of clear water increases, the parts of the hills with lower heights (parts with lower surface brightness) can be seen more clearly. In this analogy, height (from the ground) is surface brightness137 and the height of the muddy water is your surface brightness limit.

The outputs of NoiseChisel include the Sky standard deviation (\(\sigma\)) on every group of pixels (a mesh) that were calculated from the undetected pixels in each tile, see Tessellation and NoiseChisel output. Let’s take \(\sigma_m\) as the median \(\sigma\) over the successful meshes in the image (prior to interpolation or smoothing).

On different instruments, pixels have different physical sizes (for example in micro-meters, or spatial angle over the sky). Nevertheless, a pixel is our unit of data collection. In other words, while quantifying the noise, the physical or projected size of the pixels is irrelevant. We thus define the Surface brightness limit or depth, in units of magnitude/pixel, of a data-set, with zeropoint magnitude \(z\), with the \(n\)th multiple of \(\sigma_m\) as (see Flux Brightness and magnitude):

$$SB_{\rm Pixel}=-2.5\times\log_{10}{(n\sigma_m)}+z$$

As an example, the XDF survey covers part of the sky that the Hubble space telescope has observed the most (for 85 orbits) and is consequently very small (\(\sim4\) arcmin\(^2\)). On the other hand, the CANDELS survey, is one of the widest multi-color surveys covering several fields (about 720 arcmin\(^2\)) but its deepest fields have only 9 orbits observation. The depth of the XDF and CANDELS-deep surveys in the near infrared WFC3/F160W filter are respectively 34.40 and 32.45 magnitudes/pixel. In a single orbit image, this same field has a depth of 31.32. Recall that a larger magnitude corresponds to less brightness.

The low-level magnitude/pixel measurement above is only useful when all the datasets you want to use belong to one instrument (telescope and camera). However, you will often find yourself using datasets from various instruments with different pixel scales (projected pixel sizes). If we know the pixel scale, we can obtain a more easily comparable surface brightness limit in units of: magnitude/arcsec\(^2\). Let’s assume that the dataset has a zeropoint value of \(z\), and every pixel is \(p\) arcsec\(^2\) (so \(A/p\) is the number of pixels that cover an area of \(A\) arcsec\(^2\)). If the surface brightness is desired at the \(n\)th multiple of \(\sigma_m\), the following equation (in units of magnitudes per A arcsec\(^2\)) can be used:

$$SB_{\rm Projected}=-2.5\times\log_{10}{\left(n\sigma_m\sqrt{A\over p}\right)+z}$$

Note that this is just an extrapolation of the per-pixel measurement \(\sigma_m\). So it should be used with extreme care: for example the dataset must have an approximately flat depth or noise properties overall. A more accurate measure for each detection over the dataset is known as the upper-limit magnitude which actually uses random positioning of each detection’s area/footprint (see below). It doesn’t extrapolate and even accounts for correlated noise patterns in relation to that detection. Therefore, the upper-limit magnitude is a much better measure of your dataset’s surface brightness limit for each particular object.

MakeCatalog will calculate the input dataset’s \(SB_{\rm Pixel}\) and \(SB_{\rm Projected}\) and write them as comments/meta-data in the output catalog(s). Just note that \(SB_{\rm Projected}\) is only calculated if the input has World Coordinate System (WCS).

Completeness limit (of each detection)

As the surface brightness of the objects decreases, the ability to detect them will also decrease. An important statistic is thus the fraction of objects of similar morphology and brightness that will be identified with our detection algorithm/parameters in the given image. This fraction is known as completeness. For brighter objects, completeness is 1: all bright objects that might exist over the image will be detected. However, as we go to objects of lower overall surface brightness, we will fail to detect some, and gradually we are not able to detect anything any more. For a given profile, the magnitude where the completeness drops below a certain level (usually above \(90\%\)) is known as the completeness limit.

Another important parameter in measuring completeness is purity: the fraction of true detections to all true detections. In effect purity is the measure of contamination by false detections: the higher the purity, the lower the contamination. Completeness and purity are anti-correlated: if we can allow a large number of false detections (that we might be able to remove by other means), we can significantly increase the completeness limit.

One traditional way to measure the completeness and purity of a given sample is by embedding mock profiles in regions of the image with no detection. However in such a study we must be really careful to choose model profiles as similar to the target of interest as possible.

Magnitude measurement error (of each detection)

Any measurement has an error and this includes the derived magnitude for an object. Note that this value is only meaningful when the object’s magnitude is brighter than the upper-limit magnitude (see the next items in this list). As discussed in Flux Brightness and magnitude, the magnitude (\(M\)) of an object with brightness \(B\) and Zeropoint magnitude \(z\) can be written as:

$$M=-2.5\log_{10}(B)+z$$

Calculating the derivative with respect to \(B\), we get:

$${dM\over dB} = {-2.5\over {B\times ln(10)}}$$

From the Tailor series (\(\Delta{M}=dM/dB\times\Delta{B}\)), we can write:

$$\Delta{M} = \left|{-2.5\over ln(10)}\right|\times{\Delta{B}\over{B}}$$

But, \(\Delta{B}/B\) is just the inverse of the Signal-to-noise ratio (\(S/N\)), so we can write the error in magnitude in terms of the signal-to-noise ratio:

$$\Delta{M} = {2.5\over{S/N\times ln(10)}} $$

MakeCatalog uses this relation to estimate the magnitude errors. The signal-to-noise ratio is calculated in different ways for clumps and objects (see Akhlaghi and Ichikawa [2015]), but this single equation can be used to estimate the measured magnitude error afterwards for any type of target.

Upper limit magnitude (of each detection)

Due to the noisy nature of data, it is possible to get arbitrarily low values for a faint object’s brightness (or arbitrarily high magnitudes). Given the scatter caused by the dataset’s noise, values fainter than a certain level are meaningless: another similar depth observation will give a radically different value.

For example, while the depth of the image is 32 magnitudes/pixel, a measurement that gives a magnitude of 36 for a \(\sim100\) pixel object is clearly unreliable. In another similar depth image, we might measure a magnitude of 30 for it, and yet another might give 33. Furthermore, due to the noise scatter so close to the depth of the data-set, the total brightness might actually get measured as a negative value, so no magnitude can be defined (recall that a magnitude is a base-10 logarithm). This problem usually becomes relevant when the detection labels were not derived from the values being measured (for example when you are estimating colors, see MakeCatalog).

Using such unreliable measurements will directly affect our analysis, so we must not use the raw measurements. But how can we know how reliable a measurement on a given dataset is?

When we confront such unreasonably faint magnitudes, there is one thing we can deduce: that if something actually exists here (possibly buried deep under the noise), it’s inherent magnitude is fainter than an upper limit magnitude. To find this upper limit magnitude, we place the object’s footprint (segmentation map) over random parts of the image where there are no detections, so we only have pure (possibly correlated) noise, along with undetected objects. Doing this a large number of times will give us a distribution of brightness values. The standard deviation (\(\sigma\)) of that distribution can be used to quantify the upper limit magnitude.

Traditionally, faint/small object photometry was done using fixed circular apertures (for example with a diameter of \(N\) arc-seconds). Hence, the upper limit was like the depth discussed above: one value for the whole image. The problem with this simplified approach is that the number of pixels in the aperture directly affects the final distribution and thus magnitude. Also the image correlated noise might actually create certain patters, so the shape of the object can also affect the final result result. Fortunately, with the much more advanced hardware and software of today, we can make customized segmentation maps for each object.

When requested, MakeCatalog will randomly place each target’s footprint over the dataset as described above and estimate the resulting distribution’s properties (like the upper limit magnitude). The procedure is fully configurable with the options in Upper-limit settings. If one value for the whole image is required, you can either use the surface brightness limit above or make a circular aperture and feed it into MakeCatalog to request an upper-limit magnitude for it138.


Footnotes

(136)

The ground is the sky value in this analogy, see Sky value. Note that this analogy only holds for a flat sky value across the surface of the image or ground.

(137)

Note that this muddy water analogy is not perfect, because while the water-level remains the same all over a peak, in data analysis, the Poisson noise increases with the level of data.

(138)

If you intend to make apertures manually and not use a detection map (for example from Segment), don’t forget to use the --upmaskfile to give NoiseChisel’s output (or any a binary map, marking detected pixels, see NoiseChisel output) as a mask. Otherwise, the footprints may randomly fall over detections, giving highly skewed distributions, with wrong upper-limit distributions. See The description of --upmaskfile in Upper-limit settings for more.


Next: , Previous: , Up: MakeCatalog   [Contents][Index]