GNU Astronomy Utilities



4.1.5.1 Separate shell variables for multiple outputs

Sometimes your commands print multiple values and you want to use them as different shell variables. Let’s describe the problem (shown in the box below) with an example (that you can reproduce without any external data).

With the commands below, we’ll first make a noisy (\(\sigma=5\)) image (\(100\times100\) pixels) using Arithmetic. Then, we’ll measure116 its mean and standard deviation using Statistics.

$ astarithmetic 100 100 2 makenew 5 mknoise-sigma -oimg.fits

$ aststatistics img.fits --mean --std
-3.10938611484039e-03 4.99607077069093e+00

THE PROBLEM: you want the first number printed above to be stored in a shell variable called my_mean and the second number to be stored as the my_std shell variable (you are free to choose any name!).

The first thing that may come to mind is to run Statistics two times, and write the output into separate variables like below:

$ my_std=$(aststatistics img.fits --std)           ## NOT SOLUTION! ##
$ my_mean=$(aststatistics img.fits --mean)         ## NOT SOLUTION! ##

But this is not a good solution because as img.fits becomes larger (more pixels), the time it takes for Statistics to simply load the data into memory can be significant. This will slow down your pipeline and besides wasting your time, it contributes to global warming (by spending energy on an un-necessary action; take this seriously because your pipeline may scale up to involve thousands of large datasets)! Furthermore, besides loading of the input data, Statistics (and Gnuastro in general) is designed to do multiple measurements in one pass over the data as much as possible (to further decrease Gnuastro’s carbon footprint). So when given --mean --std, it will measure both in one pass over the pixels (not two passes!). In other words, in this case, you get the two measurements for the cost of one.

How do you separate the values from the first aststatistics command above? One ugly way is to write the two-number output string into a single shell variable and then separate, or tokenize, the string with two subsequent commands like below:

$ meanstd=$(aststatistics img.fits --mean --std)   ## NOT SOLUTION! ##
$ my_mean=$(echo $meanstd | awk '{print $1}')      ## NOT SOLUTION! ##
$ my_std=$(echo $meanstd | awk '{print $2}')       ## NOT SOLUTION! ##

SOLUTION: The solution is to formatted-print (printf) the numbers as shell variables definitions in a string, and evaluate (eval) that string as a command:

$ eval "$(aststatistics img.fits --mean --std \
                        | xargs printf "my_mean=%s; my_std=%s")"

Let’s review the solution (in more detail):

  1. We pipe the output into xargs117 (extended arguments) which puts the two numbers it gets from the pipe, as arguments for printf (formatted print; because printf doesn’t take input from pipes).
  2. Within the printf call, we write the values after putting a variable name and equal-sign, and in between them we put a ; (as if it was a shell command). The %s tells printf to print each input as a string (not to interpret it as a number and loose precision). Here is the output of this phase:
    $ aststatistics img.fits --mean --std \
                          | xargs printf "my_mean=%s; my_std=%s"
    my_mean=-3.10938611484039e-03; my_std=4.99607077069093e+00
    
  3. But the output above is a string! To evaluate this string as a command, we give it to the eval command like above.

After the solution above, you will have the two my_mean and my_std variables to use separately in your pipeline:

$ echo $my_mean
-3.10938611484039e-03
$ echo $my_std
4.99607077069093e+00

This eval-based solution has been tested in in GNU Bash, Dash and Zsh and it works nicely in them (is “portable”). This is because the constructs used here are pretty low-level (and widely available).

For examples usages of this technique, see the following sections: Extracting a single spectrum and plotting it and Pseudo narrow-band images.


Footnotes

(116)

The actual printed values by aststatistics may slightly differ for you. This is because of a different random number generator seed used in astarithmetic. To get an exactly reproducible result, see Generating random numbers

(117)

For more on xargs, see https://en.wikipedia.org/wiki/Xargs. It will take the standard input (from the pipe in this scenario) and put it as arguments of the next program (printf in this scenario). In other words, it is good for programs that don’t take input from standard input (printf in this case; but also includes others like cp, rm, or echo).