After a channel of raw data has been downloaded and converted into an *.IMG file, we need one more conversion before cleanup of the image can begin.

The EDR_Stats pipeline creates a *.cub file from the *.IMG file. These cube files are the file type used in ISIS 3.0, an image processing software package provided for planetary science missions by the United States Geological Survey (USGS). This package contains an entire suite of useful tools, many of which are used by our pipelines.

During the creation of a cube, a variety of statistics are gathered. For example, the number of gaps, saturated pixels, calibration pixels, and other pixels are counted. Image mean, standard deviation, and other statistics are also calculated. EDR_Stats takes these results and uploads them to our database. The resulting cube is archived in our storage directory.

The final EDR_Stats pipeline step lets the next pipeline – HiCal – know that an image channel cube file is ready for calibration processing. Let the cleanup of image data begin!

The Planetary Society has an excellent article entitled “HiRISE Image Processing” based on Tuvas’ HiBlog post “Processing images at HiROC“. Both articles explain the EDRgen pipeline very well.

It is important to note that while there are a multitude of image formats available, Experimental Data Records (EDRs) are a standardized way of packaging planetary science data sets for release to the world while ensuring future access to said data. In the case of HiRISE images, there are two components to an EDR product: (1) the image data and (2) the label.

The EDRgen pipeline uses a program called HiRISE_Observation to create an EDR from the original channel raw data. The image data is converted into a file type with the extension *.IMG and important information about the observation is attached to this *.IMG file in the form of a text label. This label includes information about this mission; the observation name, commanding, time, and temperatures parameters; and other useful information.

After the EDR is created, it is archived in our storage directory hierarchy (we follow a hierarchy that includes mission phase, orbit range, and observation ID). Finally, the database sources table for the next pipeline – EDR_Stats – is updated with the location of the new EDR. Further processing of this EDR, in a different format, is necessary to start cleaning up the image.

How long do each of these pipelines take? HiDog generally downloads a new channel file in a few minutes or less. EDRgen can create a new *.IMG file in a few minutes or less, and we have a few EDRgen pipelines working in parallel. The fact is, most of the pipelines are incredibly fast on our processing cluster. Later pipelines that stitch and mosaic take significantly longer, but rapid progress in computer technology have blown away early conservative estimates of how long HiRISE image processing would take.