The HiRISE project has developed a fairly significant amount of software. I’ve been privileged to play a part in that development, which continues even as we get deeper into the primary mission. So, rather than space science or operations, this post will discuss one of the nittier, grittier aspects of our work.

The processing pipelines have been introduced in earlier entries. Thanks to the efforts of HiRISE developers (mostly before my time with the project) these have provided a very solid foundation for our automated ground data system. There has been very little need for trouble-shooting or fine-tuning of the core software.

One issue that did come up earlier in PSP however was a strange failure that happened periodically, though not predictably. If you are a programmer, there is nothing so dreadful as a bizarre, non-repeatable bug… not counting Monday morning meetings, of course.

(more…)

The Planetary Society has an excellent article entitled “HiRISE Image Processing” based on Tuvas’ HiBlog post “Processing images at HiROC“. Both articles explain the EDRgen pipeline very well.

It is important to note that while there are a multitude of image formats available, Experimental Data Records (EDRs) are a standardized way of packaging planetary science data sets for release to the world while ensuring future access to said data. In the case of HiRISE images, there are two components to an EDR product: (1) the image data and (2) the label.

The EDRgen pipeline uses a program called HiRISE_Observation to create an EDR from the original channel raw data. The image data is converted into a file type with the extension *.IMG and important information about the observation is attached to this *.IMG file in the form of a text label. This label includes information about this mission; the observation name, commanding, time, and temperatures parameters; and other useful information.

After the EDR is created, it is archived in our storage directory hierarchy (we follow a hierarchy that includes mission phase, orbit range, and observation ID). Finally, the database sources table for the next pipeline – EDR_Stats – is updated with the location of the new EDR. Further processing of this EDR, in a different format, is necessary to start cleaning up the image.

How long do each of these pipelines take? HiDog generally downloads a new channel file in a few minutes or less. EDRgen can create a new *.IMG file in a few minutes or less, and we have a few EDRgen pipelines working in parallel. The fact is, most of the pipelines are incredibly fast on our processing cluster. Later pipelines that stitch and mosaic take significantly longer, but rapid progress in computer technology have blown away early conservative estimates of how long HiRISE image processing would take.

We mention our automated pipelines a lot, so I might as well jump in and provide some more information about them, on top of what might already have been mentioned before. I will start with the first one – HiDog – in a moment, but first, let me introduce Watchdog.

You should know by now the route our data takes: from the HiRISE camera on MRO to storage to spacecraft radiation to the Deep Space Network radio telescopes here on Earth to the ground data system network to JPL in Pasadena, CA to the University of Arizona campus network to our servers in the HiRISE Operations Center. Our Watchdog software, well, watches the JPL servers for new HiRISE raw image data. When it sees a new raw channel file (2 channels per CCD, up to 14 CCDs per observation), Watchdog flags that file as ready to be downloaded by HiDog.

HiDog is the first automated pipeline. It wakes up every few minutes to see if the Watchdog has flagged any new files (basically, it is checking a sources table in our database). If there is nothing new in the sources table, then it goes back to sleep. If there is something new, HiDog wags its tail, rapidly downloads the file, checks to see if there are any gaps in the data, and then tells the next pipeline that a new image channel has arrived in Tucson, ready for further processing. Then, it checks to see if there are any more files ready for downloading, and goes back to sleep if there are not. Sweet dreams, little doggy.

Over and over again, 24 hours a day, 7 days a week, the Dogs are ready and waiting for the latest HiRISE data from Mars.

Next time…the EDRgen pipeline.