We have now released all HiRISE images taken prior to August’s spacecraft safe mode event! Here are some statistics about our October 2009 release, which includes the images the HiRISE camera took of the Martian surface between Mars Reconnaissance Orbiter (MRO) orbits 14,200 to 14,499 (August 6, 2009 – August 26, 2009):

• 446 RDRs, 0.18 TB
• 6238 EDRs, 0.18 TB
• 5126 RDR Extras, 0.28 TB
• 12,464 EDR Extras 2.5 GB
• 16 Anaglyphs 0.001 TB

Totals for this release: 24,274 images, 0.62 TB

This brings our total released product numbers and data volume to:

• 23,122 RDRs, 12.2 TB
• 323,358 EDRs, 10.6 TB
• 196,058 RDR Extras, 15.6 TB
• 625,233, EDR Extras, 0.1 TB
• 1,192 Anaglyphs 0.5 TB

Total: 1,167,771 images, 37.7 TB

Just because we are not currently taking images does not mean we are slacking off. The Downlink team is busy reprocessing and validating all ESP observations. After reprocessing, these observations will all benefit from the same improvements we have made to our processing pipelines over the past several months. I also recently started reprocessing PSP observations, which is a much larger data set that will sync improvement to our processing pipelines made over the past few years! We are keeping busy and we are even getting help from the Uplink team while they wait for the go ahead to start taking new images of the Martian surface. Of course we all want that to happen as quickly as possible!

The HiRISE project has developed a fairly significant amount of software. I’ve been privileged to play a part in that development, which continues even as we get deeper into the primary mission. So, rather than space science or operations, this post will discuss one of the nittier, grittier aspects of our work.

The processing pipelines have been introduced in earlier entries. Thanks to the efforts of HiRISE developers (mostly before my time with the project) these have provided a very solid foundation for our automated ground data system. There has been very little need for trouble-shooting or fine-tuning of the core software.

One issue that did come up earlier in PSP however was a strange failure that happened periodically, though not predictably. If you are a programmer, there is nothing so dreadful as a bizarre, non-repeatable bug… not counting Monday morning meetings, of course.

(more…)

To conclude our exploration of the pipelines that take raw channel files and create a beautiful, unmapped mosaic, let me introduce the Stitch pipelines: HiStitch and HiccdStitch.

The HiStitch pipeline combines the matching HiCal products for the same CCD into one more-or-less lined up CCD cube file. HiccdStitch combines these HiStitch cubes into RED, IR, and BG mosaics.

Both pipelines take some time, as overlapping pixels are accounted for and brought together. After these mosaics are created, additional steps create smaller jpeg files for easier viewing, and full-sized jpeg2000 files. We use these jpeg2000 files for validating our images.

There are later pipelines, but we first validate the HiccdStitch products: Did the previous pipelines work correctly? Did the uplink team command the camera correctly? Is there haze or clouds obscuring our view of the surface?

If everything looks good, and we have received the correct reconstructed SPICE ephemeris data, then the geometry pipelines are invoked. These pipelines project the images mathematically to a model of Mars and add geometry data to the images so that each pixel becomes a point on Mars with latitude and longitude coordinates.

The raw HiRISE image data needs to be cleaned up, and the HiCal pipeline is where this work is done. Now that the raw image data has been converted to a *.cub file format, ISIS tools can be used. These include hiclean, hipical, and hidestripe.

Hiclean does just what is says. Noise introduced into the image data by spacecraft electronics is corrected. Noise can show up as vertical and horizontal lines in the raw image and other periodic manifestations.

Hipical is a newer tool that performs calibration on the image data. For example, flatfield and gain corrections are performed by hipical. Hipical will be upgraded as we learn more about our instrument in its environment around Mars.

Hidestripe corrects a known striping pattern in HiRISE images.

We use other tools to collect even more statistical data about the newly calibrated image data. The HiCal pipeline will continue to be upgraded as our software matures. New statistics will be collected while corrections are added or improved.

After cleanup has been completed and a new *.hical.cub channel product created, HiCal creates a variety of jpeg browse and thumbnail images. The cleaned up channels are large, and for quick previews, these smaller jpegs come in handy.

Finally, HiCal lets the next pipeline – HiStitch – know that cleaned up channels are ready to be stitched together into CCD products.

Below is an example of raw data, prior to going through the HiCal pipeline. This image sample was taken from TRA_000873_1780; “Victoria Crater” at Meridiani Planum.

Below is the same image sample after going through the HiCal pipeline (notice that the bright vertical line in the center and the faint vertical lines throughout the image have been correctly removed by HiCal):

After a channel of raw data has been downloaded and converted into an *.IMG file, we need one more conversion before cleanup of the image can begin.

The EDR_Stats pipeline creates a *.cub file from the *.IMG file. These cube files are the file type used in ISIS 3.0, an image processing software package provided for planetary science missions by the United States Geological Survey (USGS). This package contains an entire suite of useful tools, many of which are used by our pipelines.

During the creation of a cube, a variety of statistics are gathered. For example, the number of gaps, saturated pixels, calibration pixels, and other pixels are counted. Image mean, standard deviation, and other statistics are also calculated. EDR_Stats takes these results and uploads them to our database. The resulting cube is archived in our storage directory.

The final EDR_Stats pipeline step lets the next pipeline – HiCal – know that an image channel cube file is ready for calibration processing. Let the cleanup of image data begin!

The Planetary Society has an excellent article entitled “HiRISE Image Processing” based on Tuvas’ HiBlog post “Processing images at HiROC“. Both articles explain the EDRgen pipeline very well.

It is important to note that while there are a multitude of image formats available, Experimental Data Records (EDRs) are a standardized way of packaging planetary science data sets for release to the world while ensuring future access to said data. In the case of HiRISE images, there are two components to an EDR product: (1) the image data and (2) the label.

The EDRgen pipeline uses a program called HiRISE_Observation to create an EDR from the original channel raw data. The image data is converted into a file type with the extension *.IMG and important information about the observation is attached to this *.IMG file in the form of a text label. This label includes information about this mission; the observation name, commanding, time, and temperatures parameters; and other useful information.

After the EDR is created, it is archived in our storage directory hierarchy (we follow a hierarchy that includes mission phase, orbit range, and observation ID). Finally, the database sources table for the next pipeline – EDR_Stats – is updated with the location of the new EDR. Further processing of this EDR, in a different format, is necessary to start cleaning up the image.

How long do each of these pipelines take? HiDog generally downloads a new channel file in a few minutes or less. EDRgen can create a new *.IMG file in a few minutes or less, and we have a few EDRgen pipelines working in parallel. The fact is, most of the pipelines are incredibly fast on our processing cluster. Later pipelines that stitch and mosaic take significantly longer, but rapid progress in computer technology have blown away early conservative estimates of how long HiRISE image processing would take.

We mention our automated pipelines a lot, so I might as well jump in and provide some more information about them, on top of what might already have been mentioned before. I will start with the first one – HiDog – in a moment, but first, let me introduce Watchdog.

You should know by now the route our data takes: from the HiRISE camera on MRO to storage to spacecraft radiation to the Deep Space Network radio telescopes here on Earth to the ground data system network to JPL in Pasadena, CA to the University of Arizona campus network to our servers in the HiRISE Operations Center. Our Watchdog software, well, watches the JPL servers for new HiRISE raw image data. When it sees a new raw channel file (2 channels per CCD, up to 14 CCDs per observation), Watchdog flags that file as ready to be downloaded by HiDog.

HiDog is the first automated pipeline. It wakes up every few minutes to see if the Watchdog has flagged any new files (basically, it is checking a sources table in our database). If there is nothing new in the sources table, then it goes back to sleep. If there is something new, HiDog wags its tail, rapidly downloads the file, checks to see if there are any gaps in the data, and then tells the next pipeline that a new image channel has arrived in Tucson, ready for further processing. Then, it checks to see if there are any more files ready for downloading, and goes back to sleep if there are not. Sweet dreams, little doggy.

Over and over again, 24 hours a day, 7 days a week, the Dogs are ready and waiting for the latest HiRISE data from Mars.

Next time…the EDRgen pipeline.

The first HiRISE image data of the Primary Science Phase (PSP) arrived in Tucson last night sometime around 9 PM. Although we thought the first data might not arrive until early this morning, I was a little antsy and took a look from home around 9:40 PM to see a complete first observation ready for validation.

We are waiting for reconstructed SPICE ephemeris data, which comes out every Wednesday – starting next week – before sending these data through our geometry pipelines, and ultimately releasing them to the scientific community and public. Last time, we forced images through our geometry pipelines using predicted SPICE kernels; we do not want to double our workload by continuing that practice. The SPICE kernels released next Wednesday will cover some of the images captured this week.

Once the images have been visually and statistically validated and the matching SPICE kernels have arrived, one of the downlink folks will send the images through the geometry pipelines. We also need to get a select group of captions written and automatic caption information generated for the rest.

We are producing JPEG2000 products now in addition to smaller jpeg browse images, to be ready for our viewing client when it is ready for public release. However, there are many different JPEG2000 viewers and plugins already out there to start practicing with. One example is ExpressView from LizardTech.

Once we are on a roll, the data release will be steady and no one will be able to keep up with the wealth of Mars data coming in. Until the first public release of PSP images, we will try to provide here on HiBlog more details about the many tasks that must still be completed.

Some of you out there may be asking: what happens to a HiRISE image between the time that it is taken and the time that it is released to the public? Well, I’d like to give a summary here.

(more…)