• 446 RDRs, 0.18 TB
• 6238 EDRs, 0.18 TB
• 5126 RDR Extras, 0.28 TB
• 12,464 EDR Extras 2.5 GB
• 16 Anaglyphs 0.001 TB

Totals for this release: 24,274 images, 0.62 TB

This brings our total released product numbers and data volume to:

• 23,122 RDRs, 12.2 TB
• 323,358 EDRs, 10.6 TB
• 196,058 RDR Extras, 15.6 TB
• 625,233, EDR Extras, 0.1 TB
• 1,192 Anaglyphs 0.5 TB

Total: 1,167,771 images, 37.7 TB

Just because we are not currently taking images does not mean we are slacking off. The Downlink team is busy reprocessing and validating all ESP observations. After reprocessing, these observations will all benefit from the same improvements we have made to our processing pipelines over the past several months. I also recently started reprocessing PSP observations, which is a much larger data set that will sync improvement to our processing pipelines made over the past few years! We are keeping busy and we are even getting help from the Uplink team while they wait for the go ahead to start taking new images of the Martian surface. Of course we all want that to happen as quickly as possible!

The processing pipelines have been introduced in earlier entries. Thanks to the efforts of HiRISE developers (mostly before my time with the project) these have provided a very solid foundation for our automated ground data system. There has been very little need for trouble-shooting or fine-tuning of the core software.

One issue that did come up earlier in PSP however was a strange failure that happened periodically, though not predictably. If you are a programmer, there is nothing so dreadful as a bizarre, non-repeatable bug… not counting Monday morning meetings, of course.

This particular bug happened when a new observation came down. Each processing pipeline needs to make a new directory, a new folder in the file system where it can store a log of its work. Starting with the transition phase, most of our pipelines have two or more instances running in parallel on a small Linux cluster, each working on part of the observation.

The bug was that, occasionally, the directory creation would fail. Someone from downlink would have to investigate and take corrective action. Or, if the way forward was unclear, to call in somebody from the software development crew.

In this case, the problems seemed to happen downstream from “my” pipelines: I was blissfully unaware that it could effect HiDog and EDRgen as well.

Then on Thursday, Rich emails to say that the workarounds put in place by two other developers seemed to have helped their pipelines, could I add them to my code as well?

A little shocked, I replied (no doubt too haughtily) that I’d rather understand the problem than apply some hasty workarounds. It had all the classic hallmarks of a multi-threaded problem: notoriously hard to repeat & difficult to debug, but always when two processes were going after the same thing.

The particular bit of code—a subroutine shared by each pipeline to create a log directory—looked absolutely fine on first inspection. It uses the perl language and invokes a function called mkdir. That really should be no problem, I thought. Perhaps if there were a permissions problem. But this succeeded more often than not.

So that evening, I decided to try a little test: a small program that I’d run simultaneously in two windows. Each one would try to make the same directory, whose name would contain the current time in seconds. Since the mkdir operation should take much less than a second, there should be frequent “collisions”—if that was really what the problem was.

As a developer, you grow accustomed to the fact that almost nothing works right on the first try; you have to take small steps and constantly check yourself. So I first tried a test in just a single window. It looked like this:

% while 1
? perl -e 'print "mkdir failed!" unless(mkdir $ARGV[0]);' `date '+%S'`
? end

That should succeed every time, I thought, then I’ll move on to trying it in two terminal windows at the same time. But lo and behold, I pressed return and started getting many failure messages!

In the blinding flash of the obvious, I realized what I should have known: that mkdir returns false, indicating it failed to make the directory, because it already been created. My test script was interfering with itself, succeeding when it was a new second and failing every other time.

This explained the bug. It was what is called a race condition. The condition in this case was the creation of a directory. Two pipelines were racing to create the same directory. Occasionally they were close enough that they were calling the same subroutine, and both bypassed its initial check that the directory already existed. One got to the mkdir step a fraction of a second ahead of the other, probably due to a slightly heavier load on the machine, or other variables.

Compounding the problem, the workarounds that had been added in several previous versions (retrying up to five times with random delays) failed to check properly. A single check after mkdir would remove the race condition. Testing this (in parallel) works as expected.

Or a simpler way is to call the Linux mkdir command with an option so that it does what I (and perhaps the other developers) thought it should: to return true (success) if the directory exists or was created.

Tracking down a mystery like this is a fun part of the job. But only if it happens infrequently.

The HiStitch pipeline combines the matching HiCal products for the same CCD into one more-or-less lined up CCD cube file. HiccdStitch combines these HiStitch cubes into RED, IR, and BG mosaics.

Both pipelines take some time, as overlapping pixels are accounted for and brought together. After these mosaics are created, additional steps create smaller jpeg files for easier viewing, and full-sized jpeg2000 files. We use these jpeg2000 files for validating our images.

There are later pipelines, but we first validate the HiccdStitch products: Did the previous pipelines work correctly? Did the uplink team command the camera correctly? Is there haze or clouds obscuring our view of the surface?

If everything looks good, and we have received the correct reconstructed SPICE ephemeris data, then the geometry pipelines are invoked. These pipelines project the images mathematically to a model of Mars and add geometry data to the images so that each pixel becomes a point on Mars with latitude and longitude coordinates.

Hiclean does just what is says. Noise introduced into the image data by spacecraft electronics is corrected. Noise can show up as vertical and horizontal lines in the raw image and other periodic manifestations.

Hipical is a newer tool that performs calibration on the image data. For example, flatfield and gain corrections are performed by hipical. Hipical will be upgraded as we learn more about our instrument in its environment around Mars.

Hidestripe corrects a known striping pattern in HiRISE images.

We use other tools to collect even more statistical data about the newly calibrated image data. The HiCal pipeline will continue to be upgraded as our software matures. New statistics will be collected while corrections are added or improved.

After cleanup has been completed and a new *.hical.cub channel product created, HiCal creates a variety of jpeg browse and thumbnail images. The cleaned up channels are large, and for quick previews, these smaller jpegs come in handy.

Finally, HiCal lets the next pipeline – HiStitch – know that cleaned up channels are ready to be stitched together into CCD products.

Below is an example of raw data, prior to going through the HiCal pipeline. This image sample was taken from TRA_000873_1780; “Victoria Crater” at Meridiani Planum.

Below is the same image sample after going through the HiCal pipeline (notice that the bright vertical line in the center and the faint vertical lines throughout the image have been correctly removed by HiCal):

The EDR_Stats pipeline creates a *.cub file from the *.IMG file. These cube files are the file type used in ISIS 3.0, an image processing software package provided for planetary science missions by the United States Geological Survey (USGS). This package contains an entire suite of useful tools, many of which are used by our pipelines.

During the creation of a cube, a variety of statistics are gathered. For example, the number of gaps, saturated pixels, calibration pixels, and other pixels are counted. Image mean, standard deviation, and other statistics are also calculated. EDR_Stats takes these results and uploads them to our database. The resulting cube is archived in our storage directory.

The final EDR_Stats pipeline step lets the next pipeline – HiCal – know that an image channel cube file is ready for calibration processing. Let the cleanup of image data begin!

It is important to note that while there are a multitude of image formats available, Experimental Data Records (EDRs) are a standardized way of packaging planetary science data sets for release to the world while ensuring future access to said data. In the case of HiRISE images, there are two components to an EDR product: (1) the image data and (2) the label.

The EDRgen pipeline uses a program called HiRISE_Observation to create an EDR from the original channel raw data. The image data is converted into a file type with the extension *.IMG and important information about the observation is attached to this *.IMG file in the form of a text label. This label includes information about this mission; the observation name, commanding, time, and temperatures parameters; and other useful information.

After the EDR is created, it is archived in our storage directory hierarchy (we follow a hierarchy that includes mission phase, orbit range, and observation ID). Finally, the database sources table for the next pipeline – EDR_Stats – is updated with the location of the new EDR. Further processing of this EDR, in a different format, is necessary to start cleaning up the image.

How long do each of these pipelines take? HiDog generally downloads a new channel file in a few minutes or less. EDRgen can create a new *.IMG file in a few minutes or less, and we have a few EDRgen pipelines working in parallel. The fact is, most of the pipelines are incredibly fast on our processing cluster. Later pipelines that stitch and mosaic take significantly longer, but rapid progress in computer technology have blown away early conservative estimates of how long HiRISE image processing would take.

You should know by now the route our data takes: from the HiRISE camera on MRO to storage to spacecraft radiation to the Deep Space Network radio telescopes here on Earth to the ground data system network to JPL in Pasadena, CA to the University of Arizona campus network to our servers in the HiRISE Operations Center. Our Watchdog software, well, watches the JPL servers for new HiRISE raw image data. When it sees a new raw channel file (2 channels per CCD, up to 14 CCDs per observation), Watchdog flags that file as ready to be downloaded by HiDog.

HiDog is the first automated pipeline. It wakes up every few minutes to see if the Watchdog has flagged any new files (basically, it is checking a sources table in our database). If there is nothing new in the sources table, then it goes back to sleep. If there is something new, HiDog wags its tail, rapidly downloads the file, checks to see if there are any gaps in the data, and then tells the next pipeline that a new image channel has arrived in Tucson, ready for further processing. Then, it checks to see if there are any more files ready for downloading, and goes back to sleep if there are not. Sweet dreams, little doggy.

Over and over again, 24 hours a day, 7 days a week, the Dogs are ready and waiting for the latest HiRISE data from Mars.

Next time…the EDRgen pipeline.

We are waiting for reconstructed SPICE ephemeris data, which comes out every Wednesday – starting next week – before sending these data through our geometry pipelines, and ultimately releasing them to the scientific community and public. Last time, we forced images through our geometry pipelines using predicted SPICE kernels; we do not want to double our workload by continuing that practice. The SPICE kernels released next Wednesday will cover some of the images captured this week.

Once the images have been visually and statistically validated and the matching SPICE kernels have arrived, one of the downlink folks will send the images through the geometry pipelines. We also need to get a select group of captions written and automatic caption information generated for the rest.

We are producing JPEG2000 products now in addition to smaller jpeg browse images, to be ready for our viewing client when it is ready for public release. However, there are many different JPEG2000 viewers and plugins already out there to start practicing with. One example is ExpressView from LizardTech.

Once we are on a roll, the data release will be steady and no one will be able to keep up with the wealth of Mars data coming in. Until the first public release of PSP images, we will try to provide here on HiBlog more details about the many tasks that must still be completed.

1. The image is taken by the HiRISE camera, and is stored in up to 28 channels, two for each of the 14 CCD arrays of the camera. Each channel covers about half of the image. Of the 14 CCDs, 10 are red CCDs, two are blue-green, and two are near-infrared. The color CCDs are aligned with the center red CCDs.
2. The image is placed inside a buffer on MRO, awaiting transmission to Earth, along with science data from the other instruments on MRO.
3. The image is received in packets by the Deep Space Network (DSN).
4. After 4 hours of collecting data at the DSN, the Jet Propulsion Laboratory (JPL) puts the packets together for what is known as a “quick look”. The entire image generally has not yet been received by this point in time, but it is enough of the image that it can be processed to take a quick look at it. Subsequently, JPL puts together all of the data it has received every 4 hours and makes it available to the computers at HiROC.
5. After the files have been put together by JPL, then one of the computers at HiROC looks and sees that there is data on the JPL server and copies the data to our system at HiROC. This is the start of what is known as the pipeline, the system of programs at HiROC which process the images. This usually happens either via a direct connection to JPL (slower), or through the Internet 2(Faster, but sometimes can be bogged down).
6. The images are put together into a viewable format, using the minimum processing possible, and create what’s known as an EDR, or Experimental Data Record. This is done without calibration, stitching together the channels, or any other processing, aside from putting the image together. For an image which uses all 14 CCDs, there will be 28 EDRs. These generally speaking are of mainly scientific interest, but they will be released to the general public via the Planetary Database System (PDS). They will be in the standard PDS format.
7. After the EDRs have been created, they are converted to another format for ISIS. ISIS, the Integrated Software for Imagers and Spectrometers is a suite of tools used for processing images for most interplanetary missions, that was developed by the United States Geological Society (USGS). Most of the tools that we use at HiROC for processing our images are written for ISIS files.
8. After the ISIS files have been created, they are calibrated via a program called HiCal. This reduces the inherent noise of the camera to be more consistent with what is being photographed. All digital cameras create some level of noise, and while HiRISE is an extremely good instrument, it still generates a low level of noise.
9. After the individual channels are calibrated, then they proceed to a program called HiStitch, which puts the two channels of the same CCD together. As they are a part of the same CCD, this requires little processing.
10. Next, after each CCD been stitched together, the full CCD images run through a program called HiccdStitch. This program puts the different ccds together, making a mosaic for each color band. This requires some processing, as the ccds slightly overlap, and it can sometimes be difficult to match the different arrays exactly.
11. If the image has not been completely received, then at this point, the pipeline stops, until JPL has received the entire image, or if there are a few confirmed gaps in the image which we haven’t been able to recover. Transmission over the vast distance between Earth and Mars is not easy, and even the best systems have some small error.
12. After the image has been completely stitched together, then the image is geometrically projected. To understand this, realize that the images that HiRISE takes are flat, while Mars is actually round. Geometrical Projection alters the image so that the image points in compass directions, while correcting any distortions that are created by the ellipsoidal shape of Mars. With the geometrical projection images and the right software tools, such as qview for ISIS, the exact distance can be found between two point on the image. In order for this to happen, we must wait for information to be gathered on the exact position of the spacecraft. This is done by the nagivational team, based off of the downlink frequency. This takes two weeks after the picture has been taken, so Geometric Projection might take a while. This is the longest wait point of the operation. An image can be released from predicted information, however, most images will wait for the correct SPICE kernels to be calculated, in order to get the best information. If an image is geometrically projected from predicted information, it will be calculated with the correct info after it has been received.
13. The images are then validated by a team of students known as the HiRISE Validators. They check to make sure that everything in the pipeline worked perfectly, see if there are any gaps in the images, and other similar tasks. If they notice a problem, they contact the HiRISE Operators, who will take steps to resolve the problems, which may include passing part or all of the image through the pipeline again, or tweaking the software to make it work perfectly.
14. The image is converted to a format that the general public can use. Currently that format is JPG, or TIFF, but eventually we will use JPEG 2000.
15. After all of this, the science team members of HiRISE will look at an image to see if there is anything noteworthy. If there is, it is given a caption, and perhaps a press release. If not, it will be posted on the HiRISE website. They are also posted on the MRO website, and occasionally on others.

This process may take as long as a week or two to complete, depending on the load of MRO, scheduling concerns, load at HiROC, etc. The first image took about 9 hours to be completely processed after it was taken by HiRISE. The Victoria Crater picture, taken during a much busier time on MRO, took about 36 hours to make its way to our hands. This was in part due to the larger size of the image, as well as the cache of images already awaiting transmission on MRO to earth. The captions for the images taken during Transition imaging took anywhere from a few hours to a few weeks to write, and this will likely continue to hold. We at HiROC want to release the images we take as fast as possible to the public, and we are doing everything we can to realize this goal. Several shortcuts were taken during the Transistion imaging phase that allowed for images to be released quicker. For Primary Science Phase, this will take a bit longer because these shortcuts will not be taken, but we expect that we will release most images within two weeks after them being taken, shortly after we have finished receiving, processing, and captioning the image.

There are some variations to this process, for example, the Victoria Crater picture was released in a press conference jointly with the Mars Exploration Rovers (MER) team. Also, color images require extensive calibration and take a lot more time. However, this is the general idea. Currently the entire system, except for writing the captions and adding the images to our website, is essentially completely automatic for receiving and processing HiRISE images, due to years of preparation by the HiTECH and HiOPS teams.