Here are a few examples of cool images, which were previously unreleased:

• PSP_008248_2640, Polygons and spots on defrosting dunes (right)
• PSP_008269_1395, crazy weird stuff in Hellas Planitia (be sure to look at the whole browse image on this one!)
• PSP_008322_1865, Multiple generations of slope streaks on a crater in Arabia Terra (left)
• PSP_008343_1430, Gullies on mesas in Gorgonum Chaos

I’ve only looked through the first few pages in the release. I know there are a lot more amazing images in there, so if you’re browsing through the images, post some of your favorites below!

A few days ago I realized that after more than a year, I was still in the manual stage of monitoring data quality and not making good use of existing tools to help streamline the process. All that cutting and pasting was beginning to get ridiculous, even more so during a period of high data rates and an increasing number of observations.

When we receive a product with gaps, our automated processing pipelines usually process it just fine. Sometimes, however, there are pipeline failures. I keep a list of these failures and the actions I have taken to try to correct the problem. This generally means I have to manually processing the file or repairing its metadata header. Once the channel is repaired and/or recovered, then I can feed it back into the remaining processing pipelines.

These gapped products include “_G” in their file name. In a spectacular “Duh!” moment, I realized that rather than hunting in my browser through a list of observations that might have gapped product names, and then copying and pasting any I find into Excel, I could instead just perform a command line search in Terminal for all raw data products with “_G” in their file name.

     % ls -1 */*_G*.DAT

Why didn’t I do this over a year ago!? I am embarrassed to say I have no idea.

This only provides a list of products with gaps. Missing channel products usually arrive as gapped channel products 48 hours later (due to the nature of the automated processes that handle new MRO data at JPL.) In my spreadsheet, I scan through the missing channels and determine if a gapped file has in fact arrived. If it has been longer than 48 hours I consider the data lost for good and I “force” whatever observation data we have received through the pipelines. Of course, sometimes lost data is found or reprocessed at JPL long after the 48 hours has expired, and very rarely these new data will arrive. This requires reprocessing of the entire observation so that this newly found data can be added in.

A long time ago, our database specialist wrote a Perl script to create a daily list of missing and partial observations. For whatever reason, I stopped using this tool and then promptly forgot about it. Instead I started looking through the list of observations in my browser, clicking on those that were not complete, and manually figuring out what channels were missing. Making use of this missing EDR’s tool is so much easier and faster. Again, “Duh!”

The output from these quick searches for gapped and missing products requires a little bit of tweaking to make it look nice in Excel, but then a quick sort merges the two list into observation ID order. In no time at all, I have my list of observation products to follow up on for the day. My copying, pasting, and mouse clicking madness has been vastly reduced.

Every day I see the very worst data, generally caused by data transmission problems, but these data make up only a small percentage of all HiRISE data. To try to quantify this, I counted up the number of channels I have reported with problems (they have gaps, were missing, were somehow corrupted, etc.) and then divided this number by the total number of raw data files we have received (these are transformed into EDRs after being downloaded to Tucson.) As of this morning, I have listed 5392 channels with problems, and we have received 160,858 raw channel files. This is roughly 3% of our data, although most of these have some useful data in them. Even an observation with gaps or missing a channel or two is of potential use to a scientist. If too much data has been lost, then our targeting specialists might command HiRISE to try again in a later orbit.

Over time, we have developed many procedures for dealing with all sorts of problems. Now that I have sped up my daily data quality monitoring, I will have more time to improve these procedures, partially automate as much as I can, and provide suggestions to the software developers about tools that would make my job ever more efficient.

Note: My calculation of problematic data above is not rigorous and only a rough estimate. One flaw: I am counting missing channels in the numerator but not in the denominator.

1. Validate the image data that have arrived since last time I checked. Are the raw image files we receive gap-free and are the file sizes as expected? Did the Uplink team command the HiRISE camera properly? So far, they have a perfect record!
2. Keep checking to see if new data is arriving for processing.
3. Are our automated processes running properly?
4. Is the data being stored correctly and can the team access the images in the appropriate places?
5. Finally! Actually look at the new images. In between “oohs” and “ahhs” check to see that the images look good. Did our automated software handle the data correctly? Do I need to do any manual reprocessing of image data?
6. Report my findings to the team via email.
7. Get up and see what the scientists and other team members are up to.
8. Eat some Cheetos.
9. Repeat as necessary.
10. A million other tasks.
    

By the end of the day I am covered in Cheeto dust (joking) and amazed by some new vista of Mars (seriously).

What is it we find so amazing? I can only speak for myself, but in observation TRA_000823_1720, the boulders lying about casting shadows indicate just how “Hi” resolution the HiRISE camera can go. In the second observation – TRA_000825_2665 – the stack of water ice and dust layers and the patches of water frost make for a distinctive landscape. At this resolution, there is a marked difference between the north polar region on Mars and the pictures I have seen of the Earth’s own polar regions.

To me, this is the great joy of planetary science: seeing new vistas that are at once familiar and unfamiliar, and never, ever routine.

What is downlink? Well the uplink people do all the planning and write the commands for the camera to take the pictures. Once the camera has taken the pictures, the downlink people pull the data off the spacecraft and process it.

What are we doing?

• Uplink people have planned the images in detail. We know exactly what spots on Mars we’ll be aiming the camera at. That involved a lot of work with our planning tools, using predictions of exactly where the spacecraft will be at any given time in the orbit. We wrote and delivered our command files earlier this week, so there would be time for testing. The commands were actually radiated to the spacecraft this morning, and are on board MRO right now!

• Downlink people are getting everything ready to receive and process the data. Our hardware and software needs to be configured and tested, so that tomorrow when the images start coming down, hopefully everything will go smoothly.

We’re all reviewing the test runs of the imaging as well. There is a testbed we call the “OTB” (Orbiter Test Bed). We’re able to simulate our command sequences on it in order to make sure everything is correct. We even get fake data (usually all zeros) back from it, so that we can make sure the “images” are the right size, etc.

We expect a lot of visitors at HiROC starting tomorrow. A number of our scientific co-investigators will be visiting for a few days, to see the images first-hand and help us prepare public releases. Members of the press also want to join us to capture the excitement of seeing the first images come down (and hopefully we’ll be able to convey that on this blog, too!).

So we’re all very busy, but we’ll try to keep you updated on what’s going on here!