Tomorrow night the HiRISE camera on board the Mars Reconnaissance Orbiter will start taking images of the surface of Mars again, after a 5-week break while Mars was behind the sun from our perspective. That configuration between the Earth, Sun, and Earth is called “solar conjunction”. The data rate during solar conjunction is too low to send our huge high resolution images back to the Earth, so we just don’t take any pictures during that time.

Today I am going through the descriptions of upcoming observations to fix any spelling, grammar, or other problems. The description of one of the new images to be taken – “Gullies in a crater on floor of Newton Crater” – needs a minor correction: I need to take the “a” out because we don’t want articles in our descriptions. The descriptions are supposed to be headline-like. “Gullies in crater on floor of Newton Crater” is better.

Another thing I check is that “Newton” is spelled correctly. There is a Mars Nomenclature website (http://planetarynames.wr.usgs.gov/Page/MARS/target) run by the USGS that I go to to make sure we are using the correct nomenclature and spelling for officially named Martian features. “Newton Crater” really is a crater on Mars and that is how it is spelled. And because this is a formal name, the “c” in “crater” has to be capitalized.

I have 68 more descriptions to check today. Should take me less than an hour. Here are some of our description “rules”:

• 75 characters or less
• no punctuation and special characters except for not-bare dashes (-) (i.e. dashes with white space around them are not allowed);
• Good: “2-kilometer diameter crater”
• Bad: “Dunes in Arcadia region – seasonal monitoring”

Descriptions should include:

• official Mars Nomenclature only (look up official names and spellings on the USGS Mars Nomenclature index);
• use full names of regions and features (e.g. “Meridiani” –> “Meridiani Planum”, “Victoria” –> “Victoria Crater”); if you don’t know what kind of feature or region it is, add “region” after the name
• “Giza”, “Inca City”, “Ithaca”, “Manhattan”, “Oswego”, and “Starfish” are not official Mars nomenclature but we will allow them. They are best used with the disclaimer “dubbed”, as in “Dunes with bright-dark-bright bands dubbed Buzzel”.
• Note the distinction between vallis (singular) and valles (plural)
• Note the distinction between planum (singular) and plana (plural)
• correct grammar;
• correct capitalization:
• capitalize first letter of first word
• capitalize proper names (e.g. “flow around Olympus Mons” –> “Flow around Olympus Mons”)
• “Henry crater” should be “Henry Crater”
• complete words and phrases (fix truncated words and phrases).

There might not be any new HiRISE images to release (yet) but that does not mean we have been idle here at the HiRISE Operations Center. In fact, we have been very busy improving our existing images with new calibration and a different color stretch, making sure our entire data set is consistent, and preparing new product types!

For starters, our Extended Science Phase (ESP) images are now consistent, making use of all the latest and greatest fixes and improvements to our processing pipelines. These updated products are out now. The older Primary Science Phase (PSP) observations are still reprocessing; we should have them ready for an upcoming monthly release.

Next up, we have a fantastic new product called a Digital Elevation Model. Learn more about these products in an upcoming blog post!

Finally, several of our observations have been improved with updated descriptions.  For example, an image might have been taken of a crater that did not have an official name at that time. After the crater receives an official name, we try to go back and update the description for observations of that crater. Recently we had time to do that for a lot of observations. Unfortunately, the software used to update the EDR and RDR labels inadvertently corrupted the first few image data lines. Our attentive Targeting Specialists and Student Validators spotted the minor differences between reprocessed images and older versions. We have corrected the description update tools and reprocessed the EDRs from the 745 observations affected by this problem. Anyone who has downloaded HiRISE EDRs prior to the December 2009 release should check the list below to ensure any EDRs being worked with are not on this list. If the EDRs you have are from the observations on the list, you can check the “creation date” in their label. If the creation date is before November 4, 2009 then you should download the latest version. If the creation date is after November 4, 2009 then the EDR is good. The latest version of all of the EDRs for the following observations are now available on the PDS and on our website.
(more…)

You might remember that we were planning on releasing HiRISE images to the public on a monthly basis. That plan was delayed by issues with our processing software, hardware and other events. A productive summer working on these issues culminated last week with one of our larger releases of Mars images!  Here are some statistics about our September 2009 release, which includes the images the HiRISE camera took of the Martian surface between Mars Reconnaissance Orbiter (MRO) orbits 12,600 to 14,199, or roughly April 4 through August 6, 2009:

• 2,996 RDRs, 1 TB
• 42,370 EDRs, 1 TB
• 34,481 RDR Extras, 1.6 TB
• 83,784 EDR Extras, 0.02 TB
• 636 Anaglyphs, 0.01 TB

Totals for this release: 163,631 image products, 3.6 TB

This brings our total released product numbers and data volume to:

• 22,676 RDRs, 12 TB
• 317,120 EDRs, 10.4 TB
• 192,270 RDR Extras, 15.3 TB
• 612,769 EDR Extras, 0.1 TB
• 2,892 Anaglyphs, 0.5 TB

Total: 1,148,363 images, 37.5 TB

In summary, we released nearly 1500 observations, most of those with both black & white and color RDR products. Several newer observations matched up with older observations from a slightly different angle of the same location on the surface, resulting in 636 awesome new anaglyphs. The RDRs are the fully processed, geometrically projected products best for scientific inquiry. If you really want to, though, anyone can download and process HiRISE data from scratch. You can do this using ISIS software, which is publicly available for free download. See the ISIS Web site for download information, processing instructions, and tutorials.

Starting this week, I will be looking over the observations taken August 6 through August 26 before MRO went into safe mode and make sure they are ready for release. We plan to release these images in early October. We are also in the process of reprocessing those Extended Science Phase mission images prior to all the latest processing pipeline fixes and updates.  Once we are satisfied with that data set, we will release them to the public and then start reprocessing the images from the Primary Science Phase…a major project that should keep me and the rest of Downlink busy for several months!

Remember when I said we would start releasing data monthly to the PDS and public? We decided to work on improving calibration a little bit longer before we reprocess our data and start the monthly releases. Therefore, we are not going to have a June release (except for the usual weekly captioned image releases on Wednesdays), but we are getting closer to a regular monthly release! We are still ahead in our releases per PDS requirements, and we are preparing for the big reprocessing that is coming up. For example, our software pipelines have been updated to better automate reprocessing and we have hired two new student validators who will be responsible for visually inspecting all of the images as they are reprocessed. Training started this week.

Consider the May PDS release our practice run. Thank you for your patience…monthly releases of HiRISE images are indeed coming, but not quite as soon as we expected.

The data rate from the Mars Reconnaissance Orbiter (MRO) – in orbit around Mars on the other side of the Sun from our perspective here on the Earth – is set to increase soon, marking the end of solar conjunction and the beginning of MRO’s Extended Science Phase (ESP).  HiRISE will capture the first image of our next phase in Mars exploration this weekend.  Like all first images after the HiRISE camera has been idle for awhile, we will look especially closely for, but do not expect, any issues.

In these days leading up to the beginning of ESP the downlink team is checking our processing software and tools to make sure they are ready for the observation ID prefix change from PSP_XXXXXX_XXXX to ESP_XXXXXX_XXXX. The automated processing pipelines are ready to go.  My own validation and reporting Perl scripts make use of modules that are mission phase aware and pattern match file extensions instead of observation ID prefixes, so I think I am set. Any minor tools we miss can be easily updated as necessary.

With improvements to our tools and new procedures, we can sometimes recover image data previously stuck in the original raw data files. During this quiet period, I had an opportunity to go back to old observations, some from early in the Primary Science Phase (PSP), and recover errant channels that failed our processing software at the time.   This resulted in a few new channels of image data that we will include in a later reprocessing of our images.  For example, PSP_001746_1515 was originally processed without the RED1_0 channel. You can see this channel gap if you click on the observation’s “Full image (grayscale, non-map projected)” link.  An improved mosaic will include most of RED1_0, albeit with a small image data gap near one end.  Why did we not create this channel before?  Sometimes a data gap occurs between channels, obliterating the second channel’s science header.  The software at Jet Propulsion Laboratory’s RSDS that splits our image data into separate raw data channel files cannot figure out where to split the image data correctly within this gap and so image data from two or more channels are trapped in one raw data file.  When this jumbo raw data file hits our EDRgen processing pipeline here in Tucson, Arizona, the tool called HiRISE_Observation became confused when it discovered image data from another channel near the end of the file, and the error it flags halts the EDRgen processing pipeline.

We now have improved tools and procedures for dealing with this problem.  By running HiRISE_Observation outside of the EDRgen pipeline, it will successfully produce an EDR for the first channel before it complains about the second channel’s image data.  While we would love to get at that second (or third, or later!) channel’s image data, right now we recoverthe data that is easiest to reach.  Someday, we might have a tool that will recognize a missing science channel header, reconstruct this header, and then fit the header in correctly between the first channel and the additional channels stuck in one raw data file.

When will you see the new PSP_001746_1515 products, newly improved with the recovered RED1_0 channel?  Hopefully in a few months.  While I have recovered this EDR (as well as EDRs from other observations), we would like to wait to reprocess the entire observation until we have even better image calibration built into our processing pipelines.  It makes little sense for me to reprocess this observation now when we are just going to be reprocessing it again sometime early next year. Once all of our improvements are in place in the upgraded processing pipelines, we will reprocess ALL of our observations, a huge undertaking.

Next week we will be back to our normal downlink activities: making sure the processing pipelines behave, validating new observations, providing daily reports to the rest of the team, and processing color and RDR mosaics.  We will also add in routine creation and validation of anaglyphs and preparation for our next Planetary Data System (PDS) release in March.  The quiet time during solar conjuction was nice, but we are anxious to see new images from Mars!

After the long process of creating the HiGlyph Pipeline (anaglyphs producing software), processing the images through this pipeline and having them all properly validated, we here at HiRISE are proud to present you with a whole mess of anaglyphs (362 of them, to be exact )!

And now, for all of those curious minds out there, a brief overview of the HiGlyph Pipeline:

• Anaglyphs are created in a three-step process. The first step is to take the two images of the stereo pair and map project them. This helps the pipeline determine which image will be the left image and which will be the right image in the anaglyph.

• The second step takes the two images and looks to see if there are any improvements that can be done on them before putting them together. If there are not, the images move on. Often, due to the difference in viewing angle, the two images do not have a 100% overlap. Thus, to make the image a bit neater and easier to see, we trim off the excess portion of the image (the parts that do not overlap) and then assemble them so the left image is the red and the right image is the blue/green.

• The third and final step of this image processing is simply to prepare the images you see here and to update our catalog.

Seems complicated, right? Well luckily we have wonderful programmers that create these intricate programs. All I have to do is create a list of these images and run them through this pipeline. What really makes my job interesting is the validation process!

I have had the pleasure of being able to look at all 362 of the anaglyphs we have released today. But, you might ask, aside from looking super cool in 3-D glasses , what does it take to validate these anaglyphs? Well, at the beginning of this process the student validators and I got to ask that very same question. Since HiRISE has never had software to create images like this before, we played lab rat and came up with an entirely new technique for validation.

• You may notice that when not wearing the 3-D glasses there is a bit of a horizontal shift in the anaglyph. This shift is good because it is what allows us to see the image in 3-D. But, since the map projection of this process is not always spot on, we sometimes wind up with a vertical shift too. This is bad! Since most of us do not have googly eyes, this makes the image very difficult to see. With our validation process, we have to spot this out and fix it so that you do not have to strain your eyes (well… not too much at least ) in order to see the anaglyph.

Well, with that said, I leave you to your regularly scheduled HiRISE browsing! Enjoy!

We’ve just released 1008 new HiRISE images to the PDS! (See main page, or click here for the catalog.) This release covers orbits 8200 – 9299 of the primary mission, or in other words, the end of April through the end of July. That means we’re releasing data that’s only about 6 weeks old! This is awesome – I’m so impressed with the downlink team! The amount of work required to process these images is astounding, let alone prepare and post everything for an official release.

Here are a few examples of cool images, which were previously unreleased:

• PSP_008248_2640, Polygons and spots on defrosting dunes (right)
• PSP_008269_1395, crazy weird stuff in Hellas Planitia (be sure to look at the whole browse image on this one!)
• PSP_008322_1865, Multiple generations of slope streaks on a crater in Arabia Terra (left)
• PSP_008343_1430, Gullies on mesas in Gorgonum Chaos

I’ve only looked through the first few pages in the release. I know there are a lot more amazing images in there, so if you’re browsing through the images, post some of your favorites below!

GeoTIFF is an industry standard for embedding geographic information in images. Beginning soon, HiRISE RDRs will include GeoTIFF info in the Jpeg-2000 files. All of the information about the image will continue to be in the RDR label (.LBL plain text file), but with this additional info in the JP2, image viewing software that supports GeoTIFF will be able to take advantage of it.

For example, such software could display the actual coordinates on Mars of the pixels you are looking at, allow you to measure features directly in physical units, or stitch together images based on their absolute location on the planet. A number of GIS (Geographical Information Systems) applications use GeoTIFF; many on our science team have been waiting patiently for this feature to be rolled out.

We have already begun to produce RDRs with GeoTIFF, and they will start appearing in our weekly releases. At some point, a major reprocessing effort will be underway to bring this feature (and others) to all of our pre-existing products.

This brings up the topic of versioning: namely, how to tell which version of a HiRISE product you are working with.

(more…)

Today, our software group provided a set of major updates to our downlink operations team. It was the first major update in many months. One of the most anticipated features is smarter “stretch” algorithm for our color products (RDR Extras). As discussed in a previous post, a stretch (in image processing terms), is a mapping between one range of pixel values and another. In our case, it provides our viewers with a better-looking image up-front, with less need to adjust parameters in display software such as IAS (though this is still often very helpful when zoomed in). As always, the full range of original data is preserved in the RDR JP2.

Our former algorithm for the NOMAP and Quicklook products said that the pixel values above the brightest 0.1% and below the darkest 0.1% would be mapped to the extreme values, with a linear fit in between. For a majority of images, this was a good choice that showed excellent contrast but prevented too much saturation.

However, 0.1% (a thousandth) of a two Gigapixel image is still two million pixels. So if there were a particularly bright spot, like a rocky outcrop amid a field of dunes, or a particularly dark spot, like a cavern opening in a plain of boulders, then all the saturation would occur in that one area, washing it out completely, and lowering contrast everywhere else in the image. So the algorithm needed to be more adaptable. After a good deal of experimentation, the algorithm we settled on looks at the brightest and darkest pixels in a thumbnail version of the image, and uses those values for the extremes, instead of the values at 0.1%. We shrink a copy of each color band to 1/11th the original scale. Pixel values in the original below the darkest in the thumbnail are mapped to pure black, while pixel values above the highest are mapped to pure white. The stretched bands are then merged to make the color image. Hence, a bright or dark spot smaller than 1/11th x 1/11th of the image size will no longer dominate the stretch.

What this ultimately means is, our RDR Extras now show more detail in areas that would be completely washed out by the old algorithm.

For example, in this ‘cave’ image, the left is from the original RGB.NOMAP.JP2, while the right is the same product using the new algorithm. As you can see, previously you could not tell if there was a floor to the hole or if it sloped away to greater depths.

The new algorithm is used strictly for the JP2’s; the browse and thumb are already scaled down enough that it would not make a substantial difference with them. The new algorithm went into effect today; coincidentally we just started orbit 8000. Images with the new stretch will likely appear in upcoming weekly releases and we plan to reprocess everything with this change (and improved calibration) during the summer.

(more…)

All downlink tasks I perform follow a particular development path: (1) I practice and jot down manual procedures; (2) over time I attempt to automate subtasks by using Perl or other tools, as best I can (I am not a software developer); and (3) our talented software developers write software that automates the task completely, or at the very least speeds it up considerably. Of course, this is always just in time for me to be assigned new tasks!One of my daily tasks is monitoring data quality and paying attention to missing or gapped raw image channel files (up to 28 channels per observations.) My tools: our internal reporting website HiReport, Terminal, and Microsoft Excel. I look through a list of observations in my web browser, click on those that appear to be missing channels or are flagged as “INCOMPLETE”, and copy and paste information about the problematic channels into Excel. I then add some additional notes and take any required actions.

A few days ago I realized that after more than a year, I was still in the manual stage of monitoring data quality and not making good use of existing tools to help streamline the process. All that cutting and pasting was beginning to get ridiculous, even more so during a period of high data rates and an increasing number of observations.

(more…)