• 2,996 RDRs, 1 TB
• 42,370 EDRs, 1 TB
• 34,481 RDR Extras, 1.6 TB
• 83,784 EDR Extras, 0.02 TB
• 636 Anaglyphs, 0.01 TB

Totals for this release: 163,631 image products, 3.6 TB

This brings our total released product numbers and data volume to:

• 22,676 RDRs, 12 TB
• 317,120 EDRs, 10.4 TB
• 192,270 RDR Extras, 15.3 TB
• 612,769 EDR Extras, 0.1 TB
• 2,892 Anaglyphs, 0.5 TB

Total: 1,148,363 images, 37.5 TB

In summary, we released nearly 1500 observations, most of those with both black & white and color RDR products. Several newer observations matched up with older observations from a slightly different angle of the same location on the surface, resulting in 636 awesome new anaglyphs. The RDRs are the fully processed, geometrically projected products best for scientific inquiry. If you really want to, though, anyone can download and process HiRISE data from scratch. You can do this using ISIS software, which is publicly available for free download. See the ISIS Web site for download information, processing instructions, and tutorials.

Starting this week, I will be looking over the observations taken August 6 through August 26 before MRO went into safe mode and make sure they are ready for release. We plan to release these images in early October. We are also in the process of reprocessing those Extended Science Phase mission images prior to all the latest processing pipeline fixes and updates.  Once we are satisfied with that data set, we will release them to the public and then start reprocessing the images from the Primary Science Phase…a major project that should keep me and the rest of Downlink busy for several months!

Dr. Alfred McEwen, HiRISE principal investigator, decided early on that this incredibly powerful instrument should be “The People’s Camera”. This meant, among other things, that we would endeavor to make the data returned by HiRISE available to the scientific community and public as quickly as possible. We have PDS release requirements, but our goal has always been to beat those requirements. To do so, we needed to develop automated software pipelines to take the raw data and turn them into useful calibrated and geometrically mapped products. We also needed to develop the right PDS release tools, train a talented group of operations staff to validate the data and fix problems, and develop a website to effectively and beautifully showcase HiRISE images.

We now believe we have reached the point to be able to support a monthly release of recent HiRISE images to the public! This week we released the observations HiRISE took of Mars between orbits 11,600 and 12,599, or between January 16 and April 04, 2009. This makes us the first mission to release a data set to the PDS so quickly! Here are the statistics for this release, including the number of each product type released and their respective data volumes (EDRs are the individual uncalibrated image channels and RDRs are the calibrated, mosaicked, and geometrically-projected observations):

• 1,179 RDRs, 520 GB
• 16,861 EDRs, 459 GB
• 13,512 RDR Extras, 788 GB
• 33,152 EDR Extras, 7 GB
• 342 Anaglyphs, 51 GB

Totals for this release: 64,704 image products, 1.7 TB

This brings our total released product numbers and data volume to:

• 19,667 RDRs, 11 TB
• 278,807 EDRs, 9.5 TB
• 166,816 RDR Extras, 13.7 TB
• 529,095 EDR Extras, 0.1 TB
• 2,892 Anaglyphs, 0.5 TB

Total: 993,277 images, 34 TB

Those are various products for about 9998 Mars observations, and another reason why it makes no sense to hoard our data; there is too much of it and too few of us! The team scientists have plenty to do and there are plenty of discoveries to be made, old hypotheses to update, and new mysteries to solve.  The operations staff are now hard at work getting observations from orbits 12,600 through 12,999, or between April 04 and May 5, 2009, ready for the June PDS release. This involves making sure each observation has been processed by our software pipelines correctly, fixing any problems, and checking and double checking that the relevant image products are ready for release.  Sometimes we have to manually force an observation through the pipelines because some of its channels were lost during transmission to the Earth, or we might stumble across an observation we somehow forgot to send on to the color pipelines after it had been calibrated. There are spreadsheets to maintain, lists of problematic observations to keep (see the ERRATA.TXT file), and a variety of other tasks that need to be completed before the latest data set is ready for release.

Over the next few months we will see how this goes! It is a lot of work, but our desire for you to see these beautiful images of Mars as quickly as possible is strong. No promises, but we will also explore releasing completed observations even faster!

Here are a few excerpts from yesterday’s University of Arizona story about our PDS release:

The HiRISE team has so far released a total 26.9 terabytes of data…. That amounts to more data than has been released by all previous deep space missions combined.

“If I showed each HiRISE image for 10 seconds, it would take me about 4 years to show them all,” said UA’s Alfred McEwen, HiRISE principal investigator.

…

Spacecraft motion pushes this electronic array so that it records the view down to Mars’ surface at a ground speed of about 3.2 kilometers per second, or about 7,000 miles per hour.

Skeptics doubted that a technique called “time integration delay,” needed to compensate for extremely short exposure times – about one ten-thousandth of a second per pixel – could produce sharp, unsmeared images.

But the technique has worked “wonderfully well,” thanks to accurate spacecraft pointing and stability and precise exposure time calculations, McEwen said.

Click here for full story.

Here are a few examples of cool images, which were previously unreleased:

• PSP_008248_2640, Polygons and spots on defrosting dunes (right)
• PSP_008269_1395, crazy weird stuff in Hellas Planitia (be sure to look at the whole browse image on this one!)
• PSP_008322_1865, Multiple generations of slope streaks on a crater in Arabia Terra (left)
• PSP_008343_1430, Gullies on mesas in Gorgonum Chaos

I’ve only looked through the first few pages in the release. I know there are a lot more amazing images in there, so if you’re browsing through the images, post some of your favorites below!

For example, such software could display the actual coordinates on Mars of the pixels you are looking at, allow you to measure features directly in physical units, or stitch together images based on their absolute location on the planet. A number of GIS (Geographical Information Systems) applications use GeoTIFF; many on our science team have been waiting patiently for this feature to be rolled out.

We have already begun to produce RDRs with GeoTIFF, and they will start appearing in our weekly releases. At some point, a major reprocessing effort will be underway to bring this feature (and others) to all of our pre-existing products.

This brings up the topic of versioning: namely, how to tell which version of a HiRISE product you are working with.

Every HiRISE product has a line in the PDS label with the DATA_SET_ID. Every released HiRISE product to date shows that it has a DATA_SET_ID version of 1.0. The GeoTIFF RDR’s have a DATA_SET_ID showing version 1.1. This 1.1 version of our RDR processing also contains the updated color stretches described in a previous post. Later this year, that version number will likely be bumped up again when our improved calibration algorithms are put into production.

In addition, every HiRISE PDS product also has a line in the label with the PRODUCT_VERSION_ID. And every released HiRISE product to date has a PRODUCT_VERSION_ID of 1. When we make a new version of a released product, the version number will be incremented. If that new version is then released, it will replace the older version. We will only replace products with newer versions after a process of validation.

We’re halfway through a major update to our production pipelines that allows us to create, store, and reprocess these newer versions without effecting the released versions. This version number will only increase by integer amounts, and will never “skip”.

How much data was released? 2422 observations, making up 9.9 terabytes “in over 225,599 standard PDS and extras products” according to our database specialist. This was for data between orbit ranges 4400 and 6999, or between July 05, 2007 and January 23, 2008 (which is a lot of loops around the Red Planet!)

We have now released a total of 16.8 TB worth of data, or nearly 500,000 image products. Please check out the latest images on the HiRISE website on the “March 2008: New HiRISE Images Released to the Planetary Data System” page.

These data have been processed, and reprocessed when necessary, with the latest automated pipelines on our production processing cluster. We continue to make changes to the software, however, and will have to reprocess all of these data yet again in a few months. What you see today is gorgeous and as complete as currently possible, but we always want to tweak our calibration, color, and geometry pipelines to make these even better.

This release places us very far ahead of the MRO project’s expectations for the HiRISE team. We are now working on speeding up our releases even more, so that they occur more often. That means we will probably never have such a large release again, which, as far as us downlink folks are concerned is a very good thing. Making sure 9.9 terabytes of data is ready to release is hard work. The images and new findings make it worth it, though!

• PDS Imaging Node: http://pds-imaging.jpl.nasa.gov/
• MRO PDS page: http://pds-imaging.jpl.nasa.gov/Missions/MRO_mission.html
• HiRISE Volume: http://hirise-pds.lpl.arizona.edu/PDS/

I’ve prepared this ugly table that outlines each of the products now available (excluding the raw EDRs). So reading the columns from left to right: there are three types of “NOMAP” products, two types of lossy “QLOOK” (Quicklook) RDRs, and two types of lossless RDRs.

With that as a reference, now I’ll try to define everything more precisely.

“NOMAP”

Non map-projected product. Always lossy compressed for smaller size and quicker viewing. These are not formal Planetary Data System products; they’re “special”, meaning there is no PDS label and no Software Interface Specification describing them. Available for IRB, RGB and RED.

RDR

Reduced Data Record: reduced in the sense of refined or processed, not raw data. Formal PDS products with accompanying labels and a detailed SIS document describing their format and processing steps. Available both in lossless and quicklook formats for both RED & COLOR.

“QLOOK”

Quicklook: a special product that is a lossy compressed version of the RDR. In a normal RDR, all of the original data is retained. But with a quicklook, some of the highest resolution detail is discarded to make for quicker viewing.

RED

The image obtained by the red-filtered CCDs. It will be over the full swath width, typically data from all ten red CCDs. Covers the visible wavelength band from 550 to 850 nanometers.

IR

Infrared. Covers the near-IR wavelengths from 800-1000 nanometers.

BG

Blue-Green, visible wavelengths from 400-600 nm.

COLOR

A color RDR. It contains data from the IR, BG and center RED ccds. Typically this will be a skinny strip (”center swath”) inside a skinny strip, or as I like to say, the bacon-strip effect.

IRB

An enhanced color NOMAP. It has the same color bands as the RDR: IR, RED and BG.

RGB

An enhanced color NOMAP. It contains only data from the RED and BG. The blue is derived from the difference between the RED and BG. The color bands are RED, BG and the synthetic blue.

EDR

Experiment Data Record, a formal PDS product that is raw uncompressed data with a label header.

Note: we will be working towards making all of these products available for all prior releases.

The solution for many NASA missions has been the development of the centralized Planetary Data System (PDS). The PDS is several things: a collection of websites, a search capability, an archive, a database, a learning tool, etc. The PDS Imaging Node is located at http://pds-imaging.jpl.nasa.gov/ and acts as “the curator of NASA’s primary digital image collections from past, present and future planetary missions.” These missions include Voyager, Galileo, Cassini, and many more. Now the Mars Reconnaissance Orbiter (MRO) has been added to the list, with the HiRISE team releasing our first several months of image data.

• MRO PDS page: http://pds-imaging.jpl.nasa.gov/Missions/MRO_mission.html
• MRO Product Search page: http://pds-imaging.jpl.nasa.gov/search/index.jsp
• HiRISE Volume: http://hirise-pds.lpl.arizona.edu/PDS/

What we have released is an archive of the HiRISE Experiment Data Records (EDRs) and Reduced Data Records (RDRs). EDRs are in the *.IMG file format and represent individual CCD channels (remember, there are 14 CCDs in the HiRISE camera and two channels per CCD, for a total of 28 channels). These EDRs are cleaned up, calibrated, stitched together, and mapped to Mars’ geometry, resulting in the RDR products. RDRs are in the *.JP2 and *.LBL formats. JPEG2000 is the technology that enables us to offer our gigantic images to the scientific community and the public in a timely and efficient manner. An observation’s image data are in the *.JP2 file and its meta data are in the detached *.LBL files. To view these products, JPEG2000 compatible software is required (see our site for a list of offerings).

While we have been trying to release up to five captioned images a week for the past few months, the PDS release represents several hundred images, most of them without captions. You can find them using the PDS search capabilities, and you can also find them on the new HiRISE site, unveiled today to coincide with this first PDS release. The redesigned site focuses on the images while providing, hopefully, a more user-friendly interface:

• HiRISE Site: http://hirise.lpl.arizona.edu/
• “About Our Redesign”: http://hirise.lpl.arizona.edu/profil.php
• Images released to the PDS: http://hirise.lpl.arizona.edu/pds_release.php

As word gets out about the new site and the PDS release, you may experience some site slowness. Please be patient, and thank you for your interest!

This was the third test in a series of 4 tests. The purpose of this test was to begin ramping up the volume of products we released, to test our first JPEG2000 product, and to test the ability of the node software to distribute files that are larger than 2 Gigabytes. We are still learning how to make proper JPEG2000 files, but we did learn that the PDS node server software was successfully able to serve these large files. The ability of different clients to download such large files was a little more spotty, but this was not completely unexpected since there are many clients that are incapable of downloading files this large.

This test gave us a pretty good idea about how downloading such products might work by conventional means. However, our plans are to serve our JPEG2000 products using the jpip protocol which will allow clients that understand this protocol to pan and zoom around portions of our images without downloading the entire file. In most cases, you probably do not want to even try to download such large images in the first place.

1. The image is taken by the HiRISE camera, and is stored in up to 28 channels, two for each of the 14 CCD arrays of the camera. Each channel covers about half of the image. Of the 14 CCDs, 10 are red CCDs, two are blue-green, and two are near-infrared. The color CCDs are aligned with the center red CCDs.
2. The image is placed inside a buffer on MRO, awaiting transmission to Earth, along with science data from the other instruments on MRO.
3. The image is received in packets by the Deep Space Network (DSN).
4. After 4 hours of collecting data at the DSN, the Jet Propulsion Laboratory (JPL) puts the packets together for what is known as a “quick look”. The entire image generally has not yet been received by this point in time, but it is enough of the image that it can be processed to take a quick look at it. Subsequently, JPL puts together all of the data it has received every 4 hours and makes it available to the computers at HiROC.
5. After the files have been put together by JPL, then one of the computers at HiROC looks and sees that there is data on the JPL server and copies the data to our system at HiROC. This is the start of what is known as the pipeline, the system of programs at HiROC which process the images. This usually happens either via a direct connection to JPL (slower), or through the Internet 2(Faster, but sometimes can be bogged down).
6. The images are put together into a viewable format, using the minimum processing possible, and create what’s known as an EDR, or Experimental Data Record. This is done without calibration, stitching together the channels, or any other processing, aside from putting the image together. For an image which uses all 14 CCDs, there will be 28 EDRs. These generally speaking are of mainly scientific interest, but they will be released to the general public via the Planetary Database System (PDS). They will be in the standard PDS format.
7. After the EDRs have been created, they are converted to another format for ISIS. ISIS, the Integrated Software for Imagers and Spectrometers is a suite of tools used for processing images for most interplanetary missions, that was developed by the United States Geological Society (USGS). Most of the tools that we use at HiROC for processing our images are written for ISIS files.
8. After the ISIS files have been created, they are calibrated via a program called HiCal. This reduces the inherent noise of the camera to be more consistent with what is being photographed. All digital cameras create some level of noise, and while HiRISE is an extremely good instrument, it still generates a low level of noise.
9. After the individual channels are calibrated, then they proceed to a program called HiStitch, which puts the two channels of the same CCD together. As they are a part of the same CCD, this requires little processing.
10. Next, after each CCD been stitched together, the full CCD images run through a program called HiccdStitch. This program puts the different ccds together, making a mosaic for each color band. This requires some processing, as the ccds slightly overlap, and it can sometimes be difficult to match the different arrays exactly.
11. If the image has not been completely received, then at this point, the pipeline stops, until JPL has received the entire image, or if there are a few confirmed gaps in the image which we haven’t been able to recover. Transmission over the vast distance between Earth and Mars is not easy, and even the best systems have some small error.
12. After the image has been completely stitched together, then the image is geometrically projected. To understand this, realize that the images that HiRISE takes are flat, while Mars is actually round. Geometrical Projection alters the image so that the image points in compass directions, while correcting any distortions that are created by the ellipsoidal shape of Mars. With the geometrical projection images and the right software tools, such as qview for ISIS, the exact distance can be found between two point on the image. In order for this to happen, we must wait for information to be gathered on the exact position of the spacecraft. This is done by the nagivational team, based off of the downlink frequency. This takes two weeks after the picture has been taken, so Geometric Projection might take a while. This is the longest wait point of the operation. An image can be released from predicted information, however, most images will wait for the correct SPICE kernels to be calculated, in order to get the best information. If an image is geometrically projected from predicted information, it will be calculated with the correct info after it has been received.
13. The images are then validated by a team of students known as the HiRISE Validators. They check to make sure that everything in the pipeline worked perfectly, see if there are any gaps in the images, and other similar tasks. If they notice a problem, they contact the HiRISE Operators, who will take steps to resolve the problems, which may include passing part or all of the image through the pipeline again, or tweaking the software to make it work perfectly.
14. The image is converted to a format that the general public can use. Currently that format is JPG, or TIFF, but eventually we will use JPEG 2000.
15. After all of this, the science team members of HiRISE will look at an image to see if there is anything noteworthy. If there is, it is given a caption, and perhaps a press release. If not, it will be posted on the HiRISE website. They are also posted on the MRO website, and occasionally on others.

This process may take as long as a week or two to complete, depending on the load of MRO, scheduling concerns, load at HiROC, etc. The first image took about 9 hours to be completely processed after it was taken by HiRISE. The Victoria Crater picture, taken during a much busier time on MRO, took about 36 hours to make its way to our hands. This was in part due to the larger size of the image, as well as the cache of images already awaiting transmission on MRO to earth. The captions for the images taken during Transition imaging took anywhere from a few hours to a few weeks to write, and this will likely continue to hold. We at HiROC want to release the images we take as fast as possible to the public, and we are doing everything we can to realize this goal. Several shortcuts were taken during the Transistion imaging phase that allowed for images to be released quicker. For Primary Science Phase, this will take a bit longer because these shortcuts will not be taken, but we expect that we will release most images within two weeks after them being taken, shortly after we have finished receiving, processing, and captioning the image.

There are some variations to this process, for example, the Victoria Crater picture was released in a press conference jointly with the Mars Exploration Rovers (MER) team. Also, color images require extensive calibration and take a lot more time. However, this is the general idea. Currently the entire system, except for writing the captions and adding the images to our website, is essentially completely automatic for receiving and processing HiRISE images, due to years of preparation by the HiTECH and HiOPS teams.