What do the people at the HiRISE Operations Center (HiROC) do when the MRO spacecraft safes?

When the spacecraft safes, the automatic reset sequence also turns off all the scientific instruments on board. So we were shut down early last week. We knew from preliminary investigations that HiRISE wasn’t involved with the cause of the event (whew!), so we couldn’t really help analyze the cause. This was different from some situations in the past where HiRISE caused itself to safe; in those cases, we were responsible for figuring out why. This time we just had to wait while the spacecraft engineers figured out the problem.

Once the engineers determined we could go back to normal operations, we worked with them on our instrument safe mode recovery. This involves radiating a set of pre-written commands and files to the spacecraft, verifying that the setup commands executed correctly on the instrument, and watching the instrument telemetry to make sure we power up correctly. We have a series of checks built in to this sequence. They work like a checksum, and send that information back down to Earth in the telemetry. We check those values against what we expect them to be, and that’s how we know the correct values were written to memory. Without this confirmation, we couldn’t be sure that HiRISE’s memory was uncorrupted.

Every step in the lengthy procedure executed successfully, and all of the checks matched what we expected, so we were able to confirm that HiRISE was ready to return to normal imaging. The next day, they restarted the sequence on board that calls our imaging commands, and we were back in business!

• Related HiBlog post: On the Safe Side

We had the difficult detective job of figuring out what went wrong. It was clear early on that the instrument overheated, but we couldn’t figure out why. Our tool that predicts the temperatures (”HiTemp”) didn’t predict anything that hot. We didn’t take a really large image, which would heat us up (at least, nothing bigger than normal! ). The local operations team worked with the health & safety people, the spacecraft engineers at LMA, and some of the software developers at Ball Aerospace that originally designed HiRISE. Together we all investigated the problem.

We studied the telemetry (information from the spacecraft), the details of the commands that were sent to the instrument, and we re-modeled the temperatures and memory use. The problem was complicated by several other unusual events that occurred around the same time: First, the memory on board the spacecraft (the “Solid State Recorder”, or SSR) had filled up because one of the dishes of the Deep Space Network was broken. This meant we couldn’t send data back to Earth, so it piled up in the memory until it overflowed. Second, HiRISE’s “keep-alive counter” was withheld. This is a steady heartbeat HiRISE sends to MRO that indicates HiRISE is still running. After a certain number of heartbeats are missed, MRO will safe HiRISE. Also around the same time, there were some errors in the spacecraft’s software. The timing was also mysterious: HiRISE safed about 15 minutes after an image. This is a long time afterwards – the image should have been completely done within just a few minutes. Instead, the temperature sensors showed that we continued to heat up for 15 minutes!

Finally, after a day of research, we found an answer. What happened was this: First HiRISE did all the setup steps to take an image (set the number of lines, etc.). One of these steps turns on the CCDs (sensors) in the camera. Then, right before it was about to actually expose the image, it found out that the memory was full. Since there wasn’t enough room in memory for the data, it didn’t take the image. However, everything was left turned on! So with everything powered on, we continued to heat up until we reached the limits we have set to protect the instrument. This withheld the keep-alive counter, and HiRISE safed. So in fact, the instrument worked exactly as it should have, in order to keep itself out of danger. It was just an unexpected response to this unusual situation.

With the help of the LMA engineers, we were able to power HiRISE back on the following day and start imaging again very quickly. Thankfully, we were up & running in time for another very special observation that I’ll be writing about soon….

For those interested, let me give you an idea what we are up to. The targeting specialists are starting to find the next places we’ll image. We are getting ready to start our regular imaging meetings, which will take place biweekly, to discuss the locations to image during the upcoming weeks.

HiOPS, the Operations team, is reviewing the operation of the camera, still verifying that everything was taken correctly. So far HiRISE appears to have operated nearly perfectly.

HiTECH, the technical support team, is making some adjustments to our software, tweaking it so as to make sure everything is running perfectly.

HiEST, the engineering team, is analyzing the telemetry of the pictures, and making sure everything ran as expected. Their report was that the instrument is in perfect condition.

Everyone is still looking at the images we have already taken, and the HiRISE internal email is full of cool segments of pictures, and commentary about these pictures. Everyone is very excited for the next batch we will receive!

So, we are all getting ready for Primary Science Phase. It’s amazing to think we’ll be able to get these high-quality pictures for such a long period of time! We are all just waiting to see what will happen. All that we really know is, we will learn many new and exciting things about the Red Planet, more than we have ever known before. And we can’t wait to do so!

I share an office with some of our downlink operations people, and right now I’m listening to them discuss their plans for the image processing that will take place this afternoon. They can tell you more about the details of the processing, but essentially right now they are making sure that all the plans and procedures are in place so that things go smoothly this afternoon. The last couple of days have been filled with testing, testing, and more testing.