What do the people at the HiRISE Operations Center (HiROC) do when the MRO spacecraft safes?

When the spacecraft safes, the automatic reset sequence also turns off all the scientific instruments on board. So we were shut down early last week. We knew from preliminary investigations that HiRISE wasn’t involved with the cause of the event (whew!), so we couldn’t really help analyze the cause. This was different from some situations in the past where HiRISE caused itself to safe; in those cases, we were responsible for figuring out why. This time we just had to wait while the spacecraft engineers figured out the problem.

Once the engineers determined we could go back to normal operations, we worked with them on our instrument safe mode recovery. This involves radiating a set of pre-written commands and files to the spacecraft, verifying that the setup commands executed correctly on the instrument, and watching the instrument telemetry to make sure we power up correctly. We have a series of checks built in to this sequence. They work like a checksum, and send that information back down to Earth in the telemetry. We check those values against what we expect them to be, and that’s how we know the correct values were written to memory. Without this confirmation, we couldn’t be sure that HiRISE’s memory was uncorrupted.

Every step in the lengthy procedure executed successfully, and all of the checks matched what we expected, so we were able to confirm that HiRISE was ready to return to normal imaging. The next day, they restarted the sequence on board that calls our imaging commands, and we were back in business!

• Related HiBlog post: On the Safe Side

Unfortunately, we can’t just make them all the largest size the instrument is capable of taking, because our camera will get too hot. If it overheats, the instrument will shut itself off in order to prevent any damage to the electronics. So we have to be careful, and only plan images that won’t overheat HiRISE. In order to predict those temperatures, we use a tool called HiTemp (of course!). Here’s what it looks like (click on the image to see a bigger version).

This program reads in our planning files, and then models the temperatures of two key spots on the focal plane of the camera. It’s our job to make sure we don’t go above the dotted red line – this gives us a comfortable buffer below the scary solid red line. That’s when HiRISE would shut off, or safe. We know from experience by now that this is a big pain in the neck – a lot of work is required to get us back up & running, and we miss observations while we’re turned off. So we watch our HiTemp plots!

We had the difficult detective job of figuring out what went wrong. It was clear early on that the instrument overheated, but we couldn’t figure out why. Our tool that predicts the temperatures (”HiTemp”) didn’t predict anything that hot. We didn’t take a really large image, which would heat us up (at least, nothing bigger than normal! ). The local operations team worked with the health & safety people, the spacecraft engineers at LMA, and some of the software developers at Ball Aerospace that originally designed HiRISE. Together we all investigated the problem.

We studied the telemetry (information from the spacecraft), the details of the commands that were sent to the instrument, and we re-modeled the temperatures and memory use. The problem was complicated by several other unusual events that occurred around the same time: First, the memory on board the spacecraft (the “Solid State Recorder”, or SSR) had filled up because one of the dishes of the Deep Space Network was broken. This meant we couldn’t send data back to Earth, so it piled up in the memory until it overflowed. Second, HiRISE’s “keep-alive counter” was withheld. This is a steady heartbeat HiRISE sends to MRO that indicates HiRISE is still running. After a certain number of heartbeats are missed, MRO will safe HiRISE. Also around the same time, there were some errors in the spacecraft’s software. The timing was also mysterious: HiRISE safed about 15 minutes after an image. This is a long time afterwards – the image should have been completely done within just a few minutes. Instead, the temperature sensors showed that we continued to heat up for 15 minutes!

Finally, after a day of research, we found an answer. What happened was this: First HiRISE did all the setup steps to take an image (set the number of lines, etc.). One of these steps turns on the CCDs (sensors) in the camera. Then, right before it was about to actually expose the image, it found out that the memory was full. Since there wasn’t enough room in memory for the data, it didn’t take the image. However, everything was left turned on! So with everything powered on, we continued to heat up until we reached the limits we have set to protect the instrument. This withheld the keep-alive counter, and HiRISE safed. So in fact, the instrument worked exactly as it should have, in order to keep itself out of danger. It was just an unexpected response to this unusual situation.

With the help of the LMA engineers, we were able to power HiRISE back on the following day and start imaging again very quickly. Thankfully, we were up & running in time for another very special observation that I’ll be writing about soon….