Last week we had a rare event: HiRISE turned off! We call this safe mode, because it’s a safety measure built into the instrument’s software. Whenever any of the sensors starts going out of bounds, like temperatures or voltages, the instrument powers down to prevent damage to the electronics. In this case, one temperature sensor went over its upper limit of 35 degrees Celsius. It’s pretty disconcerting when something unexpected like this happens, but at least we know the instrument is protected.

We had the difficult detective job of figuring out what went wrong. It was clear early on that the instrument overheated, but we couldn’t figure out why. Our tool that predicts the temperatures (”HiTemp”) didn’t predict anything that hot. We didn’t take a really large image, which would heat us up (at least, nothing bigger than normal! ). The local operations team worked with the health & safety people, the spacecraft engineers at LMA, and some of the software developers at Ball Aerospace that originally designed HiRISE. Together we all investigated the problem.

We studied the telemetry (information from the spacecraft), the details of the commands that were sent to the instrument, and we re-modeled the temperatures and memory use. The problem was complicated by several other unusual events that occurred around the same time: First, the memory on board the spacecraft (the “Solid State Recorder”, or SSR) had filled up because one of the dishes of the Deep Space Network was broken. This meant we couldn’t send data back to Earth, so it piled up in the memory until it overflowed. Second, HiRISE’s “keep-alive counter” was withheld. This is a steady heartbeat HiRISE sends to MRO that indicates HiRISE is still running. After a certain number of heartbeats are missed, MRO will safe HiRISE. Also around the same time, there were some errors in the spacecraft’s software. The timing was also mysterious: HiRISE safed about 15 minutes after an image. This is a long time afterwards – the image should have been completely done within just a few minutes. Instead, the temperature sensors showed that we continued to heat up for 15 minutes!

Finally, after a day of research, we found an answer. What happened was this: First HiRISE did all the setup steps to take an image (set the number of lines, etc.). One of these steps turns on the CCDs (sensors) in the camera. Then, right before it was about to actually expose the image, it found out that the memory was full. Since there wasn’t enough room in memory for the data, it didn’t take the image. However, everything was left turned on! So with everything powered on, we continued to heat up until we reached the limits we have set to protect the instrument. This withheld the keep-alive counter, and HiRISE safed. So in fact, the instrument worked exactly as it should have, in order to keep itself out of danger. It was just an unexpected response to this unusual situation.

With the help of the LMA engineers, we were able to power HiRISE back on the following day and start imaging again very quickly. Thankfully, we were up & running in time for another very special observation that I’ll be writing about soon….