All downlink tasks I perform follow a particular development path: (1) I practice and jot down manual procedures; (2) over time I attempt to automate subtasks by using Perl or other tools, as best I can (I am not a software developer); and (3) our talented software developers write software that automates the task completely, or at the very least speeds it up considerably. Of course, this is always just in time for me to be assigned new tasks!One of my daily tasks is monitoring data quality and paying attention to missing or gapped raw image channel files (up to 28 channels per observations.) My tools: our internal reporting website HiReport, Terminal, and Microsoft Excel. I look through a list of observations in my web browser, click on those that appear to be missing channels or are flagged as “INCOMPLETE”, and copy and paste information about the problematic channels into Excel. I then add some additional notes and take any required actions.

A few days ago I realized that after more than a year, I was still in the manual stage of monitoring data quality and not making good use of existing tools to help streamline the process. All that cutting and pasting was beginning to get ridiculous, even more so during a period of high data rates and an increasing number of observations.

(more…)