Unblinding test runs (sent 2017-05-13 10:41 CET)

Dear shared task participants,

I have been in touch with several teams about this but I want to make sure that everybody is aware of the option:

I can unblind your test runs so that you see the error messages output by your system.

In general I only look at evaluator runs. If everything looks OK, I mark it as “no errors” and that’s it; the evaluator run is now colored green in TIRA. You typically already know it’s OK because the evaluator did not complain; I would only contact you (or leave a note in the review) if I saw surprisingly low scores there.

However! If some or all output files are missing (by far the most frequent thing I observe), it may be difficult for you to understand what went amiss, especially if you followed our advice, ran the system over the trial data and did not observe any problem there. Quite often the problem is clearly visible in the STDERR: for example, your system could not find the file with the corresponding model, threw an exception and gave up. In that case I can “unblind” the test run for you (naturally only if I believe it does not leak the test data). Unblinding means that you can see the STDERR and STDOUT; you still cannot download the run with whatever output files it may have produced. In your TIRA interface, you can recognize unblinded test runs by the information about runtime and size (normally these fields say “hidden”).

Please watch for unblinded runs in your run overview. With growing amount of runs to review I am no longer able to contact everyone directly. Sometimes I decide to unblind a run without your asking but I don’t have time to inspect all runs and I may overlook things. If you would like to have a run unblinded, send me an e-mail.

Best,

Dan