Realize that even temperature=0 is not deterministic and think about ways to quantify the randomness this introduces and bring it into your analyses.

Realize that individual LLMs, even from the same family of models, have different biases given your fixed prompt. In response, vibe-code a human annotation tool so that you and your collaborator can label cases for prompt optimization.

Realize that both you and your collaborator are secretly getting help from Opus to do the annotations, because neither of you can (for example) evaluate FORTRAN code relative to a Portuguese specification. Throw out all your human annotations.

Realize that it would be more productive to have a family of LLMs collaborate with each other to develop an annotation protocol, with managerial (and visionary!) guidance from you and your collaborator. Quantify the LLMs’ agreement rates and find them to be more than acceptable.

Scale the annotation protocol, and then notice that the frequency of one of the labels is implausibly high. Through laborious manual review, trace this to the fact that the annotator agent is ignoring the distinction between user turns and AI turns. Stare in wonder as Opus immediately confirms this suspicion once you raise it even though Opus itself is the offending annotator.

Return to your annotator pool with much firmer guidance and a much heavier hand. Observe that this has many positive effects. Scale for a second time. Feel energized and illuminated by the response distributions, but recall that you are now a jaded skeptic about your project, and possibly about the very concept of annotation. Question even the most straightforward labels. Engage in an epic exploratory discussion/feud with Claude about whether these annotations support “a structural finding that survives many controls” or merely “tell a cautionary tale about LLM-tagging methodology”. Ultimately come to doubt all your own linguistic and scientific intuitions.

Take solace in the fact that, by all reports, AGI is just a few years away, by which point society may have been completely transformed but at least you will (probably) be able to get the annotations you need.