quorum

1200 support tickets and no labels, only the keyword rules ops already wrote. They disagree with each other. The label model works out how far to trust each one, and turns them into a training set.

GitHub

Labelling functions

The rules ops already had, plus a prompt and two heuristics. Turn one off and the label model refits on what is left.

hash_of_id labels tickets by hashing their id. It knows nothing. Switch it on and watch the two scores come apart.

refitting
98.2% corpus covered 21 tickets no LF touched
89.1% label model gold accuracy on 147 held-out tickets
89.1% majority vote level with the label model
94.7% the model you get for free TF-IDF + logistic regression trained on 1032 of these labels. Nobody hand-labelled anything.

hash_of_id is off, and the two models agree. Switch it on.

Labelling functions

Learned accuracy comes out of the fit and never saw a label. Gold accuracy is measured on the 150-ticket held-out slice and is printed here for comparison only.

labelling functioncoverageoverlapconflict emitslearned accgold acc
refund_words 0.224 0.219 0.091 billing 0.977 0.838 n=37
payment_terms 0.258 0.253 0.117 billing 0.963 0.897 n=39
crash_words 0.203 0.198 0.088 bug 0.759 0.806 n=31
error_code 0.036 0.036 0.012 bug 0.876 0.714 n=7
login_words 0.201 0.191 0.090 account 0.596 0.696 n=23
automated_sender 0.134 0.133 0.077 account 0.488 0.583 n=12
survey_channel 0.130 0.127 0.074 feedback 0.685 0.579 n=19
praise_words 0.145 0.142 0.078 feedback 0.797 0.950 n=20
version_regression 0.150 0.144 0.086 bug 0.528 0.720 n=25
llm_triage 0.897 0.779 0.292 billing, bug, account, feedback 0.872 0.896 n=134

Labelling functions that are really one voter

The label model treats these as independent given the true class. Each pair below fires on much the same tickets and says much the same thing, so it is being counted twice and the confidence on those tickets is overstated. quorum reports this and does not correct it: correlated labelling functions are out of scope.

  • refund_words and payment_terms fire together 0.66, agree 1.00