Prioritising by comparing pairs
Scoring ten things out of ten produces numbers that cluster and drift. Comparing two at a time does not.
The problem with scoring
Ask someone to rate ten features out of ten and you get a cluster between six and eight, anchored on whichever was rated first, and a different answer if you shuffle the list and ask again.
Ask which of two features matters more and the same person is quick, confident and repeatable. Pairwise comparison asks only that, and derives the numbers rather than requesting them.
What the priorities actually are
The comparisons form a reciprocal matrix: if A is three times B, then B is a third of A, so only half the grid needs filling in. The priorities are the principal eigenvector of that matrix, which gives each item a share of the total, all summing to one.
Shares rather than scores is a useful property: it makes clear that priorities are relative, and that raising one lowers the others. Which is true, and something ten-point scoring lets people forget.
Inconsistency is information
Say A beats B, B beats C, and C beats A, and no set of priorities can satisfy all three. The consistency ratio detects this by comparing your contradictions against what random judgements would produce.
A high ratio is not a failure so much as a finding. Usually it means the items are not comparable on one dimension, or somebody was clicking rather than thinking, or two items are so close that the direction flipped. All three are worth discovering.
Keep the list short
Comparisons grow as n(n−1)/2. Five items is ten comparisons; ten items is forty-five, at which point people start clicking through and the consistency ratio begins measuring fatigue rather than disagreement.
With twenty things to prioritise, group them into categories, compare the categories, then compare within each. The hierarchy costs less effort and produces better answers than one enormous grid.
Why the list has to be short
The number of comparisons is n(n−1)/2, and it grows faster than people expect. Five items is ten judgements, which is pleasant. Ten items is forty-five. Twenty items is a hundred and ninety, and nobody completes a hundred and ninety pairwise judgements with the same care they brought to the first ten.
That decay is the real limit rather than the time cost. Once attention goes, the later comparisons are effectively noise being entered with the same weight as the early ones, and the result looks rigorous while resting on guesses. If you have twenty candidates, the honest move is to cut the field to seven or eight by some cruder method first, then compare properly — not to grind through the full matrix.
Contradicting yourself is a finding
Sooner or later you will prefer A to B, B to C, and C to A. That is a cycle, it is logically impossible as a single ranking, and it is extremely common. It is not carelessness.
What it almost always means is that you are not comparing on one criterion. A beat B on cost, B beat C on speed, C beat A on how much you trust the supplier — each judgement was sound and they used different scales, so the result cannot be reconciled into one order. This is genuinely useful information: it tells you the decision has more than one dimension and that you have not yet decided which dimension governs. The fix is to name the criteria and compare on each separately, not to go back and force the cycle straight.