Making a decision matrix that means something
Any weighted matrix produces a winner. Whether that winner survives a reasonable disagreement about the weights is the only question worth asking.
The ranking is not the output
A weighted matrix always produces a leader. The useful question is how far a weight would have to move before the leader changes — because the weights are the part somebody invented.
If the answer flips when a criterion moves from 20% to 25%, no decision has been made. A preference has been recorded and dressed in arithmetic. If it holds across every plausible weighting, the matrix has genuinely told you something and the meeting can be short.
Weights get chosen to produce the answer
Worth naming plainly: people frequently arrive with a preferred option and adjust weights until the matrix agrees, usually without noticing.
The defence is procedural. Set weights before scoring, ideally with people who have not yet seen the options, and write down the reason for each. Sensitivity analysis does the rest — if a conclusion holds only at one precise weighting, that becomes visible instead of hidden.
Normalisation is not a detail
Criteria measured on different scales cannot be added together. A criterion scored out of 100 swamps one scored out of 5 no matter what weight you assign, so scores have to be normalised first.
Min-max normalisation is the obvious method and it distorts close decisions. It stretches every criterion to span the full 0-to-1 range, so the best option always scores 1 and the worst always 0 regardless of how close they really are. With two options it is completely degenerate — every criterion becomes win-or-lose, and a 4% price difference renders identically to a fourfold one.
Ratio normalisation — divide by the best value, or the best value divided by yours where lower is better — keeps a 4% difference looking like 4%. That is what you want when the whole exercise is about how much better one option is.
Beware criteria that measure the same thing
"Cost", "licence fee" and "total cost of ownership" as three criteria give cost three votes. No matrix can detect this, and the outcome is dominated by whichever consideration was described in the most ways.
Before scoring, check that each criterion could move independently of the others. If two always rise and fall together, they are one criterion wearing two hats.
When the top two are close
If the leader and runner-up are within a few percent, the matrix has told you they are equivalent within the noise of anyone's scoring. That is a real and useful finding.
The wrong response is to add criteria until one pulls ahead. The right one is to accept that this method cannot separate them, and decide on something it does not capture — who will maintain it, which team is enthusiastic, which is easier to reverse.