Methodology

How Shortlist Scores Work: Where the Numbers Come From

People keep asking what a Shortlist rating is based on, which is a fair question to ask of any number a website puts next to a film. Here is the whole answer, with nothing held back.

The short version

1.A score in the app is yours. It is built out of comparisons you made yourself. It is not a critic average, not a star rating you typed, and not a number we assigned.
2.The S to D tier lists on this site are written, not calculated. They are our argument, with a reason attached to every film, and you are meant to disagree with them.
3.Nobody pays to move up anything. There is no sponsored placement in a ranking or a tier list on Shortlist.

The score in the app, step by step

Shortlist never asks you for a number. It asks you which of two films you liked more, over and over, and the number falls out of your answers.

1.You say roughly how you felt
When you add a title you pick one of three reactions. That is the only thing you ever type in, and all it does is decide where the title starts before any comparison happens.
Loved it
starts at 1400
It was okay
starts at 1200
Not for me
starts at 1000
2.You pick a winner, a pair at a time
The new title is put up against films already on your list. Every time you choose, both titles move, using the Elo system that chess ratings use. Beating something you have rated far above it moves it a long way. Beating something you already rank below it barely moves anything, because you had already said as much.
3.That rating is rescaled to 0 through 10
The 0 to 10 score you see on a poster is the same rating on a friendlier scale. Nothing is added to it and nothing outside your own list feeds into it.
Loved it start
9.0
It was okay start
6.3
Not for me start
3.5
Those are starting points, before a single comparison. A film you loved and then lost three matchups to will not stay there.
4.Two deliberate limits
- Only your number one can be a perfect 10.0. Everything else stops at 9.9, however far ahead it is. A list with four tens in it is not a ranking.
- Scores stay hidden until you have ranked 3 titles of that kind. A score computed from one comparison looks precise and means almost nothing, so the app shows a question mark instead of publishing a number it does not believe yet.

What the score is not

It is not a critic score. No review aggregator feeds it. If a film you love is at 96% everywhere and you put it fourth, it sits fourth.

It is not a public vote. Nobody can push a title up or down on your list. Only your own comparisons move it.

It is not final. Ranking one more film can reshuffle everything under it, which is the point. A ranking that never changes is a list, not an opinion.

Where the app shows a number for a title across everyone rather than just for you, that number is the average of Shortlist rankers' own scores for it. Same method, one layer up, and it moves as people rank.

The S to D tier lists on this site are a different thing

The franchise tier lists published here are editorial. A person wrote them, and every film carries a line saying why it landed where it did. Nothing computes them and no vote decides them.

S tier does not mean best reviewed. It means it held up. That is why a film can open to the best reviews of a director's career and still start in A tier: staying power is the one thing a film cannot demonstrate on the day it opens. When we place a new release early, the page says it is provisional and says what would move it.

Within a tier, the order is not a claim. If two films are both S tier, we are saying they are both S tier, not that the first one beats the second. That is why the ranker at the foot of those pages never scores you out of five against our order: it shows your list with our tier badge beside each film, so you can see exactly where you and we disagree, and it only calls out the top pick, which is the argument people actually came to have.

Every film on those lists carries its release year, because two films sharing a title is more common than it sounds. This site publishes two called Halloween and two called Scream.

Shortlist Ratings: Frequently Asked Questions

What is Shortlist basing its ratings on?

On comparisons you make yourself. Shortlist never asks you to give a film a number. It shows you two titles you have seen and asks which you liked more, and it does that a handful of times per title. The score is the standing that falls out of those results, rescaled to a 0 to 10 number. No critic scores, no box office, no editorial thumb on the scale.

Is a Shortlist score the same as a Rotten Tomatoes or IMDb rating?

No, and they are not measuring the same thing. Those are aggregates of other people: critics in one case, a large voting public in the other. A Shortlist score is a statement about one person's own ranking, which is why the same film can sit near the top of one list and in the middle of another without either being wrong.

How does the head-to-head ranking actually calculate a score?

With Elo, the same rating system chess uses. Each title carries a rating. When you pick one over the other, the winner takes points from the loser, and how many depends on how surprising the result was: beating something you already rank far above it moves almost nothing, while beating your reigning favourite moves a lot. The 0 to 10 score on the poster is that rating on a friendlier scale.

Why does the app show a question mark instead of my scores?

Because you have not ranked enough titles of that kind yet. A score built from a single comparison looks precise and means very little, so the app keeps scores hidden until there is enough to stand behind. Rank a couple more and the numbers appear.

Are the S to D tier lists on shortlist.watch voted on or calculated?

Neither. They are written, and every film on them carries a line saying why it landed where it did. S tier on those lists means a film that held up rather than a film that reviewed well on opening weekend, which is why a well-reviewed new release is usually placed provisionally one band lower until it has had time to prove which way it goes.

The Reel

One sharp movie newsletter, every Tuesday

What to watch, what to skip, and where it lands on the rankings. Free, no spam, unsubscribe anytime.

Subscribe and get the free printable Shortlist tier sheet.

See it build a ranking

The fastest way to understand the method is to run it. Pick a franchise and answer a few head-to-head matchups in the browser, no account needed, or get the app and rank anything.

Rank a franchise in your browser