Human ranked/sorted items

I'm trying to derive an algorithm or procedure I would use to have a human interactively rank a series of items based on importance by comparing a random-pairing of items at a time.

So, given an array of let's say 10 "things", I want to present two of the things, and have the user specify which is the more important of the two. What I don't know is:

• What is the proper name for this procedure/concept?

• Would I use some form of sorting algorithm?

• How could I determine when to stop? When is the ranking "done"?

• And, importantly, I would prefer not to present the same "thing" to the user in a row. That is, if comparing Item B and Item K, user selects Item K as more important, the next iteration should not present either Item B or Item K.

What I'm after seems similar to this question: Nearest neighbour based on subjective human comparison - is this a thing?

... but I'm not interested in similarity, just relative importance. Think prioritizing a to-do list or subjectively ranking vacation destinations, etc.

The Elo Rating System may be similar to what I'm after, as well.

There are statistical models for relative ranking. I'd suggest taking a look at the Bradley–Terry model. The data is sometimes known as paired comparison data.

See also Analysis and classification based on data points, Peer grading design - choosing a graph, to get accurate rankings/ratings, https://stats.stackexchange.com/q/83005/2921.

StackExchange Computer Science Q#76352, answer score: 4