Bradley–Terry Models and Pairwise Comparisons in brms

Whew—it’s been a minute since I’ve written a blog post. But I recently published a paper that seemed like it would lend itself nicely to a new post, so here we are!

Before I move on I’ll note that this is an abbreviated version of a longer post from my personal blog. You can read the full post here. If you really want to dig into the code and the resulting figures I’d suggest going over there to see it. I’m trying to get back to posting here at the Quantitative Peace more regularly, so I’ll put some of the substantive front-end subject matter here. But WordPress isn’t great for the code chunks and execution, so I’ll leave that stuff for the other blog page. Please excuse some of the residual formatting issues leftover from Quarto.

The goal of this post is to help readers to understand how to fit a Bradley-Terry model for pairwise comparison data using the {brms} package in R. Bradley-Terry models are useful for generating rankings of players, groups, teams, etc., in cases where users might be interested in some kind of underlying latent attribute or characteristic. For example, we might be interested in generating a latent measure of a team’s skill, or someone’s knowledge of a certain topic. Bradley-Terry models are particularly useful when we have data on pairwise comparisons, such as win/loss data from sports matches, or preference data from surveys.

Below I’ll provide a brief overview of Bradley-Terry Models, what they are, why we might use them, and why we might want to fit one using {brms}, specifically. Then I’ll walk through a simple example of how to fit a Bradley-Terry model using {brms} in R.

What are Bradley-Terry Models?

I’m not intending to provide a comprehensive history of the subject, but it’s maybe worth discussing the development of these models a little bit. Bradley–Terry models, as a broad class of models, generally trace their roots to Bradley and Terry’s (1952) Rank Analysis of Incomplete Block Designs: The Method of Paired Comparisons. I say “as a broad class of models” because there are a number of different approaches to dealing with paired comparisons, which, according to Bradley and Terry’s own article, go back to at least the 1920s.

These models are generally focused primarily on two goals. First, what is the probability that a given item (e.g. item $i$) ranks higher than, or “wins”, when compared with another item (e.g. item $j$) in a paired comparison? The second goal is to estimate the “rank” of individual items in relation to one another—essentially a latent dimension.

You can see the two components in the equation above. The left-hand side of the equation shows the first use—the probability that $i$ beats $j$. The right-hand side shows the second part, which essentially represents the latent traits or characteristics of item $i$ and item $j$.

These models have a wide range of uses. If you’re new to these and do a quick search you’ll probably stumble on something that uses team sports as an example. For example, what’s the probability that a given soccer team, football team, baseball team, etc., is going to win a given match? We might also be interested in assessing more general latent traits, like a team’s overall “skill” level, by looking at the performance of teams over a given season and using individual matches along with corresponding wins/losses/ties as the basis for the paired comparisons.

It’s worth noting that we’re not limited to these applications, and the paired comparisons framework can be used to address lots of different problems. Interestingly, @Guttman1946 uses a more policy-relevant issue of scoring US Army demobilization score cards at the end of World War II as a motivating example. The central issue was how to weight five different factors on a survey sent to US soldiers that would determine their order or priority for demobilization and discharge from the armed services.

In this example, the specific factors in question were 1) length of time in the Army, 2) the amount of time spent overseas, 3) the amount of combat experience, 4) the soldier’s age, and 5) the number of children the soldier had. I don’t intend to go into a lot of depth on this particular paper, but it’s especially interesting because it sounds like soldiers ultimately didn’t like making paired comparisons between these individual attributes in isolation of the others because they felt that the omitted information was potentially vital to their decision. In this case, the paired comparisons become multidimensional, and the solution looks an awful lot like a conjoint experiment that we’d see used today to compare the influence of different individual variables when presented in combination with one another.

Another area where there is variation in approaches is how to handle ties. Earlier versions of these models (like @Guttman1946) typically assume away the presence of equivalent judgments, or ties. That’s fine, but sometimes we might care about cases where teams are tied, evaluators are ambivalent between two choices, etc. Later work by @Davidson1970 provides one method for dealing with this. This is also something I’ll address below briefly, but just be aware that there are options here.

Why Would We Use Them?

As with other cases where we might want to compare individuals, items, teams, etc., there are some problems with some of the more basic methods of comparison.

When comparing teams, for example, wins and losses can be potentially misleading if one team has an easier schedule than another. Comparing cumulative scores might also give an inaccurate picture of team performance if teams are competing under more difficult conditions than others (like bad weather or muddy fields).

Similarly, individuals’ test scores might not be a reliable way to evaluate intelligence or ability. Individual test takers can arrive at a comparable or identical score through a wide range of different pathways. Two students might both receive an A grade on an exam with one student missing a mixture of difficult and easy questions and the other missing only the most difficult. There may be good reasons for not wanting to treat these cases as equivalent.

Given the problems with “simpler” alternatives, researchers have developed a range of options for estimating the underlying trait of interest. Education and Psychology for example have developed a variety of different models under the banner of Item Response Theory (IRT) to do just this. These models let us take observational data, which might be complex and difficult to evaluate using simpler methods, and estimate underlying traits of interest using some fairly simple methods.

Estimating Paired Comparisons

Existing Packages and Approaches

There are a number of websites and packages that you can use to estimate Bradley–Terry models. The BradleyTerry2 package is a great starting point for anyone interested in tinkering with these models. Andrew Mack also provides a walk through for how to write up your own Bradley-Terry model. These are both great places to start if you’re looking to familiarize yourself with the models and how they work under the hood.

There are also some useful alternatives to be aware of. The {bpcs} package (which stands for Bayesian Paired Comparison) uses Stan as a backend to estimate paired comparisons. It provides users with a range of options for things like how to handle ties, modeling home advantage, etc.

Why would you choose one package over the other? The Bayesian option is useful because we’re often interested in estimating latent attributes, like skill or knowledge, for the various items or players in our data. Traditionally these models have been estimated using maximum likelihood, but this approach can have limitations in certain contexts.

Mattos and Silva Ramos, the creators of the {bpcs} package, provide a nice overview of the advantages of the Bayesian approach. For example, in some cases the maximum likelihood estimate isn’t available and models won’t converge. Sometimes this can arise as a result of not having connectivity between individual units through overlapping pairings. Packages like {BradleyTerry2} will also produce latent estimates without corresponding estimates of uncertainty, as you can see in @tbl-bt-example-1. The Bayesian approach, on the other hand, provides users with posterior distributions for each individual unit, which can be converted into point estimates, used to generate credible intervals, or sampled for downstream analyses.

Takeaway

You can look over at my personal blog for more info if you’d like to dig deeper, but I’ll leave this post here for now because the rest gets more densely populated with code and examples. So go check it out!