Introduction to Akaike Information Criterion

The fancy term Akaike Information Criterion has been out there for quite some time now, and it is about time every one of you knows what AIC is. Having that said, this blog covers the following things about Akaike Information Criterion.

• What exactly is it?
• What should researchers use it? How to evaluate the outcomes?
• What are some of the drawbacks of AIC?

Let’s start with the introduction to AIC, then

What is the Akaike Information Criterion?

The Akaike information criterion (AIC), according to Wikipedia, is a measure of out-of-sample prediction error and, as a result, of statistical model quality for a given set of data. AIC measures the quality of each model in relation to the other models given a set of data models. As a result, AIC can be used to pick a model.

So, in simpler words, the AIC is a single numerical score that can be used to evaluate which of several models is most likely to be the best for a given dataset. It estimates models relative to other AIC scores for the same dataset; hence AIC scores are only helpful when compared to other AIC scores for the same dataset. It is preferable to have a lower AIC score.

AIC is most commonly employed in cases where traditional machine learning methodology makes it difficult to test the model’s performance on a test set (small data or time series). Because the most valuable data in time series analysis is frequently the most recent, which is locked in the validation and test sets, AIC is very useful.

As a result, compared to standard train/validation/test model selection methods, training on all data and employing AIC can result in superior model selection.

AIC evaluates the model’s fit on the training data and adds a penalty term to account for the model’s complexity (similar fundamentals to regularization). The goal is to determine the AIC with the lowest value, which shows the best balance of model fit and generalizability. This helps achieve the ultimate goal of improving fit on out-of-sample data.

Here,

L is the likelihood and
k is the # of parameters

Having the log-likelihood of your model, you can simply calculate AIC by hand; however calculating log-likelihood is difficult! AIC can usually be calculated using most statistical software.

The Use of the AIC Model: When to Use it?

The most commonly used model selection method in statistics is the AIC model. You can determine the best fit for the data by calculating and comparing the AIC scores of various different models.

When testing a hypothesis, you may collect data on factors about which you are unsure, especially if you are experimenting with a novel concept. You want to determine which of your independent factors accounts for the variation in your dependent variable.

Creating a collection of models, each including a different mix of the independent variables you have measured is a nice method to find out.

The combinations must base on the following factors:

1. Your understanding of the research system – do not use parameters that are not logically connected, as you can establish misleading connections between virtually everything!
2. The experimental design — suppose you have divided two treatments among test subjects, then there is usually no purpose to test for a treatment interaction.

After you have created a few different models, you can compare them using AIC. AIC scores with fewer parameters are better, and AIC reprimands models with more parameters. When two models explain the same amount of variation, the one with fewer parameters has a lower AIC score and is the better-fit model.

Drawbacks of Akaike Information Criterion

Remember that the AIC only assesses the relative quality of models. This means that all of the models evaluated may still be unfit. As a result, further metrics are required to demonstrate that the outputs of your model meet an acceptable absolute standard.

AIC is also a pretty simple calculation that has been improved upon and surpassed by other generalized measures that are more computationally demanding but often more accurate. WAIC (Watanabe-Akaike Information Criterion), DIC (Deviance Information Criterion), and LOO-CV are some examples (Leave-One-Out Cross-Validation, which AIC asymptotically lines with large samples).

You can choose between AIC and one of the newer, more difficult computations, depending on how much accuracy vs computational strain (and simplicity of the calculation, given your software package’s capabilities). Ben Lambert provides a clear and concise video explanation of the distinctions between AIC, DIC, WAIC, and LOO-CV.

To Conclude

When there is enough data, the best (and simplest) way to reliably verify your model’s performance is to employ a train, validation, and test set, which is a standard machine learning procedure. However, in situations where this is not practicable (such as with tiny data or time series analysis), AIC may be a better option.

Frequently Asked Questions