Interpretability and explainability

Machine learning algorithms are about finding correlation in data. In contrast, Interpretable machine learning is about understanding causality.

Applying domain knowledge requires understanding of the model and how to interpret its output.

Often, we need to understand individual predictions a model is making.

For instance, a model might:

• Recommend a treatment for a patient or estimate disease to be likely. However, the doctor needs to understand the reasoning behind the prediction to trust it.
• Classify a user as a scammer, but the user disputes it. The fraud analyst needs to understand why the model made the calssification to justify it.

Usefulness of interpretability

We need to understand differences on a datasets level:

• why the product release receive worse feedback than previous updates
• why are the grain yields in ore region higher in another?

Model debugging. We might want to understand why a model that worked previously are not working when applied to new data.

Dimensions of interpretability

• Prediction-level interpretability

  Explain why $x$ is predicted as $y$ by $f(x)$

• Model-level interpretability

  What does a pattern belonging to our model look like?

• Data-level interpretability

  What is the most important dimension of our data?

What about decision trees and rules?