The need to interpret the predictions obtained by machine learning models has become ever more important over the last two decades, mainly due to the omnipotent potential of such models, particularly deep learning models, to reach unprecedented levels of accuracy and high levels of performance. Several papers have been previously proposed which aim at explaining the predictions of automated decisionmaking systems, particularly those based on machine learning. One question that commonly arises upon the practical use of these explanations is whether the corresponding predictions are correct in the first place. This usually comes along with the correlated issue of the degree of uncertainty involved within such predictions. In case the prediction is wrong, or at least highly uncertain, is it worth finding an explanation for? In this work, we aim to tackle the practically actionable issue that should be addressed when such a situation arises, which is based on the following caveat: Rather than explaining a prediction which is of little-to-no impact given the relatively high likelihood that it is incorrect, we address the more applicable issue of explaining the reasons for the high uncertainty. We do so via highlighting parts of the input data which are believed to be the most responsible for such a high predictive uncertainty. Our method is based on information-theoretic computations where entropy is utilised as a proxy to evaluate the predictive uncertainty, simultaneously while performing the optimisation. This ultimately leads to finding a similar input which correspondingly produces an output with lower predictive uncertainty (compared to the original input). We empirically demonstrate the impact of the method in identifying the input components most responsible for high predictive uncertainty by conducting experiments on two tabular datasets (Titanic and Pima Indians Diabetes) as well as an image dataset (Fashion-MNIST).