https://github.com/deepmind/neural_testbed
Introduction
Posterior predictive distributions quantify uncertainties ignored by point estimates. The neural_testbed provides tools for the systematic evaluation of agents that generate such predictions. Crucially, these tools assess not only the quality of marginal predictions per input, but also joint predictions given many inputs. Joint distributions are often critical for useful uncertainty quantification, but they have been largely overlooked by the Bayesian deep learning community.
This library automates the evaluation and analysis of learning agents:
- Synthetic neural-network-based generative model.
- Evaluate predictions beyond marginal distributions.
- Reference implementations of benchmark agents (with tuning).
the first paper to propose a concrete evaluation procedure for the quality of joint predictions in neural network classification.
Neural Tesbed defines 420 classification problems where each problem is identified by a string identifier called `problem_id`.
paper:
Evaluating High-Order Predictive Distributions in Deep Learning
https://openreview.net/forum?id=rFb8y8Io5e9
好的预测对好的决策至关重要。至关重要的是，这些决 策的质量取决于联合预测的质量，而不仅仅是边际预测。
