|
A Reputation System for Uncertain Assertions
May 2009
Mark Kramer, The MITRE Corporation
Arnon Rosenthal, The MITRE Corporation
ABSTRACT
We investigate reputation systems that rate the performance of
analysts who make uncertain assertions (claims accompanied by estimated
probabilities). Accuracy metrics (based on the fraction correct) are fair only if
all analysts handle identical or statistically similar cases. Furthermore, accuracy
metrics discourage analysts from offering predictions on difficult-to-predict
events. Because of these difficulties, we develop a class of performance scoring
functions that are maximized when the analyst provides accurate probabilities,
especially when these probabilities differ from the norm. Under these metrics,
the disincentives to forecast low-probability events is removed and analysts are
rated fairly, independent of the base event probabilities of the cases they
consider. Reputation systems built around these metrics can support
productivity management and increase manipulation resistance when
information providers are not trustworthy. An application to citizen event
reporting is presented.
Additional Search Keywords
uncertainty, reputation, scoring function, citizen event reporting, counter insurgency
|
