On the unreliability of bug severity data

摘要: Severity levels, e.g., critical and minor, of bugs are often used to prioritize development efforts. Prior research efforts have proposed approaches automatically assign the severity label a bug report. All prior verify accuracy their using human-assigned reports data that is stored in software repositories. However, all assume such reliable. Hence perfect automated approach should be able same as repository --- achieving 100% accuracy. Looking at duplicate (i.e., referring problem) from three open-source systems (OpenOffice, Mozilla, Eclipse), we find around 51 % inconsistent labels even though they refer problem. While our results do indicate unreliable labels, believe send warning signals about reliability full including non-duplicate reports). Future explore if findings generalize dataset. Moreover, factor nature data. Given data, classical metrics assess models/learners not for assessing assigning label. Hence, propose new performance models. Our assessment shows current perform well 77-86 agreement with labels.