WiDS checker: combating bugs in distributed systems

摘要: Despite many efforts, the predominant practice of debugging a distributed system is still printf-based log mining, which both tedious and error-prone. In this paper, we present WiDS Checker, unified framework that can check systems through simulation reproduced runs from real deployment. All instances be executed within one process, multiplexed properly to observe "happensbefore" relationship, thus accurately reveal full state. A versatile script language allows developer refine properties into straightforward assertions, checker inspects for violations. Combining these two components, are able otherwise impossible check. We applied Checker over suite complex found non-trivial bugs, including in previously proven Paxos specification. Our experience demonstrates usefulness us gain insights beneficial future research area.