Extending XData to kill SQL query mutants in the wild

Abstract

SQL queries are usually tested for correctness by executing them on one or more datasets, to see if they give the desired results on each dataset. Erroneous queries are often the result of small changes, or mutations, of the correct query. Earlier work on the XData system showed how to generate datasets that kill all mutations in a class of mutations that included join type and comparison operation mutations. However, the system could not handle a number of commonly used SQL features. In this paper we extend the XData data generation techniques to handle features such as null values, string constraints, aggregation with constraints on aggregation results, and a class of subqueries, amongst others. We present a study of the effectiveness of our data generation approach for correcting student SQL assignments that were part of a database course. The datasets generated by XData outperform publicly available datasets, as well as manual grading done earlier by teaching assistants.