A paper entitled Software Systems as Cities: A Controlled Experiment (Slides) by Richard Wettel, Michele Lanza and Romain Robbes at ICSE 2011 presents a user experiment with CodeCity. A detailed technical report also exists.
The results of the experiment which involved 41 participants (21 academic, 20 industry professionals) show hat CodeCity improved in both correctness (+24%, i.e. more correct answers) and completion time (-12%, i.e. faster) over the state-of-the-practice exploration tools (Eclipse plus Excel spreadsheet) for 10 questions related to program comprehension tasks. They proved that these results are statistically significant.
Another contribution of the paper is an experimental design wish list for software visualization user experiments:
- Choose a fair baseline for comparison
- Involve participants from industry
- Take into account the range of experience level of the participants
- Provide a tutorial of the experimental tool to the participants
- Find a set of relevant tasks
- Include tasks which may not advantage the tool being evaluated
- Limit the time allowed for solving each task
- Choose real-world systems
- Include more than one subject system in the experimental design
- Provide the same data to all participants
- Report results on individual tasks
- Provide all the details needed to make the experiment replicable