Software Systems as Cities: A Controlled Experiment

A paper entitled Software Systems as Cities: A Controlled Experiment (Slides) by Richard Wettel, Michele Lanza and Romain Robbes at ICSE 2011 presents a user experiment with CodeCity. A detailed technical report also exists.

The results of the experiment which involved 41 participants (21 academic, 20 industry professionals) show hat CodeCity improved in both correctness (+24%, i.e. more correct answers) and completion time (-12%, i.e. faster) over the state-of-the-practice exploration tools (Eclipse plus Excel spreadsheet) for 10 questions related to program comprehension tasks. They proved that these results are statistically significant.

Another contribution of the paper is an experimental design wish list for software visualization user experiments:

  • Choose a fair baseline for comparison
  • Involve participants from industry
  • Take into account the range of experience level of the participants
  • Provide a tutorial of the experimental tool to the participants
  • Find a set of relevant tasks
  • Include tasks which may not advantage the tool being evaluated
  • Limit the time allowed for solving each task
  • Choose real-world systems
  • Include more than one subject system in the experimental design
  • Provide the same data to all participants
  • Report results on individual tasks
  • Provide all the details needed to make the experiment replicable

About Craig Anslow

Craig Anslow is a Lecturer in Software Engineering at Victoria University of Wellington, New Zealand, with research interests in software visualization.
This entry was posted in Papers, Research and tagged , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s