OSU Software Testing and Analysis Group


Overview:

This is the webpage for the OSU testing and analysis group advised by Alex Groce. Our group investigates a wide range of topics related to testing, verifying, and analyzing software for correctness. We focus on testing real-world software systems, including C compilers, file systems, JavaScript engines, and other systems software. Our group has a strong emphasis on effective random testing, combining machine learning with testing, and applications of delta-debugging.

Projects:

In cases where we have basic paper(s) on a topic, links lead to those papers best for getting an overview of the work.


Publications:

Journal Papers

  1. Cause Reduction: Delta Debugging, Even Without Bugs

    Alex Groce, Mohammad Amin Alipour, Chaoqiang Zhang, Yang Chen, and John Regehr. "Cause Reduction: Delta Debugging, Even Without Bugs." Software Testing, Verification and Reliability, accepted for publication.

  2. Guidelines for Coverage-Based Comparisons of Non-Adequate Test Suites

    Milos Gligoric, Alex Groce, Chaoqiang Zhang, Rohan Sharma, Mohammad Amin Alipour, and Darko Marinov. "Guidelines for Coverage-Based Comparisons of Non-Adequate Test Suites." ACM Transactions on Software Engineering and Methodology, accepted for publication.

  3. Establishing Flight Software Reliability: Testing, Model Checking, Constraint-Solving, Monitoring and Learning

    Alex Groce, Klaus Havelund, Gerard Holzmann, Rajeev Joshi, and Ru-Gang Xu. "Establishing Flight Software Reliability: Testing, Model Checking, Constraint-Solving, Monitoring and Learning." Annals of Mathematics and Artificial Intelligence, Volume 70, Number 4, pages 315-348, April 2014.


  4. You Are the Only Possible Oracle: Effective Test Selection for End Users of Interactive Machine Learning Systems

    Alex Groce, Todd Kulesza, Chaoqiang Zhang, Shalini Shamasunder, Margaret Burnett, Weng-Keen Wong, Simone Stumpf, Shubhomoy Das, Amber Shinsel, Forrest Bice, and Kevin McIntosh. "You Are the Only Possible Oracle: Effective Test Selection for End Users of Interactive Machine Learning Systems." IEEE Transactions on Software Engineering, Volume 40, Number 3, pages 307-323, March 2014.


  5. Swarm Verification Techniques

    Gerard Holzmann, Rajeev Joshi, and Alex Groce. "Swarm Verification Techniques." IEEE Transactions on Software Engineering, Volume 37, Number 6, pages 845-857, November 2011.


Conference and Workshop Papers

  1. Can Testedness be Effectively Measured?

    Iftekhar Ahmed, Rahul Gopinath, Caius Brindescu, Alex Groce, and Carlos Jensen. "Can Testedness be Effectively Measured?" In ACM SIGSOFT Symposium on the Foundations of Software Engineering, accepted for publication, Seattle, Washington, November 2016.


  2. Mitigating (and Exploiting) Test Reduction Slippage

    Josie Holmes, Alex Groce, and Mohammad Amin Alipour. "Mitigating (and Exploiting) Test Reduction Slippage." In 7th Workshop on Automated Software Testing (A-TEST), accepted for publication, Seattle, Washington, November 2016.


  3. A Method Dependence Relations Guided Genetic Algorithm

    Ali Aburas and Alex Groce. "A Method Dependence Relations Guided Genetic Algorithm." In International Symposium on Search-Based Software Engineering, accepted for publication, Raleigh, North Carolina, October 2016 ().


  4. Evaluating Non-Adequate Test-Case Reduction

    Mohammad Amin Alipour, August Shi, Rahul Gopinath, Darko Marinov, and Alex Groce. "Evaluating Non-Adequate Test-Case Reduction." In IEEE/ACM International Conference on Automated Software Engineering, accepted for publication, Singapore, Singapore, September 2016.


  5. Generating Focused Random Tests Using Directed Swarm Testing

    Mohammad Amin Alipour, Alex Groce, Rahul Gopinath, and Arpit Christi. "Generating Focused Random Tests Using Directed Swarm Testing." In ACM International Symposium on Software Testing and Analysis, pages 70-81, Saarbrucken, Germany, July 2016.


  6. On the Limits of Mutation Reduction Strategies

    Rahul Gopinath, Mohammad Amin Alipour, Iftekhar Ahmed, Carlos Jensen, and Alex Groce. "On the Limits of Mutation Reduction Strategies." In ACM/IEEE International Conference on Software Engineering (ICSE), pages 511-522, Austin, Texas, May 2016.


  7. Topsy-Turvy: a Smarter and Faster Parallelization of Mutation Analysis

    Rahul Gopinath, Carlos Jensen, and Alex Groce. "Topsy-Turvy: a Smarter and Faster Parallelization of Mutation Analysis." In ACM/IEEE International Conference on Software Engineering (ICSE), pages 740-743, Austin, Texas, May 2016.
    (ICSE 2016 Distinguished Poster Award)


  8. How Verified is My Code? Falsification-Driven Verification

    Alex Groce, Iftekhar Ahmed, Carlos Jensen, and Paul E. McKenney. "How Verified is My Code? Falsification-Driven Verification." In IEEE/ACM International Conference on Automated Software Engineering, accepted for publication, Lincoln, Nebraska, November 2015.


  9. How Hard Does Mutation Analysis Have to Be, Anyway?

    Rahul Gopinath,Mohammad Amin Alipour, Iftekhar Ahmed, Carlos Jensen, and Alex Groce. "How Hard Does Mutation Analysis Have to Be, Anyway?" In IEEE International Symposium on Software Reliability Engineering, accepted for publication, Gaithersburg, Maryland, November 2015.


  10. TSTL: A Language and Tool for Testing (Demo)

    Alex Groce, Jervis Pinto, Pooria Azimi, and Pranjal Mittal. "TSTL: A Language and Tool for Testing (Demo)." In ACM International Symposium on Software Testing and Analysis, pages 414--417, Baltimore, Maryland, July 2015.


  11. A Little Language for Testing

    Alex Groce and Jervis Pinto. "A Little Language for Testing." In NASA Formal Methods Symposium, pages 204-218, Pasadena, California, April 2015.


  12. Taming a Fuzzer Using Delta Debugging Trails

    Yuanli Pei, Arpit Christi, Xiaoli Fern, Alex Groce, and Weng-Keen Wong. "Taming a Fuzzer Using Delta Debugging Trails." In 3rd International Workshop on Software Mining, Shenzen, China, December 2014.


  13. Mutations: How Close Are They to Real Faults?

    Rahul Gopinath, Carlos Jensen, and Alex Groce. "Mutations: How Close Are They to Real Faults?" In IEEE International Symposium on Software Reliability Engineering, accepted for publication, Naples, Italy, November 2014.

  14. Coverage and Its Discontents

    Alex Groce, Mohammad Amin Alipour, and Rahul Gopinath. "Coverage and Its Discontents." In ACM Symposium on New Ideas in Programming and Reflections on Software, Onward! Essays, part of SPLASH (ACM SIGPLAN Conference on Systems, Programming, Languages and Applications: Software for Humanity) , accepted for publication, Portland, Oregon, October 2014.


  15. An Improved Memetic Algorithm with Method Dependence Relations (MAMDR)

    Ali Aburas and Alex Groce. "An Improved Memetic Algorithm with Method Dependence Relations (MAMDR)." In International Conference on Quality Software , accepted for publication, Dallas, Texas , October 2014.


  16. Using Test Case Reduction and Prioritization to Improve Symbolic Execution

    Chaoqiang Zhang, Alex Groce, and Amin Alipour. "Using Test Case Reduction and Prioritization to Improve Symbolic Execution." In ACM International Symposium on Software Testing and Analysis (ISSTA), pages 60-70, San Jose, California, July 2014.


  17. MuCheck: an Extensible Tool for Mutation Testing of Haskell Programs

    Duc Le, Mohammad Amin Alipour, Rahul Gopinath, and Alex Groce. "MuCheck: an Extensible Tool for Mutation Testing of Haskell Programs." In ACM International Symposium on Software Testing and Analysis (ISSTA), pages 429-432, Tools and Demonstration Track, San Jose, California, July 2014.


  18. Code Coverage for Suite Evaluation by Developers

    Rahul Gopinath, Carlos Jensen, and Alex Groce. "Code Coverage for Suite Evaluation by Developers." In ACM/IEEE International Conference on Software Engineering (ICSE), pages 72-82, Hyderabad, India, May-June 2014.


  19. Cause Reduction for Quick Testing

    Alex Groce, Amin Alipour, Chaoqiang Zhang, Yang Chen, and John Regehr. "Cause Reduction for Quick Testing." In IEEE International Conference on Software Testing, Verification and Validation (ICST), pages 243-252, Cleveland, Ohio, March-April 2014.
    (ICST 2014 Best Paper Award)


  20. Finding Model-Checkable Needles in Large Source Code Haystacks: Modular Bug-Finding via Static Analysis and Dynamic Invariant Discovery

    Amin Alipour, Alex Groce, Chaoqiang Zhang, Anahita Sanadaji, and Gokul Caushik. "Finding Model-Checkable Needles in Large Source Code Haystacks: Modular Bug-Finding via Static Analysis and Dynamic Invariant Discovery." In International Workshop on Constraints in Formal Verification (CFV), San Jose, California, November 2013.


  21. Help, Help, I'm Being Suppressed! The Significance of Suppressors in Software Testing

    Alex Groce, Chaoqiang Zhang, Amin Alipour, Eric Eide, Yang Chen, and John Regehr. "Help, Help, I'm Being Suppressed! The Significance of Suppressors in Software Testing." In IEEE International Symposium on Software Reliability Engineering (ISSRE), pages 390-399, Pasadena, California, November 2013.


  22. Comparing Non-adequate Test Suites using Coverage Criteria

    Milos Gligoric, Alex Groce, Chaoqiang Zhang, Rohan Sharma, Amin Alipour, and Darko Marinov. "Comparing Non-Adequate Test Suites using Coverage Criteria." In ACM International Symposium on Software Testing and Analysis (ISSTA), pages 302--313, Lugano, Switzerland, July 2013.


  23. Taming Compiler Fuzzers

    Yang Chen, Alex Groce, Chaoqiang Zhang, Weng-Keen Wong, Xiaoli Fern, Eric Eide, and John Regehr. "Taming Compiler Fuzzers." In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), pages 197--208, Seattle, Washington, June 2013.


  24. Lightweight Automated Testing with Adaptation-Based Programming

    Alex Groce, Alan Fern, Jervis Pinto, Tim Bauer, Amin Alipour, Martin Erwig, and Camden Lopez. "Lightweight Automated Testing with Adaptation-Based Programming." In IEEE International Symposium on Software Reliability Engineering (ISSRE), pages 161--170, Dallas, Texas, November 2012.


  25. Finding Common Ground: Choose, Assert, and Assume

    Alex Groce and Martin Erwig. "Finding Common Ground: Choose, Assert, and Assume." In International Workshop on Dynamic Analysis (WODA), pages 12--17 Minneapolis, Minnesota, July 2012.


  26. Extended Program Invariants: Applications in Testing and Fault Localization

    Amin Alipour and Alex Groce. "Extended Program Invariants: Applications in Testing and Fault Localization." In International Workshop on Dynamic Analysis (WODA), pages 7--11, Minneapolis, Minnesota, July 2012.


  27. Swarm Testing

    Alex Groce, Chaoqiang Zhang, Eric Eide, Yang Chen, and John Regehr. "Swarm Testing." In ACM International Symposium on Software Testing and Analysis (ISSTA), pages 78--88, Minneapolis, Minnesota, July 2012.


  28. Bounded Model Checking and Feature Omission Diversity

    Amin Alipour and Alex Groce. "Bounded Model Checking and Feature Omission Diversity." In International Workshop on Constraints in Formal Verification (CFV), San Jose, California, November 2011.


  29. Coverage Rewarded: Test Input Generation via Adaptation-Based Programming

    Alex Groce. "Coverage Rewarded: Test Input Generation via Adaptation-Based Programming." In ACM/IEEE International Conference on Automated Software Engineering (ASE), pages 380--383, Lawrence, Kansas, November 2011.


  30. Mini-crowdsourcing End-user Assessment of Intelligent Assistants: A Cost-benefit Study

    Amber Shinsel, Todd Kulesza, Margaret M. Burnett, William Curran, Alex Groce, Simone Stumpf, and Weng-Keen Wong. "Mini-crowdsourcing End-user Assessment of Intelligent Assistants: A Cost-benefit Study." In IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), pages 47--54, Pittsburgh, Pennsylvania, September 2011.


  31. Where Are My Intelligent Assistant's Mistakes? A Systematic Testing Approach

    Todd Kulesza, Margaret Burnett, Simone Stumpf, Weng-Keen Wong, Shubhomoy Das, Alex Groce, Amber Shinsel, Forrest Bice, and Kevin McIntosh. "Where Are My Intelligent Assistant's Mistakes? A Systematic Testing Approach." In International Symposium on End-User Development (EUD), pages 171--186, Brindisi, Italy, June 2011.


  32. From Scripts to Specifications: the Evolution of a Flight Software Testing Effort

    Alex Groce, Klaus Havelund, and Margaret Smith. "From Scripts to Specifications: the Evolution of a Flight Software Testing Effort." In ACM/IEEE International Conference on Software Engineering (ICSE), pages 129--138, Cape Town, South Africa, May 2010.


Funding:


Group members:


Other faculty and groups we work with: