Test data generation for program units using a game with a purpose

Document Type : Article

Authors

Faculty of Computer Science and Engineering, Shahid Beheshti University, Tehran, Iran

Abstract

One of the most important, but tedious and costly tasks of the software testing process is test data generation. The challenge is finding approaches in which humans could generate test data through more attractive, faster, and cheaper ways. One approach is using Game with A Purpose in the process of test data generation. In our previous work, we introduced two games called Rings and Greenify, by which many inexpensive players, with no special technical abilities, become engaged in test data generation. Despite the promising results of Rings and Greenify, they have certain limitations and issues. In this paper, we present a new GWAP for test data generation, called QOTE, in order to improve the application of GWAP in test data generation for program units. QOTE provides a different game-play and has certain advantages compared to prior gamesc. Experimental results have shown that QOTE outperforms prior games from two aspects: game quality and capability of test data generation. We have conducted another experiment based on mutation analysis to further evaluate test data generation capabilities of QOTE compared to four automatic approaches and show that the test data generated by QOTE can reveal more failures compared to the mentioned automatic approaches.

Keywords

Main Subjects

References

References:
1. Ammann, P. and Offut, J., Introduction to Software Testing, In Cambridge University Press (2016).
2. Jain, N. and Porwal, R. "Automated test data generation applying heuristic approaches-a survey", In Software Engineering, Springer, Singapore, pp. 699- 708 (2019).
3. Valueian, M., Attar, N., Haghighi, H., et al. "Constructing automated test Oracle for low observable software", Scientia Iranica, 27(3), pp. 1333-1351 (2020).
4. Ouni, A. "Search based software engineering: challenges, opportunities and recent applications." In Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, pp. 1114-1146 (2020).
5. Kirkpatrick, G. "Welcoming All Gods and Embracing All Places": Computer Games As Constitutively Transcendent of the Local.", In Game History and the Local. Palgrave Macmillan, Cham, pp. 199-219 (2021).
6. Zachos, G., Paraskevopoulou-Kollia, E.A., and Anagnostopoulos, I. "Social media use in higher education: A review", Education Sciences, 8(4), p. 194 (2018).
7. Von Ahn, L. and Dabbish, L. "Designing games with a purpose", Communications of the ACM, 51(8), pp. 58-67 (2008).
8. Siu, K.A. "Design and Evaluation of Intelligent Reward Structures in Human Computation Games", PhD Thesis, Georgia Institute of Technology (2021).
9. Amiri-Chimeh, S., Haghighi, H., Vahidi-Asl, M., et al. "Rings: A game with a purpose for test data generation", Interacting with Computers, 30(1), pp. 1-30 (2017).
10. Moosavi, Sh., Haghighi, H., Sahabi, H., et al. "Greenify: A game with the purpose of test data generation for unit testing", In Fundamentals of Software Engineering: 8th International Conference, FSEN 2019, Tehran, Iran, Springer International Publishing, pp. 77-92 (2019).
11. Gaurav, D., Kaushik, Y., Supraja, S., et al. "Empirical study of adaptive serious games in enhancing learning outcome", International Journal of Serious Games, 9(2), pp. 27-42 (2022).
12. Tillmann, N., De Halleux, J., Xie, T., et al. "Pex4fun: Teaching and learning computer science via social gaming". In 2012 IEEE 25th Conference on Software Engineering Education and Training, pp. 90-91 (2012).
13. Tillmann, N., Bishop, J., Horspool, N., et al. "Code hunt: Searching for secret code for fun", 7th International Workshop on SearchBased Software Testing, pp. 23-26 (2014).
14. Fava, D., Shapiro, D., Osborn, J., et al. "Crowdsourcing program preconditions via a classification game", 38th International Conference on Software Engineering, pp. 1086-1096 (2016).
15. Rojas, J.M. and Fraser, G. "Code defenders: A mutation testing game", 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 162-167 (2015).
16. Chen, N. and Kim, S. "Puzzle-based automatic testing: Bringing humans into the loop by solving puzzles", 27th IEEE/ACM International Conference on Automated Software Engineering, IEEE, pp. 140-149 (2012).
17. Michael, C.C., McGraw, G., and Schatz, M.A. "Generating software test data by evolution", IEEE Transactions on Software Engineering, 27(12), pp. 1085-1110 (2001).
18. Walkinshaw, N. and Fraser, G. "Uncertainty-driven black-box test data generation", 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST), pp. 253-263 (2017).
19. Papadakis, M., Kintis, M., Zhang, J., et al. "Mutation testing advances: an analysis and survey", Advances in Computers, Elsevier, 112, pp. 275-378 (2019).
20. Coles, H., Laurent, T., Henard, C., et al. "Pit: a practical mutation testing tool for java", 25th International Symposium on Software Testing and Analysis, ACM, pp. 449-452 (2016).
21. Malte, M. and Howar, F. "An ensemble of tools for dynamic symbolic execution on the Java Virtual Machine (competition contribution)", In Tools and Algorithms for the Construction and Analysis of Systems: 28th International Conference, TACAS 2022, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2022, Munich, Germany, Cham: Springer International Publishing (2022).
Scientia Iranica
Volume 30, Issue 5
Transactions on Computer Science & Engineering and Electrical Engineering (D)
September and October 2023
Pages 1670-1686

Files

History

  • Receive Date: 06 April 2022
  • Revise Date: 13 November 2022
  • Accept Date: 11 April 2023

Share

How to cite

Statistics