Constructing automated test oracle for low observable software

Document Type : Article

Authors

Faculty of Computer Science and Engineering, Shahid Beheshti University, G.C, Tehran, P.O. Box 1983963113, Iran

Abstract

Using machine learning techniques for constructing automated test oracles have been successful in recent years. However, existing machine learning based oracles have deficiencies when applied to software systems with low observability, such as embedded software, cyber-physical systems, multimedia software programs, and computer games. This paper proposes a new black box approach to construct automated oracles which can be applied to software systems with low observability. The proposed approach employs an Artificial Neural Network (ANN) algorithm which uses input values as well as corresponding pass/fail outcomes of the program under test, as the training set. To evaluate the performance of the proposed approach, we have conducted extensive experiments on several benchmarks. The results manifest the applicability of the proposed approach to software systems with low observability as well as its higher accuracy in comparison to a well-known machine learning based method.
We have also assessed the effect of different parameters on the accuracy of the proposed approach.

Keywords

References

References:
1. Shahamiri, S.R., Wan-Kadir, W.M., Ibrahim, S., and Hashim, S.Z.M. "Artificial neural networks as multinetworks automated test oracle", Automated Software Engineering, 19(3), pp. 303-334 (2012).
2. Valizadeh, M., Tadayon, M., and Bagheri, A. "Making problem: A new approach to reachability assurance in digraphs", Scientia Iranica, 25(3), pp. 1441-1455 (2018).
3. Rezaee, A. and Zamani, B. "A novel approach to automatic model-based test case generation", Scientia Iranica, 24(6), pp. 3132-3147 (2017).
4. Barr, E.T., Harman, M., McMinn, P., Shahbaz, M., and Yoo, S. "The oracle problem in software testing: A survey", IEEE Transactions on Software Engineering, 41(5), pp. 507-525 (2015).
5. Ammann, P. and Offutt, J., Introduction to Software Testing, Cambridge University Press (2016).
6. Almaghairbe, R. and Roper, M. "Automatically classifying test results by semi-supervised learning", In 27th IEEE Int. Symp. on Software Reliability Engineering, pp. 116-126 (2016).
7. Lo, D., Cheng, H., Han, J., Khoo, S.-C., and Sun, C. "Classification of software behaviors for failure detection: a discriminative pattern mining approach", In 15th ACM SIGKDD int. Conf. on Knowledge Discovery and Data Mining, pp. 557-566 (2009).
8. Yilmaz, C. and Porter, A. "Combining hardware and software instrumentation to classify program executions", In 18th ACM SIGSOFT Int. Symp. on Foundations of Software Engineering, pp. 67-76 (2010).
9. Freedman, R.S. "Testability of software components", IEEE Transactions on Software Engineering, 17(6), pp. 553-564 (1991).
10. Vliegendhart, R., Dolstra, E., and Pouwelse, J. "Crowdsourced user interface testing for multimedia applications", In ACM Multimedia 2012 Workshop on Crowdsourcing for Multimedia, pp. 21-22 (2012).
11. Jan, S.R., Shah, S.T.U., Johar, Z.U., Shah, Y., and Khan, F. "An innovative approach to investigate various software testing techniques and strategies", International Journal of Scientific Research in Science,Engineering and Technology, 2(2), pp. 2395-1990 (2016).
12. Gholami, F., Attar, N., Haghighi, H., Vahidi-Asl, M., Valueian, M., and Mohamadyari, S. "A classifier-based test oracle for embedded software", In 2018 IEEE Real- Time and Embedded Systems and Technologies, pp. 104-111 (2018).
13. Shahamiri, S.R., Kadir, W.M.N.W., Ibrahim, S., and Hashim, S.Z.M. "An automated framework for software test oracle", Information and Software Technology, 53(7), pp. 774-788 (2011).
14. Peters, D.K. and Parnas, D.L. "Using test oracles generated from program documentation", IEEE Transactions on Software Engineering, 24(3), pp. 161-173 (1998).
15. Gargantini, A. and Riccobene, E. "Asm-based testing: Coverage criteria and automatic test sequence generation", Journal of Universal Computer Science, 7(11), pp. 1050-1067 (2001).
16. Cadar, C., Ganesh, V., Pawlowski, P.M., Dill, D.L., and Engler, D.R. "Exe: automatically generating inputs of death", ACM Transactions on Information and System Security, 12(2), p. 10 (2008).
17. Shrestha, K. and Rutherford, M.J. "An empirical evaluation of assertions as oracles", In 4th IEEE Fourth Int. Conf. on Software Testing, Verification and Validation, pp. 110-119 (2011).
18. Ricca, F. and Tonella, P. "Detecting anomaly and failure in web applications", IEEE MultiMedia, 13(2), pp. 44-51 (2006).
19. Walsh, T.A., Kapfhammer, G.M., and McMinn, P. "Automated layout failure detection for responsive web pages without an explicit oracle", In 26th ACM SIGSOFT Int. Symp. on Software Testing and Analysis, pp. 192-202 (2017).
20. Carver, R. and Lei, Y. "Stateless techniques for generating global and local test oracles for messagepassing concurrent programs", Journal of Systems and Software, 136, pp. 237-265 (2018).
21. Zhou, Z.Q., Zhang, S., Hagenbuchner, M., Tse, T., Kuo, F.C., and Chen, T.Y. "Automated functional testing of online search services", Software Testing, Verification and Reliability, 22(4), pp. 221-243 (2012).
22. Simons, A.J. "Jwalk: a tool for lazy, systematic testing of java classes by design introspection and user interaction", Automated Software Engineering, 14(4), pp. 369-418 (2007).
23. Goffi, A. "Automating test oracles generation", Ph.D. Thesis, Universit'a della Svizzera italiana (2018).
24. Carzaniga, A., Goffi, A., Gorla, A., Mattavelli, A., and Pezze, M. "Crosschecking oracles from intrinsic software redundancy", In 36th Int. Conf. on Software Engineering, pp. 931-942 (2014).
25. Goffi, A., Gorla, A., Ernst, M.D., and Pezze, M. "Automatic generation of oracles for exceptional behaviors", In 25th Int. Symp. on Software Testing and Analysis, pp. 213-224 (2016).
26. Pezze, M. "Towards cost-effective oracles", In 10th IEEE/ACM Int. Workshop on Automation of Software Test, pp. 1-2 (2015).
27. Goffi, A. "Automatic generation of cost-effective test oracles", In 36th ACM Int. Conf. on Software Engineering, pp. 678-681 (2014).
28. Segura, S., Fraser, G., Sanchez, A.B., and Ruiz- Cortes, A. "A survey on metamorphic testing", IEEE Transactions on Software Engineering, 42(9), pp. 805- 824 (2016).
29. Goffi, A., Gorla, A., Mattavelli, A., Pezze, M., and Tonella, P. "Searchbased synthesis of equivalent method sequences", In 22nd ACM SIGSOFT Int. Symp. on Foundations of Software Engineering, pp. 366-376 (2014).
30. Mattavelli, A., Goffi, A., and Gorla, A. "Synthesis of equivalent method calls in guava", In Int. Symp. on Search Based Software Engineering, pp. 248-254 (2015).
31. Ernst, M.D., Cockrell, J., Griswold, W.G., and Notkin, D. "Dynamically discovering likely program invariants to support program evolution", IEEE Transactions on Software Engineering, 27(2), pp. 99-123 (2001).
32. Ernst, M.D., Perkins, J.H., Guo, P.J., McCamant, S., Pacheco, C., Tschantz, M.S., and Xiao, C. "The daikon system for dynamic detection of likely invariants", Science of Computer Programming, 69(1-3), pp. 35-45 (2007).
33. Elyasov, A., Prasetya, W., Hage, J., Rueda, U., Vos, T., and Condori-Fernandez, N. "Ab = ba: execution equivalence as a new type of testing oracle", In 30th Annual ACM Symp. on Applied Computing, pp. 1559- 1566 (2015).
34. Feldt, R. "Generating diverse software versions with genetic programming: an experimental study", IEE Proceedings-Software, 145(6), pp. 228-236 (1998).
35. Di Lucca, G.A., Fasolino, A.R., Faralli, F., and De Carlini, U. "Testing web applications", In Int. IEEE Conf. on Software Maintenance, pp. 310-319 (2002).
36. Memon, A.M., Pollack, M.E., and Soffa, M.L. "Automated test oracles for GUIs", In 8th ACM SIGSOFT Int. Symp. on Foundations of Software Engineering: Twenty-First Century Applications, pp. 30-39 (2000).
37. Last, M., Friedman, M., and Kandel, A. "Using data mining for automated software testing", International Journal of Software Engineering and Knowledge Engineering, 14(4), pp. 369-393 (2004).
38. Zheng, W., Ma, H., Lyu, M.R., Xie, T., and King, I. "Mining test oracles of web search engines", In 26th IEEE/ACM Int. Conf. on Automated Software Engineering, pp. 408-411 (2011).
39. Vineeta, Abhishek Singhal, Abhay Bansal "Generation of test oracles using neural network and decision tree model", In 5th Int. Conf.- Con fluence The Next Generation Information Technology Summit, pp. 313- 318 (2014).
40. Wang, F., Yao, L.-W., and Wu, J.-H. "Intelligent test oracle construction for reactive systems without explicit specifications", In 9th IEEE Int. Conf. on Dependable, Autonomic and Secure Computing, pp. 89-96 (2011).
41. Vanmali, M., Last, M., and Kandel, A. "Using a neural network in the software testing process", International Journal of Intelligent Systems, 17(1), pp. 45-62 (2002).
42. Zhang, R., Wang, Y.-W., and Zhang, M.-Z. "Automatic test oracle based on probabilistic neural networks", In Recent Developments in Intelligent Computing, Communication and Devices, pp. 437-445 (2019).
43. Almaghairbe, R. and Roper, M. "Building test oracles by clustering failures", In 10th IEEE Int. Workshop on Automation of Software Test, pp. 3-7 (2015).
44. Almaghairbe, R. and Roper, M. "Separating passing and failing test executions by clustering anomalies", Software Quality Journal, 25(3), pp. 803-840 (2017).
45. Shahamiri, S.R., Kadir, W.M.N.W., and bin Ibrahim, S. "An automated oracle approach to test decisionmaking structures", In 3rd IEEE Int. Conf. on Computer Science and Information Technology, pp. 30-34 (2010).
46. Shahamiri, S.R., Kadir, W.M.W., and Ibrahim, S. "A single-network annbased oracle to verify logical software modules", In 2nd IEEE Int. Conf. on Software Technology and Engineering, pp. 272-276 (2010).
47. Gurney, K., An Introduction to Neural Networks, CRC press (2014).
48. Graupe, D., Principles of Artificial Neural Networks, World Scientific (2013).
49. "Mathworks", https://www.mathworks.com/ (2018).
50. Standard, D.E., Federal Information Processing Standards Publication, National Bureau of Standards, US Department of Commerce, 4 (1977).
51. "Itu-t technical paper telecommunication standardization sector of ITU", Technical report, Transmission Systems and Media, Digital Systems and Networks Digital Sections and Digital Line System (2010).
52. "Generic sound activity detector (GSAD)", https:// www.itu.int/pub/T-REC-USB-2020.
53. Lee, E., Embedded Software, University of California at Berkeley, Berkeley (2001).
54. Livadas, L.J.C. and Lynch, N.A. "High-level modeling and analysis of tcas", In 20th IEEE Real-Time Systems Symp., pp. 115-125 (1999).
55. Shahamiri, S.R., Kadir, W.M.N.W., and Mohd-Hashim, S.Z. "A comparative study on automated software test oracle methods", In 4th Int. Conf. on Software Engineering Advances, pp. 140-145 (2009).
Scientia Iranica
Volume 27, Issue 3
Transactions on Computer Science & Engineering and Electrical Engineering (D)
June 2020
Pages 1333-1351

Files

Share

How to cite

Statistics