Macrophage+: A game with a purpose for applying human intelligence in control mechanisms

Document Type : Article

Authors

Department of Computer Science and Engineering, Shahid Beheshti University G. C., Tehran, Iran

Abstract

Originally, control mechanisms were proposed to replace the need for human intervention in operational environments, and thus, enhance the precision and reaction time. Nowadays, new requirements in computer systems such as adaptation have made the design of control mechanisms more challenging. The Observer/Controller pattern is one of the control mechanisms proposed to control many interacting independent elements by making intelligent decisions. An important challenge in designing these mechanisms is that the knowledge needed for decision making is provided by experts; therefore, the process becomes time consuming and costly, depending on the availability of experts and their costs. In this paper, we hypothesize that employing a Game With A Purpose can help to improve the process of providing knowledge in such control mechanisms by using crowd-sourcing and involving non-expert humans in an enjoyable manner. This hypothesis has been investigated by Macrophage+, a Game With A Purpose implemented for this goal. We conducted experiments evaluating Macrophage+, focusing on both its applicability and effectiveness in the context of the observer/controller pattern as well as its enjoyability for the players. The results show that Macrophage+ is a successful Game With A Purpose that involves non-expert humans in the application of the observer/controller pattern.

Keywords


  1. References:

    1. Shaw, M., Andersson, J., Litoiu, M., Schmerl, B., Tamura, G., Villegas, N.M., Vogel, T., Weyns, D., Baresi, L., Becker, B., Bencomo, N., Brun, Y., Cukic, B., Desmarais, R., Dustdar, S., Engels, G., Geihs, K., Goschka, K.M., Gorla, A., Grassi, V., Inverardi, P., Karsai, G., Kramer, J., Lopes, A., Magee, J., Malek, S., Mankovskii, S., Mirandola, R., Mylopoulos, J., Nierstrasz, O., Pezz_e, M., Prehofer, C., Schafer, W., Schlichting, R., Smith, D.B., Sousa, J.P., Tahvildari, L., Wong, K., andWuttke, J. Software engineering for selfadaptive systems: A second research Roadmap", De Lemos, R., Giese, H., Muller, H.A., Shaw, M., Editors, In Software Engineering for Self-Adaptive Systems II. Lecture Notes in Computer Science, 7475, pp. 1{32, Springer, Berlin, Germany (2013).
    2. Nafz, F., Steghofer, J.P., Seebach, H., et al. Formal modeling and veri_cation of self-* systems based on observer/controller architectures", In Camara J., de Lemos R., Ghezzi, C., and Lopes A., Editors, Assurances for Self-Adaptive Systems, Lecture Notes in Computer Science, 7740, pp. 80{111, Springer (2013).
    3. Salehie, M. and Tahvildari, L. Self-adaptive software: Landscape and research challenges", Trans. Auton. Adapt. Syst., 4(2), pp. 1{42 (2009).
    4. Mac__as-Escriv_a, F.D., Haber, R., del Toro, R., et al. Self-adaptive systems: A survey of current approaches, research challenges and applications", Expert Systems with Applications, 40(18), pp. 7267{7279 (2013).
    5. Kephart, J.O. and Chess, D.M. The vision of autonomic computing", Computer, 36(1), pp. 41{50 (2003).
    6. Muller-Schloer, C. and Tomforde, S., Organic Computing - Technical Systems for Survival in the Real World, Springer, Berlin, Germany (2017). 7. Muller-Schloer, C., Schmeck, H., and Ungerer, T., Editors, Organic Computing - A paradigm Shift for Complex Systems, 1st Edn., Springer, Berlin, Germany (2011). 8. Seebach, H., Ortmeier, F., and Reif, W. Design and construction of organic computing systems", In Evolutionary Computation, CEC 2007, pp. 4215{4221, Singapore (2007). 9. Krupitzer, C., Roth, F.M., VanSyckel, S., et al. A survey on engineering approaches for self-adaptive systems", Pervasive and Mobile Computing, 17, pp. 184{206 (2015). 10. Momtazi, S. and Moradiannasab, O. A statistical approach to knowledge discovery: Bootstrap analysis of language models for knowledge base population from unstructured text", Scientia Iranica, 26, pp. 26{39 (2019). 11. Winter, S., Richter, K.F., Baldwin, T., et al. Location-based mobile games for spatial knowledge acquisition", In Proc. of the Workshop on Cognitive Engineering for Mobile GIS, Belfast, Ireland (2011). 12. Ponsen, M., Spronck, P., Munoz-Avila, H., et al. Knowledge acquisition for adaptive game AI", Science of Computer Programming, 67(1), pp. 59{75 (2007). A. Tarihi et al./Scientia Iranica, Transactions D: Computer Science & ... 27 (2020) 2985{3004 3001 13. Bimba, A.T., Idris, N., Al-Hunaiyyan, A., et al. Towards knowledge modeling and manipulation technologies: A survey", Intl. Journal of Information Management, 36(6), pp. 857{871 (2016). 14. Von Ahn, L. Games with a purpose", Computer, 39(6), pp. 92{94 (2006). 15. Cooper, S., Khatib, F., Treuille, A., et al. Predicting protein structures with a multiplayer online game", Nature, 466(7307), pp. 756{760 (2010). 16. Law, E.L.M., Von Ahn, L., Dannenberg, R.B., et al. Tagatune: A game for music and sound annotation", In Proc. 8th Intl. Conf. on Music Information Retrieval (ISMIR), 3, pp. 361{364, Vienna, Austria (2007). 17. Quinn, A.J. and Bederson, B.B. Human computation: A survey and taxonomy of a growing _eld", In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems (CH'11), pp. 1403{1412, New York, USA (2011). 18. Sabou, M., Bontcheva, K., Scharl, A., et al. Games with a purpose or mechanized labor?: A comparative study", In Proc. of the 13th Intl. Conf. on Knowledge Management and Knowledge Technologies, p. 19, Graz, Austria (2013). 19. Wang, A., Hoang, C.D.V., and Kan, M.K. Perspectives on crowdsourcing annotations for natural language processing", Language Resources and Evaluation, 47(1), pp. 9{31 (2013). 20. Weng, L., Schifanella, R., and Menczer, F. Design of social games for collecting reliable semantic annotations", In 16th Intl. Conf. on Computer Games (CGAMES), pp. 185{192, Louisville, KY, USA (2011). 21. Reason, J., Human Error, Cambridge University Press (1990). 22. Rasmussen, J. Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models", Systems, Man and Cybernetics, IEEE, Transactions on, 3, pp. 257{266 (1987). 23. Lalanda, P., McCann, J.A., and Diaconescu, A. Autonomic computing", In Principles, Design and Implementation, 1st Edn., Springer, Germany (2013). 24. Garlan, D., Schmerl, B., and Cheng, S.W. Software architecture-based self-adaptation", In Autonomic Computing and Networking, pp. 31{55, Springer, Germany (2009). 25. Garlan, D., Cheng, S.W., Huang, A.C., et al. Rainbow: Architecture-based self-adaptation with reusable infrastructure", Computer, 37(10), pp. 46{54 (2004). 26. Oreizy, P., Medvidovic, N., and Taylor, R.N. Architecture-based runtime software evolution", In Proc. of the 20th Intl. Conf. on Software Engineering, pp. 177{186, Kyoto, Japan (1998). 27. Canal, C., Poizat, P., and Salaun, G. Model-based adaptation of behavioral mismatching components", IEEE Trans. on Software Engineering, 34(4), pp. 546{ 563 (2008). 28. Filieri, A., Ho_mann, H., and Maggio, M. Automated design of self-adaptive software with control theoretical formal guarantees", In Proc. of the 36th Intl. Conf. on Software Engineering, pp. 299{310, Hyderabad, India (2014). 29. Nafz, F., Ortmeier, F., Seebach, H., et al. A generic software framework for role-based organic computing systems", In Software Engineering for Adaptive and Self-Managing Systems, SEAMS'09. ICSE Workshop on, pp. 96{105, Cape Town, South Africa (2009). 30. Fischer, P., Nafz, F., Seebach, H., et al. Ensuring correct self-recon_guration in safety-critical applications by veri_ed result checking", In Proc. of the 2011 Workshop on Organic Computing, OC'11, pp. 3{12, New York, USA (2011). 31. Kaiser, G., Parekh, J., Gross, P., et al. Kinesthetics extreme: an external infrastructure for monitoring distributed legacy systems", In Autonomic Computing Workshop, pp. 22{30, Seattle, Washington, USA (2003). 32. Georgas, J.C. and Taylor, R.N. Towards a knowledgebased approach to architectural adaptation management", In Proc. of the 1st ACM SIGSOFT Workshop on Self-managed Systems, pp. 59{63, Newport Beach, California, USA (2004). 33. Fox, K.L., Henning, R.R., Farrell, J.T., et al. System and method for assessing the security posture of a network using goal oriented fuzzy logic decision rules", April 19 2005. US Patent 6,883,101 (2005). 34. Berry, D.M., Cheng, B.H.C., and Zhang, J. The four levels of requirements engineering for and in dynamic adaptive systems", In 11th Intl. Workshop on Requirements Engineering Foundation for Software Quality (REFSQ'05), p. 5, Porto, Portugal (2005). 35. Cheng, B.H.C., Sawyer, P., Bencomo, N., et al. A goal-based modeling approach to develop requirements of an adaptive system with environmental uncertainty", In Intl. Conf. on Model Driven Engineering Languages and Systems, pp. 468{483, Denver, Colorado, USA (2009). 36. Van Lamsweerde, A. and Letier, E. Handling obstacles in goal-oriented requirements engineering", IEEE Transactions on Software Engineering, 26(10), pp. 978{1005 (2000). 37. Sykes, D., Heaven, W., Magee, J., et al. From goals to components: a combined approach to selfmanagement", In Proc. of the 2008 Intl. Workshop on Software Engineering for Adaptive and Selfmanaging Systems, pp. 1{8, Leipzig, Germany (2008). 38. Menasc_e, D.A., Ewing, J.M., Gomaa, H., et al. A framework for utility-based service oriented design in sassy", In Proc. of the First Joint WOSP/SIPEW Intl. Conf. on Performance Engineering, pp. 27{36, San Jose, California, USA (2010). 3002 A. Tarihi et al./Scientia Iranica, Transactions D: Computer Science & ... 27 (2020) 2985{3004 39. Shen, C.C., Srisathapornphat, C., and Jaikaeo, C. An adaptive management architecture for ad hoc networks", IEEE Communications Magazine, 41(2), pp. 108{115 (2003). 40. Esfahani, N., Elkhodary, A., and Malek, S. A learning-based framework for engineering featureoriented self-adaptive software systems", IEEE Transaction on Software Engineering, 39(11), pp. 1467{1493 (2013). 41. Brinkschulte, U., Pacher, M., and Von Renteln, A. An arti_cial hormone system for self-organizing realtime task allocation in organic middleware", In Organic Computing, pp. 261{283, Springer, Berlin, Germany (2009). 42. Lodish, H., Berk, A., Kaiser, C.A., Krieger, M., Bretscher, A., Ploegh, H., Amon, A., Martin, K.C., Molecular Cell Biology, 8th Edn, W.H. Freeman and Company, New York, USA (2008). 43. Alberts, B., Bray, D., Hopkin, K., Johnson, A.D., Lewis, J., Ra_, M., Roberts, K., Walter, P., Essential Cell Biology, 4th Edn, Garland Science, Taylor & Francis Group, New York, USA (2013). 44. Sugeno, M. and Nishida, M. Fuzzy control of model car", Fuzzy Sets Systems, 16(2), pp. 103{113 (1985). 45. Schmeck, H., Muller-Schloer, C., Cakar, E., et al. Adaptivity and self-organization in organic computing systems", ACM Trans. Auton. Adapt. Syst., 5(3), pp. 1{10 (2010). 46. Richter, U., Mnif, M., Branke, J., et al. Towards a generic observer/controller architecture for organic computing", GI Jahrestagung, 93, pp. 112{119 (2006). 47. Komann, M. and Fey, D. Marching pixels-using organic computing principles in embedded parallel hardware", In Intl. Symposium on Parallel Computing in Electrical Engineering (PARELEC'06), pp. 369{ 373, Bialystok, Poland (2006). 48. Hartmann, J., Stechele, W., and Maehle, E. Selfadaptation for mobile robot algorithms using organic computing principles", In Kub_atov_a, H., Hochberger, C., Dan_ek M., Sick B., Editors, In Intl. Conf. on Architecture of Computing Systems (ARCS'13), pp. 232{243, Lecture Notes in Computer Science, 7767, Springer, Berlin, Germany (2013). 49. Branke, J., Mnif, M., Muller-Schloer, C., et al. Organic computing-addressing complexity by controlled self-organization", In Second Intl. Symposium on Leveraging Applications of Formal Methods, Veri _cation and Validation (Isola 2006), pp. 185{191, Paphos, Cyprus (2006). 50. Roth, M., Schmitt, J., Kiefhaber, R., et al. Organic computing middleware for ubiquitous environments", In Organic Computing A Paradigm Shift for Complex Systems, pp. 339{351, Springer, Berlin, Germany (2011). 51. Murch, R., Autonomic Computing. Information Management, IBM Press, California, USA (2004). 52. Gudemann, M., Nafz, F., Ortmeier, F., et al. A speci_cation and construction paradigm for organic computing systems", In Second IEEE Intl. Conf. on Self-Adaptive and Self Organizing Systems, pp. 233{ 242, Venezia, Italy (2008). 53. Kasinger, H. and Bauer, B. Combining multi-agentsystem methodologies for organic computing systems", In 16th Intl. Workshop on Database and Expert Systems Applications (DEXA'05), pp. 160{164, Copenhagen, Denmark (2005). 54. Tomforde, S., Rudolph, S., Bellman, K., et al. An organic computing perspective on self-improving system interweaving at runtime", In IEEE Intl. Conf. on Autonomic Computing (ICAC), pp. 276{284, Wurzburg, Germany (2016). 55. Herzig, P., Ameling, M., and Schill, A. A generic platform for enterprise gami_cation", In Joint Working IEEE/IFIP Conf. on Software Architecture and European Conf. on Software Architecture, pp. 219{223, Helsinki, Finland (2012). 56. Khanzadia, M., Shahbazia, M.M., Arashpourb, M., et al. Lean design management using a gami_ed system", Scientia Iranica, 26(1), pp. 15{25 (2019). 57. Ido, G., Hashavit, A. and Corem, Y. Games for crowds: A crowd sourcing game platform for the enterprise", In Proc. of the 18th ACM Conf. on Computer Supported Cooperative Work and Social Computing, CSCW'15, pp. 1860{1871, Vancouver, British Columbia, Canada (2015). 58. Rani, P., Sarkar, N., and Liu, C. Maintaining optimal challenge in computer games through real-time physiological feedback", In Proc. of the 11th Intl. Conf. on Human Computer Interaction, 58, pp. 22{27, Bonn, Germany (2005). 59. Walsh, G. and Golbeck, J. Curator: A game with a purpose for collection recommendation", In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems, pp. 2079{2082, Atlanta, Gorgia, USA (2010). 60. Chen, L.J., Wang, B.C., and Zhu, W.Y. The design of puzzle selection strategies for esp-like gwap systems", IEEE Trans. on Computational Intelligence and AI in Games, 2(2), pp. 120{130 (2010). 61. Kerler, S., Vilsmeier, J., Edenhofer, S., et al. Pheromander: Real-time strategy with digital pheromones", In 8th Intl. Conf. on Games and Virtual Worlds for Serious Applications (VS-GAMES), pp. 1{4, Wurzburg, Germany (2016). 62. Ho, C.J., Chang, T.H., Lee, J.C., et al. Kisskissban: A competitive human computation game for image annotation", ACM SIGKDD Explorations Newsletter, 12(1), pp. 21{24 (2010). 63. Syu, Y.S., Yu, H.H., and Chen, L.J. Exploiting puzzle diversity in puzzle selection for esp-like gwap systems", In IEEE/WIC/ACM Intl. Conf. on Web Intelligence and Intelligent Agent Technology, 1, pp. 468{475, Washington, District of Columbia, USA (2010). A. Tarihi et al./Scientia Iranica, Transactions D: Computer Science & ... 27 (2020) 2985{3004 3003 64. Otterbacher, J. Crowdsourcing stereotypes: Linguistic bias in metadata generated via gwap", In Proc. of the 33rd Annual ACM Conf. on Human Factors in Computing Systems, pp. 1955{1964, Seoul, Republic of Korea (2015). 65. Harris, C. Cluemein: Enhancing the esp game to obtain more speci_c image labels", In Proc. of the 2018 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts, CHI PLAY '18 Extended Abstracts, pp. 447{452, Melbourne, Victoria, Australia (2018). 66. Ashraf, G., Why, Y.P., and Islam, T. Mining human shape perception with role playing games", GSTF Journal on Computing (JoC), 1(1), pp. 58{64 (2018). 67. Amiri-Chimeh, S., Haghighi, H., Vahidi-Asl, M., et al. Rings: A game with a purpose for test data generation", Interacting with Computers, 30(1), pp. 1{30 (2017). 68. Kondreddi, S.K., Trianta_llou, P., and Weikum, G. Combining information extraction and human computing for crowdsourced knowledge acquisition", In IEEE 30th Intl. Conf. on Data Engineering, pp. 988{ 999, Chicago, Illinois, USA (2014). 69. Von Ahn, L., Kedia, M., and Blum, M. Verbosity: A game for collecting common-sense facts", In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems (CHI '06), pp. 75{78, Montreal, Quebec, Canada (2006). 70. Bellotti, F., Berta, R., De Gloria, A., et al. Exploring gaming mechanisms to enhance knowledge acquisition in virtual worlds", In Proc. of the 3rd Intl. Conf. on Digital Interactive Media in Entertainment and Arts, pp. 77{84, Athens, Greece (2008). 71. Fisch, D., Janicke, M., Muller-Schloer, C., et al. Divergence measures as a generalized approach to quantitative emergence", In Organic Computing-A Paradigm Shift for Complex Systems, pp. 53{66, Springer Berlin, Germany (2011). 72. Seebach H., Nafz F., Steghofer J.P., et al. How to design and implement selforganising resource-ow systems", In Organic Computing-A Paradigm Shift for Complex Systems, pp. 145{161, Springer Berlin, Germany (2011). 73. Quinlan, J.R. Induction of decision trees", Machine Learning, 1(1), pp. 81{106 (1986). 74. Kohavi, R. and Quinlan, J.R. Data mining tasks and methods: Classi_cation: decision-tree discovery", In Handbook of Data Mining and Knowledge Discovery, pp. 267{276. Oxford University Press, Inc. Oxford, United Kingdom (2002). 75. Nakamura, S., Nakagawa, H., Tahara, Y., et al. Towards solving an obstacle problem by the cooperation of uavs and ugvs", In Proc. of the 28th Annual ACM Symposium on Applied Computing, pp. 77{82, Coimbra, Portugal (2013). 76. Venkatesh, S., Lobo, R., and Sugunan, N. Controlling unmanned ground vehicle using stationary airborne system", Intl. Journal of Innovation and Applied Studies, 8(3), p. 958 (2014). 77. Grocholsky, B., Keller, J., Kumar, V., et al. Cooperative air and ground surveillance", IEEE Robotics & Automation Magazine, 13(3), pp. 16{25 (2006). 78. MacArthur, E.Z., MacArthur, D., et al. Use of cooperative unmanned air and ground vehicles for detection and disposal of mines", In Proc. of Intl. Society for Optics and Photonics, 5999, pp. 94{101 (2005). 79. Liu, Y. and Nejat, G. Robotic urban search and rescue: A survey from the control perspective", Journal of Intelligent & Robotic Systems, 72(2), pp. 147{165 (2013). 80. Anderson, B. and Crowell, J. Workhorse auv-a costsensible new autonomous underwater vehicle for surveys/ soundings, search & rescue and research", In Proc. of OCEANS 2005 MTS/IEEE, pp. 1{6, Washington, District of Columbia, USA, IEEE (2005). 81. Mayer, I., Bekebrede, G., Harteveld, C., et al. The research and evaluation of serious games: Toward a comprehensive methodology", British Journal of Educational Technology, 45(3), pp. 502{527 (2014). 82. Brockmyer, J.H., Fox, C.M., Curtiss, K.A., et al. The development of the game engagement questionnaire: A measure of engagement in video game-playing", Journal of Experimental Social Psychology, 45(4), pp. 624{634 (2009). 83. Norman, K.L. Geq (game engagement/experience questionnaire): a review of two papers", Interacting with Computers, 25(4), pp. 278{283 (2013). 84. Von Ahn, L. and Dabbish, L. Designing games with a purpose", Commun. ACM, 51(8), pp. 58{67 (2008). Biographies1. Shaw, M., Andersson, J., Litoiu, M., Schmerl, B., Tamura, G., Villegas, N.M., Vogel, T., Weyns, D., Baresi, L., Becker, B., Bencomo, N., Brun, Y., Cukic, B., Desmarais, R., Dustdar, S., Engels, G., Geihs, K., Goschka, K.M., Gorla, A., Grassi, V., Inverardi, P., Karsai, G., Kramer, J., Lopes, A., Magee, J., Malek, S., Mankovskii, S., Mirandola, R., Mylopoulos, J., Nierstrasz, O., Pezz_e, M., Prehofer, C., Schafer, W., Schlichting, R., Smith, D.B., Sousa, J.P., Tahvildari, L., Wong, K., andWuttke, J. Software engineering for selfadaptive systems: A second research Roadmap", De Lemos, R., Giese, H., Muller, H.A., Shaw, M., Editors, In Software Engineering for Self-Adaptive Systems II. Lecture Notes in Computer Science, 7475, pp. 1{32, Springer, Berlin, Germany (2013). 2. Nafz, F., Steghofer, J.P., Seebach, H., et al. Formal modeling and veri_cation of self-* systems based on observer/controller architectures", In Camara J., de Lemos R., Ghezzi, C., and Lopes A., Editors, Assurances for Self-Adaptive Systems, Lecture Notes in Computer Science, 7740, pp. 80{111, Springer (2013). 3. Salehie, M. and Tahvildari, L. Self-adaptive software: Landscape and research challenges", Trans. Auton. Adapt. Syst., 4(2), pp. 1{42 (2009). 4. Mac__as-Escriv_a, F.D., Haber, R., del Toro, R., et al. Self-adaptive systems: A survey of current approaches, research challenges and applications", Expert Systems with Applications, 40(18), pp. 7267{7279 (2013). 5. Kephart, J.O. and Chess, D.M. The vision of autonomic computing", Computer, 36(1), pp. 41{50 (2003). 6. Muller-Schloer, C. and Tomforde, S., Organic Computing - Technical Systems for Survival in the Real World, Springer, Berlin, Germany (2017). 7. Muller-Schloer, C., Schmeck, H., and Ungerer, T., Editors, Organic Computing - A paradigm Shift for Complex Systems, 1st Edn., Springer, Berlin, Germany (2011). 8. Seebach, H., Ortmeier, F., and Reif, W. Design and construction of organic computing systems", In Evolutionary Computation, CEC 2007, pp. 4215{4221, Singapore (2007). 9. Krupitzer, C., Roth, F.M., VanSyckel, S., et al. A survey on engineering approaches for self-adaptive systems", Pervasive and Mobile Computing, 17, pp. 184{206 (2015). 10. Momtazi, S. and Moradiannasab, O. A statistical approach to knowledge discovery: Bootstrap analysis of language models for knowledge base population from unstructured text", Scientia Iranica, 26, pp. 26{39 (2019). 11. Winter, S., Richter, K.F., Baldwin, T., et al. Location-based mobile games for spatial knowledge acquisition", In Proc. of the Workshop on Cognitive Engineering for Mobile GIS, Belfast, Ireland (2011). 12. Ponsen, M., Spronck, P., Munoz-Avila, H., et al. Knowledge acquisition for adaptive game AI", Science of Computer Programming, 67(1), pp. 59{75 (2007). A. Tarihi et al./Scientia Iranica, Transactions D: Computer Science & ... 27 (2020) 2985{3004 3001 13. Bimba, A.T., Idris, N., Al-Hunaiyyan, A., et al. Towards knowledge modeling and manipulation technologies: A survey", Intl. Journal of Information Management, 36(6), pp. 857{871 (2016). 14. Von Ahn, L. Games with a purpose", Computer, 39(6), pp. 92{94 (2006). 15. Cooper, S., Khatib, F., Treuille, A., et al. Predicting protein structures with a multiplayer online game", Nature, 466(7307), pp. 756{760 (2010). 16. Law, E.L.M., Von Ahn, L., Dannenberg, R.B., et al. Tagatune: A game for music and sound annotation", In Proc. 8th Intl. Conf. on Music Information Retrieval (ISMIR), 3, pp. 361{364, Vienna, Austria (2007). 17. Quinn, A.J. and Bederson, B.B. Human computation: A survey and taxonomy of a growing _eld", In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems (CH'11), pp. 1403{1412, New York, USA (2011). 18. Sabou, M., Bontcheva, K., Scharl, A., et al. Games with a purpose or mechanized labor?: A comparative study", In Proc. of the 13th Intl. Conf. on Knowledge Management and Knowledge Technologies, p. 19, Graz, Austria (2013). 19. Wang, A., Hoang, C.D.V., and Kan, M.K. Perspectives on crowdsourcing annotations for natural language processing", Language Resources and Evaluation, 47(1), pp. 9{31 (2013). 20. Weng, L., Schifanella, R., and Menczer, F. Design of social games for collecting reliable semantic annotations", In 16th Intl. Conf. on Computer Games (CGAMES), pp. 185{192, Louisville, KY, USA (2011). 21. Reason, J., Human Error, Cambridge University Press (1990). 22. Rasmussen, J. Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models", Systems, Man and Cybernetics, IEEE, Transactions on, 3, pp. 257{266 (1987). 23. Lalanda, P., McCann, J.A., and Diaconescu, A. Autonomic computing", In Principles, Design and Implementation, 1st Edn., Springer, Germany (2013). 24. Garlan, D., Schmerl, B., and Cheng, S.W. Software architecture-based self-adaptation", In Autonomic Computing and Networking, pp. 31{55, Springer, Germany (2009). 25. Garlan, D., Cheng, S.W., Huang, A.C., et al. Rainbow: Architecture-based self-adaptation with reusable infrastructure", Computer, 37(10), pp. 46{54 (2004). 26. Oreizy, P., Medvidovic, N., and Taylor, R.N. Architecture-based runtime software evolution", In Proc. of the 20th Intl. Conf. on Software Engineering, pp. 177{186, Kyoto, Japan (1998). 27. Canal, C., Poizat, P., and Salaun, G. Model-based adaptation of behavioral mismatching components", IEEE Trans. on Software Engineering, 34(4), pp. 546{ 563 (2008). 28. Filieri, A., Ho_mann, H., and Maggio, M. Automated design of self-adaptive software with control theoretical formal guarantees", In Proc. of the 36th Intl. Conf. on Software Engineering, pp. 299{310, Hyderabad, India (2014). 29. Nafz, F., Ortmeier, F., Seebach, H., et al. A generic software framework for role-based organic computing systems", In Software Engineering for Adaptive and Self-Managing Systems, SEAMS'09. ICSE Workshop on, pp. 96{105, Cape Town, South Africa (2009). 30. Fischer, P., Nafz, F., Seebach, H., et al. Ensuring correct self-recon_guration in safety-critical applications by veri_ed result checking", In Proc. of the 2011 Workshop on Organic Computing, OC'11, pp. 3{12, New York, USA (2011). 31. Kaiser, G., Parekh, J., Gross, P., et al. Kinesthetics extreme: an external infrastructure for monitoring distributed legacy systems", In Autonomic Computing Workshop, pp. 22{30, Seattle, Washington, USA (2003). 32. Georgas, J.C. and Taylor, R.N. Towards a knowledgebased approach to architectural adaptation management", In Proc. of the 1st ACM SIGSOFT Workshop on Self-managed Systems, pp. 59{63, Newport Beach, California, USA (2004). 33. Fox, K.L., Henning, R.R., Farrell, J.T., et al. System and method for assessing the security posture of a network using goal oriented fuzzy logic decision rules", April 19 2005. US Patent 6,883,101 (2005). 34. Berry, D.M., Cheng, B.H.C., and Zhang, J. The four levels of requirements engineering for and in dynamic adaptive systems", In 11th Intl. Workshop on Requirements Engineering Foundation for Software Quality (REFSQ'05), p. 5, Porto, Portugal (2005). 35. Cheng, B.H.C., Sawyer, P., Bencomo, N., et al. A goal-based modeling approach to develop requirements of an adaptive system with environmental uncertainty", In Intl. Conf. on Model Driven Engineering Languages and Systems, pp. 468{483, Denver, Colorado, USA (2009). 36. Van Lamsweerde, A. and Letier, E. Handling obstacles in goal-oriented requirements engineering", IEEE Transactions on Software Engineering, 26(10), pp. 978{1005 (2000). 37. Sykes, D., Heaven, W., Magee, J., et al. From goals to components: a combined approach to selfmanagement", In Proc. of the 2008 Intl. Workshop on Software Engineering for Adaptive and Selfmanaging Systems, pp. 1{8, Leipzig, Germany (2008). 38. Menasc_e, D.A., Ewing, J.M., Gomaa, H., et al. A framework for utility-based service oriented design in sassy", In Proc. of the First Joint WOSP/SIPEW Intl. Conf. on Performance Engineering, pp. 27{36, San Jose, California, USA (2010). 3002 A. Tarihi et al./Scientia Iranica, Transactions D: Computer Science & ... 27 (2020) 2985{3004 39. Shen, C.C., Srisathapornphat, C., and Jaikaeo, C. An adaptive management architecture for ad hoc networks", IEEE Communications Magazine, 41(2), pp. 108{115 (2003). 40. Esfahani, N., Elkhodary, A., and Malek, S. A learning-based framework for engineering featureoriented self-adaptive software systems", IEEE Transaction on Software Engineering, 39(11), pp. 1467{1493 (2013). 41. Brinkschulte, U., Pacher, M., and Von Renteln, A. An arti_cial hormone system for self-organizing realtime task allocation in organic middleware", In Organic Computing, pp. 261{283, Springer, Berlin, Germany (2009). 42. Lodish, H., Berk, A., Kaiser, C.A., Krieger, M., Bretscher, A., Ploegh, H., Amon, A., Martin, K.C., Molecular Cell Biology, 8th Edn, W.H. Freeman and Company, New York, USA (2008). 43. Alberts, B., Bray, D., Hopkin, K., Johnson, A.D., Lewis, J., Ra_, M., Roberts, K., Walter, P., Essential Cell Biology, 4th Edn, Garland Science, Taylor & Francis Group, New York, USA (2013). 44. Sugeno, M. and Nishida, M. Fuzzy control of model car", Fuzzy Sets Systems, 16(2), pp. 103{113 (1985). 45. Schmeck, H., Muller-Schloer, C., Cakar, E., et al. Adaptivity and self-organization in organic computing systems", ACM Trans. Auton. Adapt. Syst., 5(3), pp. 1{10 (2010). 46. Richter, U., Mnif, M., Branke, J., et al. Towards a generic observer/controller architecture for organic computing", GI Jahrestagung, 93, pp. 112{119 (2006). 47. Komann, M. and Fey, D. Marching pixels-using organic computing principles in embedded parallel hardware", In Intl. Symposium on Parallel Computing in Electrical Engineering (PARELEC'06), pp. 369{ 373, Bialystok, Poland (2006). 48. Hartmann, J., Stechele, W., and Maehle, E. Selfadaptation for mobile robot algorithms using organic computing principles", In Kub_atov_a, H., Hochberger, C., Dan_ek M., Sick B., Editors, In Intl. Conf. on Architecture of Computing Systems (ARCS'13), pp. 232{243, Lecture Notes in Computer Science, 7767, Springer, Berlin, Germany (2013). 49. Branke, J., Mnif, M., Muller-Schloer, C., et al. Organic computing-addressing complexity by controlled self-organization", In Second Intl. Symposium on Leveraging Applications of Formal Methods, Veri _cation and Validation (Isola 2006), pp. 185{191, Paphos, Cyprus (2006). 50. Roth, M., Schmitt, J., Kiefhaber, R., et al. Organic computing middleware for ubiquitous environments", In Organic Computing A Paradigm Shift for Complex Systems, pp. 339{351, Springer, Berlin, Germany (2011). 51. Murch, R., Autonomic Computing. Information Management, IBM Press, California, USA (2004). 52. Gudemann, M., Nafz, F., Ortmeier, F., et al. A speci_cation and construction paradigm for organic computing systems", In Second IEEE Intl. Conf. on Self-Adaptive and Self Organizing Systems, pp. 233{ 242, Venezia, Italy (2008). 53. Kasinger, H. and Bauer, B. Combining multi-agentsystem methodologies for organic computing systems", In 16th Intl. Workshop on Database and Expert Systems Applications (DEXA'05), pp. 160{164, Copenhagen, Denmark (2005). 54. Tomforde, S., Rudolph, S., Bellman, K., et al. An organic computing perspective on self-improving system interweaving at runtime", In IEEE Intl. Conf. on Autonomic Computing (ICAC), pp. 276{284, Wurzburg, Germany (2016). 55. Herzig, P., Ameling, M., and Schill, A. A generic platform for enterprise gami_cation", In Joint Working IEEE/IFIP Conf. on Software Architecture and European Conf. on Software Architecture, pp. 219{223, Helsinki, Finland (2012). 56. Khanzadia, M., Shahbazia, M.M., Arashpourb, M., et al. Lean design management using a gami_ed system", Scientia Iranica, 26(1), pp. 15{25 (2019). 57. Ido, G., Hashavit, A. and Corem, Y. Games for crowds: A crowd sourcing game platform for the enterprise", In Proc. of the 18th ACM Conf. on Computer Supported Cooperative Work and Social Computing, CSCW'15, pp. 1860{1871, Vancouver, British Columbia, Canada (2015). 58. Rani, P., Sarkar, N., and Liu, C. Maintaining optimal challenge in computer games through real-time physiological feedback", In Proc. of the 11th Intl. Conf. on Human Computer Interaction, 58, pp. 22{27, Bonn, Germany (2005). 59. Walsh, G. and Golbeck, J. Curator: A game with a purpose for collection recommendation", In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems, pp. 2079{2082, Atlanta, Gorgia, USA (2010). 60. Chen, L.J., Wang, B.C., and Zhu, W.Y. The design of puzzle selection strategies for esp-like gwap systems", IEEE Trans. on Computational Intelligence and AI in Games, 2(2), pp. 120{130 (2010). 61. Kerler, S., Vilsmeier, J., Edenhofer, S., et al. Pheromander: Real-time strategy with digital pheromones", In 8th Intl. Conf. on Games and Virtual Worlds for Serious Applications (VS-GAMES), pp. 1{4, Wurzburg, Germany (2016). 62. Ho, C.J., Chang, T.H., Lee, J.C., et al. Kisskissban: A competitive human computation game for image annotation", ACM SIGKDD Explorations Newsletter, 12(1), pp. 21{24 (2010). 63. Syu, Y.S., Yu, H.H., and Chen, L.J. Exploiting puzzle diversity in puzzle selection for esp-like gwap systems", In IEEE/WIC/ACM Intl. Conf. on Web Intelligence and Intelligent Agent Technology, 1, pp. 468{475, Washington, District of Columbia, USA (2010). A. Tarihi et al./Scientia Iranica, Transactions D: Computer Science & ... 27 (2020) 2985{3004 3003 64. Otterbacher, J. Crowdsourcing stereotypes: Linguistic bias in metadata generated via gwap", In Proc. of the 33rd Annual ACM Conf. on Human Factors in Computing Systems, pp. 1955{1964, Seoul, Republic of Korea (2015). 65. Harris, C. Cluemein: Enhancing the esp game to obtain more speci_c image labels", In Proc. of the 2018 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts, CHI PLAY '18 Extended Abstracts, pp. 447{452, Melbourne, Victoria, Australia (2018). 66. Ashraf, G., Why, Y.P., and Islam, T. Mining human shape perception with role playing games", GSTF Journal on Computing (JoC), 1(1), pp. 58{64 (2018). 67. Amiri-Chimeh, S., Haghighi, H., Vahidi-Asl, M., et al. Rings: A game with a purpose for test data generation", Interacting with Computers, 30(1), pp. 1{30 (2017). 68. Kondreddi, S.K., Trianta_llou, P., and Weikum, G. Combining information extraction and human computing for crowdsourced knowledge acquisition", In IEEE 30th Intl. Conf. on Data Engineering, pp. 988{ 999, Chicago, Illinois, USA (2014). 69. Von Ahn, L., Kedia, M., and Blum, M. Verbosity: A game for collecting common-sense facts", In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems (CHI '06), pp. 75{78, Montreal, Quebec, Canada (2006). 70. Bellotti, F., Berta, R., De Gloria, A., et al. Exploring gaming mechanisms to enhance knowledge acquisition in virtual worlds", In Proc. of the 3rd Intl. Conf. on Digital Interactive Media in Entertainment and Arts, pp. 77{84, Athens, Greece (2008). 71. Fisch, D., Janicke, M., Muller-Schloer, C., et al. Divergence measures as a generalized approach to quantitative emergence", In Organic Computing-A Paradigm Shift for Complex Systems, pp. 53{66, Springer Berlin, Germany (2011). 72. Seebach H., Nafz F., Steghofer J.P., et al. How to design and implement selforganising resource-ow systems", In Organic Computing-A Paradigm Shift for Complex Systems, pp. 145{161, Springer Berlin, Germany (2011). 73. Quinlan, J.R. Induction of decision trees", Machine Learning, 1(1), pp. 81{106 (1986). 74. Kohavi, R. and Quinlan, J.R. Data mining tasks and methods: Classi_cation: decision-tree discovery", In Handbook of Data Mining and Knowledge Discovery, pp. 267{276. Oxford University Press, Inc. Oxford, United Kingdom (2002). 75. Nakamura, S., Nakagawa, H., Tahara, Y., et al. Towards solving an obstacle problem by the cooperation of uavs and ugvs", In Proc. of the 28th Annual ACM Symposium on Applied Computing, pp. 77{82, Coimbra, Portugal (2013). 76. Venkatesh, S., Lobo, R., and Sugunan, N. Controlling unmanned ground vehicle using stationary airborne system", Intl. Journal of Innovation and Applied Studies, 8(3), p. 958 (2014). 77. Grocholsky, B., Keller, J., Kumar, V., et al. Cooperative air and ground surveillance", IEEE Robotics & Automation Magazine, 13(3), pp. 16{25 (2006). 78. MacArthur, E.Z., MacArthur, D., et al. Use of cooperative unmanned air and ground vehicles for detection and disposal of mines", In Proc. of Intl. Society for Optics and Photonics, 5999, pp. 94{101 (2005). 79. Liu, Y. and Nejat, G. Robotic urban search and rescue: A survey from the control perspective", Journal of Intelligent & Robotic Systems, 72(2), pp. 147{165 (2013). 80. Anderson, B. and Crowell, J. Workhorse auv-a costsensible new autonomous underwater vehicle for surveys/ soundings, search & rescue and research", In Proc. of OCEANS 2005 MTS/IEEE, pp. 1{6, Washington, District of Columbia, USA, IEEE (2005). 81. Mayer, I., Bekebrede, G., Harteveld, C., et al. The research and evaluation of serious games: Toward a comprehensive methodology", British Journal of Educational Technology, 45(3), pp. 502{527 (2014). 82. Brockmyer, J.H., Fox, C.M., Curtiss, K.A., et al. The development of the game engagement questionnaire: A measure of engagement in video game-playing", Journal of Experimental Social Psychology, 45(4), pp. 624{634 (2009). 83. Norman, K.L. Geq (game engagement/experience questionnaire): a review of two papers", Interacting with Computers, 25(4), pp. 278{283 (2013). 84. Von Ahn, L. and Dabbish, L. Designing games with a purpose", Commun. ACM, 51(8), pp. 58{67 (2008).