Trends and research outcomes of technology-based interventions for complex thinking development in higher education: A review of scientific publications

Trends


INTRODUCTION
Complex thinking is a desired competency in university students and future workers to face the challenges of the 21st-century.The post-truth era we currently live in poses numerous difficulties, compelling individuals to deal with extensive volumes of false information (Valladares, 2021).As noted by Chinn et al. (2020), the widespread dissemination of false information and disinformation is accompanied by its public acceptance, and the denial of established scientific claims and facts.Plus, the constant emergence of new technologies highlights the need for digitally skilled citizens capable of overcoming contemporary societal challenges.pedagogically aligned with the intended learning outcomes and evaluation methods (Biggs, 1996;Biggs & Tang, 2011).Consequently, technology-based interventions should offer learning resources and activities that foster interactions and learning processes aligned with the desired learning outcomes.Instructors need to possess the ability to align the capabilities of learning technologies with the intended objectives, selecting suitable technologies and activities that effectively facilitate interactions supporting the envisioned learning objectives.There is a need for educational interventions that support the development of complex thinking in university students through effective learning technologies.This research aims to identify technologymediated educational strategies for complex thinking development in higher education through a systematic literature review.

BACKGROUND AND RELATED RESEARCH Teaching and Learning Strategies for Complex Thinking Development
Previous studies addressing complex thinking provide a series of strategies for training university students in reasoning for complexity skills.For instance, Gómez-Franciso et al. (2019) argues that questioning is an effective methodology based on the Socratic method for the development of complex thinking in university students.According to Silva Pacheco (2020), the development of complex thinking and cognition occurs through action and interaction with the environment.Zalba-Azzoni and Orta-Martínez (2011) identified nine general strategies for developing complex thinking: problem-based learning, case-studies, academic literacy, experiential learning, e-learning, interdisciplinary learning, task-based learning, and research-based learning.Also, Velducea et al. (2019) conducted a systematic review aiming to analyze intervention strategies with a complex thinking approach that have been implemented in university education.Their findings indicate that implemented strategies for complex thinking development in university students include case study, problem-solving, the use of digital platforms (i.e., Moodle), conceptual cartography, problem-based learning, project tasks and the use of the Internet.
Other empirical studies were based on the concept of complex thinking to propose technological solutions.Such is the case of Zadi et al. (2020) who presented a proposal for an educational digital game for teaching English as a foreign language based on the pedagogical approach of complex thinking.In their proposal, the role-playing game (RPG) integrates a chatbot as an interlocutor with the aim of maintaining students' motivation and contributing to the teaching of English language skills.Sung et al. (2019) conducted a study aiming to identify long readers (low frequency, high duration) and short readers (high frequency, low duration); to then examine to what extent they made different contributions to collaborative discussions.In their study, university students participating in a virtual internship had to read engineering resources and collaborate with other students through an online chat to design a mechanical device.According to their findings and to scientific literature in computer-supported collaborative learning (CSCL), people learn complex thinking through interactions with people (e.g., discussions) and tools.Finally, Goicovic Madriaza (2015) conducted a hermeneutic documentary research, where she analyzed twenty-five manuscripts in which Leonardo da Vinci wrote observations, reflections, hypotheses and conclusions, interconnecting several disciplines.Her study proposes a speculative interpretation based on the principles of complexity defined by philosopher Morin (1986), where she uses the three levels of semiotics (i.e., syntactic, semantic, and pragmatic) as a technique of analysis.This study, based on the work of Leonardo da Vinci who is considered a reference of complex thinking, reveals that transdisciplinary makes it possible to establish better conditions for the development of complex thinking and the creative process.

Technology-Based Interventions and Education 4.0
Education 4.0 refers to a new era in education characterized by the integration of advanced technologies, such as artificial intelligence (AI), machine learning, and the Internet of Things (IoT), into the learning process (Miranda et al., 2019).This approach to education can contribute to meet the needs of the 21st-century by providing students with the skills and knowledge required for a rapidly evolving digital world (Rodríguez-Abitia et al., 2020).Technology-based educational interventions might play a crucial role in achieving the goals of Education 4.0 since they encompass the design, development, and implementation of products, services, and processes through the application of scientific knowledge to meet a need, provide new solutions, or add value.They provide students with access to cutting-edge technologies and digital tools, which can be used to enhance the learning experience, promote collaboration and creativity, and improve student outcomes (Zhou et al., 2020).
A technology-based model contemplates the design, development, and production of products, services, and processes, through applying scientific knowledge to meet a need, provide new solutions, or add value.An example of a technology-based model is presented by Hightow-Weidman et al. (2018) with iTech, which aims to advance HIV prevention and care for young people through technology-based, youth-relevant interventions that maximize adaptability and sustainability.Miranda et al. (2019) provide a framework for companies to project a new generation of primarily technology-based products to deliver solutions to contemporary social problems and respond to changing consumer demands.Already in practice, these technology-based models can maximize their potentiality if accompanied by proactive, personalized, and regular support (Oreopoulos & Petronijevic, 2018), equally contemplating a practical, low-cost, scalable approach (Tofighi et al., 2019).The central point of the models should be the contributions they provide for new solutions supported by technology.
Different tools and strategies can support technology-based models.For example, some technologies used in health interventions have relied on social networks to promote human papillomavirus vaccination among college students (Brandt et al., 2020).Properly, technology-based intervention strategies are integrated into the educational field, such as those conducted by Reyneke et al. (2018).They implemented the interactive online homework system Aplia™ and the flipped classroom, which positively affected the success rates of statistics students in college.Another technology-based intervention strategy was conducted by Katsioudi and Kostareli (2021) with the "sandwich model" strategy combined with personal response systems on student learning, engagement, and satisfaction through a randomized trial in a large biomedical/medical sciences undergraduate class.Similarly, Zha et al. (2021) conducted a WeChat-based educational intervention on HIV and AIDS to improve knowledge and attitudes and reduce stigma among college students in China.The outcomes of technology-based models can be new processes, products, and services, in the realm of change and complexity.
There is a need for educational interventions that support the development of complex thinking in university students through effective learning technologies.Previous studies of literature reviews addressing the topic of technology-based interventions were analyzed and found to focus mainly on health areas.For example, Zebrack and Isaacson (2012) reviewed evidence-based psychosocial support interventions for adolescents and young adults with cancer and other chronic or life-threatening illnesses and technologybased interventions.Similarly, Brown (2013) found in the literature review that using technology-based interventions, such as the Internet, can be an effective tool for tobacco treatment interventions, especially for college students.In the same vein, Farrer et al. (2013)  Also, Wang et al. (2019), in a review of 21 eligible articles to evaluate telerehabilitation, game-based, or web-based treatment, located moderate quality evidence showing that technology-assisted rehabilitation, in particular, telerehabilitation, results in statistically significant improvement in pain.In agriculture, Ahmed et al. (2022) conducted a review to identify gaps and suggested a technology-based intervention that can support rain-fed agriculture under changing climatic conditions.These literature reviews have expanded knowledge in health and agriculture, but what about the area of education with literature reviews on technology-based interventions?To the best of our knowledge, only the study of Velducea et al. (2019) focused on identifying strategies for complex thinking development through a systematic literature review.Yet, their study did not aim to identify the tech-based interventions.The current context of digital transformation in education and the absence of studies analyzing tech-based interventions for complex thinking skills have motivated the study presented here.This paper presents a systematic review that investigates the effectiveness of technology-mediated interventions for complex thinking development.The literature review method, results, and data are presented and discussed in the light of the study questions to provide evidence of effective technology-based interventions to develop complex thinking skills in university students.It concludes with answers to the questions that guided the study; limitations are stated, implications for practice and research, and suggestions for future studies.

METHODOLOGY
This study was undertaken as a systematic literature review based on the guidelines of PRISMA statement.A systematic literature review is a method used to identify, evaluate, and interpret available research data relevant to a particular research question or topic within a time range (Brereton, 2007;Page et al., 2021).This method has been carried out to document the effectiveness of technology-mediated interventions for complex thinking development.The following subsections detail the methodology of the systematic literature review process implemented in this study, including the research questions, search strategy, inclusion/exclusion criteria, data extraction, and data synthesis.

Research Questions
We established the research questions for the study based on the primary goal of this systematic literature review, which was to document the effective technology-mediated interventions aimed at developing complex thinking skills in university students.It was then possible to formulate the following research questions: RQ1.What effective, technology-mediated educational strategies for complex thinking development in university students are described in the scientific literature?
RQ2.What learning-delivery modalities for complex thinking development are reported in the study?
RQ3.What teaching and learning materials are used for complex thinking development reported in the studies?

RQ4.
On what 4.0 components do the publications focus?

Search Strategy
In this systematic review, we first developed a search string to extract the primary studies on the topic under consideration.To create the search string, we first selected the search terms, namely, "complex thinking," "educational intervention," and "emerging technologies."The structure of the search string was formulated according to the approach deployed by Brereton et al. (2007), where a Boolean operator AND is deployed to link the major terms, and the operator OR is used to look for related terms.The articles involved in this study were retrieved through a database search in Scopus and Web of Science.The search was undertaken in June 2022 (Table 1).

Inclusion and Exclusion Criteria
The objective of this systematic literature review was to investigate the contemporary technologymediated pedagogical approaches for developing complex thinking in university students.To achieve this goal, the inclusion criteria were established to consider only the studies published within the last five years.The inclusion and exclusion criteria for selecting the primary studies were specified according to the systematic literature review methodology.In this study, the primary criteria for inclusion were studies that used technologies to develop complex thinking skills in university students.The following inclusion criteria were defined and guided by the research questions.
1. Studies that describe educational interventions for complex thinking development using technologies.
2. Studies that describe educational interventions involving university students in formal context settings.
3. Peer-reviewed studies published between January 1, 2018, and June 2022, including papers, conference papers, and systematic literature reviews.
4. The full text is available in English, French, or Spanish.
Following exclusion criteria were.Studies meeting these criteria were removed from our database.
1. Studies do not document interventions to develop complex thinking skills.
2. Studies do not include the use of technologies for learning.
3. Publication is grey literature, such as professional magazine articles, white papers, project reports, and technical reports.
4. Publications are books, or book reviews.

Data Selection and Analysis
151 documents identified in the search were reduced to a final number of relevant documents for answering the research questions considering inclusion and exclusion criteria.From every paper, different data types were extracted and recorded in Excel.Extracted data from each of the articles included: The process of data selection took place in four stages.First, screening was carried out to remove duplicates.Second, the first analysis of abstracts was carried out to collect standard information from the papers and the inclusion-exclusion criterion types of data.Third, a more careful examination of each full text (TITLE-ABS-KEY ("Complex thinking" OR "Complex thought" OR "Reasoning for complexity" OR "Pensamiento complejo" OR "Pensée complexe" OR "scientific thinking" OR "systemic thinking" OR "innovative thinking" OR "Critical thinking") AND TITLE-ABS-KEY ("Educational strategies" OR "Teaching strategies" OR "Learning strategies" OR "Pedagogical intervention" OR "Pedagogical strategies") AND TITLE-ABS-KEY ("Emerging technologies" OR "Education 4.0" OR "Disruptive technology" OR "Educational technology" OR "technology")) AND PUBYEAR>2017 AND PUBYEAR >2017 Web of Science (ABSTRACT ("Complex thinking" OR "Complex thought" OR "Reasoning for complexity" OR "Pensamiento complejo" OR "Pensée complexe" OR "Scientific thinking" OR "systemic thinking" OR "innovative thinking" OR "Critical thinking") AND ABSTRACT ("Educational strategies" OR "Teaching strategies" OR "Learning strategies" OR "Pedagogical intervention" OR "Pedagogical strategies") AND ABSTRACT ("Emerging technology" OR "Education 4.0" OR "Disruptive technology" OR "Educational technology" OR "technology") AND PUBLICATION DATE<Last 5 years) was performed to apply inclusion and exclusion criteria.Finally, in the fourth round of selection, an in-depth analysis was carried out to collect information to answer the research questions of this study (Figure 2).

Data Synthesis
32 selected studies (listed in Appendix A) were analyzed to answer the research questions.We aimed to identify effective technology-mediated interventions to develop complex thinking skills in university students.The authors thoroughly reviewed the chosen publications, creating summaries that aligned with the research questions of the review.The full texts were coded by the authors and notes were organized into common themes, categories, and subcategories.As the search was intended to identify the effective tech-based strategies to train university students in complex thinking skills, publications meeting the inclusion criteria were examined regarding The thematic synthesis involved analyzing the content of the examined publications to group them together based on their similarities and differences.

RESULTS
Based on the selected 32 studies published regarding technology-based interventions for complex thinking development in university students from 2018 to 2022, this section reports the findings of this systematic review answering the specified research questions.Out of the 32 studies reviewed, 21 of them reported positive research outcomes, while the results of 11 studies were either inconclusive or not clearly stated.The effective technology-mediated educational strategies documented in this systematic review are based on constructivist and active learning strategies such as problem-based, case-based, collaborative, discussion-based, project-based, assessment and feedback-based activities, mind mapping techniques, and other strategies.
Three studies reported the effective use of problem-based learning strategies for developing complex thinking or its sub-competencies.First, Utami et al. (2020) implemented a learning strategy called strategy meets augmented reality technology using problem-based learning (SMART-PBL) to improve metacognitive and problem-solving skills in preservice teachers in Indonesia.SMART-PBL strategy consisted of 10 phases: attentional focusing and motivational process, problem orientation, student orientation, fact identification, hypothesis generation, identification of knowledge deficiencies, knowledge creation process, generation process, new knowledge application, and abstraction.
Several phases were designed to enhance problem-solving skills individually and collaboratively.Their findings indicate that implementing SMART-PBL strategy improved metacognitive problem-solving, scientific reasoning, critical thinking, and communication skills.Second, Jiménez et al. (2021) describe the use of a pedagogical process in which students were confronted with problems of different levels of difficulty and content.First, they were presented with a list of problems, filtered by title, topic, and recommendation, and provided with resources adapted to their learning style.Then, in the activation phase, the students analyzed the problem and used tools such as brainstorming, concept or mind maps.Then, during the application phase, students used an integrated development environment (IDE) to solve the presented problem and manage knowledge collaboratively.Finally, in the evaluation phase, the students evaluated the solved problem.This process is designed to be repeated with new problems in order to develop complex thinking in students.Finally, Bangun and Pragholapati (2021) documented a variety of educational strategies for critical thinking development that include problem-solving tasks allowing students to organize and articulate knowledge to solve complex problems.
Other studies reported the use of case-based learning strategies to improve complex thinking and complex thinking sub-competencies development.For example, Lisperguer Soto et al. (2021) implemented case-based learning activities that allowed 22 physical therapy students enrolled in a clinical reasoning course to hone their critical thinking skills.A case-solving digital resource was designed to present clinical cases to students.Each clinical case scenario had to be solved within 20 minutes.Half of the students participating in the study reported not having enough time to solve the case.Yet, 58% of students reported that critical thinking was one aspect improved by using the resource.In the same sense, Vedi and Dulloo (2021) found that online case-based teaching (CBT) is beneficial for medical students, increasing their interest and critical thinking skills.According to the authors, this teaching strategy allows students to learn at their own pace and covers all aspects of medicine, while not interfering with traditional classes.Similarly, Bangun and Pragholapati (2021) documented educational strategies to develop critical thinking skills in nursing students including case studies, small group activities, debates, simulations, and writing assignments.
Collaborative and discussion-based activities were documented as effective strategies to train students in complex thinking sub-competencies.For instance, Banerjee et al. (2018) conducted a study that aimed to improve students' critical thinking skills through collaboration and oral presentations.In the study, the students received a lecture on DNA replication, repair, and its consequences.They were divided into two groups to present case reports on autosomal disorders and the exploitation of DNA repair mechanisms in cancer therapy.Then, the groups conducted an extensive literature search, selected articles, and prepared presentations.The presentations were delivered orally and followed by a discussion facilitated by a representative from each group, focusing on the suitability of the article, general criticism of the article, and correlation with the concepts presented in the lecture.Similarly, the study of Styers et al. (2018) documented the use of questions and discussions to develop critical thinking in students.In their study, a teaching method based on the use of targeted critical thinking exercises was employed in three courses.These exercises were designed to help students develop their critical thinking skills in a step-by-step manner.The exercises were Contemporary Educational Technology, 2023 Contemporary Educational Technology, 15(4), ep447 9 / 25 introduced by presenting students with a problem or data set and then guiding them through the analysis process with a series of questions.The critical thinking exercises were used as in-class activities to develop critical thinking skills and as questions on quizzes and exams.Finally, Hawkins et al. (2022) conclude that peersupported tutoring can support student academic success and foster critical thinking.
Project-based activities were also documented as effective strategies for complex thinking subcompetencies development.Such is the case of the study conducted by Mshayisa (2022), where food science and technology teachers aimed to develop critical thinking, teamwork, and self-directed learning skills in students through group assignments involving various instructional materials such as online journal reflections, video screencasts, group assignments, and crossword puzzles.In another study, Chin et al. (2019) presented two cases of project-based strategies used to train Education students in science, mathematics, and sustainable development in a blended learning context.In the first case, students developed critical thinking skills through discussions on Facebook and hands-on experience when completing sustainability projects in primary schools.Assessment and feedback-based activities were also identified as effective methods for fostering critical thinking and higher order thinking skills in university students.The integration of the e-Assessment TeSLA system into traditional in-person and hybrid teaching environments in higher education has been analyzed by Durcheva et al. (2019).The system was used to assess students in knowledge of physical phenomena and related objects, concepts, and laws; application of simple mathematical formulas and relationships; in-depth understanding of studied material and practical application in new areas and cases; and use of knowledge and skills to solve problems that require abstract logical thinking, discovery of analogies, and fostering imagination and creativity.
They found that using this technology for student assessment in subjects like electronics, engineering, physics, and mathematics promotes the adoption of modern and mobile communication devices for discipline-related learning gains and skills acquisition, leading to a transformation in the learning approach towards student-centered methods such as collaboration, problem-based, and inquiry-based learning.Additionally, Fang et al. (2021) conducted research to evaluate the impact of peer assessment on critical thinking.They proposed the collaborative feedback-based peer-assessment (CFPA) approach and developed a learning system to measure its effectiveness through a quasi-experiment with pre-service teacher training students.The experiment was conducted with two groups of students, with one using CFPA approach and the other using the non-collaborative peer assessment (NCPA) approach.The results showed that students who used CFPA approach demonstrated a higher tendency towards critical thinking than those in NCPA group.
According to the analyzed publications within this systematic literature review, mind mapping techniques were used to enhance critical thinking skills among undergraduate students.Seckman and van de Castle (2020) suggest that mind mapping techniques can promote critical thinking and sharing of ideas related to digital health technologies.In their study, 163 nursing practice students completed four major projects using mind maps to examine digital health technologies and data sources.Their findings showed that the mind mapping techniques allowed for critical thinking and a deeper understanding of the complexities of the healthcare environment.It was also found that the mind mapping projects were relevant, appropriate, and meaningful to the students' learning.In the same school of thought, Carvalho et al. (2020) compared the critical thinking skills of undergraduate nursing students after a training intervention using concept maps.The experimental, randomized study involved 77 students in a five-week cardiology course, where the intervention involved the construction of four concept maps.The results showed that the evaluation skill score increased significantly in the experimental group compared to the control group.The authors conclude that teaching strategies using constructivist theories, such as concept maps, can improve critical thinking and student learning.
One of the less common strategies included visual thinking strategies such as critical observation, deduction, group discussions and interpretation of works of art to support complex thinking training among the participants of a MOOC, as reported in the study conducted by Poce (2021).For this author, visual thinking strategies are effective in promoting critical thinking when combined with group discussion activities within an Inquiry-based learning approach.In his study, the intervention method involved the educator encouraging learners to examine a museum object, make connections, interpret it, consider multiple perspectives, and support their observations with what they were able to see.Even though the 32 selected studies reported using strategies for complex thinking skills and subskills development, not all included specific information about the research outcomes.Some researchers like York and Conley (2020) reported inconclusive results indicating that future research is needed, while others did not provide detailed information because their results focused on other discipline-related skills rather than higher order thinking skills.Such is the case of authors like Hu (2022), who reported findings regarding the acceptance of the devices used for the intervention without indicating the outcomes related to the development of higher-order thinking skills.Similarly, Pacioni et al. (2020) documented using teaching and learning strategies for critical thinking development, one of the complex thinking subskills, but did not specifically indicate the research outcomes regarding critical thinking development.This was common in studies, where critical-thinking-reported gains were unintended during interventions to develop other discipline-related skills.The type of publication and the aims of each study might have influenced the inconclusive findings reported by some authors.For example, in their systematic literature review on mobile learning technologies, Chang andHwang (2018a, 2018b) indicated that further research is needed to determine the role of game-based mobile learning interventions on higher order thinking skills development since their analysis included few studies on that subject and reported inconclusive results.Even if the implemented tech-based strategies and interventions of those few studies were described in their review, the findings were inconclusive.The categories to classify the learning delivery modalities reported in the analyzed publications are based on the work of Miranda et al. (2021) regarding the core components of Education 4.0.According to them, there are three delivery modalities: face-to-face learning, online distance learning, and hybrid learning modalities.In this systematic literature review, face-to-face learning modality refers to the use of online technologies available in the classroom settings during face-to-face in-person sessions.Online distance learning modality refers to tech-based interventions that take place entirely online without face-to-face sessions on campus.Finally, hybrid modality refers to the combination of online and face-to-face learning sessions.
Figure 3 presents the number of papers reporting face-to-face (n=2), online (n=5), hybrid (n=12), and unspecified (n=13) learning modalities.It is important to note that, even though all documented interventions were technology-based, 13 publications did not include an explicit description of the delivery modality.Those publications were classified as unspecified.
Few face-to-face tech-based interventions were documented in this study.In the work of York and Conley (2020), the authors adopted an experimental approach to engage undergraduate STEM students in anticipatory ethical reasoning, and ethical reasoning applied to analyze the implications and outcomes of technological innovation.Also, Srimadhaven et al. (2020) adopted virtual reality as a technology with the potential to shift the traditional classroom paradigm with the new innovative teaching styles.They exploited smart classrooms equipped with available, free, online ICT tools and cloud services.Some interventions were entirely online.For instance, in the study of Hawkins et al. (2022), the online learning modality was valuable and necessary for the 86 undergraduate students enrolled in nursing programs who participated in online peer-tutoring strategies during the COVID-19 pandemic.This was similar to the study of Vedi and Dulloo (2021), where 84 first-year undergraduate medical students indicated that online case-based learning activities enhanced critical thinking, one of the complex-thinking sub-skills.Given that the learning delivery modality was not part of the extracted information in the systematic literature reviews on mobile learning technologies performed by Chang andHwang (2018a, 2018b), their studies were classified as unspecified.Other studies did not include detailed information regarding the teaching delivery modality.Per the provided descriptions, the technology-based interventions reported in those studies could be any of the three categories.For this reason, studies not indicating the modality were also classified into the "unspecified" category.

RQ3. What Teaching and Learning Materials Are Used for Complex Thinking Development Reported in the Studies?
Initially, we attempted to categorize instructional materials based on the categories in Ololube et al. (2015), including print, audio, visual, and multimedia materials (e.g., videos).
However, difficulties arose in classifying the data due to inconsistent interpretations of materials among the researchers of each study.As a result, we considered Gao et al. (2020)'s classification of pedagogical and operational technologies with subcategories of tool-based and program-based.This classification also faced compatibility issues with the authors' interpretations in the analyzed publications.Therefore, we decided to present the summary of the learning materials and resources as described by the authors in the analyzed studies, which can be found in Figure 4.
Reading assignments, videos, and eliciting/reflective questions were the most-used teaching and learning materials in the analyzed technology-based interventions.These were followed by evaluations, discussions, and case studies.Many teaching and learning materials supported communication, teamwork, and reflection.Reflection was supported through reading and writing assignments.Regarding reading assignments, Baguzina (2020) reported using course books, course reading materials, and texts as part of the pedagogical resources during the educational interventions.In the study of Mshayisa (2022), brochures and PDF documents were used, while York and Conley (2020) provided students with diverse reading materials to promote group discussions.Ganieva et al. (2020) and Morais et al. (2019) described the use of online reading materials such as online documents and e-books.On the other hand, writing assignments were also implemented.For example, one of the writing tasks implemented by Mshayisa (2022) was students' online journal reflections, while Bangun and Pragholapati (2021) and Durcheva et al. (2019) provided students with writing assignments like essays and reports.
Among the documented pedagogical resources, crossword puzzles were used to improve students' science vocabulary (Bangun & Pragholapati, 2021;Mshayisa, 2022); others reported implementing activities such as classroom discussions and lectures (Banerjee et al., 2018;Styers et al., 2018).Discussions, debates, oral presentations, collaborative assignments, forums, and chats supported communication and knowledge exchange among students and between students and tutors or facilitators.Ilma et al. (2021) reported the use of class discussion and online debates, while Crispin et al. (2021) and Poce (2021) implemented chat and forum discussions.Poce (2021) also implemented works of art.That intervention required students to analyze the works of different artists to develop higher-order thinking skills.Their "see/think/wonder" activity promoted an attitude of exploration of relevant information to solve specific problems.

RQ4. On What 4.0 Components Do the Publications Focus?
After carefully reading and analyzing the technological solutions enabling the documented tech-based interventions for the development of reasoning-for-complexity skills, the technologies at use were classified into two categories: technology-based; and tools and platform solutions.The "technology-based" category refers to the technological solutions that incorporate the working principles of emerging technologies and modern techniques such as IoT, machine learning, big data, or AI for educational purposes (Miranda et al., 2021).On the other hand, the category "tools and platforms" refers to technological solutions that combine different 4.0 technologies to improve teaching and learning processes, for example, web-based technologies, blogs, wikis, learning management systems, virtual learning environments, and videoconferencing tools (Miranda et al., 2021).The results indicate that the documented tech-based interventions relied mostly on the use of existing tools and platforms rather than custom-built technology-based software (Figure 5).

Technology-based solutions
The results indicate that four publications documented the use of custom-built software solutions aimed at enhancing the learning experience of students and the development of complex thinking skills.The technology-based solutions developed for educational interventions include a virtual reality game, an educational chatbot, an e-Assessment system, and a chemical augmented reality technology.Srimadhaven et al. (2020) documented the design and use of a virtual reality game that provided learning analytics to better understand the experience of students enrolled in a programming course.The game requires the use of head-mounted display (HMD) equipment for each student to be given an opportunity to experience the virtual reality based mobile games, test their programming code, debug it, and submit it for the evaluation.Another example of a custom-built technology-based solution was the educational chatbot used during the intervention reported by Hu (2022).This chatbot represented teachers and guided students in self-adjusting their learning, reminded students via mobile messages of the objectives they had previously set and notified them of the release of the latest weekly information in their learning journey.
The e-Assessment initiative documented by Durcheva et al. (2019) constitutes a third example of technology-based solutions.Their TeSLA system was designed as a plug-in for a virtual learning environment server and provided interfaces for instructors and students.Teachers were able to select the authentication and authorship verification instruments during the setup of an activity in the virtual learning environment, and the students were able to see their results after finishing the activity.Before starting a learning activity, students had to sign in and create profiles.Then, data such as images, audio recordings, and typing style was collected and compared to the profiles.The results available to the teacher showed the level of matching for the selected instruments both individually and as a whole.
Finally, Utami et al. (2020) implemented a chemical augmented reality technology combined with assignments as a method to explore prior knowledge and build new expertise in problem-solving skills.The augmented reality technology used in their study took the form of a mobile application on a smartphone equipped with cards containing chemical element information.Their intervention involved students using the mobile augmented reality technology to scan the cards in order to display 3D visuals representing the physical form of chemical elements (solid phase, gas phase, liquid phase), their atomic number, symbols of the elements, and the general properties of the elements accompanied by audio recordings.They concluded that chemical augmented reality technology can evoke student motivation for learning chemistry by engaging students in meaningful learning activities that enhance their learning experience.The results show that technology-based custom-built software can play a significant role in enhancing the learning experience and developing complex thinking skills of students.The four documented studies indicate that custom-built technology-based solutions have been implemented to provide valuable learning analytics, facilitate self-adjustment in learning, provide timely reminders, provide an engaging experience, and allow for the collection of data to assess student progress.

Tools and platforms
Technological tools such as mobile apps, social media, and online and virtual platforms were most frequently used for pedagogical interventions intended to develop complex thinking.These technological tools include mobile apps, social media, and online and virtual platforms that provide innovative ways for individuals to interact with and understand information.
Several virtual learning environments were used to develop complex thinking skills during the documented tech-based interventions.To name a few, Fang et al. (2021), documented an intervention involving 97 sophomore preservice teachers in a university in China that used the resources available in a learning management system.In the same vein, the study of Ilma et al. (2021) examined the learning habits of 30 students enrolled in the physics education program using Google Classroom for online learning.The authors concluded that feedback from lecturers is needed to achieve 21st-century student competencies such as creative, critical thinking, and collaborative skills.
Social media and web-based technologies were also part of the strategies for complex thinking skills development; for example, Facebook and Edmodo were used in the intervention documented by Chin et al. (2019).In their study, preservice teachers justified their opinions on issues raised on Facebook.The social media platform was used to communicate, provide arguments, and exchange ideas, which led to critical thinking development as perceived by the students.Similarly, Carvalho et al. (2020) and Seckman and van de Castle (2020) documented the use of digital mind mapping tools such as Coogle, GitMind, and Canvas to promote critical thinking development in undergraduate and graduate nursing students.In the study conducted by Vedi and Dulloo (2021) Google group technology was found to be an easy and cost-effective tool that can be used by a large number of students with one instructor to promote higher order thinking skills such as critical thinking.Conde-Caballero et al. (2019) evaluated the use of a public WordPress-based blog for a nutrition and dietetics module in a nursing undergraduate program at a Spanish university.The blog was initially created as a way to increase student engagement and interest in the subject, but also to introduce future health professionals to the concept of blogging.After analyzing the benefits and drawbacks of using blogs in university health sciences education, the authors concluded that blogs can be a valuable tool for the development of critical thinking skills and knowledge acquisition, but there are also challenges of low student motivation and participation.Finally, Morais et al. (2019) documented the use of Kahoot and Socrative to present problem-solving activities and case studies.
Complex thinking development strategies were delivered using a variety of technologies such as online platforms, social media, and mobile applications.These technologies provided users with new and innovative ways to interact with and understand information.For instance, online platforms enabled people to access and exchange large amounts of information and connect with those who had differing opinions (Lisperguer Soto et al., 2021).Social media provided users with a space to express their ideas and participate in discussions with others (Chin et al., 2019), which could foster the growth of critical thinking skills as they weigh various perspectives.Mobile apps and educational games offered problem-solving activities that encouraged users to think creatively and develop innovative solutions to complex problems (Srimadhaven et al., 2020).Overall, the use of these technologies in the analyzed studies shows that they have the potential to enhance complex thinking development by providing engaging methods for individuals to access, process, and apply information.

DISCUSSION
This study presents an analysis of the scientific publications in Web of Science and Scopus databases that refer to technology-based educational strategies for developing complex thinking.Even though these databases have broad coverage, other high-quality articles may have been left aside.In general, the 32 Contemporary Educational Technology, 2023 Contemporary Educational Technology, 15(4), ep447 17 / 25 included studies highlighted interventions supporting the development of complex thinking or one of its subcompetencies.Most of the reviewed publications report positive research outcomes (Table 2).This study's methodology and results reflect the intention to contribute to educational technology research, especially in identifying effective technology-based educational interventions to develop complex thinking in university students.
Active learning strategies including problem-based and case-based learning strategies were primarily used to develop complex thinking skills and subskills.Table 2 summarizes the strategies and the teaching and learning materials employed during technology-based interventions for developing complex thinking skills.These results are related to the findings reported by Zalba-Azzoni and Orta-Martínez (2011) regarding the strategies for complex thinking development identified in their work.According to Zalba-Azzoni and Orta-Martínez (2011), problem-based learning and case-based learning are the top two strategies for training students in complex thinking skills.This is not surprising since problem-based learning is one of the innovative learning models that present real-world or concrete cases to facilitate learning through a student-centered approach (Salinitri et al., 2015).Our results are similar to the findings of Velducea et al. (2019), who listed a series of strategies for complex thinking development based on active pedagogies.While the work of Velducea et al. (2019) mainly focused on the pedagogical dimension and did not include detailed information on the technologies used, our approach allowed us to document the technologies enabling the delivery of pedagogical strategies.
The analyzed papers documented teaching and learning materials supporting reflection, teamwork, argumentation and communication.Figure 4 shows that reading assignments, videos, and eliciting/reflective questions were the most used teaching and learning materials in the documented tech-based interventions.
These results are related to the work of Sung et al. (2019) who indicated that written texts are a prominent feature of many computer-supported and face-to-face learning environments.As for the eliciting/reflective questions results, our findings are in accordance with the work of Degener and Berne (2017) and Gómez-Francisco et al. (2019) who consider questioning as an efficient methodology for training university students in complex thinking skills; a methodology based on the Socratic method that provokes the learners to reflect and enlighten their own thinking.
Our study found that instructional materials commonly used for promoting higher order thinking skills such as complex and critical thinking are also utilized in the documented tech-based interventions.Discussions, debates, problem-solving assignments, case studies, and teamwork (Figure 4) facilitated the exchange of ideas among students and the collaborative construction of knowledge, which aligns with the work of Dewey (1910), who postulated that the thinking process could be stimulated by presenting a problem, cases, questions, conflict, or confusion about something.This finding aligns with current scientific literature on the subject, as it suggests that traditional instructional materials are still effective in fostering complex thinking when delivered through technology.Reflection and collaborative construction of knowledge seem important for complex thinking skills development, and they are enabled by technological tools and platforms such as videoconferencing tools, virtual environments, forums and chats.
Hybrid learning interventions dominate the documented technology-based educational strategies for complex thinking skills development.As shown in Figure 3, hybrid learning was the second most common learning delivery modality and face-to-face tech-based interventions were scarce.This might be related to the fact that the analyzed studies were published between 2018 and 2022, including the COVID-19 pandemic years, where online learning became a trend and necessary to comply with lockdown regulations.The mandatory lockdown forced educational institutions and society to explore online alternatives (Slack & Priestley, 2022).This might have influenced the lack of face-to-face interventions and the abundance of hybrid interventions in the post-lockdown times.
Technologies like online platforms, social media, and mobile applications were used to deliver complex thinking development strategies.These technologies offered opportunities for users to engage with and process information in new and innovative ways.For example, online platforms allow individuals to access and share vast amounts of information and connect with others who may have different perspectives and knowledge (Lisperguer Soto et al., 2021).Social media provides a platform for users to express their ideas and engage in discussions with others (Chin et al., 2019), which can lead to the development of critical thinking skills as users consider and evaluate different viewpoints.Mobile applications offer a variety of educational and problem-solving games and activities that challenge users to think creatively and develop new strategies for solving complex problems (Srimadhaven et al., 2020).Overall, these technologies were implemented in the analyzed studies and authors conclude that they have the potential to enhance complex thinking development by providing new and engaging ways for individuals to access, process, and apply information.
While inconclusive and unspecified research outcomes and results are primarily associated with the lack of evidence regarding measuring reasoning skills in university students, it is possible that the technologies employed were not pedagogically aligned with the learning objectives.In some cases, the principle of constructive alignment (Biggs & Tang, 2011) was not clearly identified in the analyzed technology-based interventions.For instance, a chatbot reminding students of the pending tasks or the new assignments published in the course learning environment are not necessarily aligned with the learning strategies that can potentially facilitate higher order thinking skills development like reflection, questioning, discussion and knowledge exchange (Gómez-Francisco et al., 2019: Sung et al., 2019;Velducea et al., 2019).Future research should focus on studying the effectiveness of tech-based interventions, considering the constructive alignment of technologies used to train students in reasoning skills.

CONCLUSIONS
This systematic review documented a series of effective technology-based educational strategies for complex thinking skills development.The data revealed that (A) the tech-based strategies for complex thinking development encompass a diverse array of approaches, including problem-based learning, case-based learning, collaboration-driven and discussion-based learning, project-based learning, assessment-and feedback-oriented activities, and mind mapping techniques, which were widely employed by practitioners and researchers in the 32 analyzed studies; (B) while several studies did not describe the teaching delivery modality of the tech-based interventions, most of the documented strategies were implemented in hybrid contexts combining in-person, face-to-face session with online synchronous and asynchronous learning activities; (C) traditional instructional materials commonly used for promoting higher order thinking skills such as reading assignments, videos, and eliciting/reflexive questions are still effective in fostering complex thinking when delivered through technology; and (D) custom-built technological development for complex thinking development software that incorporates emerging technologies such as IoT, machine learning, big data, or AI is somewhat scarce at present.By identifying learning theories and approaches underpinning technology-based interventions targeting the development of complex thinking development, this review offers practitioners and researchers researchinformed insights.These insights can be utilized to design learning resources that align the capabilities of learning technologies with learning strategies to effectively foster interactions that support the envisioned learning objectives.
The search strategy of this systematic literature review was performed in Scopus and Web of Science databases since they have a large and diverse range of indexed journals and publications, and they have been used extensively in literature reviews, which might increase the chances of capturing most of all relevant studies in our field of study.Nonetheless, the use of only two databases as a source of literature for this systematic literature review is a limitation of the study since relevant studies may not have been captured by these databases and the findings may not be exhaustive and may not represent the entire body of literature on the topic.It is recommended to consider conducting a broader search strategy in future studies to mitigate this limitation.Although this study presented a first analysis of tech-based interventions for developing complex thinking, findings are not generalizable since we only covered peer-reviewed publications from 2018 to 2022 and it is possible to miss other relevant studies.Further literature review studies are suggested to map empirical studies concerning other inclusion and exclusion criteria.
This study provided information on technology-based interventions and instructional materials used for training university students in complex thinking and its sub competencies.By leveraging the benefits of technology, instructional materials can be tailored to meet the unique needs of students, leading to improved outcomes and the development of higher order thinking skills.Our findings contribute to the growing body of research that highlights the importance of technology in promoting complex thinking and support the use Contemporary Educational Technology, 2023 Contemporary Educational Technology, 15(4), ep447 19 / 25 of technology as a tool for enhancing education.However, it was not clarified how the utilized instructional materials affect the motivation and the level of complex thinking competency in students.Further research is needed to assess the pedagogical alignment of technologies used for training students in reasoning skills.
Given that many studies lacked detailed information on the teaching delivery modality of the educational interventions, researchers and practitioners might focus on providing more details when presenting their work regarding technology-based pedagogical strategies for developing complex thinking skills.Furthermore, it might be interesting to examine the effectiveness of technology-based pedagogical strategies for complex thinking development delivered in different teaching modalities.Research efforts must focus on identifying effective strategies supported by technologies and best practices for improving the reasoning-for-complexity competency in university students.In conclusion, given the extent of the documented interventions, it can be argued that the findings of this study can provide a panorama of recent technology-based interventions that might inform the design of teaching and learning strategies for complex thinking development and future studies on the subject.

Figure 1 .
Figure 1.Complex thinking (Source: Authors) (a) the findings or results of the documented tech-based interventions, (b) the learning delivery modalities of the documented strategies, (c) the instructional materials and teaching and learning resources at use, and (d) the technologies enabling the documented interventions.

Figure 2 .
Figure 2. Results from the search & selection process (PRISMA flow diagram) (Source: Authors, based on the recommendations provided by Page et al., 2021)

Figure 4
presents the complete list of teaching and learning materials used during tech-based interventions to develop complex thinking skills.

Figure 3 .
Figure 3. Learning delivery modalities reported in the studies (Source: Authors, using Tableau)

Figure 4 .
Figure 4. Teaching & learning materials reported in the studies (Source: Authors, using Tableau)

Table 1 .
Search string for each database

Table 2
presents the summary of the studies reporting positive research outcomes.

Table 2 .
Effective tech-based educational strategies for complex thinking skills development

Table 3
summarizes the tech-based interventions that provide inconclusive or unspecified research outcomes regarding developing complex thinking skills.

Table 3 .
Unspecified research outcomes of tech-based interventions on complex thinking development

Table 3 (
continued).Unspecified research outcomes of tech-based interventions on complex thinking development Inconclusive Note.LDM: Learning delivery modality & RRO: Results & research outcomes