A Model of Learning Technology Partnership with Vendors: A Systematic Process

Higher education institutions partner with technology providers to stay competitive in the fast-evolving innovations of technological advancements. The purpose of this study is to reflect on and investigate technology integration cases at a mid-sized public university in the U.S. Taking a qualitative research approach, this study utilizes critical reflection and multiple case studies (Stake, 2006). After reviewing the data, a technology partnership model was created that is composed of five phases: (1) analysis, (2) negotiation, (3) pre-implementation, (4) implementation, and (5) evaluation. The findings indicate that a systematic process must be utilized to ensure efficiency in vendor–university partnerships. Discussions, limitations, and implementation are provided.


INTRODUCTION
Despite the rapid growth of online education in recent years, many higher education institutions continue to face difficulties keeping up with technological advancements (Ortagus & Tanner, 2019). To ensure efficiency and the use of newer technological innovations, several institutions have begun to partner with technology Based on our personal experiences regarding learning technology partnerships, the current study proposes a model for partnerships with technology providers. We focused on case studies where learning technologies were implemented in the context of universities with scaled online programs. We also concentrated on selecting sample technologies that were crucial for operating the full aspect of the online learning programs. The model presented in this study focuses on three factors: (1) process, (2) principles, and (3) outcomes. We have been significantly involved in the development and maintenance of an online education program and, as such, have used selected examples from our professional experiences with educational technology partnerships in the current study to create a systematic manual for all stakeholders involved in learning technology partnerships.

METHODS
Our study is framed in such a way as to develop a model by focusing on case studies of learning technology partnerships and, as such, we utilized the critical reflection approach to identify conceptual components that could be included in the proposed process model. Critical reflection, which is widely used in professional settings to help practitioners enhance professional practices (Fook, 2011), is composed of different perspectives and theoretical underpinnings, such as reflexivity (Taylor, 2000) or reflection-in-learning and reflection-on-learning (Schön Schon, 1984). According to Brookfield (1994), critical reflection includes four major activities: (1) assumption analysis, (2) contextual analysis, (3) imaginative speculation, and (4) reflective skepticism. To stay updated in a fast-paced work environment, practitioners tend to make quick decisions based on their psychological models-a structured knowledge base of a domain. Critical reflection allows practitioners to (1) engage in transformative learning by helping to identify the reasons and emotions involved (Taylor, 2001); (2) situate themselves within a broader social context; (3) understand the underlying values, beliefs, and biases that affect the process; (4) work through conflicting feelings, reactions, and perspectives to enhance relationships with stakeholders; and (5) assess their learning methods and results to enhance the practices used.
Critical reflection, in this study, follows a specific process of examining relevant cases related to technology partnerships. Several scholars have proposed various models of critical reflection. For example, Hampton (2001) stated that the reflection process must provide information regarding the following aspects: (1) description, (2) interpretation, and (3) outcomes. Lay and McGuire (2010) suggested the use of a similar process; however, they highlighted the significance of the lessons learned. The steps include (1) describing the experience, (2) examining the experience, and (3) highlighting the lessons learned from it. By synthesizing existing reflection models, we examined case studies regarding technology partnerships that included 3 / 19 information regarding various factors, such as technology and major functions, context, audience (endusers), partnership process, major takeaways (i.e., lessons learned), and outcomes. Synthesizing the literature, we employed a process that involved (1) describing the cases, (2) interpreting the outcomes, and (3) highlighting the lessons learned.
Moreover, in the current study, we used the multiple case study approach to identify case studies regarding technology partnerships and determine the notable themes within them. We applied the multiple case study approach to ensure that the analyses could be conducted within each setting, as well as across various settings with differing contexts. Using this approach, five technology partnership cases were analyzed to determine their similarities and differences. This analysis was based on that presented by Yin (2017), who stated that the multiple case study approach can help to predict similar (literal replication) or contrasting results for predictable reasons (theoretical replication). Baxter and Jack (2008) stated that if more than one case study regarding an event or phenomenon is available for analysis, then conducting a multiple case study is advisable.
The current study aims to address the following research questions: 1. What was the context of the technology partnerships with the providers? 2. What was the process used for the technology partnerships?
3. What were the outcomes of the technology partnerships? 4. Which solutions will ensure sustainable innovations for technology partnerships?
Based on our perspectives regarding the selected case studies, a model is proposed to address sustainable and effective educational technology partnerships.

Study Background
In the current study, we focused on technology partnership cases that occurred at a university located in the Pacific Northwest region that has an annual enrollment of approximately 25,000 students. Due to the rapid growth of online education in recent years, the university has been busy implementing new learning technologies, specifically those aimed at online learning. To help fulfill the university's new mission to make higher education increasingly affordable, the Office of Information Technology has implemented Open Educational Resources (OER) and other relevant technology, such as Pressbooks. To maintain the quality of learning for the scaled online programs, the university recently purchased a learning analytics tool called Analytics for Learn (A4L). In addition to these new technologies, the university has maintained the use of its foundational tools, such as the Learning Management System (LMS) Blackboard. In this study, we primarily focused on online learning technologies.

Participants
Three online learning scholar-practitioners participated in this study. The following is a list of the participants and their primary responsibilities.
• Participant 1: The first participant-researcher is an educational researcher and practitioner whose job is to evaluate and implement online learning technologies as well as provide technology and research consultations to the university faculty. Participant 1 also designs and develops online courses and programs at the university. Recently, the participant has been working on identifying directions and strategies for scaling online programs.
• Participant 2: The second participant-researcher is an instructional technologist whose job is to evaluate, pilot, and support technologies at the enterprise level. The participant's primary area of expertise is the learning management system, video tools, and web conferencing.
• Participant 3: The third participant-researcher is a project specialist whose job is to help implement new and support existing assessment-related technologies. This includes the improvement of assessment 4 / 19 processes (both online and face-to-face) and providing training for the university faculty. Participant 3 coordinates between vendors and internal stakeholders to ensure that the desired objectives are fulfilled.

Sample and Selection Criteria
In this study, we utilized criterion sampling (Patton, 2002), wherein the participants provided data that were most relevant to the context-technology partnerships (Merriam, 1988). Criterion sampling involves selecting cases that meet some predetermined criterion of importance to a study (Patton, 2002). The selection criteria for the case studies analyzed in this study included (1) recent examples proving its relevance, (2) a wide coverage of operations for scaled online education programs (e.g., online proctoring, digital textbooks), (3) coverage for both faculty and students as end-users of these technologies, and (4) technologies that the university funded and supported. Therefore, we identified five cases in the following five fields: learning analytics, videoconferencing, digital text and OER, online video platform, and online proctoring. We believe these technologies cover a wide spectrum of online learning activities.

Data Collection
The three participant-researchers, who are also the authors of the current study, provided critical reflections for this study. These participant-researchers are university professionals, who are currently serving as learning technology specialists, educational researchers, and faculty, respectively. Eleven Reflective dialogues were conducted among the three participant-researchers to determine relevant cases and experiences. Reflective dialogue is a key component of communities of reflective practice, especially for professionals (Wenger, 1998). The reflective dialogues were the primary data sources for the investigation into the technology partnerships and clarifications on the technology integration processes used in this study. These dialogues were documented to enable further analysis. During instances wherein their opinions conflicted, the researchers determined a consensus by revisiting the concepts and experiences in retrospect. Specific data sources included (1) the backlog data, (2) a series of meeting and dialogue notes, and (3) relevant vendor-institution documents and communications to ensure accuracy during this process.

Data Analysis
We conducted a thematic analysis for within-and cross-cases (Braun & Clarke, 2006) and referred to the Coding Manual for Qualitative Researchers that was developed by Saldaña (2015). The coding was conducted after the data collection and involved the following processes. First, we carefully read the meeting notes, dialogue transcripts, and documentation relevant to partnerships with technology providers (e.g., contracts) several times in order to code each paragraph and/or sentence. Second, we codified the data. Third, we categorized the codes. Fourth, we organized the data into categories and sub-categories. As a result, we created five categories (phase) and 27 sub-categories (stage). These themes were used to create a comprehensive and holistic model at the end of the paper in an organized manner.
For the cross-case analysis, we followed an analysis process that was proposed by Stake (2006). According to Stake (2006), researchers attempting to merge their findings must conduct a comparative analysis on each case study. Therefore, we presented our reflections in a shared document, and compared the five case studies to determine whether any patterns could be highlighted. Lay and McGuire (2010) stated that critical reflection should result in the interpretation of an experience and the lessons learned from it. To fulfill the authors' recommendations by examining each of the categorized codes and raw data connected to them, we organized our findings in terms of the six major aspects of our technology integration experiences: (1) technology and primary function, (2) context, (3) audience, (4) partnership process, (5) primary findings, and (6) outcomes. Finally, we synthesized the reflections into a partnership model from a research-oriented perspective of educational technology researchers who have engaged in the fields of open education, learning analytics, and technology integration.

Technology and primary function
A4L is a learning analytics tool that has been integrated into the Blackboard environment. It extracts detailed user interaction data from the LMS. A4L fulfills multiple functions, thereby allowing LMS administrators, program coordinators, and individual faculty members to observe data across courses, departments, and the LMS environment. For faculty members, data is available through pre-defined reports within the LMS, whereas program coordinators and administrators can access more pre-defined reports within Microsoft SQL Server Reporting Services. Administrators can also manipulate raw data in custom reports within enterprise business intelligence software, such as Pyramid Analytics. The availability of such extensive data allows stakeholders to analyze the data required to aid decisions regarding faculty course design and student engagement.
This service allows stakeholders to view data through several types of interactions within a course, including students' assignment submissions, grades, logins, and time spent studying in each course. Although a significant amount of this data can be accessed by the faculty members in the LMS through the grade center or legacy course reports, the LMS does not provide the option to quickly display a visualization of this data. This data is packaged into four integrated course reports, a feature that most faculty members consider to be increasingly accessible and intuitive. Providing faculty members with the ability to quickly identify at-risk students, it therefore allows them to potentially intervene before a problem occurs; this can serve to strengthen models aimed at supporting students in online education.
In addition, data on faculty members' course designs, their use of LMS features, and the incorporation of third-party tools is available to LMS administrators. Based on this data, stakeholders can gauge information regarding the adoption rates and value of tools, and determine the allocation of support resources for extensively-used features and tools. This feature ensures that faculty members are provided with the appropriate infrastructure to administer their online courses.

Context
The university purchased A4L in early 2017 when it purchased its standard Blackboard Learn LMS license with the primary aim of determining the scale of student disengagement in online courses. When it was first purchased, the university focused on Blackboard's role as the vendor providing the A4L service because, when compared with utilizing several different vendors for different services, bundling services with a flagship vendor can lead to cost savings for the university.

Audience
The primary users of A4L are faculty members, with the group of fully online faculty members being highlighted among the audience. Faculty members can utilize four integrated Blackboard reports to track their students' participation and performance in the online courses. The aggregated data within the courses feature color-coded indicators throughout the reports, thereby providing a visual representation of the students with which the faculty members should pursue interventions. However, program leadership and coordinators use A4L to assess their students at scale. This audience can simultaneously observe data from multiple courses and identify trends related to student performance. Although LMS administrators have also been provided with this ability, they prefer to use A4L to understand the utilization of the software environment and allocate support for features and tools that are extensively used.

Partnership process
The university purchased A4L in late 2016, and began a partnership with Blackboard for a roll-out of the tool in Spring 2017. Staff from the university's learning technologies and online education groups attended a series of virtual demonstrations conducted by the Blackboard staff. These sessions focused on both the end-6 / 19 user perspectives, and taught university staff about the methods by which to administer the service and produce reports. The university's existing relationship with Blackboard was crucial during the implementation of the A4L service. The provision of one dedicated central service manager indicated that any road bumps encountered during technical implementation can be brought up not only to Blackboard staff who were managing the project, but also to a member of Blackboard who was responsible for overseeing the relationship with the university. Partnering with a third-party vendor for analytics services would potentially lead to only having one escalation path, and coordination between the two vendors occurring through the university staff.

Primary findings
Although the implementation of A4L added value to the online learning experience at the university, this does not indicate that the process was perfect or seamless. The university faced technical issues that continued well into the summer-weeks after the university had planned to fully implement the online programs. To resolve these issues, the university was charged engagement fees for services, such as a short email response-a stipulation that many members involved in the implementation of the project were unaware. This led to more work being done asynchronously in silos, and the dissolution of a true partnership between the vendor and university to formally launch the product. Blackboard ceased updates for A4L in October 2017, merely five months after the university implemented the tool. Although the university continues to maintain a contract and access to the existing reports, no new functionality has been added for A4L since 2017. Based on this case study, we can determine that higher education institutions must seek to communicate with vendors about their long-term roadmaps and product strategies during initial discussions and negotiations so that situations like this do not occur in the future to other organizations.

Outcomes
The implementation of A4L at the university has yielded some positive results, particularly for students in specific online degree programs. The ability to receive reports regarding student activity in courses has helped enhance students' success rates, even in programs that already featured relatively high student performances. The university's project with its multidisciplinary studies program identified at-risk students by using the A4L tool, which led to increased student retention.

Technology and primary function
Zoom is a worldwide market leader in web-based video and phone conferencing. This tool allows users at the university to meet synchronously over a distance, and share camera, video, and screen content to broad audiences. High-definition video conferencing through Zoom provides users with the closest replication for face-to-face class discussions in the realm of online learning. To provide a seamless connection between the LMS and web conferences, Zoom follows the Learning Tools Interoperability standard. Instructor and student accounts are automatically provisioned through the integration. Moreover, simply clicking on the provided link in the LMS automatically signs users in to Zoom, thereby creating a seamless experience for all stakeholders. To conduct group work, Zoom provides meeting hosts (instructors and teaching assistants) with the ability to divide their online meetings into smaller groups called breakout rooms. This function allows students to easily conduct group projects in an online course, with the instructor having the ability to instantly move between the rooms. By replicating a significant portion of the functionality of device-based student response systems and integrating them with the features of a traditional face-to-face classroom, Zoom allows instructors to poll students at any time during a meeting. Polls can either be pre-populated ahead of time, if the instructor has predetermined plans regarding what they want to quiz students about, or they can be given on the fly.

Context
The university first selected Zoom in 2016 with the aim to replace appliance-based video conferencing infrastructure at the university. However, in Spring 2018, as Zoom's LMS integration began to mature and the university started experiencing performance issues with its previous provider, the university's learning technologies team began to consider the replacement of the licensed and supported web conferencing tool.

Audience
The primary audience for the switch in web conferencing providers were faculty members and staff in online courses. However, instructional designers, course builders, learning technologists, and help desk support staff were also considered stakeholders in this process.

Partnership process
Although the partnership between the university and Zoom began in 2016, the initial contract included only a small number of licenses. As the contract with the previous provider continued until the following fiscal year, the university initially piloted Zoom in online classes with only a small number of faculty members. However, in 2018, learning technology staff, in collaboration with online education staff, conducted a thorough evaluation process to ensure that they were partnering with the appropriate vendor for the following years.
The learning technology staff first identified the population (under 40 faculty members) that had been using the former vendor's web conferencing product and conducted a needs assessment. Based on the interview responses, the features of the former vendor were compared with that of Zoom and Google Hangouts. These findings were summarized in a report submitted to governance groups on campus and, by the start of the Fall semester, all faculty members had transitioned to the new product.
Starting with the approximately 40 faculty members, the proliferation increased rapidly at the university, both due to word-of-mouth among faculty members, and through recommendations by the online learning staff. After one and a half years, the university now has more than 240 faculty members utilizing Zoom for various degrees of online education. Throughout this rapid growth in the utilization of web conferencing in online education, the university's partnership with Zoom has played a crucial role. Throughout the process, Zoom's customer support managers, sales representatives, and technical support staff guided the university through the uncharted waters of rapid adoption.

Primary findings
The university's partnership with Zoom serves as a prime example of a healthy, mutually beneficial relationship between a vendor and client. Considering not only the health of the company at the time of the purchase, but also its reputation for product stability and the quality of the support provided from the start of the partnership, ensured that this partnership would be deemed successful.

Outcomes
Rapid increases in the popularity of web conferencing in online education at the university level can be attributed to both the quality of the product and the successful partnership with the vendor. The increased prevalence of web conferencing in online classes has served to increase the admission rate of these courses.

Technology and primary function
Pressbooks is a content publishing platform, based on WordPress architecture. It allows users to create textbooks, e-books, and print books and is used extensively for the production of OER and its associated materials. Although the concept of OER is widely known and accepted among stakeholders in higher education institutions, few platforms allow faculty members to author or re-mix materials. Pressbooks offers an open-source platform to help faculty members produce these materials and share them on a broad scale. An important consideration while creating openly licensed materials is to ensure that all students are able to access the materials. Pressbooks fulfills the WCAG 2.0 AA standards, the level to which the university requires content to adhere. Since Pressbooks is based on the WordPress platform, it also benefits from WordPress' accessibility coding standards. Instead of merely creating a plain textbook, Pressbooks offers access to more than 20 themes to ensure users can tailor their content to the aesthetic they wish to portray.

Context
Although some work had been conducted in the realm of OER for many years at the university, 2017 marked the year in which that activity ramped up. An OER group, consisting of interested faculty members and staff, was formed to coordinate support for various initiatives and provide recommendations to campus leadership. In this group, the idea of purchasing and supporting an authoring platform for faculty members was introduced.

Audience
The audience for Pressbooks was almost entirely composed of faculty members, along with some consideration for stakeholders in the online course design.

Partnership process
The university leveraged existing relationships with the OER community in the effort to determine what OER publishing platform to use. The two chairs of the OER group also served as OER research fellows and maintained a good connection with the community. They used their knowledge about the product landscape to select Pressbooks. After the learning technology and online learning staff collaborated to implement and support Pressbooks, both groups began to regularly meet with the product manager for Pressbooks. This provided insight regarding how to socialize the tool among the faculty members and support them in their utilization of it.

Primary findings
The partnership between the university and Pressbooks was unique due to the currently niche nature of the product. The university had numerous experts in the field, and those experts knew Pressbooks to be the best platform for addressing the university's requirements. The connection with the community allowed the relationship between the vendor and the university to grow and expand the use of the tool.

Outcomes
Thus far, the university's faculty members and staff have authored over a dozen textbooks on the platform, including recreating content from other sources, such as Rice University's OpenStax. This content would have otherwise remained in lower quality and less accessible formats or would not have been published at all.

Technology and primary function
Panopto is an all-in-one enterprise video recording and hosting platform that is utilized by the university to fulfill both academic and operational functions. Panopto utilizes advanced automated speech recognition and optical character recognition to accurately identify the words uttered and displayed in videos. Students and faculty members alike can then search within a video, or an entire folder of videos, for key terms and are provided with time-stamped results, thereby taking them directly to the content they want to see. Panopto is integrated with Blackboard on several levels. Provisioning courses within Blackboard creates a Panopto folder that is unique to the class.
When instructors record to this folder, the students are automatically provided with permission to view the content. This provision also allows faculty members to use increasingly advanced features, such as student 9 / 19 video recording assignments and embedded quiz questions, that link directly to the Blackboard Grade Center. Faculty members who have a licensed Zoom account can opt to have their meeting recordings automatically moved to their Panopto account. Instead of being forced to view a simple picture-in-picture recording of a screen capture and webcam, Panopto provides users with ultimate flexibility in terms of their choices of recording inputs. Users can attach a variety of video device to their computers and simultaneously record multiple feeds. In addition, mobile devices can be used to join the same recording session, thereby allowing a faculty member to capture his/her screen, webcam, and a third feed, such as a whiteboard, in one recording package.

Context
Prior to Panopto being selected as the enterprise video product used by the university, the university had a history of frequent and disruptive changes to its supported video technologies, at times supporting multiple options simultaneously and to varying degrees. This led to the consolidation of classroom lecture video capture with the TechSmith Relay video product in 2016. However, the features that led the learning technologies team to select the product (its simplicity and lack of advanced features, such as automated scheduling) eventually became the key factors influencing the faculty members in calling for a replacement to the tool.

Audience
The audience for Panopto included the university's faculty members, students, and staff.

Partnership process
The process for initiating the university's eventual partnership with Panopto began by cataloguing faculty complaints, support tickets, and documented bugs with the current vendor. The learning technology staff then reached out to faculty members who were considered "power users" of the current tool, and determined the most extensive users from each academic college at the university. These faculty members were brought together in Spring 2019 and requested to list the features that would be present in an ideal video recording and hosting product. These features were compiled by the learning technology staff and, thus, became the requirements for a proposal call that was sent out to vendors. The learning technology staff then vetted these proposals to ensure they fulfilled the requirements set by the faculty group.
After attending a series of presentations by the qualifying vendors, the faculty group and other stakeholders voted to pursue a partnership with Panopto, instead of continuing a relationship with the current vendor. Based on this change, the learning technology team engaged in contract-related discussions with the vendor. Although Panopto was a feature-packed platform that offered a substantial number of features, compared with that of the current vendor, Panopto came with an initial quote that far exceeded the university's allotted budget. The learning technology staff conveyed the truth about their budget limitations to the vendor and were, thus, able to settle for a mutually agreeable sum to set up a contract and subsequent partnership. Throughout the early stages of the partnership, the vendor and client met on a weekly basis to establish a relationship and ensure that all aspects of the migration and an adoption campaign were covered. As the university began to transition, these meetings focused on tying up loose ends, client suggestions for the product, and the vendor informing the client about upcoming features and future plans for the product roadmap.

Primary findings
Although the university's relationship with Panopto is in the initial stage, it has become a key factor for scaling online education, particularly in the midst of many courses moving to distance learning modalities in response to COVID-19. The establishment of Panopto as a key partner, and not merely a vendor for services, has aided the university in incorporating the use of videos into these online courses.

Outcomes
Due to the adoption of Panopto, the use of video resources in courses at the university has steadily increased. This opens the possibility for increasingly engaging course experiences for online students.

Technology and primary function
ProctorU is a virtual proctoring service that offers cheating detection technology assisted by artificial intelligence (AI). It allows students to take exams with a personal computer and monitors the exam session by using the device's built-in (or peripheral) hardware. The AI component of the tool analyzes the students' on-screen behaviors, as well as webcam footage, thereby flagging instances of suspicious movement, lighting changes, noises, and frequent looking off-screen during sessions. Upon the completion of the exam, a proctor inspects the flagged footage to determine whether any violations occurred. In addition, some service plans include live proctoring, wherein an exam proctor monitors the students and can intervene in real-time.

Context
The majority of the online courses offered at the university rely on exams to assess the students' learning outcomes throughout the term. Prior to finalizing a partnership with ProctorU, eCampus staff coordinated with other universities to allow remote students to test at a local testing center. In addition, other proctoring solutions, including ProctorU, were used by various instructors to a lesser extent. In the absence of a centralized process, hundreds of exam administrations per semester were individually coordinated, which led to a significant amount of additional transactional work. Based on predictions that the use of online instruction is projected to grow, the university outsourced proctoring to a vendor that offered an efficient, scalable solution.

Audience
The primary stakeholders of the ProctorU implementation were the instructors and students of online courses. The secondary stakeholders include the instructional designers and support staff who help acclimatize new faculty to the technology tools available at the university.

Partnership process
Prior to formalizing the partnership with ProctorU, the university relied on various virtual proctoring services. While evaluating the existing tools in the market, the stakeholders requested trial access for these tools and compared them against the outlined requirements. A staff member from the university's central online learning department directly worked with instructors to outline the desired outcomes for a proctoring solution. Moreover, although most platforms offered the same baseline functionalities, ProctorU was ultimately selected due to its increasingly advanced detection algorithms, as well as its easier-to-use interface, compared with that of its competitors. ProctorU signaled increased stability in scaling to accommodate the projected growth.

Primary findings
We learned from our experience with ProctorU that institutions can benefit from vendors' expertise in implementing their products. Although ProctorU has been an instrumental online learning tool at the university, its implementation required some instructors to reevaluate the assessment design. For example, in cases where students are permitted to use spreadsheet applications with their tests, the instructors expressed concerns that file deletion cannot be guaranteed after the exam. The instructors worked with project specialists and ProctorU representatives to address this issue. The specialists and vendor recommended creating a higher degree of variety in specific question pools, thereby placing less emphasis on how students use the spreadsheet application. Allowing students to use spreadsheets as an optional tool during the exam helped to negate the 'payoff' of sharing the response files. It is important to remember that 11 / 19 vendors amass a significant amount of experience because they serve various clients. Leveraging this experience during implementation can lead to meaningful process improvements within the institution.

Outcomes
Based on the ongoing growth of online instruction, outsourcing the proctoring of online students to this vendor proved to be successful. The vendor has allowed for the standardization of proctoring at the university's eCampus. Over the past four years, the delivery of virtually proctored exams has significantly increased; it increased from 383 in 2016 to 517 in 2017 and rose sharply to 845 in 2018 before reaching 1179 in 2019. As a scalable service, ProctorU has been able to maintain pace with the demand.

A Model of Technology Partnership
Based on the analysis of the aforementioned cases, we developed a holistic and comprehensive model for a technology partnership. The model is composed of five phases: (1) Analysis, (2) Negotiation, (3) Preimplementation, (4) Implementation, and (5) Evaluation. We included only the major components and processes that typically occur during a technology partnership. Figure 1 presents a summary of the five-phase model. It is important to note that the processes are intertwined and iterative in nature, instead of being one-time events. Both technology partners and universities play crucial roles in this process.

Analysis
The analysis phase is similar to the needs assessment process (Dick, Carey, & Carey, 2005) and the changes that theorists and scholars have proposed. The first step is primarily focused on denoting the problems with the current technology, which may help practitioners develop a long-term plan to address these problems (see Kaplan, 2005). In the context of a university-wide technology partnership, some LMS do not provide sufficient video tools. For example, we started exploring alternative video-conferencing tools to facilitate high-quality interactions in online courses. To prepare for meetings and negotiation processes, the university professionals needed to analyze their technological infrastructure by identifying the tools currently used by Figure 1. A model of a sustainable technology partnership the university, and determining the gaps that new technology could address. Feasibility is also a crucial factor in this process. Through this process, university professionals confirm that a strategy, plan, or design is coherent and can be conducted. Purchasing new technology is only the beginning-once this technology is implemented, the university should determine whether sufficient human and technological resources exist by which to sustainably help the users; otherwise, the university must address these issues with the stakeholders. The implementation of new technology requires substantial time and effort. By conducting a sustainability analysis, the university can enhance its understanding about the sustainability of the new technology and, thus, ensure that it is better prepared for the partnership. The audience is also an important factor in this process. Institutions must identify the differences in the use of the technology between user groups (e.g., faculty, students, staff). Understanding the users' natures can help stakeholders establish expectations. These activities are summarized as follows: 1. Problem analysis 2. Technology infrastructure analysis 3. Feasibility analysis 4. Support analysis 5. Suitability analysis 6. Audience analysis

Negotiation
A negotiation is a continuous effort that stakeholders must commit to throughout the technology partnership process. Due to the nature of the products, negotiation applies to each aspect of the technology's implementation and partnership. For example, pricing is negotiable to some extent, based on a vendor's willingness to provide trials or the newness of the technology. The implementation of technology must feature a support system for users and an enhanced foundation as well as considerable human resources and finances. It is critical to be aware of one's financial and resource situations before the negotiation process begins. The purchased technologies may be centrally updated and controlled by the providers.
However, it is important to confirm whether this is true or universities may wish to negotiate how updates and maintenance issues will be managed. Pricing can vary based on the products purchased. Some providers want to charge per-user pricing, whereas some providers pursue an enterprise license. While reviewing one's current situation, this is an issue wherein decision-makers must provide various perspectives and discuss their options with practitioners. The terms and conditions, as well as the contract details, can be jointly discussed. Based on our cumulative experiences, it is advisable to start with a short-term contract, particularly when the technology is new and has not been proven effective in similar situations.
Finally, leaving some room for flexibility in the case of unforeseen circumstances is a crucial factor in the negotiation process. The decision-makers and learning technology staff cannot test every aspect of the technology within a short period. Both parties must also discuss potential solutions in scenarios for when glitches occur and assistance is required from the providers. These activities are summarized as follows:

Pre-implementation
Pre-implementation is highly recommended pilot process that occurs before scaling the technology. In the pre-implementation stage, it is crucial to identify the potential 'power users' for the product so that you can solicit feedback and determine the pros and cons of the technology. Using a systematic plan for piloting, project leaders allow power users to beta-test the technology and share their findings by conducting a usability test. Once the usability test is complete, the project leads must follow-up with the technology provider and university stakeholders to share feedback that is intended to promote mutual benefits for both parties. Although the pre-implementation process may take longer than usual, this process is crucial for ensuring an efficient partnership before making a long-term commitment. The activities are summarized as follows: 1. Identifying power users 2. Conducting a pilot test with power users 3. Conducting a usability test

Implementation
After the completion of a pilot test, the technology must be implemented on a larger scale. The first step is to diffuse and promote the use of the new technology. It is useful to inquire with the provider whether it can contribute to this activity: sometimes the partners have existing promotional materials as well as job aids that can be used. Using the new technology for the first time typically generates unexpected glitches and errors. Therefore, stakeholders on both sides must create an efficient feedback system that allows end-users to troubleshoot and consultants to address and preserve data regarding these issues. This feedback system should not remain limited to troubleshooting, but should also serve as an evaluation repository for future reference. It can also be used to help make important decisions, such as contract renewals. In particular, troubleshooting should be handled by help-desk assistants. Ideally, the help desk must be operated 24/7 due to the nature of online education; however, these hours may depend on the individual factors of the institutions in the partnership. Feedback can inform the universities how their users are utilizing the technology, thereby resulting in the removal or addition of specific functions. Consolidating feedback, university professionals can optimize the technology to best suit the needs and context of the university. For example, to ensure seamless integration of a video-based discussion tool, an online education center at the university has integrated the video-based tool within the Blackboard environment. The following four stages are involved in the implementation phase: 1. Performing diffusion and promotion 2. Establishing a feedback system 3. Establishing help desk guidelines 4. Optimizing and customizing the technology

Evaluation
The evaluation phase occurs after the implementation of a new technology. Many institutions often disregard the evaluation phase after a new technology is implemented; however, this evaluation can inform and potentially influence each of the aforementioned phases. Therefore, the implementing university must develop an evaluation system (e.g., university-wide survey) for collecting both summative and formative feedback. This evaluation differs from the feedback provided in the previous phase because it occurs after 14 / 19 the users have had a chance to fully experience the technology for at least a full semester. Along with the data, technology facilitators should share the evaluation feedback with the technology providers so as to address identified areas for improvement. This step will help stakeholders make informed decisions regarding contract renewals as well as adjustments in the implementation of the product(s), support services, or the need for updates. The activities in this phase are summarized as follows:

DISCUSSION
A common characteristic of successful partnerships with technology providers is the implementation of scalable solutions that can ensure rapid growth. The providers and clients must be prepared and remain flexible in case of sudden shifts in priorities and the importance of online education. When pursuing new products for their educational technology portfolios, universities must be cognizant of a provider's reputation, financials, and product roadmaps. Understanding the progress made by the vendor, as well as its future plans, is a key factor in building a successful partnership. Emerging vendors may be crucial when piloting products with and set the course for future territory; however, a similar university will require to focus on vendors established in their domains to ensure a successful partnership at scale. The five phases included in the proposed model reflect the partnership processes highlighted in prior studies.
During the analysis phase, a thorough assessment of stakeholders, budgets, processes, and support channels must be conducted to set the high-level strategies and overall direction of the project. Although technology partnership processes have not been a primary concern in the field of education, theoretical and empirical evidence provided by research in other disciplines (e.g., management) have shed light on the importance of IT partnerships, which gives direction for practice in higher education. For example, Brcar and Bulovec (2013) emphasized that the perception about IT outsourcing is largely dependent on the thorough analysis of the expectations of the target technologies, such as cost, quality, and performance. These aspects were regarded as important factors from our reflections on the cases and, thus, included in the analysis step in the proposed model.
Throughout the negotiation process, it is imperative to ensure that all uncertainties and potential issues are accounted for (González et al., 2016). Ideally, negotiations should consider additional flexibility to account for any unforeseen problems. Clarity is particularly crucial in this phase, during which a shared understanding between the internal stakeholders and the vendor begins to form. As the partnership heads into the implementation process, power users must be identified among the stakeholders to ensure that they can assist the university with usability tests and provide feedback regarding tool effectiveness. The roles of negotiation described in our model also correspond to suggestions made by Yao et al. (2010). The authors stated that it is important for clients to effectively communicate with providers in all phases of the partnership so as to make the partnership sustainable. They stressed that timely negotiations should be steered by clients based on collective information (e.g., potential risks, additional costs) available in the different phases of the partnership. They also suggested that IT staff be aware of the technical support and flexibility provided by the technology vendor during the negotiation phase.
The importance of pre-implementation, as shown in the proposed model, was emphasized by Bii and Rukwaro (2019). They examined the factors that led to the pre-implementation of outsourced technology at a university and found that a lack of user participation, a lack of administrative support, and poor communication with stakeholders resulted in the failure of a successful and sustainable technology implementation. The same finding was reported by Veenstra et al. (2009), who claimed that the preimplementation of technology is the most important step in the implementation process as it allows the implementer to identify potential conflicts and users' resistances. Recognizing the suggestions made in prior studies as well as our reflections, we believe that a pilot test involving power users and communication with IT stakeholders is key to a sustainable partnership.
At the implementation stage, a feedback system must be deployed to provide support and ensure the ideal optimization of the new tool. The vendor must be consulted regarding the promotion of the tool because it may be able to provide template materials for this purpose. According to Poleto et al. (2020), it is important to manage new technology solutions at the administration level within an educational institution. The implementation phase in the proposed model champions previous findings in that it shows that the promotion of new technology and the management of feedback from end users is critical.
After the implementation stage, the tool should be evaluated. This phase differs from the feedback system and is intended to provide stakeholders with sufficient time to become familiar with the tool before probing for areas of improvement. Al-Ahmad and Al-Ogaili (2013) warned that tasks that need to be accomplished in the post-outsourcing phase are often neglected; they asserted that clients should evaluate their positions in their partnerships based on the results of constant monitoring and managing after implementing the technology. Universities must remember that vendors are also invested in the success of the project. In some cases, the vendors can offer insights to overcome the challenges noted during previous implementations. By listing the outcomes that need to be examined after the implementation, our model provides direction for sustainable partnerships.

CONCLUSIONS
As the rapid growth of online learning propels it to unprecedented heights, many higher education institutions are increasingly relying on partnerships with technology vendors to ensure cost-effectiveness and sustainable growth in this area. However, as scholar-practitioners, we noted that few studies have focused on providing a systematic direction for practitioners in charge of such partnerships. This study utilized critical reflection as a research method, with the goal of developing a process model for enhancing efficiency in partnerships with technology providers. Multiple case studies of technology partnerships were analyzed to identify similarities and differences, as well as comprehend emergent themes.
Our findings showed that, if a university has a mature process for evaluating and implementing tools, then it sets the stage for selecting appropriate partners and systematic processes, thereby leading to sustainable partnerships with technology providers. It is crucial that the evaluation process addresses all pertinent questions. The key findings from the study highlight the importance of (1) an analysis for understanding the key end-users and the benefits of using a technology, (2) negotiation for setting the ground rules between the vendors and institutions as to the financial model, future support, troubleshooting, and the degree of flexibility, (3) pre-implementation for identifying power users and testing the tool for optimal scalability, (4) implementation for the dissemination of the technology; and (5) evaluation for understanding whether the end-users are satisfied with the tool and identifying areas for improvement.
We anticipate that the proposed five-phase partnership model and activities will serve as a starting point of reference, particularly for stakeholders involved in higher education innovation. In this study, we presented the key points of our long-term partnership experiences with technology providers.
In conclusion, although technology continues to evolve, staff seldom experience difficulties in keeping up with emerging needs. The nature of fast-changing technological environments may lead to slow adoption processes, thus leading the university to become less competitive in providing quality educational services for stakeholders. From a financial perspective, partnerships are an increasingly viable option compared with the self-development of technological tools. Based on a strong relationship and clear shared understanding, visions, and missions with technology partners, it is possible to achieve sustainable growth in online education.

Limitations and Future Research Suggestions
This study employed a critical reflection and multiple case study approach. Some crucial study limitations include the use of purposive sampling, thus rendering this method non-representative of the entire population. As the multiple case study approach involves researchers' reflections and interpretations, it is plausible that this study may feature the researchers' biases and presumptions. Some potential biases may have been unknowingly included based on the researchers' subjective life experiences, educational backgrounds, cultures, ages, work experiences, and social relationships. Qualitative studies can be affected by these factors, particularly when the data involves interpretations from the researchers. As discussed earlier, this study used reflections and interpretations as well as data collection and data analysis (White, 2001). Reflective dialogue was used as the major tool of data collection and the process of reflective dialogues was conducted based on the perspectives of the researchers. How participants engage in a reflective process varies with their degrees of experience and job responsibilities (Killeavy & Moloney, 2010). For novice participant-researchers, we would like to recommend conducting a structured dialogue to analyze cases and interpret their experiences, then they are more likely to challenge their professional practices and consider new possibilities (Berry et al., 2010;Danielson, 2009).
The results of this study showed that understanding technology providers and collaborative work relationships impact the outcomes of successful technology/innovation integrations. Future research on vendors' prior analyses, such as market, budget, and resource, are recommended. In addition, this study showed that user participation and administration support were crucial in creating sustainable and scalable technology partnerships; thus, we recommend future research should investigate strategies or models for sustainable and effective educational technology partnership that focus on these factors.