How research informs educational technology decision‑making in higher education: the role of external research versus internal research.

Fiona Hollands ( fmh7@tc.columbia.edu) and Maya Escueta (mme17@tc.columbia.edu ). Education Tech Research Dev https://doi.org/10.1007/s11423-019-09678-z

Abstract
Research use in educational decision-making has been encouraged and well documented at the K-12 education level in the United States but not in higher education, or more specifically for educational technology. We conducted a qualitative study to investigate the role of research in decisions about acquiring and using educational technology for teaching and learning in higher education. Results from 45 interviews of decision-makers in higher education show that they engage in different types of research activities throughout the decision-making process, but that in most cases the research is lacking in methodological rigor.

Externally-produced, scientifically-rigorous research was mentioned in less than 20% of interviews. Decision-makers often conduct their own internal investigations on educational technology products and strategies producing locally-relevant, but usually less-than rigorous, evidence to inform decisions about continuing use of the technology or scaling up.

Main purpose of the article
This study addresses this gap in the literature by asking the question: Do educational technology decision-makers in higher education use research to inform decisions about acquiring and using educational technology to facilitate teaching and learning and, if so, how?

Some other interesting remarks of the authors  (the literature can be found in the article on Internet (https://link.springer.com/article/10.1007%2Fs11423-019-09678-z )

1. There is a general consensus that integrating externally-produced research into decision-making is difficult (Neal et  al. 2018) since rigorous research about the educational program or practice in question may not be available. Even when it is, the findings need to be contextualized with “local data analyses, organizational history, and practice experience” (Tseng and Nutley 2014, p. 170) to apply it to the decision-maker’s own situation.
    
2. Locally-relevant, internal research, such as faculty and student surveys or pilot studies, may be more feasible to implement and may provide more timely information, especially to answer questions about whether an educational technology tool or strategy is meeting local needs. However, internal research may be less reliable for providing solid answers to questions about effectiveness for improving academic outcomes. There is substantial evidence that involving stakeholders in identifying educational needs and goals and in designing and conducting locally-relevant research and evaluation increases the likelihood that the findings are used for decision-making (Anderson and Shattuck 2012; Coburn and Penuel 2016; Dede 2005; Lewin 1946; Penuel et al. 2015; Penuel and Farrell 2017; The Design-Based Research Collective 2003). But the jury is still out as to whether evidence-based decision-making in education leads to improved student outcomes (Heinrich and Good 2018).
    
3. How decisions are made?
   As investments in educational technology increase (Morrison 2017), it is important to understand how educational technology decision-makers in higher education make decisions about acquiring and using educational technology for the purposes of teaching and learning, whether and how they use research, and how to improve these practices to ensure positive academic returns on educational technology investments.
    
4. A number of useful strategies
   A responsible approach to educational technology decision-making might involve combining a number of the strategies that surfaced in this study:

• regular needs assessments,
• staying abreast of technology developments,
• match-making between solutions and needs,
• gathering evidence commensurate with the tier of investment,
• systematic stakeholder engagement,
•  supportive leadership,
• educational technology professional development for faculty and
• continuous improvement.

But,
we also urge initial attention to the underlying theory of change for any educational technology tool or strategy  to ensure there is a plausible mechanism by which it can lead to the intended outcomes.
Once the tool or strategy is implemented, periodic reassessment of efficacy is warranted to determine whether the theory of change holds up and endures in practice.