014227 Evaluating Learning and Innovation

Warning: The information on this page is indicative. The subject outline for a particular session, location and mode of offering is the authoritative source of all information about the subject for that offering. Required texts, recommended texts and references in particular are likely to change. Students will be provided with a subject outline once they enrol in the subject.

Subject handbook information prior to 2024 is available in the Archives.

UTS: Education: Professional Learning
Credit points: 6 cp
Result type: Grade, no marks

Requisite(s): 014220 Learning and its Trajectories AND 014222 Designing Innovative Learning AND 014225 Leading Learning AND 014221 Learning in the Digital Age
These requisites may not apply to students in certain courses.
There are course requisites for this subject. See access conditions.
Anti-requisite(s): 013227 Evaluating Learning and Innovation

Description

This subject provides students with an overview of contemporary evaluation theories, approaches and practices relevant to evaluating learning programs and innovative initiatives. Students explore key aspects of current evaluation approaches through identifying a scenario for an evaluation and through the opportunities to develop an evaluation plan for a project relevant to their professional context. Through this experience they also explore the ethical and cultural aspects of evaluation.

Subject learning objectives (SLOs)

a.	Critically define and frame the scope of an evaluation process relevant to a professional context
b.	Critically analyse contemporary evaluation theories, approaches and practices
c.	Develop evaluation plan relevant to a professional context
d.	Identify the critical and ethical issues in planning an evaluation relevant to a professional context
e.	Identify key issues in evaluating program/ initiatives relevant to Aboriginal and Torres Strait Islander students/users and communities
f.	Communicate appropriately using a range of genres and technologies

Course intended learning outcomes (CILOs)

This subject engages with the following Course Intended Learning Outcomes (CILOs), which are tailored to the Graduate Attributes set for all graduates of the Faculty of Arts and Social Sciences.

• Synthesise advanced knowledge of complex concepts to make research and theory informed judgements about a broad range of professional learning and / or leading practices (1.1)
• Lead, develop and evaluate innovative learning in professional environments (1.3)
• Engage respectfully with Aboriginal and Torres Strait Islander cultures, knowledges, histories, policies and priorities and their implications for learning, leading or research (4.1)
• Apply and critically evaluate principles of social justice, impact and engagement to be responsible, ethical and accountable learning professionals (5.1)
• Apply strong communication and interpersonal skills to engage diverse audiences around complex professional practice issues (6.1)

Teaching and learning strategies

Students are guided through a series of six online modules, over a seven-week teaching session. Each module contains rich content that is organised around a series of topics and includes activities that help students actively engage with the content and undertake the practice–based tasks. Students engage interactively through online discussion boards, embedded comments, and activities and are scaffolded to develop an evaluation plan relevant to a scenario in their professional context.

This subject provides a structure and scaffolding for students to define and frame an evaluation scenario, develop a theory of change and an evaluation plan relevant to the scenario in their professional context. Through these activities students critically explore the ethical and key issues in evaluation. As part of this process, students have the opportunity to receive feedback from their teachers and from their peers.

Synchronous online seminars provide an opportunity to revise key concepts, further develop connections with other students, and to prepare for assessments. Students receive formative feedback on assignment tasks both from their lecturers and peers, including early formative feedback.

To ensure practice-relevant and authentic outcomes, teaching and learning strategies for this subject involve students customising learning to suit their professional practice context. This occurs through the ‘Capability Wrap’ process. Students develop a ‘Subject Learning Plan’ as a way of customising assessments and their learning in the subject, to their professional context. As a first step in this process, students must complete their ‘Course Learning Plan’ and Portfolio (Part A – if this is the student’s first subject in the course) and their ‘Subject Learning Plan’ (Part B), which are compulsory (non-graded) tasks and constitute minimum requirements for this subject.

Students also complete a ‘subject wrap-up’ on conclusion of the subject by reflecting on their learning in relation to their Course and Subject Learning Plans. This also contributes to the ongoing ‘Capability Wrap’ process.

Content (topics)

This subject explores key concepts and debates regarding evaluating learning and innovation. Students begin by interrogating key concepts, particularly the difference between evaluation and assessment. Following this, students explore evaluation approaches and critically evaluate a selection of evaluation models and their relative strengths and limitations in evaluating learning programs and initiatives. Building on this, students are introduced to an evaluation framework that shapes their learning for the remainder of the subject. Student learning is structured around the development of aspects of an evaluation plan– from identifying an evaluation scenario for a learning program or initiative, through to planning their approach. Throughout, students base their work on contemporary evaluation, data, and learning analytics research.

Content includes evaluation approaches, theories and practices; power and interests and ethical issues in evaluation; data and learning analytics and telling stories through data. Students examine the ethical issues in designing evaluations for Aboriginal and Torres Strait Islander students/ users and communities.

Assessment

Assessment task 1: Identify and articulate an evaluation scenario

Objective(s):	a and f
Weight:	15%
Length:	500 words
Criteria linkages:	Criteria Weight (%) SLOs CILOs a. Clarity of the articulation of a professionally relevant evaluation scenario 70 a 1.3 b. Appropriateness of evaluation approach for the identified evaluation scenario supported by reference to the relevant literature 20 a 1.3 c. Clarity of expression 10 f 6.1 SLOs: subject learning objectives CILOs: course intended learning outcomes

Assessment task 2: Define and frame the evaluation

Objective(s):	a, b and e
Weight:	35%
Length:	1000 words
Criteria linkages:	Criteria Weight (%) SLOs CILOs a. Comprehensiveness and relevance of theory of change/program theory 40 a, b 1.3 b. Relevance and depth of the key assumptions and issues identified 20 a, b 1.1 c. Relevance and depth of issues relating to Aboriginal and Torres Strait Islander peoples identified 10 a, b, e 4.1 d. Appropriateness of key evaluation questions to the scenario 20 a, b 1.3 e. Clarity of expression and logical structuring of argument 10 SLOs: subject learning objectives CILOs: course intended learning outcomes

Assessment task 3: Develop an evaluation plan

Objective(s):	c, d, e and f
Weight:	50%
Length:	2000 words
Criteria linkages:	Criteria Weight (%) SLOs CILOs a. Comprehensiveness of the plan for the evaluation scenario 40 c 1.3 b. Depth of justification for plan, using relevant literature, and identification of ethical and other key issues 30 d 5.1 c. Clarity and relevance of identified ethical and other key issues for evaluating programs / initiatives Aboriginal and Torres Strait Islander students/ users and/or communities 20 e 4.1 d. Clarity of expression and logical structuring of argument 10 f 6.1 SLOs: subject learning objectives CILOs: course intended learning outcomes

Minimum requirements

The satisfactory completion of the Course Learning Plan (Part A) (if not previously completed) and the Subject Learning Plan (Part B) are minimum requirements for this subject as they are key components of the Capability Wrap process and a defining feature of this course. Failure to meet this requirement will result in the final assessment not being graded.

Required texts

There are no required texts for this subject. Recommended readings will be available via UTS Library and Canvas.

References

Rousselle, A. and Buregeya, J-M. (2018). Theory-based evaluations: Framing the existence of a new theory in evaluation and the rise of the 5th generation. Evaluation, 24(2), 153–168. https://doi.org/10.1177/1356389018765487

Buckingham Shum, S. and Luckin, R. (2019). Learning analytics and AI: Politics, pedagogy and practices. British Journal of Educational Technology, 50 (6), pp. 2785-2793. doi 10.1111/bjet.12880

Caley, L., Williams, S. J., Spernaes, I., Thomas, D., Behrens, D., & Willson, A. (2021). Frameworks for evaluating education programmes and work-related learning: A scoping review. Journal of Workplace Learning, 33(6), 486–501. https://doi.org/10.1108/JWL-09-2020-0157

Dietz, B., & Woods, D. (2018). An introduction to learning analytics. In R. A. Reiser & J. V. Dempsey (Eds.), Trends and issues in instructional design and technology (4th ed, pp. 104–111). Pearson.

Funnell, S. C. and Rogers, P. (2011). Purposeful program theory: effective use of theories of change and logic models. Jossey-Bass.

Grammatikopoulos, V. (2012). Integrating program theory and systems-based procedures in program evaluation: a dynamic approach to evaluate educational programs. Educational Research and Evaluation, 18(1), 53–64. https://doi.org/10.1080/13803611.2011.640874

Hathaway, D. and Norton, P. (2018). Evaluating Learning Outcomes. In P. Norton (Ed.), Understanding problems of practice: A case study in design research (pp. 51–61). Springer.

Hill. L.H. (2020). Assessment and evaluation in adult and continuing education. In T. Rocco, T. S. Rocco, M. C. Smith, R. C. Mizzi, L. R. Merriweather, & J. D. Hawley (Eds.), The Handbook of Adult and Continuing Education pp. 140–149). Stylus Publishing.

Pammer-Schindler, V. and Rosé, C. (2021). Data-Related Ethics Issues in Technologies for Informal Professional Learning. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-021-00259-x

Pawson, R. and Tilley, N. (1997). An Introduction to Scientific Realist Evaluation.. In E. and S. Chelimsky, Evaluation for the 21st century: A handbook (pp. 405–418). SAGE Publications, Inc.

Perrin, B. (2002). How to - and how not to - evaluate innovation. Evaluation, 8(1), 13–28. https://doi.org/10.1177/1358902002008001514

Picciotto, R. (2019). Is evaluation obsolete in a post-truth world? Evaluation and Program Planning, 73, 88–96. https://doi.org/10.1016/j.evalprogplan.2018.12.006

Pozzi, F. and Sarti, L. (2018). Evaluating innovation injection into educational contexts. Journal of E-Learning and Knowledge Society, 14(1), 83–94. Retrieved August 18, 2021 from https://www.learntechlib.org/p/182313/.

Prieto-Alvarez, C.G., Martinez-Maldonado, R. and Dirndorfer Anderson, T. (2018). Co-designing learning analytics tools with learners (chap 7). In J. M. Lodge,(Ed). Learning analytics in the classroom: Translating learning analytics research for teachers (pp. 93–110). Routledge.

Thompson, K., Alhadad, S.S.J., Buckingham Shum, S., Howard, S., Knight, S., Martinez-Maldonado, R. and Pardo, A. (2018). Connecting expert knowledge in the design of classroom learning experiences (chap 8). In J. M. Lodge,(Ed). Learning analytics in the classroom: Translating learning analytics research for teachers (pp. 111–128). Routledge.

Williams, M. (2018). Ngaa-bi-nya Aboriginal and Torres Strait Islander program evaluation framework. Evaluation Journal of Australasia, 18(1), 6-20. https://doi.org/10.1177/1035719X18760141