Evaluation of Learning
The evaluation process is important to make decisions about effective learning from the objectives intended to be mastered as well as improvements in instruction. The evaluation process is intended to determine the merit, worth, and value of things and evaluations are the products of that process according to R. Burke Johnson and Walter Dick in Chapter 10 of Trends and Issues in Instructional Design and Technology. Formative assessments are designed to support decisions about student successes on the objectives they were taught and be able to demonstrate the skills involved. Summative evaluations are used to make decisions in lesson planning and curriculum instruction to increase student success.
According to Johnson and Dick, CIPP model or Context, Input, Process, and Product, are distinct types of evaluations that can be done alone or in a single comprehensive evaluation. The Context Evaluation is a needs assessment. I would use this model to plan lessons to determine where students were and what their specific needs were BEFORE I began the assignment. Input refers to the necessary resources or program structure decisions. Process Evaluation examines ways the program is developed, implemented and effectiveness before and after revisions. Product Evaluation includes measuring outcomes. It is used to make implementation decisions. As a Librarian, it would be important to know what the stakeholders (parents, teachers, students, administrators, etc.) knows about the library, how it is organized and the expectations. All stakeholders would need to know how the library is structured for maximum utility and effectiveness not just the library in and of itself, but also am I effective as a librarian?
When looking at the Five-Domain Evaluation Model, we look at specific needs to be evaluated. According to Johnson and Dick, there are five domains: Needs Assessment (Is the program needed?), Theory Assessment (Does the program work the way it is intended to?), Implementation Assessment (Was the program implemented properly?), Impact Assessment (Did the program impact stakeholders?) and Efficiency Assessment (Is the program cost effective?). As a librarian, this evaluation would help assess my goals for the library and the full gamut its effectiveness for the community of stakeholders. In order for me to build accountability, I would need to be able to strengthen library use and use each of the five domains to assess and evaluate the program's desired results using attendance in the library, use of the library and book checkouts to see if there is a return on investments in the program.
Kirkpatrick's Training Evaluation Model motivate training directors to realize the importance of evaluation and increase the training program evaluations, according to Johnson and Dick. Evaluations may occur at the early steps or all four levels. Level 1 is Reaction, which refer to the learner's reactions or attitudes towards the learning experience which is used to evaluate instruction. Level 2 is Learning or what did the participants learn or attitude change or did their skill increase. Level 3 concerns Behavior or Transfer of Training. Can/Will/Are the skills be used on the job and does the environment affect the transfer of training. Level 4 equates to the final results or outcomes of the organization's performance. I would actually use this evaluation to determine the effectiveness of the Library and Information Systems at TAMUC (Texas A & M University Commerce) as it relates to my understanding of becoming a Librarian. When I look back on the Bachelor's of Science degree in education I received to become a teacher (even as I keep in mind I did not Student Teach), there was little correlation to the experiences in class to the experiences in the classroom. But I can already tell that I will use what I have learned so far because I find myself cataloguing personal books, becoming more techno savvy due to the increase use of technology required by my instructors.
Participatory/Collaborative Evaluation or the emphasis of participatory/collaborative forms, according to Preskell, of evaluation is engaging stakeholders in the evaluation process, so they may better understand evaluation and the program being evaluated and ultimately use the evaluation findings for decision-making purposes. This evaluation tool is similar to Formative Assessments. It determines what each stakeholders believes is necessary for the programs and determines whether each stakeholder sees value in the process or product. This type of evaluation determines the success and continuation of a program or the dissolution of a program according to the stakeholders. As a Librarian, I can see this as being an effective tool simply because if the library is not useful to the stakeholders, there is no point in having it as a full time position in a school.
Success Case Method, according to Preskell, approach to evaluation focuses on the practicalities of defining successful outcomes and success cases (Brinkerhoff, 2003). It uses some of the processes from theory-driven evaluation to determine the linkages, which may take the form of a logic model, an impact model, or results map. Evaluators using this approach gather stories within the organization to determine what is happening and what is being achieved. The major question this approach asks is, “What is really happening?” When we look at the library and the Librarian individually we determine success outcomes. If students are visiting the library, what is the cause? Is it the library itself or is it the inability of the librarian to make the library inviting? We must ask, was there a time when the library was used readily? If so why? If not, why? If we get to the root of any issues that keep the library or librarian from successfully instituting a Library Program that involves all stakeholders, then we can determine how success can be achieved.
When we refer to instructional design evaluation, we shouldn't just look at learner outcomes, but also teacher effectiveness. Teacher effectiveness isn't always related to learner outcomes. Students in Honors and AP classes have parent support and a natural ability to strive for excellence. But students who are not in these programs rarely have this. It is therefore imperative that teacher evaluations are included. We should ask such questions as: Are you making the lessons accessible to all students? Are you encouraging/helping all students become successful? Are you including interactive, interesting and a variety of assessment tools for all learners? Teaching and Learning affects both the teacher and the learner and assessments and evaluations must mirror this concept.
One particular non-instructional problem is discipline (although technically students act out because they are unfamiliar with the information being taught). Teachers can not be effective whole class if one or two students have disciplinary issues. Math teachers across the hall from me are told to have students spend more time discussing the Math and are given calculators to do simple calculations. They are not required to "think through" the math, therefore they have an inability to settle down during the actual individual work time to do the work. They become discipline problems. I contend, if we look at these students individually and teacher approaches individually, it is possible to end the discipline issues that relate to instructional weaknesses as as well as student's internal motivation. If we address students as individuals and focus on their individual performance system, we can combat any issues students have. If we analyze student performance and evaluate the causes of their performance, then we can intervene.
According to Johnson and Dick, CIPP model or Context, Input, Process, and Product, are distinct types of evaluations that can be done alone or in a single comprehensive evaluation. The Context Evaluation is a needs assessment. I would use this model to plan lessons to determine where students were and what their specific needs were BEFORE I began the assignment. Input refers to the necessary resources or program structure decisions. Process Evaluation examines ways the program is developed, implemented and effectiveness before and after revisions. Product Evaluation includes measuring outcomes. It is used to make implementation decisions. As a Librarian, it would be important to know what the stakeholders (parents, teachers, students, administrators, etc.) knows about the library, how it is organized and the expectations. All stakeholders would need to know how the library is structured for maximum utility and effectiveness not just the library in and of itself, but also am I effective as a librarian?
When looking at the Five-Domain Evaluation Model, we look at specific needs to be evaluated. According to Johnson and Dick, there are five domains: Needs Assessment (Is the program needed?), Theory Assessment (Does the program work the way it is intended to?), Implementation Assessment (Was the program implemented properly?), Impact Assessment (Did the program impact stakeholders?) and Efficiency Assessment (Is the program cost effective?). As a librarian, this evaluation would help assess my goals for the library and the full gamut its effectiveness for the community of stakeholders. In order for me to build accountability, I would need to be able to strengthen library use and use each of the five domains to assess and evaluate the program's desired results using attendance in the library, use of the library and book checkouts to see if there is a return on investments in the program.
Kirkpatrick's Training Evaluation Model motivate training directors to realize the importance of evaluation and increase the training program evaluations, according to Johnson and Dick. Evaluations may occur at the early steps or all four levels. Level 1 is Reaction, which refer to the learner's reactions or attitudes towards the learning experience which is used to evaluate instruction. Level 2 is Learning or what did the participants learn or attitude change or did their skill increase. Level 3 concerns Behavior or Transfer of Training. Can/Will/Are the skills be used on the job and does the environment affect the transfer of training. Level 4 equates to the final results or outcomes of the organization's performance. I would actually use this evaluation to determine the effectiveness of the Library and Information Systems at TAMUC (Texas A & M University Commerce) as it relates to my understanding of becoming a Librarian. When I look back on the Bachelor's of Science degree in education I received to become a teacher (even as I keep in mind I did not Student Teach), there was little correlation to the experiences in class to the experiences in the classroom. But I can already tell that I will use what I have learned so far because I find myself cataloguing personal books, becoming more techno savvy due to the increase use of technology required by my instructors.
Participatory/Collaborative Evaluation or the emphasis of participatory/collaborative forms, according to Preskell, of evaluation is engaging stakeholders in the evaluation process, so they may better understand evaluation and the program being evaluated and ultimately use the evaluation findings for decision-making purposes. This evaluation tool is similar to Formative Assessments. It determines what each stakeholders believes is necessary for the programs and determines whether each stakeholder sees value in the process or product. This type of evaluation determines the success and continuation of a program or the dissolution of a program according to the stakeholders. As a Librarian, I can see this as being an effective tool simply because if the library is not useful to the stakeholders, there is no point in having it as a full time position in a school.
Success Case Method, according to Preskell, approach to evaluation focuses on the practicalities of defining successful outcomes and success cases (Brinkerhoff, 2003). It uses some of the processes from theory-driven evaluation to determine the linkages, which may take the form of a logic model, an impact model, or results map. Evaluators using this approach gather stories within the organization to determine what is happening and what is being achieved. The major question this approach asks is, “What is really happening?” When we look at the library and the Librarian individually we determine success outcomes. If students are visiting the library, what is the cause? Is it the library itself or is it the inability of the librarian to make the library inviting? We must ask, was there a time when the library was used readily? If so why? If not, why? If we get to the root of any issues that keep the library or librarian from successfully instituting a Library Program that involves all stakeholders, then we can determine how success can be achieved.
When we refer to instructional design evaluation, we shouldn't just look at learner outcomes, but also teacher effectiveness. Teacher effectiveness isn't always related to learner outcomes. Students in Honors and AP classes have parent support and a natural ability to strive for excellence. But students who are not in these programs rarely have this. It is therefore imperative that teacher evaluations are included. We should ask such questions as: Are you making the lessons accessible to all students? Are you encouraging/helping all students become successful? Are you including interactive, interesting and a variety of assessment tools for all learners? Teaching and Learning affects both the teacher and the learner and assessments and evaluations must mirror this concept.
One particular non-instructional problem is discipline (although technically students act out because they are unfamiliar with the information being taught). Teachers can not be effective whole class if one or two students have disciplinary issues. Math teachers across the hall from me are told to have students spend more time discussing the Math and are given calculators to do simple calculations. They are not required to "think through" the math, therefore they have an inability to settle down during the actual individual work time to do the work. They become discipline problems. I contend, if we look at these students individually and teacher approaches individually, it is possible to end the discipline issues that relate to instructional weaknesses as as well as student's internal motivation. If we address students as individuals and focus on their individual performance system, we can combat any issues students have. If we analyze student performance and evaluate the causes of their performance, then we can intervene.
J Johnson, Burke R. and Dick, Walter. Evaluation in Instructional Design: A comparison
of Evaluation Models. Chapter 10. Trends and Issues in Instructional Design and
Technology. New York: NY. (3 ed)
Preskill. (2004). Building Evaluation Capacity. Evaluation Models, Approaches and
Design. http://www.sagepub.com/sites/default/files/upm-binaries/
5068_Preskill_Chapter_5.pdf
Ethel,
ReplyDeleteI think you did a great job giving a summary of each evaluation model and applying it to how you could use it in your position. I think all of the evaluation models have good components so I like the way you were able to make them fit you.
Thank you for giving the information on the two evaluation models that you researched. I did not research those so it was great to get to learn about them.
I especially enjoyed reading about how to address discipline. I agree with you that teachers should focus on the student's individual performance system to try to find ways to help them. If discipline is not taken care of, it is hard for learning to take place.
I found the Success Case Method really interesting in my research as well. However, I could find any specific examples that related to education in order to include that in my posts. But I do think it is a really good evaluation tool and would like to spend more time reading about it.
ReplyDeleteSuch a great point you bring up with behavior issues related to academic weaknesses. I certainly saw this with one of my students last year. It can become a vicious cycle unless we can get ahead of it.
Great post!
We are on the same page when it comes to evaluation. In your blog you mention that when we refer to instructional design evaluation, we shouldn't just look at learner outcomes, but also at teacher effectiveness. Teacher effectiveness isn't always related to learner outcomes but at a minimum a "good" teacher is willing to look at herself or himself to see if there is a relation. I love your original quote, "When in the course of human development we must expand, explore and then revise all that we have expanded and explored." I read it twice and it has stuck with me. Keep them coming!
ReplyDeleteEthel,
ReplyDeleteI agree with your statement about the correlation between teacher effectiveness and learner outcomes. As you stated, students with more home support may have more drive, but I don't believe that should lower the effort of the teacher. I think that all teachers should have expectations of effectiveness regardless of student background, Homelife, socio-economic status, etc. There is always a student or students that you can reach with your efforts.
I also believe that discipline/behavior issues arise when students are not engaged in their learning. As teachers, that is something we must keep at the top of the list when lesson planning. Keeping students engaged in learning, diminishes many behavior problems.
Great post!