Section 3: Evaluating, Implementing and Managing Instructional Programs and Projects

1. Two Models for Evaluation

Flashlight Triad Model

Virginia Tech uses the Flashlight Triad Model of evaluation for technology oriented evaluations. The Flashlight Model comprises five steps:

1. Overview and Confronting the Blog: This step involves brainstorming to create a comprehensive list of elements that “feed into and flow” from a lesson or course. The examples that were given were: faculty development and preparation, student prerequisites, budget, assistance with course materials development, and student attitudes and satisfaction.

2. From Blob to Issue: Consider the purpose of the evaluation. Two questions should be answered:

1) What are the three most crucial things you fear might happen as a result of using the technology?

2) What are the three most crucial things you hope will occur?

One or two top issues should be selected from the answers and become the focus of the evaluation.

3. From Issue to Triad: For each issue that was selected, several triads are created. “A triad consists of the type of technology you might employ (T), a specific activity that the technology enables (A), and outcomes expected from that type of activity (O).” After several triads are created, the ones that will be focused on are selected.



Sample Triad
http://www.edtech.vt.edu/edtech/id/eval/eval_models.html



4. From Triad to Data: For each triad that was selected, questions are generated to gather data through surveys, focus groups, interviews, etc. At least five types of questions can be generated: 1) Technology (related to technical infrastructure, training, function, etc.), 2) Interaction of Technology and Activity (related to technical systems and their ability to support the specific activity), 3) Activity (related to the nature of the student tasks), 4) Interaction of Activity and Outcomes (related to the capability for technology-based activities to generate predicted outcomes), and 5) Outcomes (related to student learning, satisfaction, and student suggestions)

5. From Data to Next Steps: Based on the evaluation data, decisions will be made about modification and improvement for the course/lesson.

Source: http://www.edtech.vt.edu/edtech/id/eval/eval_models.html

My Implementation of the Flashlight Triad Model:  Since technology implementation is such an important part of instruction, this is an excellent tool to evaluate the use of those technologies in the learning process. I have just started to use blogging in my classroom as a means for students to express their ideas and opinions on specific topics. In my evaluation of the use of blogging using this model, I would first focus on student attitudes and satisfaction. For the issues, I would select student motivation and active learning. The triad I would use is as follows:

Technology	Activity	Outcomes
Blogging	Expression of ideas, thoughts, and reflection to improve writing; sharing ideas to support different perspectives	Students demonstrate more ownership in their writing; are more aware of checking their writing for grammatical and spelling errors before posting; gain different perspectives of ideas through reading other blogs.

From this triad, I would generate survey questions to ask students through an online survey. Based on the data that is collected, I can make a decision as to whether blogging is being beneficial for students in my classroom.

Dessinger-Moseley Full Scope Evaluation Model

The Dessinger-Moseley (1998) model of evaluation integrates evaluation throughout the ISD model. It includes six integrated evaluation activities:



Recreation of Figure 1-1 from Confirmative Evaluation: Practical Strategies for
Valuing Continuous Improvement by Dessinger and Moseley (2004)

 

• Formative Evaluation focuses on the need for training, instructional design and development processes and products, reaction during and after training, and accomplishment during training.
• Summative Evaluation focuses on the immediate results of program implementation: reaction during and after training, accomplishment during training, and the self-reported expectation that new knowledge, skills, and attitudes with transfer to the job and affect workplace performance.
• Confirmative Evaluation focuses on the program’s continuing impact and value, as well as the long-term transfer of knowledge and skills to the job and workplace.
• Meta Evaluation focuses on all the evaluation types—their inputs, outputs, outcomes, and focus.

Source: Dessinger, J., & Moseley, J. (2004). Confirmative Evaluation: Practical Strategies for Valuing Continuous Improvement. Pfeiffer & Co.

My Implementation of the Dessinger-Moseley Full Scope Evaluation Model:

The use of wikis has been something that I have experimented with using in the classroom as a collaboration tool.

• For the formative evaluation, I wanted to create a field test evaluation that would allow me to try using a wiki on a small scale. I started with a small project where students had to create an online survey using SurveyMonkey.com. They then had to post their survey link in a wiki for other students to easily find and take the surveys. I wanted to see if the students could manage the wiki, see its relevance in use, and find areas to improve its use in the future.
• For the summative evaluation, I asked student groups what they liked and disliked about using the wiki during the project. I had made the wiki editable only by members. So each student had to create a log-in in order to edit. Many students forgot their passwords (which seems to be the case even after we discuss the creation of log-ins and passwords), which took up time to reset passwords. I will have to have a better system in place for students to create log-ins and passwords. I am about to implement the use of wikis once again for students to create Excel charts to share with the whole class. Students will create an Excel chart comparing educational, income, and other statistics of Sulphur Springs, TX with two other cities of their choice. They will post their Excel charts to their class wiki to share and discuss findings. At this point, I will continue summative evaluations having students answer an online survey to find their reaction to the use of the wiki. I will be able to evaluate their accomplishment of using the wiki when all students successfully post and respond to the charts.
• For the confirmative evaluation, I will create projects that will give students a choice of collaboration methods (with the wiki being one of the choices) to see if students will choose to use this as a means of group collaboration.
• For the meta evaluation, I can use data collected from all evaluations made during the course of using class wikis to determine whether wikis should be a tool that I encourage students to use in the future.

2. Technological Innovation — Perceived Attributes



Microsoft Office Image



A technological innovation that was introduced last semester at our school was an online tardy tracker. The tardy tracker was developed by our technology specialist as a better way of keeping track of student tardies. We have used an online attendance method for several years, but teachers were still using a paper based method of tracking tardies. Teachers had to write down students’ names when they were tardy and physically take the tardy sheet to the office at the end of each day. Secretaries in the office had to then manually enter the tardies for the day into a database to be able to generate reports for the administrator that handles student tardies.

According to the concept of perceived attributes (Rogers, 2003), there are five key attributes that potential adopters will base their feelings about an innovation. They are:

Relative Advantage: In the case of the new online tardy tracking system, the immediate relative advantage was less paperwork for the teacher. Teachers were very responsive to using the new tardy system because of this attribute of the innovation.

Compatibility:  The new tardy tracking system is easy to access online through our local area network. Teachers are able to open the tardy system and our attendance system to easily work between in recording student attendance and tardies each class period.

Complexity:  The online tardy system is very simple and easy to use. It requires the same information that was needed in paper form, but is much easier to complete since a database with student information is connected to the system. The date for that day is automatically selected when the system is opened. As a teacher begins typing the student’s last name, a list of possible students appears. After selecting the correct student, the student’s ID number and grade are added to the form automatically. The teacher then selects the class period, enters the time the student arrived in class, and hits submit.

Trialability: We were not really given a trial period before the online tardy system was adopted. However, teachers had been requesting a way to enter tardies online for some time. With this request, our technology specialist created this program and tested it thoroughly before it was implemented.  

Observability: The observed benefits was not only less paperwork for teachers, but reduced work for the secretaries who had previously had to manually enter all paper tardy slips turned in by the teachers each day. The new system is accessible to everyone on the network. Reports with accumulated student tardies can be automatically generated for each student per class period and per day for the administrator to issue punishments for excessive tardies. Teachers can also look back at their own tardy report at any time to see the tardies that were entered for the day.

3. Situational Leadership

http://managementlessons.wordpress.com/2008/06/04/situational-leadership-model/

As a teacher, I completely understand resources being scarce. With budget cuts, resources are becoming slimmer and slimmer for educators.  I actually see administrators turning more to “in-house” training and staff developments to cut costs. If I were assigned the task of developing a series of professional development sessions focusing on technology use in the classroom for teachers, I would facilitate the project using the Situational Leadership model. My basic personality behavior is that of a follower. I really do not like to lead, but I love working with a team and have good interpersonal skills. From what I understand, the Situational Leadership model is based more on a side-by-side team approach to management rather than a top-down approach. To be effective, the manager has the task of achieving organizational goals by adapting the leadership style and resources based on the analysis of specific situations.  In developing the professional development sessions, I would proceed as follows:

Build the Team: First I would ask other interested colleagues to join the team. Based on the type of training, I would try to pull in those people with experience in those areas.  My team for the professional development sessions based on technology use in the classroom would consist of:



Project Team Communication Pattern
Created using PowerPoint Diagrams

 Phase 1:  After talking with each team member, I would need to decide the experience and confidence level of the team as a whole.  I feel that most of my team members are experienced and have the confidence because of their experiences to be able to work efficiently towards meeting the goals. If however in the initial planning meeting I sense that there is some uncertainty, I must assume the leadership style of being a directive and thorough manager without being too demanding.

Phase 2:  As we continue to meet to plan, organize, and develop the professional development sessions, the team should build confidence and begin to “buy-in” to the goals of the team.  In this phase, I will assume a more coaching leadership style by explaining, advising, and supporting decisions made by the team.

Phase 3:  In this phase, focusing on results is critical. My style of leadership must move to being that of a supporting motivator—encouraging the team and rewarding them for effort and production.  I would ask the client for team members to have additional time during the school day to work together as well as provide lunch for team members.

Phase 4: The team at this point should feel comfortable in working together. As the manager, I should now be able to step back and take the role of a monitor. The decision making and problem solving aspects of finalizing the project can now be proficiently completed by the team members.