Assignment 5

 Original Survey   Click here for version 1

 Revised Survey  Click here for version 2

 

Survey Explanation

In planning an evaluation design for the Teen Homework Help Program, the ‘Survey of Homework Skills and Study Habits’ was created. This measure is intended to be used as both a pre- and post-test assessment and will serve two purposes. Foremost, the tool will provide the evaluation team with evidence of the program’s success. One of the primary goals of the program is to improve students’ homework and study habits, therefore designing a measure that assesses the extent of this effect is the survey’s primary intent. Secondly, the survey questions are designed to provide client information that will assist tutors in designing a suitable program for their students.  Although this last element is not productive to the evaluation itself, I believe the types of questions will be invaluable for tutors, beneficial for stakeholders, and the questions are well-suited to this type of introductory survey.   

The questionnaire was tested with four individuals and their feedback was helpful in ensuring that the items met the survey’s goals.  In summary, their suggestions assisted in identifying flaws, ensured that the multiple choice options were more focused, and made certain that the short answer questions allowed for the identification of the student’s difficulties.   

The following changes were made to the initial survey:

SECTION: ABOUT YOU

Question 5: Do you currently receive resource assistance?

The term ‘resource assistance’ was unknown to two of the individuals who took the survey, therefore, the question was changed to:  Do you currently receive extra support at school? (Resource assistance, tutoring, etc.)

SECTION: STUDY AND HOMEWORK HABITS

Question 3: I complete my homework assignments on time.

A sixth option was added to the selections – In only the subjects that I like.

Question 6: I begin a long term assignment shortly after it is given to me instead of waiting until the night before.

Since this question only provided the student with two options, it was changed to: I begin a long term assignment...(2 or more weeks before it is due, a week before, a few days before, a night before.)

Question 8: I get help at home staying organized and completing homework.

This question was asking the student to comment on two aspects of their habits, so the ‘staying organized’ was taken out.

Question 9: I get help from teachers staying organized and completing my homework.

As with number 8, the ‘staying organized’ was taken out and the words ‘get help’ were changed to seek help.

Question 12: I know what my homework is.

This question was simply changed to I know what my homework assignments are.  

Question 15: I can find things in my locker.

This question was taken out entirely, as the individuals who took the survey suggested that the answer to this question would not provide the interpreter with useful information regarding the student’s homework skills and study habits.

Question 16: I ask friends to study or do homework with me.

This question was also removed, as asking friends to study with you may not necessarily be a good study habit. Instead, to get information regarding the student’s study skills, the question was changed to: What study skills do you regularly use? Check all the apply. (highlighting, writing notes, flash cards, rereading text, study groups, other)

Questions added to this section:

Two questions were added to this section in order to get more specific information on how the students felt about their homework skills and study habits.

Question 18: How effective do you believe your study skills are? (very effective, mostly effective, somewhat effective, not effective)

Question 19: Do you wish you had more effective homework skills and study habits? (yes, no)

 

SHORT ANSWER

Question 2: What subject or aspect of studying and completing homework do you struggle the most with?

Since this original item was asking two questions, it was broken into two questions: What subject do you struggle the most with? What aspect of studying and completing homework do you struggle the most with?

Question 3: Why are you seeking help from the homework program?

This question seemed to have an obvious answer, and would not provide the tutor with useful information regarding the student’s habits. The question was changed to: Describe your study routine.

Question 4: What do you hope to achieve by receiving tutoring assistance?

The word ‘specifically’ was added to this question as well as some sample descriptors in brackets. What do you specifically hope to achieve by receiving tutoring assistance? (improved grades in a subject area, improved study skills)

Seeing as some of the questions were adapted from an on-line survey, the revised copy of the questionnaire acknowledged the source of the survey at the bottom of the final page.    

 

  

Assignment 3 and 4

 

Logic Model Explanation

The Teen Homework Help Program is conducted in partnership between Saskatoon’s Frontier College and the Saskatoon Public Library. The purpose of the program is to provide support to learners ages 12-18 through the use of volunteer tutors. There are three goals attached to the program. 1) Through one-to-one tutoring, volunteers will assist students who are having difficulty with reading, writing, math, comprehension, completing assignments and studying. 2) To match youth learners with volunteers from post-secondary institutions as well as dedicated community members. 3) To familiarize students with Saskatoon Public Library resources and services that can assist them in their studies.

The program is offered in seven of the Saskatoon Public Library’s eight locations. Trained, screened (by Frontier College) volunteer tutors are matched with registered (by Frontier College) students to meet at one of the libraries on specific days of the week at specific times. Seeing as the tutors are primarily university students, the program follows the U of S academic calendar and runs from October to March.

The evaluation assessment will be focused on the immediate results, or outputs of the program. In relation to the project activities, the evaluation will be seeking to assess the impact that the program has on student learning. This will entail tracking participant attendance and seeking evidence of improved academic grades and skills related to homework and studying. In addition, the assessment will be judging the opinions of teachers, parents, tutors, and participants themselves regarding the youth’s success.

Looking to the intermediate results of the program (1 to 3 years later), the outcomes are directly related to the effects on participants. It is the hope of those involved in the program that the youth have maintained their study habits and skills, that they have become patrons of the library and view it as a resource for their learning, and that the participants more readily seek community support when required.

The long-term impact of the Teen Homework Help Program is focused on the effects that the program has had on the community and the services it provides its youth. One of the positive impacts of the program will be an increase in community services that meet youth needs, providing them with opportunities for academic support, growth and development.

 

Click here for my Logic Model

Click here for the Program Evaluation Worksheet

 

Assignment 2

Assignment 2: Program Case Study

            The case study for evaluation is a prenatal fitness program designed to promote regular physical activity in Aboriginal women, in hopes of reducing their risk of developing gestational diabetes mellitus (GDM).  69 women from Saskatoon participated in the program, some of whom had a previous case of GDM. The program was held once a week and provided participants with free fitness instruction, as well as the opportunity to socialize with other mothers and receive educational materials and advice on health related issues.    

            To evaluate this program, I would utilize Stufflebeam’s CIPP model.  This would be a summative report as the program was completed in 1997 and the purpose of the evaluation would be to assess whether the program succeeded in meeting its objectives. Stufflebeam’s model is comprehensive and seeks information regarding a program’s context, input, process, and product. It is for these qualities and specific framework that I believe this model would be best suited in supplying stakeholders with the details needed to make informed decisions regarding the program’s merit.      

Evaluation Outline – CIPP Framework

Context – to assess the needs and problems associated with pregnant Aboriginal women and clearly understand the objectives for the program, I would interview program leaders and stake holders to gain further insight into the problem.  In addition, I would assess and confirm the background information that was collected on the women regarding their medical history and risk factors.  It would be valuable if baseline data regarding the women’s fitness levels was collected in order to compare to at the completion of the program.

Input – to determine the cost-effectiveness of the program and evaluate its strategies for implementation, a cost-benefit analysis would have to be conducted in order to determine if the program could be sustained in the long-term. To evaluate the program plans I would seek out similar programs to compare the model to.

Process – to evaluate the program’s activities I would interview or create a questionnaire for the beneficiaries and program facilitators in order to receive their assessment of the program’s implementation.  Questions that I would pose in the assessment would include: How did participants hear about the program? Did the once a week session meet their needs or should it have been held more often?  Did women benefit from the structure of the program including the educational materials provided and the opportunity to socialize with other mothers?      

Product – to identify and assess short and long term outcomes, I would collect both qualitative and quantitative data through interviews, questionnaires, and/or surveys. If baseline data was conducted at the onset of the program, the women would be given a post-test to determine if their fitness level had indeed improved.  The following questions would assist in this product evaluation:  Did the women continue to be physically active following the program’s completion? Did they develop GDM or type 2 diabetes? Did their children? Permitting time, cost, and resources, incorporating the use of a control group would be helpful in increasing the validity of the evaluation.

References

Dyck, R. F. (2003). Description and evaluation of a prenatal exercise program for urban Aboriginal women. Canadian Journal of Diabetes, 27(3), 231-238.

Stufflebeam, D. L. (2003, March). The CIPP Model for Evaluation: An update, a review of the model’s development, a checklist to guide implementation. In Annual Conference of the Oregon Program Evaluators Network (OPEN) Portland, Oregon.

             

Assignment One

ECUR 809

Program Evaluation Module 1: Assignment 1

Lauren Willey

P.A.R.T.Y. Program Evaluation 2006-2008

Program Overview

The Saskatoon Health Region P.A.R.T.Y. (Prevent Alcohol and Risk-Related Trauma in Youth) Program is a prevention and awareness program that is directed at youth to reduce preventable injuries and death.  With a goal of promoting smart choices and safe risks in high school students, this one day, in-hospital program exposes teens to the dangers associated with drinking and driving, riding without use of a seat belt, and not wearing a bike helmet. Put on by the Acquired Brain Outreach Team, the P.A.R.T.Y. Program runs in approximately 30 communities across Saskatchewan.   

Evaluation Model

            Following Stufflebeam’s CIPP model of evaluation, the purpose of this formative assessment was to determine the program’s effectiveness in relation to their short-term goals.  While the CIPP model defines evaluations as providing information regarding context, input, process, and products, this particular evaluation focused solely on the program’s product.  Stufflebeam and McKee (2003) noted that product evaluations could be divided into subcategories. In the case of the P.A.R.T.Y Program, evaluators used an effectiveness model to judge the success of the program. Motivated to assess outcomes and gage the program’s effectiveness in meeting their needs, the evaluators sought to decipher whether students’ attitudes and knowledge around risk taking behavior changed after attending the one day session. It was determined that the P.A.R.T.Y. Program is beneficial in improving students’ attitudes and risk taking behavior in the short term, but that further evaluations would have to be conducted in order to effectively assess the intermediate and long-term outcomes of the program. 

Method

            In order to assess the short-term outcomes of the program, the evaluators chose a quasi-experimental, pre-post design, with no control group.  408 grade 10 students from both urban and rural high schools were given a questionnaire one week prior to attending the program and were provided the same survey one week following their participation. The questions were a mix of attitudinal and knowledge-based questions surrounding injury and risk, with both qualitative and quantitative data being collected.

Strengths of Evaluation

            Although there are few strengths apparent in this program assessment, evaluators attempted to meet certain objectives suggested for an effectiveness evaluation. The first of these is that the results of the questionnaires were analyzed using statistically sound methods, helping contribute to the validity of the results. Secondly, the evaluators compared assessments of other P.A.R.T.Y. programs in an attempt to judge the effectiveness of the Saskatoon Health Region’s.  A last strength is that the evaluators acknowledged the weakness of their design and included recommendations for future modifications.    

Weaknesses of Evaluation

There are many changes that would need to be made to the evaluation and approach in order to make it reliable and beneficial to stakeholders. The primary weakness of the evaluation is that there were no suggestions made for strengthening and improving the program, a vital component of a formative assessment. Instead, the conclusion of the evaluation described the modifications that would need to be made to the assessment’s approach in order to increase the validity of the findings and provide a more accurate appraisal of the program’s outcomes.  These suggestions included (a) using a control school to compare results to; (b) using a one year follow up time frame instead of two weeks, and (c) going beyond the questionnaire by assessing behavioral change using the driving records of those who participated in the program.  The latter recommendation would be beneficial as changes to students’ attitudes and knowledge surrounding risk-taking behaviors does not necessarily translate into a reduction in risk taking behavior.  A final weakness to the report is that there was no mention of the program’s stakeholders or who the audience of the assessment would be.

In conclusion, the evaluation was not well planned or executed. To provide a more reliable and valuable assessment in the future, the evaluation should adhere to the recommendations provided, as well as offer stakeholders feedback for program improvement.

References

Kershaw T., Marko J & Gerwing, J. (2010). P.A.R.T.Y. Program Evaluation 2006-2008. Saskatoon: Saskatoon Health Region.

Stufflebeam, D. (2002). The CIPP model for evaluation. Evaluation models, 279-317.

 http://www.saskatoonhealthregion.ca/your_health/documents/PHO/PartyProgramEvaluationReport.pdf