Program to be Evaluated: The Diverse Learning Teacher Program

Engage Stakeholders

Who should be involved?

Primary Stakeholders

  • School Administration team (Principal, Vice Principal, and Assistant Principal)
  • The Diverse Learning Teachers (DLT)
  • Classroom teachers
  • Teacher Assistants
  • Students directly affected by the DLT program (those on Individualized Program Plans, S.A.P.’s, etc)
  • Parents directly affected by the DLT program (those who have day-to-day interactions with DLT’s)

Secondary Stakeholders

  • Non-classroom teaching staff (option teachers, counselor, Career and Technology Studies teachers)
  • Students not directly affected by the DLT program
  • Parents not directly affected by the DLT program
  • DLT Consultants (Central Office)
  • DLT’s at other schools

How might they be engaged?

The primary and secondary stakeholders will be engaged through a series of surveys, interviews and consultations.  The data from these assessment tools will be aggregated into a final recommendation report.  Some of the primary stakeholders will also participate in the creation of the surveys and data collection.  Teaching staff will be asked for anecdotal feedback to the DLT program, and students will provide feedback on their experiences in the DLT program.  DLT Consultants may be ask to provide information on the District’s vision of how DLT’s are to operate at their schools and work-sites.  DLT’s operating at other work-sites will provide information on how the DLT program works at their school.

Focus the Evaluation

What are you going to evaluate?  Describe the program (logic model).

The program being evaluated is the Diverse Learning Teacher program that was introduced to our school September 2011.  The Calgary Catholic School District recently decided to adopt a path that includes Universal Design for Learning (UDL) and the Alberta Education document “Making a Difference: Meeting Diverse Learning Needs with Differentiated Instruction”.  The DLT program replaced the former model of the Resource Teacher, the English as a Second Language teacher, and any other ‘resource’ teacher with a single, unified position – the Diverse Learning Teacher.  Replacing the former model with the new model required individual schools to restructure the way support services were delivered to teachers and students.  Each school chose to model the DLT program differently – keeping their unique situations in mind.  In our school, the changes were far-reaching.  There was a change in physical space (where the DLT’s were housed), time-table changes, and most importantly, a change in “the way we do business”.  The DLT’s were given extensive professional development at the beginning of the school year.  One DLT is the DLT Coordinating Teacher (DLT-CT) and is required to attend a matrix meeting once in a 6-day cycle.  Information gathered at these meetings is shared amongst the other DLT’s at the school.

One far-reaching change from the previous year is that the barriers between Resource Teachers and English as a Second Language (ESL) teachers have been removed.  The DLT’s are now in a position to support ALL students, regardless of their needs.  These staff members also support the classroom teacher to deliver quality programs that adopt the philosophy of Universal Design for Learning (UDL).  The end-goal is to meet the unique requirements of every student – not just those who have been identified as ‘ESL’ or coded.

What is the purpose of the evaluation?

The purpose of this evaluation is to gather feedback on current practice – the ‘what is going on right now’ at the school.  The evaluation will gather qualitative and quantitative data and make recommendations on what is working best, what is not working, and what is being done that can be done better.

Who will use the evaluation?  How will they use it?

The evaluation will be used by the Administration Team and the DLT team at the school to make revisions and changes to the program for the 2012-2013 school year.  It will be used to improve DLT service provided to students, classroom teachers, and parents.

1. Administration – Administration will use the report to make staffing and timetabling decisions.  They wil also use the information to make decisions regarding office space, student/teacher workspace, and will have input as to how the over-all program is deliver at our local school.

2. DLT – DLT’s will use the recommendations to keep or change current practices.  This information will help in their day-to-day interactions with teachers, set-up Teacher Assistant schedules, student and classroom timetabling, and to stream-line and make their work more efficient.

3. The DLT Consultants and the District – This group will use the information in the report to see how things are working at our school – to gather a snapshot of information.  The results in the report may be shared with other schools via DLT consultants to make improvements and enhancements both at the local and district level.

What questions will the evaluation seek to answer?

1. Is the current set-up and DLT program working?

2. Are the DLT’s properly scheduled?

3. Do the DLT’s feel they have the proper resources to do their jobs as expected?

4. What improvements, if any, can be made to the existing model of DLT?

5. Do students feel supported in their learning?

6. Do parents feel their children (especially those with special needs or ESL) feel supported by the school?

7.  What recommendations can be made for next year?

What information do you need to answer the questions? What will I need?

  • District
    •  Initiatives mandated from the District
    • Directives coming down from central office
    •  Best-practices and current research in UDL
    • What other DLT models are operating at schools throughout the district
    • Staffing (the number of DLT’s assigned to our school)
  • School Administration
    • Scheduling/Time-tabling
    • Space allocations
    • Support staffing allocations
    • Their vision and understanding of the DLT program
    • Supports in place for the classroom teacher and professional development in UDL
  • DLT
    • Current time-tables
    • A list of current duties
    • Minutes from meetings
    • Schedules
    • Anecdotal notes
    • Student records (SRT’s, IPP’s, SAP’s)
  • Teachers
    • Support schedules
    • Anecdotal notes
    • Teacher plans
    • An assessment of technology skills
  • Parents
    • Informal discussions
    • Feedback forms (parent forms from student program plans)
  • Students
    • Informal discussion
    • Student records
    • Anecdotal notes from teachers and DLT’s

When is the evaluation needed?

The evaluation needs to be completed (including a recommendation report) by the end of April.  Administration and DLT’s will need the information so that proper planning, staffing, and accommodations can be made for the 2012-2013 school year.

What evaluation design will you use?

The evaluation will be formative.  This is the first year of the DLT program and it is scheduled to continue for the foreseeable future.  As this is the rookie year for the program, a proper evaluation will help guide the DLT program for the 2012-2013 school year.

The model that I will use is Sufflebeam’s CIPP model.  This is an ongoing process and because it continues for the next school year, the program needs to be evaluated and recommendations need to be made.  The following questions will be used to guide the evaluation.

“What needs to be done?”

“How should it be done?”

“Is it being done?”

“Did it succeed?”

Collect the Information

What sources of information will you use?

  • Existing Information
    • Previous Resource/ESL models
    • Current DLT models at other schools
    • Resources put out by Alberta Education on UDL and Differentiated Instruction (DI).
  • People
    • DLT’s
    • Administration
    • Teachers
    • Students
    • Parents
  • Pictorial records and Observations
    • Observations of DLT and student interactions
    • Sit in on meetings between DLTs and administration
    • Possibly attend a Day 6 Matrix meeting of DLT Coordinating Teachers (DLT-CTs)
    • Observations of DLT / Teacher planning.

What data collection methods will you use?

  • Surveys
  • Interviews (both informal and formal (video recorded))
  • Observations
  • Photos
  • Document Review
  • Journal/Reflection

Instrumentation: What is needed to record the information?

Surveys

  1. Staff surveys
    1. Teachers
    2. Support staff
  2. Student survey (general student population)
  3. Parent survey (students directly impacted by the DLT program).

Interviews with:

  1. Administration
  2. DLT’s
  3. Students directly impacted by the DLT program

Observations/Photos of:

  1. Student performance
  2. Classroom operations
  3. Teachers teaching
  4. Teacher/DLT interactions
  5. Student/DLT interactions

Document Review:

  1. Individualized Program Plans (IPPs)
  2. English as a Second Language folders (SAP’s)
  3. Student Resource Team meeting notes (SRT’s) and SRT forms
  4. District job descriptions for DLTs and DLT-CT’s.

Journal/Reflections

  1. DLT entries
  2. Student entries
  3. Teacher entries
  4. Administration entries

When will you collect data for each method you’ve chosen?

  • Surveys
    • Start: Mid-March  End: End of March
  • Interviews
    • Mid-March
  • Observations
    • On-going  – Beginning of March to end of March
  • Document Review
    • Beginning of March to Mid-March
  • Journal Entries/Reflections
    • End of March – Beginning of April

Will a sample be used?

A sample will not be used.  The program is systemic and district wide.  It will be impossible to find a control group.  The assessment tools will be constructed to evaluate the entire population they are suited for (ex. The student survey will be made available to all students in the school).

Pilot testing: when, where, how?

There will be no pilot testing.  All assessments will take place in the general population.  The assessment tools will be designed specifically for our DLT program, however, certain items (such as survey questions, or interview questions) can be dropped so that the assessment tools can be used by other schools to evaluate their DLT programs.

Analyze and Interpret

How will the data be analysed?

  • Surveys  – Results will be tabulated.  Data will be synthesized into graphs and examined for trends.
  • Interviews – Interview responses will be collated into a report.  Trends will be monitored.
  • Observations – Notes will be taken during observations.  This data will be collated into a report and cross-referenced with interviews.  Trends will be monitored.
  • Document review – Documents will be organized, read, and summarized.  These documents will be compared and contrasted with the interview and observation results to find commonalities and discrepancies.
  • Journal Entries – the evaluator will review entries and reflections. The evaluator will look for commonalities and discrepancies between the journal entries, observations, and survey data.  Data retrieved from the Journal entries will help guide the evaluator in making recommendations for future planning.

Who is responsible to analyze the data? 

Because of the sensitive and far-reaching nature of this program, only the evaluator will analyze the data and make recommendations.

How will the information be interpreted?

The information will be interpreted by the evaluator.  Other program evaluators may be asked to help review the data and the final report.

 

Use the Information

How will the evaluation be communicated and shared?

The information will be presented in 2 phases.  Phase 1 will be presenting the information to the DLTs and Administration.  The report will be presented in a private room with these two groups in late-April (when the report is complete).  The presentation will display data from the surveys.  Results from the individual responses will be presented as quotations.

Phase 2 will involve presenting the results (data and anecdotal notes) to the general teaching staff.  This will be done at a staff meeting (end-April to mid-May).

 

Manage the Evaluation

Human Subject Protection

All data being collected is being completed by district staff and the evaluation is being completed for district staff.  Since all information is being kept internal, no release forms will be necessary.  Should there be a need to release the information beyond the confines of the Calgary Catholic School District, release forms will be completed.

Time-Line

  • February – Complete the Planning Program Evaluation Worksheet
  • Late-February – Develop Surveys and Interview questions
  • Early-March – Administer surveys, conduct interviews
  • Late-March – Conclude and tabulate surveys, synthesize interviews
  • Late-March – Early-April – Complete report
  • Early-April – Share evaluation.

Responsibilities

Data collection, analysis, and the final report are the responsibility of the evaluator.  Students surveys may be administered by homeroom teachers, and staff and parent surveys can be administered by school administration.

Budget

As this is a school based evaluation being completed by school personnel, no budget is being provided for the evaluation.

 

Standards

Utility

This program evaluation can have far-reaching impact.  The DLT program is new in our district this year and seeing “What we’re doing right” and “What we can do better” can provide strong guidance to the staff at my school and result in meaningful impact in our students’ academics and well-being.

Feasibility

This is a large undertaking and may require more than one evaluator.  As further materials are developed (surveys and interview questions) it may become more evident that the evaluation may need to take a narrow focus.

Propriety

All materials being developed, evaluation, and results will remain property of the Calgary Catholic School District.

Accuracy

Assessment materials will be looked over not only by the evaluator but also by a fellow program evaluator for the “sober second thought”.  These reviews will be completed before the assessment materials are administered to staff.  This will help minimize any evaluator bias.