The United States Department of Education (USDE) requires the state to develop an oversight program to monitor each of its local education agencies (LEAs). The Oklahoma State Department of Education (OSDE), Office of Accountability and Assessments, maintains the responsibility for the general supervision of compliance with Federal and State requirements for administering the State required assessments. In order to be in compliance with both state and federal statute and regulations, AAMP was implemented.
The academic assessment monitoring focus for the OSDE, in collaboration with the LEAs, is on the improvement of the educational performance for all students in Oklahoma and compliance with regulations as established by the USDE, Title I, Part A.
Monitoring is a regular and systematic examination of the LEAs efforts to implement critical requirements required by federal and state academic assessments. These quality assurance observations are intended to:
The monitoring program will apply to all public schools, districts, and other educational entities within the state that participate in the OSTP. These rules will affect the following groups:
The USDE requires the OSDE to observe/monitor at least 95 school sites annually for quality assurance. It is expected, that the OSDE will be monitoring approximately 106 districts and as many as 318 school sites per year.
To fulfill the quota established by the USDE, districts will be selected for monitoring based on a five-year random, generated list, using the web-based site, RANDOM.ORG. The service has existed since 1998 and was built and is being operated by Mads Haahr of the School of Computer Science and Statistics at Trinity College, Dublin in Ireland. In reality, most random numbers used in computer programs are pseudo-random, which means they are a generated in a predictable fashion using a mathematical formula and are often used in scientific studies <http://www.random.org/sequences/>.
Sequence boundaries for selection were set using the smallest value of 1 and the largest value of 532. 532 is the number of districts within the state at the time the five-year cycle was established. Generated integers were formatted in five columns, with each column representing one year of the cycle sequence. Using the randomized sequence order ensures duplication of values does not occur. A hand verification of the randomized order was conducted by the OSDE staff to ensure there was no duplication of values.
The numerical sequence was then matched to the alphabetical database listing of districts maintained by the OSDE in a spiraling fashion to determine the year of monitoring within the five year cycle.
In addition to the randomly generated list, the following can trigger selection of a district for placement on the monitoring schedule:
The OSDE may at any time, determine that a compliance review of the LEA is warranted to verify progress and data due to complaints or concerns.
Once selected for monitoring, OSDE personnel can utilize several types of monitoring, including, but not limited to, on-site and desk monitoring.
For on-site monitoring the Office of Accountability and Assessments staff will travel to the district which may include a visit with the LEA and will use the on-site monitoring checklist to conduct an observation and review of the district/site pre-administration, administration, and/or post-administration procedures of the assessments. This type of monitoring will direct its attention towards test administration procedures, including, but not limited to: security, staff training, administration of the assessments, review of student’s records who receive accommodations and modifications, or alternate testing, review of administrative records, or staff interviews.
Desk monitoring will not involve an on-site visit to the LEA. The OSDE will conduct this type of monitoring utilizing data and evidence submitted to the OSDE by the LEA, testing vendors, and/or OSDE department. Part of the review will involve comparison of district-reported information to state or vendor reported data.
To coordinate the on-site visit logistics, the district/site must provide the Office of Accountability and Assessments with a copy of their testing schedule five days prior to the opening of the testing window. At a minimum, the State will monitor one elementary school, one middle school, and one high school site within the same district to maximize resources and improve the quality of improved results. Charter schools within a district may also be chosen for monitoring.
On-site monitoring timeline/protocol:
a. The Office of Accountability and Assessments will notify the district ten (10) “state” working days prior to the opening of the testing window in which the district will be monitored and will include the type of monitoring as on-site monitoring.
b. The district/site has five (5) “state” working days to submit the site testing schedule to the Office of Accountability and Assessments. The schedule may be submitted through either e-mail or fax.
c. The Office of Accountability and Assessments will assign monitors and an observation period, notifying the site prior to the observation. Monitors will follow these guidelines when conducting the monitoring visits:
1. Assessment monitors will check into the site office, presenting proper identification.
2. When observing assessment activities, OSDE monitors will practice the principle of observation from a distance, with the understanding that the site staff needs to go about performing their job tasks while taking little or no notice of their observers, who likewise should be able to conduct their observation without being asked to participate in the administration in any way.
3. The majority of the assessment day activities will be easily visible to observers. Before and after the administration, the observer may walk amongst thedistrict/ site assessment personnel to view their work.
4. The State observers may request access to view documentation for students who are receiving accommodations on the assessments.
5. During the assessment, the observers will try to seat themselves where they can observe all assessment activities and complete the observation checklist while maintaining a comfortable distance between themselves and the site assessment personnel.
6. The observation may be extended after the conclusion of the assessment so that post assessment activities can be observed.
7. If district/site staff are not following assessment protocol, this will be noted on the observation checklist. The observer will only correct site staff or make comments about task performance prior to a testing session in order to insure proper testing procedures occur.
8. At the conclusion of the visit, observation feedback will be submitted to the OSDE, Office of Accountability and Assessments, using the checklist document (i.e., hardcopy or electronic version).
9. The section for Other comments may include the observers thoughts regarding: administration of the assessment, such as appropriate tone, management, and monitoring of the session; provision for security and confidentiality of test materials, and school and student information; any information which might require action during this assessment cycle; and overall impressions of the assessment administration.
10. Following a testing session, if an observation has been made that needs immediate attention, the Building Test Coordinator (BTC) or DTC will be notified (e.g., invalidation, cheating, improper test administration)
11. Completed checklists will be submitted to the OSDE, Office of Accountability and Assessment in a timely manner.
e. The Office of Accountability and Assessments will have 45 working days from the last day of the End-of-Instruction (EOI) online testing window to analyze submitted documentation and provide feedback to the district/site. These steps will be taken to review evidence received:
Desk monitoring timeline/protocol:
a. The Office of Accountability and Assessments will notify the district ten (10) “state” working days prior to the opening of the testing window in which the district will be monitored and will identify the type of monitoring as desk monitoring.
b. The district/site documentation must be received in the OSDE Office of Accountability and Assessments, within twenty (20) “state” working days from the last testing day of the EOI online assessment window. Documentation can be submitted in a hardcopy format attached to the Desk Monitoring Form.
c. The Office of Accountability and Assessments will provide monitoring results to the district superintendent within forty-five (45) “state” working days from the desk monitoring document twenty-day submission. These steps will be taken to review evidence received:
The Office of Accountability and Assessments will provide monitoring results to the District Superintendent. A letter sent to the district will inform the district of the monitoring status. The status, “in compliance” will be given if all requirements have been met and the evidence is complete and valid. If any monitored area is found deficient with additional monitoring or documentation being required, a “noncompliant” status will be indicated.
Districts/sites that provide complete and valid evidence of successful accountability during the monitoring process will return to the scheduled five-year cycle.
Districts/sites that do not provide complete and valid evidence of successful accountability will have some form of monitoring until evidence is obtained. Repeated noncompliance with State and Federal regulations may result in: repeated on-site visitations, suspension or loss of accreditation, and/or other sanctions as determined appropriate by the OSDE.
The checklist section titled, “District-Provided Documentation” must be completed by the district or site. The section titled, “State-Collected Documentation” will be completed by the Office of Accountability and Assessments.
Desk monitoring evidence can be submitted in either a hardcopy format or in an electronic format (CD or DVD) and as a site packet. A district/site may choose which format to submit documentation, yet ALL documentation must be in the same format.
Hardcopy format: A cover page should be included in the file. It should be on school letterhead and indicate (1) who has compiled the evidence, (2) the summer contact information for responsible person, and (3) the district/school’s contact information (if not on the letterhead). The written descriptions and evidence should appear in the same order as the items appear on the checklist. Send copies of sign-in sheets, security documents, etc., keeping the originals for your files.
Electronic format: A cover page should be included in the file. It should be on school letterhead and indicate (1) who has compiled the evidence, (2) the summer contact information for responsible person, and (3) the district/school’s contact information (if not on the letterhead). The electronic file must be submitted on a CD or DVD. Documents will not be accepted via email. The full County District code (6 characters) must be visible on the front of any disc, along with the district’s name.
Documents may be saved using any of the Microsoft Office 2007 or earlier file extensions. (Word 2007 or earlier, Excel 2007 PowerPoint 2007) Scanned images which contain signatures may only be saved as a pdf file. Documentation submitted as a TIFF or JPEG will not be acceptable. Districts/sites which do not use Microsoft Office may call the OSDE regarding file submission. (405) 521-3341.
Documentation should be saved within files. Files must be labeled according to the order an item appears on the Desk Monitoring Checklist. The file listing below, is an example of how to indicate file names which provide both a description of documents within the file and would appear in the order used by the checklist.
CAUTION: if a document is two sided, the two pages must appear in consecutive order within the same pdf file. It is the responsibility of the district/site to ensure all signatures on scanned documentation is visible, legible. Forms which are double sided include, but not limited to: Test Administrator/Proctor Security Form and the Student Cumulative Record forms.
Evidence should be specific enough to provide the reader with an accurate description of what procedures are in place. Providing only a copy of the State's policy and that it will be in effect is incomplete. Complete and valid evidence would include a written description indicating how the site implements the policy. If appropriate, steps can be bulleted or numbered instead of written in a paragraph. While not an all inclusive list, several examples are given below to help you think through how specific a description should be to provide complete and valid evidence.
Example #1: A description for providing procedures for use of calculators to ensure policy compliance would provide answers to these key questions:
1. Who will oversee the implementation of the calculator policy?
2. Which tests will you permit calculators to be used on?
3. Are the students using their own calculators, site provided calculators, or the online provided calculator tool?
4. If students are using a calculator other than the online version:
Example #2: A written description of a communication plan in case of emergencies would provide answers to these key questions:
1. If a student becomes ill during a testing session:
2. If there is a loss of power failure or technology problems during a testing session:
3. If an evacuation of the school facilities becomes necessary during a testing session:
Example #3. A written description of the procedure and testing plans for students who need testing accommodations would provide answer to these key questions:
1. If students needed additional time to complete the test:
2. If the student needs special accommodations:
Once the State notifies the district that it will be monitored, it is the decision of the district whether to inform the site that it has been selected for monitoring, the type of monitoring that will be conducted, and when monitoring will occur.
It is anticipated that most on-site visits will last one day.
If a district will be receiving an on-site monitoring visit, the district will receive a letter indicating which specific items on the checklist are to be provided in a hardcopy format to the OSDE monitor. Examples include, but are not limited to: sign-in sheets for training, written communication plans; a complete testing schedule for the site; security checklists; and nondisclosure forms, copys of student cummulative folders.
Additional resources that may assist in completing the process of your district’s review can be found on the OSDE’s Web site <www.ok.gov/sde>.
• Select “Site Index.”
• Click on “Assessment Administrator Resources”
• Click on “Academic Assessment Monitoring Program" (AAMP) in the blue quick link tab.