AAMP - Frequently Asked Questions


Frequently Asked Questions (FAQ)
Academic Assessment Monitoring Program (AAMP)

Subchapter 13 Oklahoma School Testing Program (OSTP)
Section 21 Monitoring

1. Why has the OSDE implemented an assessment monitoring program?

The United States Department of Education (USDE) requires the state to develop an oversight program to monitor each of its local education agencies (LEAs).  The Oklahoma State Department of Education (OSDE), Office of Accountability and Assessments, maintains the responsibility for the general supervision of compliance with Federal and State requirements for administering the State required assessments.  In order to be in compliance with both state and federal statute and regulations, AAMP was implemented.

2.  What are the specific Federal and State regulations which guide the monitoring program?
  • Student Achievement and School Accountability Programs (SASA) for Formula Grant Programs;
  • Elementary Secondary Education Act (ESEA) Section 9304(a)(1) requires that a State Education Agency (SEA) ensure that programs authorized under the ESEA are administered in accordance with all applicable statutes, regulations, program plans, and applications;
  •  ESEA Section 9304(a)(3)(B) requires that an SEA adopt and use proper methods of administering each ESEA program, including the correction of deficiencies in program operations that are identified through audits, monitoring, or evaluation.
  • Section 80.40 of the Education Department General Administrative Regulations (EDGAR) requires an SEA to monitor grant and sub-grant activities to ensure compliance with applicable Federal requirements;  and
  • Title 70 O.S., § 1210.505 Student Assessment, Oklahoma School Testing Program (OSTP), and Oklahoma Administrative Code (OAC) 210:10-13-2,4, 6, 7, 9, 10, 11, 18, 20, and 21 the OSDE shall establish and implement the AAMP to evaluate school district implementation of both Federal and State required academic assessments.
3.  What is the intended effect of the program? 

The academic assessment monitoring focus for the OSDE, in collaboration with the LEAs, is on the improvement of the educational performance for all students in Oklahoma and compliance with regulations as established by the USDE, Title I, Part A.

Monitoring is a regular and systematic examination of the LEAs efforts to implement critical requirements required by federal and state academic assessments. These quality assurance observations are intended to:

  • Ensure that all eligible students are tested, proper training of school district staff is conducted, test security is maintained, and assessments are administered consistently and in a standardized manner as mandated in the OAC 210:10-13-2, 4, 6,, 7, 9, 10, 11, and 21;
  • Ensure that the OSDE receives from districts data of the highest quality, as mandated in the OAC 210:10-13-4, 18, and 20; and
  • Assist the staff of the OSDE to better advise and partner with districts regarding accountability and assessments.
4.  Who will be affected by the monitoring program?

The monitoring program will apply to all public schools, districts, and other educational entities within the state that participate in the OSTP.  These rules will affect the following groups:

  • OSDE, Office of Accountability and Assessments will establish the guidelines and authority to set up monitoring cycles, determine which districts will be monitored yearly, and conduct on-site visits and desktop monitoring at the district and site level.
  • District/Site Level will be held responsible for full compliance to previously established rules and regulations regarding test security, administration, and training as established by all sections of OAC 210:10-13, through accreditation sanctions.
  • District superintendents or their appointee will be responsible for submitting all required documents and evidence to the Office of Accountability and  Assessments in the correct format and by the set timeline.
5.  How many districts or schools will be monitored yearly?

The USDE requires the OSDE to observe/monitor at least 95 school sites annually for quality assurance.   It is expected, that the OSDE will be monitoring approximately 106 districts and as many as 318 school sites per year.

6.  What process will be used to determine which districts/sites are selected for monitoring?

To fulfill the quota established by the USDE, districts will be selected for monitoring based on a five-year random, generated list, using the web-based site, RANDOM.ORG.   The service has existed since 1998 and was built and is being operated by Mads Haahr of the School of Computer Science and Statistics at Trinity College, Dublin in Ireland.  In reality, most random numbers used in computer programs are pseudo-random, which means they are a generated in a predictable fashion using a mathematical formula and are often used in scientific studies <http://www.random.org/sequences/>.

Sequence boundaries for selection were set using the smallest value of 1 and the largest value of 532.  532 is the number of districts within the state at the time the five-year cycle was established.  Generated integers were formatted in five columns, with each column representing one year of the cycle sequence. Using the randomized sequence order ensures duplication of values does not occur.  A hand verification of the randomized order was conducted by the OSDE staff to ensure there was no duplication of values.

The numerical sequence was then matched to the alphabetical database listing of districts maintained by the OSDE in a spiraling fashion to determine the year of monitoring within the five year cycle.

7.  Besides being randomly selected in the five-year cycle, how might my district/site incur Out-of- Cycle monitoring?

In addition to the randomly generated list, the following can trigger selection of a district for placement on the monitoring schedule:

  • Statistical irregularities or discrepancies with student assessment data (e.g., high growth in percent of students scoring proficient, questionable erasure analysis, change in student demographics)
  • Consistent testing violations or irregularities as reported to the OSDE (e.g., vendor reports, invalidations, improper test administration, failure to attend or conduct yearly training), and
  • Documented concerns (e.g., parent and community, noncompliance issues from prior years, other technical assistance requests).

The OSDE may at any time, determine that a compliance review of the LEA is warranted to verify progress and data due to complaints or concerns.

8.  What are the different types of monitoring that will be utilized by the State?

Once selected for monitoring, OSDE personnel can utilize several types of monitoring, including, but not limited to, on-site and desk monitoring. 

For on-site monitoring the Office of Accountability and Assessments staff will travel to the district which may include a visit with the LEA and will use the on-site monitoring checklist to conduct an observation and review of the district/site pre-administration, administration, and/or post-administration procedures of the assessments. This type of monitoring will direct its attention towards test administration procedures, including, but not limited to:  security, staff training, administration of the assessments, review of student’s records who receive accommodations and modifications, or alternate testing, review of administrative records, or staff interviews. 

Desk monitoring will not involve an on-site visit to the LEA. The OSDE will conduct this type of monitoring utilizing data and evidence submitted to the OSDE by the LEA, testing vendors, and/or OSDE department.   Part of the review will involve comparison of district-reported information to state or vendor reported data.

9.  How will the state know when testing is occurring at my school in order to schedule an on-site visit?

To coordinate the on-site visit logistics, the district/site must provide the Office of Accountability and Assessments with a copy of their testing schedule five days prior to the opening of the testing window. At a minimum, the State will monitor one elementary school, one middle school, and one high school site within the same district to maximize resources and improve the quality of improved results.   Charter schools within a district may also be chosen for monitoring. 

10.  Explain the on-site monitoring timeline and protocol in place for the visits.

On-site monitoring timeline/protocol:

a. The Office of Accountability and Assessments will notify the district ten (10) “state” working days prior to the opening of the testing window in which the district will be monitored and will include the type of monitoring as on-site monitoring.
b. The district/site has five (5) “state” working days to submit the site testing schedule to the Office of Accountability and Assessments.  The schedule may be submitted through either e-mail or fax.
c. The Office of Accountability and Assessments will assign monitors and an observation period, notifying the site prior to the observation.  Monitors will follow these guidelines when conducting the monitoring visits:

1. Assessment monitors will check into the site office, presenting proper identification.
2. When observing assessment activities, OSDE monitors will practice the principle of observation from a distance, with the understanding that the site staff needs to go about performing their  job tasks while taking little or no notice of their observers, who likewise should be able to conduct their observation without being asked to participate in the administration in any way.
3. The majority of the assessment day activities will be easily visible to observers. Before and after the administration, the observer may walk amongst thedistrict/ site assessment personnel to view their work.
4. The State observers may request access to view documentation for students who are receiving accommodations on the assessments.
5.  During the assessment, the observers will try to seat themselves where they can observe all assessment activities and complete the observation checklist while maintaining a comfortable distance between themselves and the site assessment personnel.
6.  The observation may be extended after the conclusion of the assessment so that post assessment activities can be observed.
7.  If district/site staff are not following assessment protocol, this will be noted on the observation  checklist.  The observer will only correct site staff or make comments about task performance prior to a testing session in order to insure proper testing procedures occur.
8.  At the conclusion of the visit, observation feedback will be submitted to the OSDE, Office of Accountability and Assessments, using the checklist document (i.e., hardcopy or electronic version).
9.  The section for Other comments may include the observers thoughts regarding: administration of the assessment, such as appropriate tone, management, and monitoring of the session; provision for security and confidentiality of test materials, and school and student information; any information which might require action during this assessment cycle; and overall impressions of the assessment  administration.
10.  Following a testing session, if an observation has been made that needs immediate attention, the Building Test Coordinator (BTC) or DTC will be notified (e.g., invalidation, cheating, improper test administration)
11. Completed checklists will be submitted to the OSDE, Office of Accountability and Assessment in a timely manner.

e.  The Office of Accountability and Assessments will have 45 working days from the last day of the End-of-Instruction (EOI) online testing window to analyze submitted documentation and provide feedback to the  district/site. These steps will be taken to review evidence received:

  • Review and cross check all submitted documentation required by the monitoring checklist to ensure compliance with the directions in the Test Preparation Manuel, Test Administration Manuals, in-service trainings, and/or state assessment rules.
  • Collaborate with various OSDE offices to cross check data (e.g., Accreditation, Special Education Services, and Bilingual Education).
11.  Explain the desk monitoring timeline and protocol.

Desk monitoring timeline/protocol:

a.  The Office of Accountability and Assessments will notify the district ten (10) “state” working days prior to the opening of the testing window in which the district will be monitored and will identify the type of monitoring as desk monitoring.
b. The district/site documentation must be received in the OSDE Office of Accountability and Assessments, within twenty (20) “state” working days from the last testing day of the EOI online assessment window. Documentation can be submitted in a hardcopy format attached to the Desk Monitoring Form.
c. The Office of Accountability and Assessments will provide monitoring results to the district superintendent within forty-five (45) “state” working days from the desk monitoring document twenty-day submission.  These steps will be taken to review evidence received:

  • Review and analyze across a four-year period, all pertinent testng and educational data.
  • Review and cross check all submitted documentation required by the monitoring checklist to ensure compliance with the directions in the Test Preparation Manuals, Test Administration Manuals, in-service trainings, and/or state assessment rules.
  • Complete the State Collected Documentation section.
  • Collaborate with various State offices to cross check data (e.g., Special Education Services, Bilingual Education, Accreditation, and Student Transportation).
  • Collaborate with testing vendors to cross check data.
12.  Will the districts be notified of the results?

The Office of Accountability and Assessments will provide monitoring results to the District Superintendent.  A letter sent to the district will inform the district of the monitoring status. The status, “in compliance” will be given if all requirements have been met and the evidence is complete and valid.  If any monitored area is found deficient with additional monitoring or documentation being required, a “noncompliant” status will be indicated.

13.  What happens to a district that receives “in compliance” status is achieved?

Districts/sites that provide complete and valid evidence of successful accountability during the monitoring process will return to the scheduled five-year cycle.

14. What happens to a district that receives a “noncompliant” status designation?

Districts/sites that do not provide complete and valid evidence of successful accountability will have some form of monitoring until evidence is obtained. Repeated noncompliance with State and Federal regulations may result in:  repeated on-site visitations, suspension or loss of accreditation, and/or other sanctions as determined appropriate by the OSDE.

15.  What portion of the Desk Monitoring Checklist should the district complete?

The checklist section titled, “District-Provided Documentation” must be completed by the district or site.  The section titled, “State-Collected Documentation” will be completed by the Office of Accountability and Assessments.

16.  In what format and order should the evidence for desk monitoring be sent to the OSDE?

Desk monitoring evidence can be submitted in either a hardcopy format or in an electronic format (CD or DVD) and as a site packet.  A district/site may choose which format to submit documentation, yet ALL documentation must be in the same format. 

Hardcopy format:  A cover page should be included in the file. It should be on school letterhead and indicate (1) who has compiled the evidence, (2) the summer contact information for responsible person, and (3) the district/school’s contact information (if not on the letterhead).  The written descriptions and evidence should appear in the same order as the items appear on the checklist. Send copies of sign-in sheets, security documents, etc., keeping the originals for your files.

Electronic format: A cover page should be included in the file. It should be on school letterhead and indicate (1) who has compiled the evidence, (2) the summer contact information for responsible person, and (3) the district/school’s contact information (if not on the letterhead).  The electronic file must be submitted on a CD or DVD. Documents will not be accepted via email. The full County District code (6 characters) must be visible on the front of any disc, along with the district’s name.

Documents may be saved using any of the Microsoft Office 2007 or earlier file extensions. (Word 2007 or earlier, Excel 2007 PowerPoint 2007)  Scanned images which contain signatures may only be saved as a pdf file.   Documentation submitted as a TIFF or JPEG will not be acceptable. Districts/sites which do not use Microsoft Office may call the OSDE regarding file submission. (405) 521-3341.

Documentation should be saved within files.  Files must be labeled according to the order an item appears on the Desk Monitoring Checklist.  The file listing below, is an example of how to indicate file names which provide both a description of documents within the file and would appear in the order used by the checklist.

1_evidence_training
2_testing_schedule
3_security_of_materials_plan
4_transfer_of_documents
5_security_forms
6_calculator_use_compliance
7_plan_fo_add_time
8_emergency_commun_plan
9_test_accomodation_plan
10_stud_cum_folder_remediation
District_Site_Cover
11_Participation_Variance

CAUTION:  if a document is two sided, the two pages must appear in consecutive order within the same pdf file.  It is the responsibility of the district/site to ensure all signatures on scanned documentation is visible, legible.  Forms which are double sided include, but not limited to:  Test Administrator/Proctor Security Form and the Student Cumulative Record forms.

17.  How specific should the evidence providede for desk monitoring be?

Evidence should be specific enough to provide the reader with an accurate description of what procedures are in place.  Providing only a copy of the State's policy and that it will be in effect is incomplete.  Complete and valid evidence would include a written description indicating how the site implements the policy.  If appropriate, steps can be bulleted or numbered instead of written in a paragraph.  While not an all inclusive list, several examples are given below to help you think through how specific a description should be to provide complete and valid evidence.


Example #1:  A description for providing procedures for use of calculators to ensure policy compliance would provide answers to these key questions:
1.  Who will oversee the implementation of the calculator policy?
2.  Which tests will you permit calculators to be used on? 
3.  Are the students using their own calculators, site provided calculators, or the online provided calculator tool?
4.  If students are using a calculator other than the online version:

  • What type of calculators, such as a four-function or TI-183, etc?
  • When (before and after each test) and how (process to clear memory) and who will clear the calculators’ memory?
  • Will each student have his or her own calculator?
  • Who will be responsible for removing or disabling programs or applications (e.g. Polynomial Root-Finder and Simultaneous Equation                  Solver on TI-86?
  • When will this be done?
  • If applicable for the calculator, who is responsible for enabling the “Press to Test” function? When will this be done?
  • Oklahoma Modified Academic Assessment Program (OMAAP) grade 3 – 8 exams can only use a four function calculator.

Example #2: A written description of a communication plan in case of emergencies would provide answers to these key questions:
1. If a student becomes ill during a testing session:

  • Who will be notified to escort the student from the room? Who will continue to monitor the test?
  • What will happen to the student’s test document or online test?
  • Who will determine if the test needs to be invalidated and notify the state of the testing invalidation?

2. If there is a loss of power failure or technology problems during a testing session:

  • Who is included in the line of communication? What role does this person play (IT person for site or district who will  trouble shoot problem in computer lab)?
  • What will the Test Administrator (TA) and Test Monitor(TM) do in this situation?
  • How long will students be held in a secure testing situation?
  • Who makes the decision as to when the testing vendor or state is called?

3. If an evacuation of the school facilities becomes necessary during a testing session: 

  • Who is included in the line of communication?  What role does this person play?
  • What will the TA and TM do in regards to providing for the safety of the students and to
  • ensure that secure materials remain secure?
  • Will materials be locked in the classroom or will the TA collect all the documents and keep them in his or her personal possession until normal operation resumes?
  • Will the students refrain from talking to each other so that they do not share information or answers regarding the test?
  • Who makes the decision to call the state or testing vendor? When will the call be made?

Example #3. A written description of the procedure and testing plans for students who need testing accommodations would provide answer to these key questions:

1.  If students needed additional time to complete the test:

  • Who will identify those students?
  • Who would assign them special seating so as to not distract other students, if needed?
  • Who will replace the TA or TM if one of them needs to leave?
  • Will the students stay in the original testing room (paper/pencil or online exams) or will they be moved to a central location?

2.  If the student needs special accommodations:

  • Who will ensure the proper tests are ordered for each individual student?
  • Who will ensure a list of students taking the OMAAP and students taking the Oklahoma Core Curriculum Tests (OCCT) is given to the TA?
  • Will the TA check the list prior to administering the test to ensure that each student is receiving the correct test?
  • Who will ensure that approved accommodations (e.g., small group of no more than five students for a read-aloud within the same subject area, word-to-word dictionaries for ELL students) are given for each student?
18.  Is the district required to tell the site that it has been selected for on-site monitoring? 

Once the State notifies the district that it will be monitored, it is the decision of the district whether to inform the site that it has been selected for monitoring, the type of monitoring that will be conducted, and when monitoring will occur.

19.   How long will the on-site monitoring visit last?

It is anticipated that most on-site visits will last one day.

20.  What are some examples of “suggested evidence” that a district can provide during an on-site monitoring visit to indicate that assessment program requirements have been met?

If a district will be receiving an on-site monitoring visit, the district will receive a letter indicating which specific items on the checklist are to be provided in a hardcopy format to the OSDE monitor.  Examples include, but are not limited to:  sign-in sheets for training, written communication plans; a complete testing schedule for the site; security checklists; and nondisclosure forms, copys of student cummulative folders.

21. Where can I locate the additional resources, such as the monitoring schedule or checklists?

Additional resources that may assist in completing the process of your district’s review can be found on the OSDE’s Web site <www.ok.gov/sde>. 
• Select “Site Index.”
• Click on “Assessment Administrator Resources”
• Click on “Academic Assessment Monitoring Program" (AAMP) in the blue quick link tab.
 

 

 

 

Last updated on December 7, 2012