From Practice to Performance: Horizon's SAT Test Accuracy
Using descriptive statistics, split-half reliability, and negative discriminator analysis, Horizon Education evaluated its test forms in relation to College Board assessments.
Horizon Education, a leader in the development of SAT and PSAT-aligned practice tests, consistently prioritizes data-driven improvements to enhance the reliability and accuracy of its assessment tools. In the 2023-2024 school year (SY 23/24), Horizon partnered with Aurora Public Schools in Colorado to administer six PSAT/SAT practice tests. These were strategically delivered at the beginning and middle of the school year, followed by the College Board's official PSAT and SAT assessments at the end of the year. Using descriptive statistics, split-half reliability, and negative discriminator analysis, Horizon Education evaluated its test forms in relation to College Board assessments. This comprehensive analysis led to actionable insights, enabling Horizon Education to refine its assessments for the upcoming 2024-2025 school year.
Data Overview and Methodology
Horizon Education analyzed over 1,500 student records per assessment, comparing results between Horizon Education (HE) and College Board (CB) versions of the SAT, PSAT 10, and PSAT 8/9. The count of records, shown below, highlights the volume of data used in the analysis:
- SAT: HE1 (1654), HE2 (1581), CB (1842)
- PSAT 10: HE1 (1814), HE2 (1791), CB (1927)
- PSAT 8/9: HE1 (1582), HE2 (2022), CB (2092)
This volume of data provided a robust foundation for evaluating form reliability, correlation, score characteristics, and the impact of negative discriminators, all aimed at improving the reliability and accuracy of future Horizon Education test forms.
Key Analyses and Findings
Horizon Education conducted multiple statistical analyses to assess its test forms and their alignment with College Board assessments, ensuring that students receive high-quality practice tests. These analyses include:
Form Reliability and Correlation: Horizon Education compared the correlations between its assessments and the College Board versions, as well as correlations between Horizon test versions. The correlations were strong across the board, with the highest being 0.89 between HE2 SAT and CB SAT, indicating close alignment. Areas for improvement were identified in the PSAT 10 math section and PSAT 8/9 reading and writing.
Descriptive Statistics (Mean, Median, Standard Deviation):
Horizon Education examined score distributions by calculating means, medians, and standard deviations to highlight the differences between Horizon and College Board assessments. The mean score difference for the PSAT 10 total score was -136, indicating that Horizon assessments were generally lower. However, adjustments to scoring tables have been implemented to reflect more accurate and reliable conversions.
Split-Half Reliability:To ensure consistency and parallelism between test forms, Horizon Education utilized split-half reliability analysis. Results confirmed that all forms met industry standards, with reliability coefficients ranging from 0.77 to 0.89. These findings further validated the structural soundness of Horizon Education’s test forms.
Negative Discriminant Analysis:
Horizon Education identified and addressed items that negatively discriminated student performance. Through a detailed analysis of item-level response frequencies, poorly performing items were removed or revised. This effort resulted in significant improvements, with approximately 11% of PSAT 8/9 and PSAT 10 items being edited, moved, or swapped to increase overall form reliability.
Next Steps for Horizon Education
Horizon Education’s proactive approach to assessment design is anchored in continuous improvement. Based on this year’s data analysis, several key updates are planned for the 2024-2025 school year:
- Item-Level Swaps and Edits: Horizon will revise and replace items in the PSAT 10 math section and the PSAT 8/9 reading and writing sections to enhance test form correlation with College Board assessments.
- Scoring Table Modifications: Adjustments will be made to the scoring conversion tables to reflect more precise score calculations.
- Focus on Targeted Forms: While the overall reliability of Horizon Education forms is high, targeted improvements will focus on the PSAT 10 and PSAT 8/9 assessments to bring correlations in these areas up to the same level as the SAT forms.
Conclusion
Horizon Education remains committed to providing high-quality, reliable, and SAT-aligned assessments for the 2024-2025 school year. Through rigorous data analysis—including form reliability, correlations, split-half reliability, and negative discriminator analysis—Horizon Education has not only validated the strength of its assessments but also identified opportunities for targeted improvements. As a result, Horizon is well-positioned to offer students a stable and trusted suite of practice tests that mirror the structure and rigor of College Board assessments.
By continually refining its assessments, Horizon Education exemplifies its dedication to helping students succeed, positioning itself as a proactive and reliable leader in educational testing.
