Improving Students’ Testing Performance

Problem

Lebanon, which has witnessed a massive influx of refugees since the start of the Syrian crisis, is facing a challenge in providing education to all Syrian and Lebanese children. Despite efforts by national and international actors (e.g. Back to School Program by UNICEF), over half of the Syrian children in Lebanon remain out of school^[1], with many becoming ineligible to enroll in public schools.^[2]

To cater for this marginalized population, Amel Association International (Amel), in collaboration with UNICEF, ANERA, and MEHE, initiated a Youth Literacy and Numeracy (YLN) program that offers basic training in English, Arabic, Math, Computer literacy and Life-skills to youth aged 14 to 24.

A pilot conducted in 2017 revealed consistent improvement among students across all subjects, relative to their placement exams scores. However, performance remained very low, reaching an average of 34.4% following the completion of the first cycle of the program, and 43.6% following the completion of the second cycle.

As these students are considered disadvantaged and belong, for the most part, to refugee populations, their performance could be hindered by the effects of stereotype threat associated with the performance of minorities.

The effects of stereotype threat on the performance of minorities have been well documented, as well as the benefits of using values affirmation techniques to alleviate the psychological stress that results from such stereotypes. The re-affirmation of values that are important to students has been shown to increase confidence and positive self-image which can be conducive to improving performance.  ^{[3],[4],[5],[6],[7],[8]}

Against this background, Nudge Lebanon, in collaboration with Amel, implemented a behavioural intervention that consisted of asking students to complete a visual values affirmation task, before completing their end of Cycle I exams.

Intervention

A Randomized Controlled Trial (RCT) was conducted at seven of Amel’s community centers to improve the performance of students sitting for Cycle I exams. The trial consisted of a short activity that comprised of a ranking exercise whereby they have to rank a list of 10 values from most of least important to them, followed by a drawing exercise in which they have to visualize their number 1 value ^[9]. Both exercises were completed by students right before sitting for their exams.

A sample of 150 students was randomly assigned to two groups using a cluster randomization at the center level:

1. A control group, in which participants were asked to rate 10 values from most to least important to others, and then complete a random drawing task;
2. A treatment group, in which participants were asked to rate 10 values from the most to least important to them, and then draw the value most important to them;

The idea being that students who complete the values affirmation task will have reduced stereotype threat by affirming an alternative positive identity, and as a result perform better on their end of cycle exams relative to those in the control group.

The measured outcome was exam scores of participants in control and treatment groups while controlling for placement test scores and center.

Results

Results showed that students in the treatment group performed significantly better on their Arabic test than students in the control group.  More specifically, for students who completed the values affirmation task, there was an 11% increase in their Arabic test scores (B=6.89, p=0.046) compared to the control group. This finding was robust to adding covariates (age, gender and class level; B=7.20, p=0.033) and to removing outliers.

At the same time, applying the same values affirmation task to Math and English had no significant effects. This could be partly due to a discrepancy between the level of difficulty of the exams in these two subjects and the levels of the students, which a values affirmation task cannot resolve. Indeed, results show that even after the exit tests, average performance remains low for Math (45.1%) and English (38.4%), while it is higher for Arabic (65.8%). Additionally, upon looking at differences in placement test scores across subjects, we find that there is a large discrepancy: for Arabic, the average baseline score was the highest at 35% while it was substantially lower for English (4%) and Math (23%). Thus, we speculate that such low starting points in English and Math could have had adverse effects on students’ ability to completely assimilate material in these subjects in the short period of the first YLN cycle, for which a values-affirmation exercise cannot compensate. Indeed, having a second cycle of the program is much needed to strengthen the students’ competencies, especially in English and Math.

Conclusion

Re-affirming values of students who are stigmatized can bolster their sense of self-worth and consequently allows them to show their real skill level as opposed to the one that is hindered by stereotype threat.

Going forward, values affirmation tasks will be used before cycle 2 exit exams for all students in the YLN program. In addition, it is important to replicate this study with similar populations, among other things, to test out the generalizability of the results to other contexts.

Endnotes

[1] UNHCR, UNICEF, & WFP. (2015). Vulnerability Assessment of Syrian Refugees in Lebanon, 112. Retrieved from http://reliefweb.int/sites/reliefweb.int/files/resources/VASyR2015_ExecutiveSummary.pdf

[2] Due to having exceeded the enrolment age or to having been out of school for too long.

[3] Cohen, G. L., Garcia, J., Apfel, N., & Master, A. (2006). Reducing the racial achievement gap: A social-psychological intervention. science, 313(5791), 1307-1310.

[4] Miyake, A., Kost-Smith, L. E., Finkelstein, N. D., Pollock, S. J., Cohen, G. L., & Ito, T. A. (2010). Reducing the gender achievement gap in college science: A classroom study of values affirmation. Science, 330(6008), 1234-1237.

[5] Shapiro, J. R., Williams, A. M., & Hambarchyan, M. (2013). Are all interventions created equal? A multi-threat approach to tailoring stereotype threat interventions. Journal of Personality and Social Psychology, 104(2), 277.

[6] Cohen, G. L., & Sherman, D. K. (2014). The psychology of change: Self-affirmation and social psychological intervention. Annual review of psychology, 65.

[7] Shnabel, N., Purdie-Vaughns, V., Cook, J. E., Garcia, J., & Cohen, G. L. (2013). Demystifying values-affirmation interventions: Writing about social belonging is a key to buffering against identity threat. Personality and Social Psychology Bulletin, 39(5), 663-676.

[8] Aronson, J., Cohen, G., & McColskey, W. (2009). Reducing Stereotype Threat in Classrooms: A Review of Social-Psychological Intervention Studies on Improving the Achievement of Black Students. Issues & Answers. REL 2009-076. Regional Educational Laboratory Southeast.

[9] The target population is students who have been out of school for a long time, and therefore have very low literacy rates. As such, we opted for a drawing task as opposed to a written task to account for the population’s literacy challenges: instead of having to explain why the chosen value is important to them in a few sentences, participants had to illustrate it with a drawing.