Context of the Evaluation
The Cordillera region of Paraguay, located just east of the nation’s capital, reflects the diversity of the country. In Cordillera, like in much of the country, people speak Spanish, Guaraní, or a mix between the two called Jopará. Schools range from instructing in Spanish completely to not at all and the number of students in kindergarten classes ranges from one to over 40. Generally, students in small, rural, Guaraní speaking schools have lower access to educational resources and test lower than their peers, whereas students in larger, urban and Spanish speaking schools test higher. The data also showed a significant gender gap between boys and girl’s math abilities, and 90 percent of surveyed teachers responded that they were unable to teach all topics in the preschool math curriculum.
Details of the Intervention
The Paraguayan government, in an effort to increase math abilities and close achievement gaps, reviewed a series of math initiatives around the world and chose to adapt New York City’s Big Math for Little Kids program. The Ministry of Education and Culture and Inter-American Development bank adapted the curriculum to the Paraguayan context and languages, and turned it into audio lessons on CD so that it could be used in classrooms regardless of variation in individual teachers’ preparation or pedagogical skill. The program is called Tikichuela: Mathematics in My School, and teachers were trained to play the 30-40 minute lessons and in tutoring for the curriculum. The lessons were played in class four days a week, with the fifth day for the teacher to review or provide extra assistance based on observed needs. Lessons were also implemented using complementary materials such as student workbooks with exercises instructed by the characters in the lessons, along with physical objects for counting and other illustrative material to help assimilation of mathematical concepts. The Tikichuela program contrasts with the traditional curriculum which previous research has found uses long lectures focused on rote memorization of basic concepts, without attention to children internalizing an understanding of how mathematics concepts work in everyday life.3
To evaluate the new program’s effectiveness, all 265 schools in the Cordillera region with at least one single grade kindergarten (i.e. not combined with other grades) were randomized, with 131 schools assigned to receive the program and 134 continuing on as normal, to serve as the comparison group. At the beginning of 2011 a baseline survey was conducted of the students in both sets of schools evaluating basic numeracy and early math skills, along with other areas such as reading and writing, which the program could conceivably also affect. Teachers, principals and parents also answered questions on socio-demographic information for students and schools, and data was supplemented with a qualitative evaluation. In 2012 the same evaluations were conducted on the new students in the same classrooms, and a one year follow up survey was given to the original students.
Results and Policy Lessons
The 2011 implementation lasted a relatively short five months, but in that time students who received the Tikichuela program increased their performance 0.16 standard deviations in mathematics skills, an effect equivalent to moving from the 50th percentile to 57th percentile, relative to the comparison group. The effects of the program were strongest for those in schools with lower initial test scores – smaller, rural, and Guaraní-speaking schools. It benefitted the classrooms and individual students who tested lower originally, those in the bottom three quartiles at the baseline improved relative to those in the comparison group, while those in the top quartile showed no change.
The program raised math scores for both boys and girls, but boys benefitted more. Boys improved 0.21 standard deviations (about 9 percentile points), and girls 0.13 standard deviations (about 6 percentile points) over the comparison group. In light of this finding, the second year of implementation adjusted the program and preliminary analysis suggests the gender gap was closed. Analysis of the second year data is ongoing.
Pequeños Matemáticos from BIDtv.
View the Big Math section on the IDB website.