Oral Assessments at Asha Chennai Projects

Background

At Asha Chennai we have been supporting about 25 government schools spread across Tamilnadu. With this kind of scale we felt we need to develop metrics to tell us how our projects are doing and start measuring the impact we are having and fine-tune our actions so as to improve the outcome. Towards this end we started conducting assessments starting with one in late 2014-15.

Assessments Out There

We hope to get the following information from the Assessments.

1. How are our schools performing compared to broader samples of children in the district, state or the country. This is not very relevant for helping us shape our work but it would be useful for donor communication etc. This same information can also be used to compare one school against the average or against another. This will help us focus our efforts and learn from best practices.
2. Chart the progress of our schools in terms of how they are improving or becoming worse over the years.
3. Help identify the gaps in the learning. Identify specific areas of weakness that we can then try to address through our training and the materials we give to the schools.

Getting broader statistics on the performance of students in India is difficult. The only India-wide efforts that we were aware of are the ASER conducted by Pratham, and the National Achievement Survey conducted by NCERT. Both of these have some advantages but serious disadvantages as well. This is a major area that needs to be addressed as India continues improving its education system.

ASER is a very simple assessment that just groups the learning attainment of children into 4 or 5 buckets in 3 competencies – English reading/comprehension, Mother tongue reading and Arithmetic. For example in Tamil reading the children can be at a Pre-letter level, letter level, simple word level, sentences level or paragraph/story level level.  This has a large sample and covers most of the districts in India.  Its simplicity means it can easily be replicated by us for comparision. However its simplicity also means this wouldn’t serve our third need. i.e Help us identify gaps in the learning and the causes for the deficiencies in learning attainment.

There are also significant concerns over the data collected by ASER. Here are some concerns,

1. There is a significant variation year on year. In a large-scale survey such as this, there really shouldn’t be, as the data being studied would not vary suddenly or significantly. This casts doubts over their methodology. i.e. Have the assessments been conducted in entirely the same way every year and in every place to enable comparision?
2. In particular for Tamilnadu, there was a siginificant jump in their performance in 2014 over the previous years. For instance the % of 5^th std children who can do division ranged from 12.1% to 17.1% percent from 2007 to 2013. In 2014 it jumped to 25.8%.
3. Their statistics for the 1^st standard is also a little suspect. About 25% of the children in first standard can read 2 digit numbers. This totally flies against our experience which says the number is much smaller (less than 5% in November).

NAS on the other hand has full-fledged question papers in English, Maths and the mother tongue. It doesn’t just test reading in the languages but tests comprehension and grammar. Similarly in Maths it goes beyond the mechanical skills of doing the various operations to test their grasp of the concepts.

However NAS is conducted on an average once every three years and only for standards 3, 5 and 8. The thing we found most difficult in using NAS is that we couldn’t get detailed data on their statistics and methodology. For a complex assessment such as NAS, we cannot have something similar without really detailed information

Asha Chennai’s Assessments

Given this scenario, we decided to do two things.

1. Conduct Oral assessments in line with ASER. This will help us in achieving goals 1 and 2 identified above. i.e. It will enable us to compare the performance of our schools and projects as a whole against broader Tamilnadu and India numbers from ASER. It will also enable us to see how a school is doing compared to other schools. It will further enable us to see the time trends as we conduct over the years.
2. Design and conduct written assessment that will help us identify gaps in the teaching/learning process. While we do not have the data or skill to develop this thoroughly based on Item Response Theory (as NAS has done), we can certainly map this to the curriculum and assess what areas of the curriculum pose problems for the children.

We conducted detailed assessments (but just written) of all the supported schools in April of 2015. This turned out to be a disaster.

1. We had naively distributed the papers to the schools ahead of the exams. Most of the school teachers had coached the children for this test with the question paper provided! For instance in many schools all the children wrote the same three sentences for a question where we asked the student to write three sentences on a crow. What is even more telling in that often all these sentences had the same mistake! This totally defeated the purpose of our test.
2. Children were not used to writing 3 hours of test. For 4^th and 5^th standard, our tests were 1.5 hours long for each paper.
3. Written assessments for 1^st and 2^nd standards didn’t make any sense as the language abilities of the children didn’t make it possible for them to understand the question and write the answers.

We therefore changed the pattern for the tests conducted in Feb 2016. We decided we should occupy no more than 1 hour time for each class. Therefore we had significantly shorter tests. We decided to have oral tests for class 1 and class 2 – English. We had written tests in Maths for classes 2 to 5 and in English for classes 3 to 5. To ensure consistency and that the tests are conducted properly, the same set of teachers travelled to each school and conducted the tests, one day per school. The papers were taken to the school only on the day of test and we did not allow the teachers from the school (Asha teachers or govt. teachers) into the room where the tests were being conducted.

See here to see the photos from these assessments.

Here are our findings from these assessments.

1. In remote projects (Sangamam Senji and Project Pearl), the standard set of teachers who conducted the tests in the Sangamam and Poorna Vidhya schools could not travel and conduct the test there. So we had to train and use the local Asha teachers. We did ensure that teachers from the same school were not conducting the test at a school. To our utter dismay and amazement, these teachers also fudged the results very badly. We just couldn’t understand their reason for doing that except attribute it to some misplaced sense of loyalty to their project schools. We had to throw away the data from these schools!
2. Our English assessments were still too difficult for the children esp. for the 4^th and 5^th As we moved from just simple vocabulary to tougher words, little bit of grammar and reading comprehension, the children hadn’t mastered these skills at all in the 4^th and 5^th standards.
3. The oral assessment results were comparable to the ASER 2014 results. However direct comparison was not possible except of 1^st and 2^nd
4. Our Maths papers yielded a good understanding of the learning gaps and helped us give meaningful recommendations to our teachers and the schools we support. Some of the suggestions were,

• Teachers can call out the sum and the students should write and do the sums. This will help in doing the work independently. i.e. Teacher should not write the sums in the board or their notebook all the time. This will help children get their place values right.
• Please revise earlier class and term topics. Many of the students (from all schools) failed to answer easy questions that would have been covered in earlier classes.
• Basic arithmetic operations can be provided to the children in multiple formats. Children often get things wrong if the sum is given in a format they are not used to.

This year we decided to do a full oral assessment (in English, Tamil and Maths) and then written assessments (English and Maths). Further the oral assessments will be conducted in Oct/Nov so as to compare the results with ASER which are conducted at that time of the year. Note results will differ a lot if you conduct in June vs Nov vs Feb especially for smaller classes as their learnings will be growing significantly during this time.

The full blown oral assessments are conducted only at 17 of the schools. At the remaining 8 supported schools and all the schools where we only send computer teachers (about 40 schools) we will only be conducting the written tests in Feb 2016. We may also be conducting a short computer science assessment at all the schools to which we send computer teachers.

Oral Assessments 2016-17

Oral assessments were conducted at the 14 project Sangamam schools from November 8^th to 16^th. They were conducted at Project Pearl schools from November 22^nd to  December 2^nd.  At both these projects, the teachers were trained to conduct the assessment and it was ensured that the assessments were conducted similarly at all the schools.

Click here to see all the photos from the assessments at the Thiruvallur Dist schools.

The Reading (i.e. Tamil) assessments and the English assessments were the same as ASER. For Maths we felt the levels given in ASER make huge jumps in levels. So we felt we needed to introduce couple of additional levels. The levels in ASER are,

1. Reading one digit numbers.
2. Reading two digit numbers.
3. Subtraction sum with 2 digit numbers requiring borrow.
4. Division of a 3-digit number with a 1-digit number leaving a remainder.

We introduced the following levels.

• Addition of 2 2-digit numbers without carry. This would come between ASER levels 2 and 3.
• Multiplication of 2 2-digit numbers. This would come between ASER levels 3 and 4.

To ensure that the data we get can be compared with ASER data, we give the students the addition and multiplication sums only when they cannot do the next level sums. i.e. We give the addition sum only if the student could not do subtraction sum and give the multiplication sum only if the student could not do the division sum. Thus the cumulative scores for the ASER levels will not be affected. See a sample set of sums we used for these two levels.

See the graphs below that show the comparision between the average of all our schools at Thiruvallur (14 schools) and the ASER 2014 results for Tamilnadu. The full lines are our schools and the dashed lines are ASER results.

As you can see the performance of our schools is slightly better than the overall Tamilnadu schools from ASER. If we only include the schools that we have been supporting for 2+ years, the outperformance grows. Some mitigating factors to be considered are,

1. We select schools with poor teacher student ratio which tend to be far away from the urban centers, in poorer pockets of the district and thus tend to perform worse in assessments.
2. Thiruvallur as a whole does a little worse than Tamilnadu but the data for Thiruvallur is not available in this format.

We are comparing with ASER 2014 results, which saw a huge jump in the performance of Tamilnadu. Our outperformance is strong against ASER 2012 results