Problem Solving in PISA 2015

January 30, 2015

By Adrienne Hu

The year 2015 brings about the sixth administration of the Programme for International Student Assessment (PISA) – an international assessment of 15-year-olds that takes place every three years. To date, more than 70 economics have participated in the exam and results of PISA will ostensibly allows countries to see where they stand and “motivate policymakers to identify shortcomings and remedy them with proper reforms.”

PISA 2015 will focus on science and collaborative problem solving. Science is not a new thing, as PISA has measured students’ skills and knowledge in reading, mathematics and science in every cycle since 2000, while taking turns in focusing on one of these three subjects. However, problem solving is the new focus in the recent two cycles as it made the stage in PISA 2012 for the first time. Problem solving assessment in 2012 featured a set of non-traditional problems that were closely related to individual’s life experience. This year, PISA is taking one step further in the problem solving assessment: instead of assessing individuals’ problem solving skills, an assessment on collaborative problem solving (CPS) will be administered.

PISA 2012: Creative Problem Solving

Before we delve into the new possibilities, let us revisit the computer-based problem solving Assessment for individual student that took place during the previous PISA administration.

A sample item released by PISA reads like this (please also see Figure 1):

Here is a map of a system of roads that links the suburbs within a city. The map shows the travel time in minutes at 7:00 am on each section of road. You can add a road to your route by clicking on it. Clicking on a road highlights the road and adds the time to the Total Time box. You can remove a road from your route by clicking on it again. You can use the RESET button to remove all roads from your route. Question: Maria wants to travel from Diamond to Einstein. The quickest route takes 31 minutes. Highlight this route.

Figure retrieved from

The item presented above has several characteristics: first, the problem context is a real life scenario; second, the problem has multiple constraints that need to be taken into consideration. This is only an item at level 2, which means that students who get this right have the equivalence of a level 2 proficiency in problem solving on a 6-point scale. At level 3, students can handle information presented in different formats, while at the highest level, students can develop complete, coherent mental models of diverse problem scenarios to solve complex problem efficiently.

PISA Next: Collaborative Problem Solving

Pearson, one of the largest education and book publishing companies in the world, designed the CPS assessments for PISA 2015 as well as PISA 2018. This company is known as a leader in technology in education, as evidenced in their assessment design plan. In their draft assessment framework for PISA, CPS was defined as “…the capacity of an individual to effectively engage in a process whereby two or more agents attempt to solve a problem by sharing the understanding and effort required to come to a solution and pooling their knowledge, skills and efforts to reach that solution” (p. 6). However, the agents involved in the group setting are not all actual students. The CPS assessment is still a computer-based instrument of individuals in a collaborative problem-solving context, which means that a student will be collaborating with computer-based agents representing team members with a range of skills and abilities. They will “communicate” and “negotiate” via a chat window to solve a complex task and to respond to the probing questions. When the student reaches an impasse or runs out of time, a rescue agent on the computer will intervene and enable the student to proceed with the next item.

Technology is really the key in this new type of assessment in PISA 2015. The assessment is not artificial intelligence, but it is trying to interact with the students as if it was another group member in CPS. This requires a sophisticated design of the platform and anticipation of the many possible setbacks or glitches that could happen in the CPS processes. If you are interested in seeing some of the released sample tasks and the scoring methods, please refer to the document here (starting at p.51).

The Desired Citizenship as Reflected in PISA

The past two cycles of PISA have reflected the demands of the skills and competencies that many countries desire future generations will obtain. Among these competencies, students’ financial literacy, problem solving, and communication skills are emerging to be as important as their reading and quantitative literacy. However, not all schools in U.S. teach some of these things that are tested in PISA, including non-traditional problem solving and financial literacy. We are still hoping success in math will naturally lead to success in both problem solving and financial management, and to some extend it indeed helps. But are we doing enough in education if these skills sets are desirable as a citizen? Or have our students cultivated these abilities in their everyday life already? Data will tell.

In closing, PISA 2015 is a good test to President Obama’s statement in State of the Union address that American students are at their best in math and reading on record in terms of test scores. Also, it could provide other information about schooling in relation to test results. Particularly, student performance on the new assessments (financial literacy and problem solving with emphasis on communication skills) could provide Americans with more data on whether students have obtained these specific skill sets that are essential to both personal and career success. We can have good data but make no good use of them, or we can get the most out of the data to inform our research and educational policy agenda. To participate in an international assessment takes money, energy, and time, but it is worth the efforts if by doing so we can improve the educational system.


Contact Adrienne Hu: