In spring 2022, the assessment of the 8th PISA main study in the participating schools takes place. In total, 7700 students of about 270 schools in Germany complete the PISA items of the competence domains mathematics, science, reading and creative thinking on the computer. The students answer questions about their background, learning behavior, attitudes and learning conditions. School principals, teachers and parents also respond to questionnaires.
The PISA main study focuses on mathematical competences that are assessed via newly developed as well as items used in previous cycles. Germany participates again in the innovative domain: creative thinking. Science and reading literacy are assessed solely with items from previous cycles, so called trend-items.
The questionnaires assess demographic information, mathematical learning conditions and attitudes as well as general social and emotional characteristics of the students. In addition, information about the learning situation during the Covid pandemic is assessed.
The trained test administrators prepare and administer the PISA testing for the students in the schools and coordinate the testing day closely with the school coordinators to ensure a standardized procedure.
In the run-up to the main study of PISA 2022, all materials used in this context will be submitted to the Ministries of Education and Cultural Affairs of the respective federal states for a review. In addition to the testing procedures, all letters to the groups involved - students, parents, teachers, and school principals - as well as the associated context questionnaires were reviewed. The test items cannot be published due to the confidentiality of the content; however, you can get a good impression here.
Recommendations on changes in the materials from the Ministries of Education and Cultural Affairs of the federal states were implemented wherever possible. PISA focuses on the cross-national comparability of the test materials as an international comparative education study. Therefore, we paid particular attention to conformity with international specifications when making adjustments. In addition, comparability with previous PISA cycles must always be ensured.
The students’ questionnaire can be consulted on this page.
The first step of sampling consisted of drawing schools based on their school types in the federal states in Germany. Special-needs and vocational schools were specially highlighted to reflect their share in Germany as accurately as possible. In the PISA main study 2022, the sample of the schools consists of 272 participating schools in Germany. In the next step, the 15-year-old students were randomly drawn within these schools, i.e. each student has an equal probability of being included in the sample. For each school, 30 adolescents are selected.
In addition to the 15-year-old adolescents, two 9th grades were also drawn. This allows for the analyses of research questions on the class level. For this purpose, 15 randomly selected students from each of the selected 9th grades were sampled. Vocational schools were excluded from the 9th grade draw because there are usually no 9th graders in this type of school. The full sample of students comprises a total of 7700 adolescents.
In addition to the students, teachers at each school were also sampled for an online survey if they teach mathematics or another subject in the 9th or 10th grade. Here, the target numbers have been set at 10 mathematics teachers and 15 other teachers per school. Furthermore, for a national supplementary survey, all science and German teachers who teach one or both of the drawn 9th grades were also drawn to the teachers’ sample.
Based on the results of the field trial, the instruments for the main study PISA 2022 were selected. The PISA National Center reviews and checks all items of the competence domains mathematics, science, reading and creative thinking. Furthermore, all questionnaires for students, teachers, school principals and parents are reviewed and checked again.
For example, linguistic errors or inaccuracies are corrected and single items and questions are excluded. Based on the international and national results of the field trial as well as on content-related considerations, the National Center decided whether to exclude items and questions or to retain them for the main study PISA 2022 in Germany. In a final step, the items and questions for the main study PISA 2022 were finally released.
After quality checks by the international PISA consortium, the participating countries received the processed national data sets on September 30, 2021. In addition, thanks in part to the efforts of the PISA National Center in Germany, all participating states received detailed quality scores on all items.
From the international side, reports on the quality of the procedures were prepared for the questionnaires, the main domain mathematics as well as the innovative domain. These include information on task difficulty, coding reliability, potential sequence effects, task completion time, and scaling.
In addition to these international evaluations, the PISA team also conducted its own analyses using the German field trial data. Among other things, the nationally supplemented instruments on various aspects of creativity were related to the international PISA items on creative thinking.
From April to June 2021 the field trial for PISA 2022 was conducted. In total, 1218 students at 41 schools participated in Germany. The students completed the PISA items of the competence domains mathematics, science, reading and creative thinking on the computer. The students answered questions about their background, learning behavior, attitudes and learning conditions. School principals, teachers and parents responded to questionnaires.
During the field trial the questionnaires as well as the competence items were examined. The focus was on the major domain of mathematical literacy. The innovative domain of creative thinking, which was assessed for the first time, was also reviewed closely. The minor domains of science and reading were assessed re-using items from previous PISA cycles.
Among others, the field trial examined whether items varied in their difficulty in different countries. In the questionnaires various response formats and different scales were assessed.
Since the 2021 PISA study has been postponed by one year due to school closures caused by the Corona pandemic, various content and organizational adjustments were necessary.
A supplementary questionnaire module on Global Crisis will be used in the PISA study 2022. This module will survey, among other things, the circumstances in which school took place during the Corona pandemic.
For this purpose, the participants are asked, for example, about their experiences in connection with school closures. The national project management reviewed this supplementary questionnaire module with the same scrutiny as all other test materials (see description "Item review" on this page).
In addition to the scientific review of the new module, organizational adjustments were also necessary. For example, the computer platform by means of which the cognitive test items and the questionnaires are administered, had to be set up and tested again. The information materials for school administrators, teachers, parents and students also needed to be updated. Both the information letters and the new questionnaires had to go through the approval process by the Ministries of Education and Cultural Affairs again.
Furthermore, the manuals for the test administrators and school coordinators had to be adapted to the new conditions in the schools. In addition, the samples of students and teachers had to be drawn again, since the originally drawn fifteen-year-old students were no longer fifteen.
The preparations for the field trial have been largely completed. The German language instruments were finalized, the sample for the field trial was drawn, and the manuals and scripts for test administrators and school coordinators were prepared. But as a result of the school closures due to Corona pandemic, the implementation of both the field trial and the main study was postponed by one year.
In the run-up to the field test, all materials used in this context will be submitted to the Ministries of Education and Cultural Affairs of the respective federal states for review of the content as well as obligation to data protection standards. In addition to the procedures, all letters to the groups involved - students, parents, teachers, and school administrators - as well as the associated context questionnaires will be reviewed. The test items cannot be published due to the confidentiality of the content; however, you can get a good impression here.
For the 2020 field trial, none of the federal states drawn - Brandenburg, North Rhine-Westphalia, Saarland, and Schleswig-Holstein - had any data protection concerns. In addition, some of the Ministries of Education and Cultural Affairs provided feedback on the content of the materials. Where possible, we incorporated this feedback into the field trial or main survey. Since PISA, as an international comparative education study, focuses on the cross-national comparability of the test materials, we pay particular attention to conformity with international specifications when making adjustments. In addition, comparability with previous PISA cycles must always be ensured.
The first step of sampling consisted of drawing schools based on their school types in the federal states in Germany. Special-needs and vocational schools were specially highlighted to reflect their share in Germany as accurately as possible. Unlike in the main survey, not all federal states were selected in the field trial, but only a small proportion. Care was taken to ensure that the chosen states represent e. g. both small and large federal states. Also unlike in the main survey, schools were not selected for the sample according to a statistical drawing procedure, but on the basis of their logistical location, which is favorable for the survey. The sample of schools for the field trial included 46 schools in the states of Brandenburg, North Rhine-Westphalia, Saarland, and Schleswig-Holstein. In a further step, the 15-year-old students were randomly sampled within 40 of these schools, i.e. each student had an equal probability of being included in the sample. For each school, 54 adolescents were selected.
In addition to the 15-year-old adolescents, two 9th grades were also drawn. This allows for the analyses of research questions on the class level. For this purpose, 18 randomly selected students from each of the selected 9th grades were sampled. Furthermore, three schools at which only a few 15-year-old students are enrolled were additionally included in the sample. These students will receive an extra weighing factor in the main survey. Therefore, this needed to be tested in the field trial. The full sample of students comprised a total of 2451 adolescents.
In addition to the students, teachers at each school were also sampled for an online survey if they teach mathematics or another subject in the 9th or 10th grade. Here, the target numbers have been set at 10 mathematics teachers and 15 other teachers per school. Furthermore, for a national supplementary survey, all science and German teachers who teach one or both of the drawn 9th grades were also drawn to the teachers’ sample.
In reconciling, a single German version is formed out of the two initial German translations from the previous work step, taking into account technical accuracy. Since PISA 2009, there has been a translation cooperation with the other German-speaking PISA participating countries (Switzerland, Austria, Luxembourg). We create a "Common German Version" of the PISA items in a meeting lasting several days. In this process, colleagues from mathematics didactics support us in finding formulations that correspond to everyday mathematics teaching.
Finally, the Common German Version has to be verified by the international project management for comparability with the original version as well as for fairness towards translations from other countries. This verification of the Common German Version takes place at the PISA contractor cApStAn in Belgium. There, all translations are linguistically checked and revised in several loops in consultation with the national project managers. After this, the items for the cognitive domains and the questions for the context questionnaires are ready for use in the field test.
From March 2019 to June 2019, the new PISA items for Mathematics, Creative Thinking, and the questions of the context questionnaires were each translated from the original English or French version into German by two independent translators. We are pleased to work with a very competent team and would like to express our sincere thanks to:
Julia Frenzel (LF Languageservices), Ulrika Wiesner, Franziska Klos (that word), Ilse Freiburg (IKF Translations).
The first milestone for PISA 2021 concerns the preparation of the test material. In this process, we received the units and items in several batches, which were assessed by external and internal subject matter experts. In the domain of mathematics, we are very grateful for the support by the following persons:
- Dr. Götz Bieber, Berlin-Brandenburg State Institute for School and Media (LISUM).
- Prof. em. Werner Blum, University of Kassel
- Prof. Aiso Heinze, Leibniz Institute for Science and Mathematics Education at Kiel University (IPN)
- Prof. Stefan Krauss, University of Regensburg
- Prof. Dominik Leiß, Leuphana University of Lüneburg
- StD a.D. Max Schmidt, Bavarian Values Alliance Foundation
- Prof. Stefan Ufer, Ludwig Maximilian University of Munich
- Dr. Barbara Hank, State Institute for School Quality and Educational Research Munich (ISB)
With regard to the innovative domain, we sincerely thank PD Dr. Mathias Benedek, Karl-Franzens-University Graz, for his valuable input.