doi:10. 21831/cp. Cakrawala Pendidikan. Vol. No. October 2019 MEASURING TEACHERSAo PEDAGOGICAL CONTENT KNOWLEDGE USING MANY-FACET RASCH MODEL Maryati1*. Zuhdan Kun Prasetyo2. Insih Wilujeng3. Bambang Sumintono4 1,2,3 Universitas Negeri Yogyakarta. Indonesia Universiti Malaya. Malaysia *e-mail: maryati@uny. Abstract: In the context of teacher professional development and teacher certification, an objective measurement of teachersAo Pedagogical Content Knowledge (PCK) is requried. This research is a case study to measure teachersAo PCK within prospective middle school science teachers participating in an Indonesian teacher professional development program. The study used quantitative approach which involved multirater assessment analysed by Many-Facet Rasch Model (MFRM) to assess teachersAo PCK in teaching a specific topic, i. AuThe Particulate Nature of MatterAy (PNM). Researchers developed a rubric for items to measure teachersAo enactment PCK for teaching that specific topic. The MFRM analysis of the data revealed item reliability, item validity, separation and unidimensionality fall within good category. The findings show prospective teachers need to improve their PCK specifically in terms of Auknowledge of instructional strategies and representations for teaching scienceAy dimension. However, for knowledge of science curriculum and knowledge of studentsAo understanding in science dimension, the findings show that the participants were competent. Keywords: pedagogical content knowledge, assessment, many-facet rasch model, teacher professional MENGUKUR PEDAGOGICAL CONTENT KNOWLEDGE GURU DENGAN MANY-FACET RASCH MODEL Abstrak: Dalam konteks pengembangan profesi guru dan sertifikasi guru, penilaian objektif terhadap Pedagogical Content Knowledge (PCK) guru diperlukan. Penelitian ini adalah studi kasus untuk mengukur performans PCK calon guru IPA SMP peserta program pengembangan profesi guru di Indonesia. Penelitian ini menggunakan pendekatan kuantitatif yang melibatkan penilai berganda yang dianalisis dengan Many-Facet Rasch Model (MFRM) untuk menilai PCK guru dalam mengajar topik spesifik, yaitu Ausifat partikel materiAy. Peneliti mengembangkan item rubrik untuk mengukur hasil PCK calon guru dalam mengajar topik tersebut dan hasil analisis data dengan MFRM menunjukkan bahwa reliabilitas item, validitas item, separasi dan unidimensionalitasnya dalam kategori baik. Hasil analisis menemukan bahwa calon guru masih perlu meningkatkan kemampuan PCK mereka, khususnya dalam dimensi Aupengetahuan tentang strategi dan penyajian pengajaran sainsAy. Sementara itu, dalam dimensi Aupengetahuan tentang kurikulumAy dan Aupengetahuan tentang pemahaman siswa terhadap sains, hasil analisis menunjukkan mereka sudah menguasai kompetensi. Kata kunci: pedagogical content knowledge, penilaian, many-facet rasch model, pengembangan profesi INTRODUCTION Since Shulman introduced the notion of Pedagogical Content Knowledge (PCK) in 1987, studies on PCK have grown rapidly (Abell, 2. It has become the focus of teacher professional development programs, it covers teachersAo knowledge on studentsAo success and failure in learning as well as specific teaching materials (Van Driel & Berry. PCK is considered as an indicator of teachers and prospective teachersAo competences in America and some other countries . NRC, 1996. Shulman, 2015. NCATE, NSTA, 1. The concept of PCK becomes a guidance for teachers to transform contents through representation, examples, and experiences (Ball. Themes & Phelps, 2. and its not limited to teachersAo knowledge of teaching certain topics, but it also refers to Auwhat the teachers do in classrooms and their reasonsAy (Baxter & Lederman, 1999, p. This gives an indication that PCK must be explored at two levels: the planned PCK and enacted PCK, which in turn will give a better and clearer understanding of how teachers design and implement PCK in their The Planed PCK is an amalgam of teachersAo content knowledge and pedagogical knowledge of learning strategies needed, so that certain science topics can be comprehensively understood by students. The enacted PCK, on the other hand, refers to the type of PCK that can be observed during learning processes. PCK is often tacit, measuring it could be difficult, complicated, and time consuming. Some researchers make it possible to investigate with more efficient (Rohaan. Taconis, & Jochems, 2. , for instance the PCK rubric instrument for measuring purpose had been done by Park. Jang. Chen, & Jung . The 2012 PCK Summit resulted in an agreement that the PCK model that can be tested as a teacherAos professionalism is Aua model of teacher professional knowledge and skill [TPK&S] including PCKAy. This model introduces Pedagogical Content Knowledge (PCK) and skill, in which PCK is defined as both a knowledge base for planning and delivering a topic-specific instruction in a very specific classroom context and as a skill when engaged in the act of teaching. The TPK&S is actually derived from the generic teacher professional knowledge bases (TPKB) referring to topicspecific professional knowledge (TSPK). The knowledge category of TSPK explicitly contributes that: . the teaching contents occur at the topic level and not at the disciplinary . this knowledge combines subject matter, pedagogy, and context. TSPK is recognized as knowledge possessed by the profession, allowing it to assume a normative role (Gess-Newsome, 2. This concept has also been developed by Mavhunga & Rollnick . known as TSPCK (Topic-specific PCK). Obviously teacher profesional preparation programs should emphasize more on Auhow to teach a specific topicAy rather than Auhow to teach a subjectAy. This means PCK in nature is dynamic, not static (Baxter & Lederman, 1. meaning that teachersAo PCK can be developed all the time (Henze & Van Driel, 2. Therefore, novice teachers usually demonstrate low PCK (Pitjeng, 2014. Van Driel. De Jong, & Verloop, 2. Although PCK develops through teaching experiences, novice teachersAo PCK can be developed through intensive training that collaborates with experts of content and pedagogy (Williams & Lockley, 2012. Loughran. Mullhall, & Berry, 2004. Loughran. Berry, & Mulhall, 2. This is a challenging task for pre-service teacher educators (Aydeniz & Kirbulut, 2. Assessment of teacherAos PCK is a prerequisite for any teacher professional development (Henze. Van Driel & Verlop, 2008. Park & Oliver, 2008. Nilsson, 2. Much research has been carried out to capture and portray teacherAos PCK (Loughran, et. , 2. yet little has been done to assess its quality, particularly in science education (Park & Suh. Mavhunga & Rollnick, 2. This is due to the lack of clear operational definition (Smith & Banilower, 2. Some researchers such as Tamir . and Magnusson. Krajcik, & Borko . , have attempted to identify PCK components for assessing teacherAos PCK. However, the PCK model that is widely applied in science education research is the one developed by Magnusson, et. which comprises five components such as Van Dijk & Kattmann . and Kind . who applied this model to improve teacher quality. Research development programs attract researchers from different parts of the world (Postholm. Widodo & Riandi, 2. Several studies suggest that the teacher professional development programs by Indonesian government apparently cannot effectively improve teachersAo competencies (Rahman. Abdurahman. Kadariyanto, & Rusminanto, 2015. Sumintono & Subekti, 2. and teachers who have received professional allowance sometimes are still considered incompetent (De Ree. Muralidharan. Pradhan, & Rogers, 2. This is seemingly due to teachersAo lack of subject mastery or content (Al-Samarrai. Chang. Ragatz. Shaeffer. Stevenson, & De Ree, 2. Though content knowledge is prerequisite for developing TSPCK, the level of expertise of content Measuring TeachersAo Pedagogical Content Knowledge Using Many-Facet Rasch Model knowledge is not always related to high TSPCK (Davidowitza & Potgieterb, 2. The failure of teacher training programs is apparently due to the absence of problem solving related to Auwhat will be done in classroomsAy. Most teacher training programs focus more on how to help teachers to learn about broad theories of teaching, learning, or a subject matter as a discipline (Aydeniz & Kirbulut, 2. Meanwhile. Nealsen . asserted that failure was caused by teacher training programs always involves a large number of participants and a wide range of curriculum content that leads to fewer opportunities for teachers to improve the material understanding and how to teach particular topics effectively. Related to the teacher professional development program (Indonesian: Pendidikan Profesi Guru. PPG), it is necessary to measure teacherAos PCK to evaluate the implementation of teacher training program. This study employed a quantitative analysis on middle school prospective science teachersAo participating in that program. The researchers used a multi-rater assessment employing Many Facet Rasch model (MFRM) for quantitative data analysis. MFRM provides an accurate analysis result used to evaluate teachersAo performances with a small number of subjects (Linacre, 2. This quantitative study is an enacted PCK exploration focused on specific learning topics, that is Authe particulate nature of matterAy (PNM). The topic is the basis for understanding some chemical concepts and other scientific concepts (Snir. Smith, & Raz, 2003. Tsai, 1. The understanding of PNM can even improve learning achievement in chemistry (Gabel, 1. Harrison & Treagust . recommend research on this topic to investigate both to science teachers and students. The aim of this study is to portray the PCK skill of middle school prospective science teacher and discuss their strengths and weaknesses by involved multirater assessment as a practice to achieve objective measurement. Research Context This research conducted at the teacher professional development program in Indonesia who joined special program for prospective teachers . resh graduate. who have served for a year in remote area. This program held for one year . wo semester. The first semester offered a workshop on the improving science learning understanding refered to subject-specific pedagogy (SSP) while the second one was conducted teaching practices in schools. The PNM is the first chemistry topic in the new curriculum for middle schools science classes which aims to distinguish the characteristics of solids, liquids and gases both macroscopically and sub microscopically . omposition and motion of particle. followed by topics on elements, compounds and mixtures (Wahono. Rahmadiyati, & Hidayati, 2016a. Students at the middle school can start think abstract based on the piagetAos theory (Gelman & Baillargeon, 1. , where the chosen topic can be so associated with kinetic theory of matter, the composition and motion of particles in solids, liquids, and gases (Toplish, 2010. Nakhleh. Samarampungavan, & Saglam, 2005. Ayas, ynzmen, & yNalk, 2. The specialty of the new curriculum content is the emphasis on scientific approach in the learning process (Majid, 2. which to stimulate five science learning skills, namely observing, questioning, associating, experimenting, and networking. This is a good chalenge for the prospective teachers to implement effective learning strategies related to submicroscopic of matter topic. Some researchers have proven the effectiveness of learning strategies which can improve studentsAo understanding, e. using real object to describe particle motion (Boz & Boz, 2008. Tsai, 1. , role playing demonstration (Tsai, 1999. Purvis, 2. , and computer animation (Boz & Boz, 2. METHODS This is quantitative study on on teacher professional development in Indonesia which took place in a natural setting, where researchers do not manipulate the educational or pedagogical This study attempted to find out the quality of prospective science teachersAo PCK and discussed it strengths and weaknesses. Subject and Data All subjects in this study are prospective teachers who have just graduated from the science education study program from three state universities in Java. Indonesia, who have an average age of the same, i. 23-24 years old with one year teaching experience. Sample consisted of five male . ode Is. Zk. Rl. Tn and H. and Cakrawala Pendidikan. Vol. No. October 2019 doi:10. 21831/cp. eleven female (Nr. Rs. Rn. Rr. Ms. Lt. De. Na. Ot. Bt, and L. All of 16 prospective teachers . hich were ratee. divided into three groups based on their teaching practice location in three public middle schools in Surabaya. Indonesia. The researchers videotaped their classroom activities and observed while teaching the PNM topic and colected their lesson plan. Then, there were six lecturers . cted as rater. who asked to assess the 16 teachers by using a rubric instrument which consisting of 12 items. Thus, total data collected was 16 teachers x 12 items x 6 lectures = 1152 and there was no missing data in this study. Instrument The researchers developed the PCK rubric instrument which designed to measure only three key components related a particular topic, . , knowledge of science curriculum. knowledge of student understanding. of instructional strategies and representations for teaching scienc. among the five components suggested by Magnusson, et. The instrument consisting of 12 items to assess teachersAo PCK which all items were organized based on the literature review on the PNM topic teaching which is proposed by some researchers, such as Boz & Boz . Tsai . and Purvis . The descriptions of PCK components organized in rubric items are presented in the Table 1 each item in the rubric was rated on a four-point rating scale . = limited, 2= basic, 3= proficient, and 4= exemplar. Data Analysis The data collected was analyzed with MFRM, which is a development of rasch model measurement for the purpose of multi raters Firstly, the raw scores data of teachersAo PCK observation using the rubric were collected from the six raters. then the data were keyed in into Microsoft Excel, after that a specific coding to analyses the data is prepared to be calculated by Facets software version 3. The MFRM analysis included in this study are three facets raters, prospective science teachersAo PCK, and 12 items. The MFRM calibration in this study is based on severity (C) of the raters . affects the assessment of the probability of teachersAo PCK . on rubric items . to determine the threshold category . for the raters . with the following equation: Table 1. The Descriptions of the PCK Rubric Items PCK Component Description Knowledge of science Knowledge of studentsAo understanding in science Item Code Source Big idea of this topic BIT Lesson plan Understanding the Sequencing learning USL Understanding about the importance of this UIT Understanding the prior knowledge UPK Understanding the difficulty of this topic UDT Lesson plan Knowledge of Designing computer animations media instructional strategies and representations for teaching Teaching by computer animation Designing the role playing media DCA Lesson plan TCA Class observation DRP Lesson plan Teaching by role playing demonstration TRP Class observation Designing real object media DRO Lesson plan Teaching by real object observtion TRO Class observation Undertanding the weaknesses and Strengths of the implemented teaching strategies UWS Reflection Measuring TeachersAo Pedagogical Content Knowledge Using Many-Facet Rasch Model This equation encloses teachersAo PCK ( n ), rubric itemsAo difficulty level ( i ) , the difficulty level threshold (F. , and ratersAo severity (C. (Boone. Staver, & Yale, 2014. Englehard, 2. As an advancement of Rasch model. MFRM is a logistical measurement model based on probability then logarithm which is able to provide information on ratersAo accuracy, reliability, and validity, indices to be used as criteria are the value of outfit means-square (MNSQ) , a standardized outfit (ZSTD), and point measure correlation (Pt. MeaCo. (Boone, et. , 2014. Bond & Fox, 2015. Linacre, 2. RESULTS AND DISCUSSION Result The Quality of PCK Assessment Rubric Items The MFRM analysis of the data as Table 2 below presents a summary of the statistics on the reliability and separation index of both the items and the raters of the MFRM analysis results. Both of ratee and item reliability is classified as very good for a measurement, its value more 8 (Bond & Fox, 2015. Linacre, 2. The high item reliability shows that all items define the latent variable well. This means that the twelve items can be relied upon and can be used in different groups of respondents. Whereas the item separation index shows item difficulty range (Perera. Sumintono, & Jiang. In this study, the item separation index 0 and value of standard deviation clearly indicates a good distribution of item difficulty (Fisher, 2. This criterion shows that this PCK assessment instrument is suitable and reliable to identify the science teachersAos PCK. Whereas separation index for ratee indicates how well this rubric can assess AupeopleAos abilityAy in terms of latent nature, namely PCK in teaching that specific topic. Table 2. Reliability and Separation Report of MFRM Analysis Logit Mean Standar Deviation Separation Index Reliability Standard Error Ratee Item Table 3. Psychometrics Attributes of Items Infit Outfit Item Measure Model MNSQ ZSTD MNSQ ZSTD Correlation PtMea TCA DRP DCA TRP UWS DRO TRO BIT UIT USL UPK UDT Mean Cakrawala Pendidikan. Vol. No. October 2019 doi:10. 21831/cp. The overall quality of rubric items for assessing teacherAos PCK is shown at in Table 3. is found that the logit values of all items/measure fall within 2SD and -2SD . ll between 2. 88 logi. indicating that there is no outlier The Measure correlation point values fall 32 1. 44 logi. 7 items . %) in the moderate category ( 1. 44 > LVI > -1. and one items . %) in the easy category (LVI < 44 logi. Overall, raters rated the prospective teachers were good in the first two dimensions, which are in knowledge of science curriculum and knowledge of studentsAo understanding in science. but the prospective teachers still struggle in terms of knowledge of instructional strategies and representations for teaching science dimension. Figure 1. Variable Map of Ratee and Items Measuring TeachersAo Pedagogical Content Knowledge Using Many-Facet Rasch Model Table 4. PCK Item Calibration Dimension Difficulty level Difficult Moderate Easy Knowledge of science curriculum BIT. USL. UIT Knowledge of studentsAo understanding UPK UDT Knowledge of instructional strategies TCA. DRP. DCA. TRP UWS. DRO TRO Prospective teacher skills in understanding student ability show they know better, even the raters rated they capable of understanding the difficulty of PNM topic . tem UDT). This showing that the prospective teachers have study and include that in their lesson plan and teaching Understanding of the PNM subject also not having problem by the prospective teachers, they are capable of that. Meanwhile, component of Auknowledge of instructional strategies and representations for teaching scienceAy dimension is classified as difficult and moderate items. On that component, items related to the use of computer animationbased learning strategies and role playing demonstration are classified as AodifficultAo items (DCA. DRP. TRP & TCA). Two items which are assosiated to the use of learning strategies by real objects include are AomoderateAo items (DRO and TRO). and one item connected to their understanding of the weaknesses and strengths of the learning strategies (UWS) they use are also classified as AomoderateAo items. The assessment of all raters revealed these four items (DCA. DRP. TRP & TCA) are not mastered by most participants. Only two participants with high person logit . ood teacher. , i. Is and Nr master these four items . ee Figure . DCA item is related to teachersAo ability to design learning assisted with computer animation learning media. The use of learning media is to help students to comprehend the arrangement and particle movement of solids, liquids, and gases. Meanwhile. TCA is an item that represents teachersAos ability to implement computer-based animation learning in classrooms. In addition. DRP item represents teachersAos ability to design learning processes using role playing demonstration to help students in understanding the arrangement and particle movement of matter. and item TRP is used to assess the teacherAos ability to implement role playing demonstrations in their respective classes. Cakrawala Pendidikan. Vol. No. October 2019 Disscussion Teaching and learning is a complex task. thus, it is very important for prospective teachers to be able to distinguish between the different components included in teaching practice. The ability to identify each component in the teaching practice receive wide attention from educational researchers, especially in the context of science education to assess and improve the PCK of prospective science teachers (Park, et. , 2011. Park & AUAUOliver, 2. This research area is full of challenges due to their difficulties in conceptualizing PCK (Aydeniz & Kirbulut. In this study, we develop instruments to measure teachers PCK on specific topic AuThe particulate nature of matterAy (PNM) and has been proved reliable and valid after being analyzed with many facet rasch models (MFRM). Thus, it can be used to assess teachersAo PCK in different sample groups. In addition, prospective teachers and in-service teachers should pay attention to important elements for effective teaching (Grossman, 2011. Bryan & Abell, 1. PCK teachersAo skill in this research devided into three groups, i. good, moderate and poor, where two teachers in good teacher, eleven teachers in moderate and three teachers in poor All samples have the same educational background, age and teaching experience. The overall samples are dominated by female, but all five male teachers are spread almost equally across three categories . ood, mederate and This shows that male and female teachers have the same potency to develop their PCK The results of this study suggest that the prospective middle school teachersAo PCK for teaching AuThe particulate nature of matterAy or PNM is relatively low, because only 12. 5% AUAUof prospective teachers have high PCK. This result is not surprising considering that the participants are novice teachers (Baxter & Lederman, 1999. doi:10. 21831/cp. NRC, 1996. Van Driel, et. , 2. and PCK is directly related to the practice of teaching and learning which is a complex task (Grossman. Exploring the PCK development for prospective teachers who participate in a year training program is quite interesting because quality of Indonesian teachers is of great interest (De Ree, et. , 2. In addition, assessment and analysis of each PCK component in a teacher training program or teacher professional development is useful for evaluating and designing the next program. Analysis with MFRM clearly illustrates the teacherAos ability in three PCK components, i. knowledge of science curriculum, knowledge of studentsAo understanding in science and knowledge of instructional strategies and representations for teaching science (Magnusson, et. , 1. The results indicate that the majority of teachers demonstrates low ability in terms of Au knowledge of instructional strategies and representations for teaching scienceAy, while the prospective teachers have low understanding of knowledge of science curriculum and knowledge of studentsAo understanding in science. These findings suggest that teachersAo understanding of knowledge of science curriculum and knowledge of studentsAo understanding in science influence the understanding of representations and conceptual teaching strategies. This is in line with Sen. Oztekin & Demirdogen . that content knowledge supports knowledge of learning strategies and even studentsAo understanding of The MFRM analysis, revealed that they prefer the use of real objects to learn the differences on the macroscopic nature of solids, liquids and gases. They have not been able to determine and implement effective learning strategies to understand their submicroscopic properties, namely differentiating the nature of particles in solids, and liquids. In fact, understanding of the submacroscopic nature is important to be mastered by middle school students, because this is the basis in learning other science concepts (Snir, et. , 2003. Gabel. Learning the abstract nature of particles is a challenge for teachers, but middle school age students are already in the formal operational stage so teachers must be able to facilitate it (Toplish, 2010. Nakhleh, et. , 2005. Ayas, et. , 2. The very striking result in this study is the majority of prospective teachers do not master computer animation-based learning strategies to visualize the distance and motion of particles in Even though they are millennials who are already familiar with information technology. Likewise by visualizing it using real objects with role playing demonstration where both of these strategies have been proven effective to facilitate students in understanding the kinetic theory of matter related to the distance and motion of particles in solids, liquids and gases (Tsai, 1999. Purvis, 2006. Boz & Boz, 2. This case shows that most of science teachers in the case of the professional teacher development program are still struggle to teach science at middle school level. It indicated area for professional development in Indonesia (De Ree, et. , 2. Most teachers in the study need to be trained in designing and implementing effective learning strategies. They seem to be more subject to regulation and bureaucracy, so, they conducted learning in class as what is guided in the teacherAos manual on the implementation of the latest curriculum. This is in line with BjorkAos . finding that teachers in Indonesia are reluctant to improve authority and independence . eaching creativit. This is possibly due their limited competences in terms of content knowledge and pedagogical This assumption needs to be proven by further research, although some studies have shown that teacher competence in Indonesia is indeed low (De Ree, et. , 2. These results provide benefits for both prospective science teachers and teacher The prospective teachers gain insight that teaching is not simple, but there is knowledge and understanding that must be possessed. Deeper content knowledge will guide them in establishing the right learning sequence and perspective for middle school students on that In addition, the teachers will be inspired to teach the science concepts appropriately and effectively (Grossman, 2. for middle school students. The PNM topic is very useful as a basis for learning other scientific concepts (Haidar, 1997. Snir, et. , 2003. Tsai, 1. In the end, the teachers reflect on whether the learning strategies that have been implemented are effective to help students gain the right concepts of science. When both prospective Measuring TeachersAo Pedagogical Content Knowledge Using Many-Facet Rasch Model science teachers and teacher educators are aware that PCK is related to how teachers teach a specific topic to help students understand science concept comprehensively, teacher educators should more emphased on TSPCK in teacher professional development (Van Driel & Berry . Gess-Newsome, 2. Each topic has its own characteristics, so teacher training will not be effective if it is carried out en masse, both in terms of the number of training participants and in the content curriculum training (Nealsen. This study is limited to only explore the middle school science teachersAo PCK on a specific topic AuThe particulate nature of matterAy. PCK-based teacher professionalism should be directed at teaching certain topics (Gess-Mewsome, 2. Thus, this research is expected to motivate PCK research on various topics of teaching at various levels of school and other disciplines. Given the relatively low quality of Indonesian education, using PCK to improve the quality of teaching in the context of teacher professional development is important (Loughran, et. , 2004. Van Driel & Berry. CONCLUSION MFRM analysis has provided complete information related item quality, item difficulty and the ability of prospective science teachersAo PCK. Those information includes the ability of PCK in each of its components which is useful in analyzing the strengths and weaknesses of teachers of professional competence. The majority of respondent in this case, need to improve their PCK specifically in terms of Knowledge of instructional strategies and representations for teaching scienceAy. The analysis results show that most of prospective teachers are still struggle to teach science at the middle school level because they are not optimal in teaching differences in the submicroscopic nature of materr. Most of them are still dominant in teaching differences in solid, liquid and gas based on macroscopic nature. Mastery of effective learning strategies related to the arrangement and distance of particles in the matter is still low, such as the using of computer animation and role playing demonstrations to describe the distance and motion of particles in solids, liquids and gases. The ability of PCK Cakrawala Pendidikan. Vol. No. October 2019 in the components of Knowledge of science curriculumAy and AuKnowledge of studentsAo understanding in science is also only at a moderate level. ACKNOWLEDGEMENTS This research funded by the Director General of Research and Development Strengthening the Ministry of Research. Technology and Higher Education of the Republic of Indonesia No. 3 / E / KPT / 2018. REFERENCES