PROCEEDING
Al Ghazali Internasional Conference The Future is Now: Adaptation to the WorldAos Emerging Technologies e-issn.
IS THE STUDENT WITH THE HIGHEST SCORE CONSIDERED THE SMARTEST?
MAKING INDIVIDUALISED EVALUATIVE DECISIONS BASED ON DIGITAL
COGNITIVE DIAGNOSTIC ASSESSMENT IN READING
Giati Anisah Universitas Nahdlatul Ulama Sunan Giri Email: giati@unugiri.
Abstract This research highlights the importance of evaluating students' reading skills holistically, not just based on final grades.
A digital cognitive diagnostic assessment was used to detect students' accuracy and tendency in answering questions, helping to identify students who answered guesswork despite achieving high scores.
Using a descriptive quantitative method, a reading test was conducted on 70 students through a digital diagnostic assessment application.
The results were analysed using the RASCH model.
This study found that some students with high scores showed a pattern of guessing.
In the context of differentiated learning, this finding proposes that high-scoring but less conscientious students can be grouped separately from more able students.
In conclusion, this in-depth assessment is important for more accurate evaluative decision-making so that teachers can provide appropriate interventions for students' reading development.
Keywords: reading learning, cognitive diagnostic assessment, grades.
Introduction Assessment of students' reading skills is often based on the final score or grade they get in a test.
However, high scores do not always reflect real reading comprehension and skills, because there is a possibility that students answer questions without thoroughness and just guess, which results in their scores appearing high.
This situation raises concerns about the accuracy of assessing students' abilities that solely depend on the final score.
Therefore, digital cognitive diagnostic assessment is a promising alternative because it can diagnose students' reading ability more comprehensively.
This assessment not only records students' answers, but is also able to make quick corrections, monitor answer patterns, and provide more accurate data in determining the level of accuracy of students in answering questions.
Cognitive diagnostic assessments have emerged as a powerful tool for evaluating reading comprehension skills, providing detailed feedback on students' strengths and weaknesses (Li et , 2.
(Mirzaei et al.
, 2.
(Toprak & Cakir, 2.
These studies have developed and validated cognitive diagnostic assessment frameworks for various contexts.
This research has consistently identified several attributes involved in reading comprehension weaknesses (Li et , 2.
(Mirzaei et al.
, 2.
Advanced modelling techniques, such as the G-DINA model and log-linear cognitive diagnostic modelling, have been used to analyse and refine these assessments (Li et al.
, 2.
(Toprak & Cakir, 2.
In addition, recent developments in cognitive diagnostic computerised adaptive testing (CD-CAT) have shown promise in improving measurement precision and efficiency while reducing test processing burden (Li et al.
, 2.
Indonesia, various previous studies examined the relevance of diagnostic assessments in measuring students' academic ability.
The first study by Santoso .
showed that high scores in reading tests can be obtained without deep comprehension, indicating the inaccuracy of PROCEEDING Al Ghazali Internasional Conference Volume 1.
Desember 2024
PROCEEDING
Al Ghazali Internasional Conference The Future is Now: Adaptation to the WorldAos Emerging Technologies e-issn.
evaluating only from scores.
Secondly, research by Hasanah et al.
, .
found that digital assessments provide an advantage in assessing students' thinking processes directly through their answer patterns.
Third.
Yuliani's research .
highlighted the effectiveness of digital cognitive assessments in detecting student inattention that is not detected in traditional Fourth, a study by Nugraha .
found that students with low accuracy but high grades hindered the effectiveness of differentiated learning in the classroom.
Fifth, a study by Rahmawati & Sutrisno .
proved that digital diagnostic assessment allows teachers to map students' abilities more deeply and accurately.
However, these studies have not been able to show how to make evaluative decisions that are truly individualised for each student.
This research emphasises the novelty of using digital diagnostic assessments to make individualised evaluative decisions on reading learning based on the accuracy and pattern of students' answers.
The focus of this research is to make individualised evaluative decisions on students' reading ability based on digital cognitive diagnostic assessments.
It aims to describe students' abilities more accurately, identify students who guess, and determine appropriate grouping strategies in differentiated learning.
The benefit of this research is that it provides a more holistic assessment of students' abilities, which is not only fixated on final grades.
By detecting patterns of guessing, this research helps teachers to provide more appropriate interventions, such as grouping according to ability and providing special treatment.
The findings are expected to help teachers make more accurate evaluative decisions that benefit the overall development of students' abilities in the classroom.
Literature Review Digital Diagnostic Cognitive Assessment A review of the literature on digital cognitive diagnostic assessments shows that they are evolving as an important tool in education to evaluate students' overall academic ability beyond just the final score (Li et al.
, 2.
Cognitive-based digital assessments allow for deeper detection of aspects such as rigour and guesswork, making it a very useful diagnostic tool in differentiated The digitisation of assessments allows educators to make more precise measurements, as it combines real-time data from studentsAo answers with the ability to automatically adjust according to the difficulty of the questions, and therefore has a higher accuracy in identifying studentsAo learning difficulties than traditional methods.
Many studies confirm the effectiveness of digital assessments in detecting student errors and inaccuracies.
Research by Hasanah et al.
, .
found that digital-based assessments provide an opportunity to understand how students process questions directly, which was previously difficult to do through traditional tests.
Another study conducted by Rahmawati & Sutrisno .
shows how digital cognitive diagnostic technology enables a more thorough and accurate mapping of student abilities, identifying the need for early intervention for students who often guess rather than understand questions.
In an educational context, these digital systems not only offer a more in-depth evaluation and enhance students' learning experience through immediate feedback integrated with assessments, allowing teachers to immediately identify students who require additional guidance.
Some studies suggest that immediate feedback from digital assessments can improve students' skills in the long run, especially in areas that require rigour and deep understanding.
The above explanations show that digital cognitive diagnostic assessments have an important role in detecting studentsAo abilities comprehensively, beyond the final test results, and focusing more on studentsAo cognitive patterns.
These assessments are beneficial in differentiated learning by providing in-depth, real-time data for teachers to make more accurate evaluative decisions, enabling interventions to be tailored to individual students' learning needs.
PROCEEDING Al Ghazali Internasional Conference Volume 1.
Desember 2024
PROCEEDING
Al Ghazali Internasional Conference The Future is Now: Adaptation to the WorldAos Emerging Technologies e-issn.
Reading Learning Reading is a process of interpreting written text that involves comprehension of written symbols and cognitive processing.
Reading is an activity that combines decoding and comprehension processes through interaction between the text and the reader's background It involves the integration of prior knowledge with new information in the text.
This process is called interactive because the reader actively connects the information obtained with the knowledge already possessed.
Reading is also understood as a multidimensional activity that includes literal, critical and creative comprehension (Snowling et al.
, 2.
This activity involves basic skills to the ability to analyse more complex texts.
Based on the various views above, reading can be summarised as an interpretative and interactive activity that involves linguistic and cognitive abilities to understand the meaning of texts.
This definition reflects the complex and layered nature of reading.
Reading learning is important for students' academic and social development.
Good reading skills can help students understand various disciplines and improve critical thinking In learning to read, assessment is needed.
Reading assessment is the process of evaluating students' reading ability that includes literal, inferential, and critical comprehension (Afflerbach.
This assessment is important to assess the effectiveness of reading learning and identify students' learning needs.
Traditional methods of measuring reading ability often use written tests that focus on literal comprehension.
However, these assessments are less effective in detecting reading inattention or guessing behaviour.
Conventional assessments are often unable to identify cognitive errors that may arise during the reading process.
Technologies such as computerised assessment help track patterns of errors in reading and identify the origin of guessing.
This allows teachers to analyse students' thought processes and provide appropriate interventions.
Students with low comprehension tend to guess when answering questions.
This behaviour can interfere with the interpretation of assessment results and lead to misunderstandings regarding students' reading ability.
(Kramer et al.
, 2.
states that accurate assessment in reading should be able to detect students' true level of comprehension, including identifying inattention and guesswork.
This is necessary so that teachers can provide more appropriate support.
According to (Black & Wiliam, 2.
, formative assessment plays an important role in monitoring students' progress at regular intervals, helping educators to identify and address comprehension issues before it is too late.
The feedback provided in reading assessments can help students improve their comprehension.
Hattie & Clarke, .
found that constructive feedback significantly improved students' text comprehension skills.
Diagnostic assessments play a role in identifying specific aspects of reading learning that need attention.
Research by (Fletcher et al.
, n.
) shows that quality assessments can significantly improve students' reading and comprehension skills by providing educators with insights to design appropriate interventions.
Comprehension-based teaching methods are more effective in improving students' reading skills than text-based methods.
Technology-based assessment development offers a more flexible and adaptive approach to assessing reading skills.
According to (Koenig & Martin, 2.
technology allows more accurate analysis of students' errors in Reading learning requires comprehensive and accurate assessments to detect errors or guesswork that may arise.
By adopting technology-based assessments and adaptive approaches, educators can gain deeper insights into students' understanding of texts.
Research Method PROCEEDING Al Ghazali Internasional Conference Volume 1.
Desember 2024
PROCEEDING
Al Ghazali Internasional Conference The Future is Now: Adaptation to the WorldAos Emerging Technologies e-issn.
This research uses descriptive quantitative methods.
Descriptive quantitative research aims to collect numerically measurable data to describe existing characteristics or phenomena without manipulating variables.
In this study, the main objective is to identify and analyse students' reading ability based on quantitative data collected through digital diagnostic cognitive The population in this study were all phase D students at MTs Abu Darrin.
Bojonegoro, totalling 630 students.
From this population, a sample of 70 students was taken randomly .
andom The random sampling technique is used so that each student in the population has the same opportunity to be selected as a sample, so that the research results can represent the population more objectively.
Descriptive analysis was conducted using the Rasch method, specifically through the scalogram analysis technique.
The Rasch method is an analytical technique that is often used in educational measurement because it can provide accurate estimates of individual abilities based on the answers given.
By using this model, the research can detect inaccuracy or guessing behaviour in students' answers in reading assessment.
The scalogram in the Rasch model is able to show the pattern of students' answers in detail.
This pattern helps in identifying whether the student understands the question or just guesses.
Scalogram analysis works by sorting students' answers based on the difficulty level of the question, so that inconsistent response patterns can be seen.
If students tend to answer correctly for easy questions but incorrectly on difficult questions, this indicates a logical pattern of However, if there is a discrepancy, such as answering correctly on difficult questions but incorrectly on easy questions, this could be an indication of guesswork or Discussion In traditional education, grades .
n the form of number.
are often used to measure how well students understand a subject matter or master a competency.
From this perspective, grades reflect a student's achievement or level of success in meeting set learning objectives.
Grades are also often considered an indicator of students' academic competence.
High grades can be interpreted as evidence that students master certain material or skills, while low grades are considered to indicate a lack of such understanding or skills.
This perspective focuses on measurable outcomes, such as test scores, standardised test scores or project work.
In a more inclusive approach to learning, grades reflect not only the end result, but also the student's effort, perseverance and learning process.
This approach prioritises valuing student development, measuring growth from starting point to achievement.
Such an approach recognises that each student has a different learning journey and values the process, not just the end result.
In some contexts, grades are used to detect students' strengths and weaknesses in a competency (Jang & Sinclair, 2.
For example, diagnostic tests provide grades designed to indicate areas where students need additional guidance or have demonstrated competence.
From this perspective, grades become an evaluation tool that assists in the development of more individualised learning plans.
However, can the final grade be a complete reflection of the student's ability? This research attempts to uncover how teachers cannot trust grades as the only sign of student ability.
This research makes use of digital cognitive diagnostic assessment on reading skills and then analyses studentsAo work using RASCH modelling through WinStep application to find out the true distribution of studentsAo abilities.
Scalogram is a tool used to visualise the results of item response theory (IRT)-based diagnostic analysis or cognitive diagnostic analysis (Sumintono.
PROCEEDING Al Ghazali Internasional Conference Volume 1.
Desember 2024
PROCEEDING
Al Ghazali Internasional Conference The Future is Now: Adaptation to the WorldAos Emerging Technologies e-issn.
& Widhiarso, 2.
This feature serves to assess learners' competencies based on their answer patterns to a series of questions, showing the extent to which learners have mastered the various sub-competencies being measured.
In this analysis, question items and student responses are categorised based on the level of difficulty and dimension of competency being measured.
After the analysis is conducted.
Winstep compiles a scalogram in the form of a matrix diagram that shows students' answers .
orrect or incorrec.
in relation to each item.
These patterns show the extent to which students succeed or fail to master the competencies tested.
The patterns also show students who simply answered despite their high scores.
There are three types of texts tested, namely explanation text, exposition text, and discussion text.
The three texts were selected based on the analysis of learning outcomes in the independent curriculum Stage D.
The question indicators of the 24 reading questions are as Table 1.
Question indicator Text Type Question Question indicator Explanation 1 describe the main idea of an explanatory text relate the content of the explanatory text with common sense choose the reason for the formation of the idea in the explanatory Classify the correct information based on the content of the explanatory text Give examples of things described in the text in real life Reveal unexplained information from the text by relating it to other Make predictions about future conditions based on the information in the explanatory text interpret the content of the text in the form of a chart Exposition describe the author's idea that the reader wants to be convinced of argue about the logicality of the argumentation in the exposition interpret the author's purpose conveyed through argumentation in exposition texts critique the appropriateness of the conclusion's generalisation to the thesis and argument give examples of the impact of not implementing the ideas in the find evidence used to support the author's arguments compare the emotive meaning of a word in the text with its denotative meaning correcting the appropriateness of the author's expertise and background to the theme of the text outline the statement that is the topic of the text PROCEEDING Al Ghazali Internasional Conference Volume 1.
Desember 2024 The Future is Now: Adaptation to the WorldAos Emerging Technologies PROCEEDING Al Ghazali Internasional Conference Discussion e-issn.
categorise the arguments of the pro and con sides validate the information in the text using other relevant information give examples of consequences that can occur if the ideas in the text are violated summarise the text that has been read diagnose the cause and effect of pro and con ideas select the better arguments between the pro and con sides by finding additional information from other texts provide arguments about the course of the discussion in the text In table 1.
It can be seen that there are eight questions for each type of text tested.
The eight questions were developed based on the Wiggins and Tighe taxonomy.
The Wiggins and Tighe taxonomy was developed as part of the Understanding by Design (UbD) framework, emphasising deep understanding in the learning process.
Wiggins and Tighe's approach helps students develop critical and analytical thinking skills, which are essential in reading nonfiction texts.
Students are not only asked to understand the content, but also encouraged to develop meaningful understanding and the ability to apply what they learn in other contexts (Wiggins, 2.
The following presents the 12 highest scores from the scalogram analysis of the cognitive diagnostic assessment of 70 students in the reading activity.
Order of questions from easy to difficult Order of low to high Figure 1.
Scalogram Test Results In the results of the Scalogram analysis in Figure 1 above, it appears that in the item section, the easiest questions .
ar lef.
to the most difficult questions .
ar righ.
Below that, in the person section, you can see the position of the students, with the highest score to the lowest.
There are 24 questions tested, based on the results of the scalogram analysis, it can be seen that the order of the easiest to the most difficult questions is as follows.
5 7 1
2 6 4 1
Easiest Hardest PROCEEDING Al Ghazali Internasional Conference Volume 1.
Desember 2024 The Future is Now: Adaptation to the WorldAos Emerging Technologies PROCEEDING Al Ghazali Internasional Conference e-issn.
Figure 2.
Order of easiest to hardest questions The order of difficulty of the questions is seen from the number of students who can answer.
With this data, it can be concluded that question number 9 was answered correctly by the most students, while question number 23 was answered correctly by the least students.
The student with the highest score is student number 45, he was able to answer correctly 21 out of 24 questions.
He was only wrong on questions number 2, 4, and 23.
The questions that were answered incorrectly were questions that were in the difficult category, so he could be said to be the most capable student in the class.
Students with numbers 3 and 9 get the same score and the same distribution of errors, namely in questions number 3, 13, 8, and 10.
Although they get good scores, these two students can be indicated as cheating in the form of cheating on each other because the scores and the location of the errors are exactly the same.
Indications of cheating or cheating on each other are also seen in students with serial numbers 56, 59, 65, and 68.
Their answers are exactly the same, both the location of the questions answered correctly and the questions answered incorrectly.
Teachers can consider the accuracy of their abilities apart from their scores.
Student numbers 10, 21 and 30 are students who fall into the category of less careful students, this can be seen from their ability to answer difficult questions, but incorrectly answer easier questions.
Students number 38 and 44 are medium ability students.
They were able to answer easy questions but had difficulty answering difficult questions.
From the sample of 12 answers the teacher can make groupings as follows.
38,44
10,21,30
56, 59,
65, 68
Most able student Students need the most help Figure 3.
Grouping of students based on CDA results PROCEEDING Al Ghazali Internasional Conference Volume 1.
Desember 2024
PROCEEDING
Al Ghazali Internasional Conference The Future is Now: Adaptation to the WorldAos Emerging Technologies e-issn.
Based on this grouping, teachers can provide appropriate treatment to students.
Students with high, medium, and low abilities can be given material according to their abilities, while students with cheating cases need to be given special treatment.
This is because their abilities are not reflected through the scores they get, so the teacher needs to give a retest to find out their original score.
The same method can be used for all students who take the diagnostic CDA Digital (Cognitive Diagnostic Assessmen.
is a technology-based evaluation system designed to provide an in-depth analysis of students' strengths and weaknesses.
Through comprehensively designed diagnostic tests.
CDA Digital can evaluate students' abilities in various aspects of reading.
Using data-driven analytic techniques, the system can provide relevant feedback to students, helping them to understand which areas need improvement and which are already well mastered (Sahin & Yildirim, 2.
For example, if a student shows weakness in understanding basic theory but excels in problems involving practical applications.
Digital CDA can highlight the theory topic as an area for improvement.
In addition.
CDA Digital also enables personalised learning tailored to each student's test results.
Based on the analysis of test results, the system can identify individual students' areas of strength and weakness and provide relevant learning materials to address those For example, if the system detects that a student has difficulty in understanding the main idea of a text, the teacher can provide additional training on the subject to the student.
This allows for a more efficient learning experience that focuses on the specific needs of the student, improving the overall quality of learning.
Tracking student progress is another important aspect of using Digital CDA.
The system not only identifies a student's weaknesses and strengths at any one time, but can also monitor changes in student performance throughout a given period.
By comparing test results taken at various times.
Digital CDA can assess if there is progress in areas that were previously weaknesses, or if there are still difficulties that need more attention.
This approach helps in designing appropriate and timely educational interventions, ensuring each student receives the support needed to reach their full potential (Tsai & Chou, 2.
Therefore.
Digital CDA serves as an effective tool in supporting individualised learning that is more adaptive and responsive to student development.
CDA provides deep insights into students' understanding and skills on various dimensions of learning, which allows for more targeted and focussed teaching.
As a result, teachers can obtain more detailed information that allows them to design more specific and adaptive teaching according to students' individual needs (Yan & Boud, 2.
As such.
Digital CDA focuses not only on the end result, but more on the process and progression of students' learning, which can help motivate them to keep improving.
By analysing students' weaknesses and strengths in more detail.
CDA allows educators to tailor the instruction they provide to students, especially in the context of differentiated Differentiated learning demands the use of various approaches to meet students' individual learning needs.
Based on the results of CDA analysis, teachers can determine more precisely which areas require further reinforcement or understanding, allowing them to design more effective strategies for students with different needs (Choi & Lee, 2.
For example, students who struggle with basic concepts can get additional practice or teaching with a more visual or practical approach, while advanced students can be given more complex challenges.
This differentiated treatment is not based on grades alone, but should show the students' true abilities by RASCH analysis.
In addition, the results of CDA analysis help in designing more productive and adaptive learning groups.
In differentiated learning, student grouping strategies are crucial to ensure that PROCEEDING Al Ghazali Internasional Conference Volume 1.
Desember 2024
PROCEEDING
Al Ghazali Internasional Conference The Future is Now: Adaptation to the WorldAos Emerging Technologies e-issn.
each student works at the appropriate ability level.
CDA provides data that allows teachers to group students based on their strengths and weaknesses, creating groups that can support each other or enrich the learning process.
This not only enhances students' learning experience, but also accelerates the achievement of their learning goals (Kim & Lee, 2.
Thus.
CDA analysis facilitates more dynamic and effective classroom management, where each student can learn in a way that best suits their learning style and pace.
Another impact of using CDA in differentiated learning is the improvement of teachers' ability to provide more specific and constructive feedback.
Based on the diagnostic test results, teachers can provide feedback that is more focused on areas that need improvement, improving the overall learning process of students.
This relevant and data-based feedback helps students to better understand their strengths and weaknesses, as well as knowing the concrete steps that need to be taken to improve their performance.
This is in line with the principle of differentiated learning that emphasises on personalised learning, where each student gets attention and support according to their learning needs (Ng & Tan, 2.
In other words.
CDA analysis enables the implementation of differentiated learning that is more effective and responsive to students' individual needs.
While Digital CDA offers many advantages, implementing this technology in the classroom is not without its challenges.
One of the main challenges is accessibility to adequate technological devices for both students and teachers.
Many schools, especially in resourceconstrained areas, struggle to provide the hardware and software needed to use Digital CDA effectively (Sahin & Yildirim, 2.
In addition, the unstable quality of internet connections can also hinder the use of these systems, necessitating a more integrated solution to address these infrastructure issues.
Therefore, access to the necessary technology is a significant constraint in the implementation of Digital CDA in some educational settings.
Another challenge faced in the use of Digital CDA is adequate teacher training.
Not only does it require an understanding of the technology itself, teachers must also be trained to interpret the data generated by the system and use it to effectively support student learning.
Without sufficient training, teachers may find it difficult to adjust learning based on the results of CDA analysis, so the full potential of this technology cannot be optimally utilised (Kim & Lee, 2.
Therefore, it is important for educational institutions to provide comprehensive training for teachers so that they can utilise this technology to support differentiated and personalised learning strategies.
Digital CDA can change the way teachers perceive students' intelligence and abilities.
Traditionally, intelligence is often assessed only based on academic ability as seen from test However, with Digital CDA, teachers are provided with a broader view of students' various cognitive abilities, including aspects that are often not reflected in conventional tests, such as critical thinking skills or the ability to solve problems creatively.
This leads to a more holistic understanding of intelligence, encompassing broader dimensions of student potential (Tsai & Chou, 2.
Along with these developments, teachers can design more inclusive learning experiences, which honour students' diverse ways of learning and motivate them to develop their full potential.
Conclusion This research confirms that high scores obtained by students in reading tests do not always reflect their true abilities.
There are various factors that can affect test results, such as inaccuracy in answering, reliance on guesswork, as well as cheating practices that often occur in conventional Therefore, relying solely on test scores as an indicator of students' intelligence or academic ability can be misleading.
In this context, the use of a standardised Cognitive Diagnostic PROCEEDING Al Ghazali Internasional Conference Volume 1.
Desember 2024
PROCEEDING
Al Ghazali Internasional Conference The Future is Now: Adaptation to the WorldAos Emerging Technologies e-issn.
Assessment (CDA) can provide a more accurate picture of students' cognitive abilities.
CDA not only identifies areas of strength, but also reveals weaknesses that may go undetected in conventional assessments.
Furthermore, the results from CDA analysis allow teachers to make more personalised evaluation decisions in learning.
By understanding students' real abilities, teachers can design more targeted differentiated learning strategies, as well as group students according to their needs and abilities.
This will lead to a more effective approach to learning and minimise potential errors in the assessment of students' abilities.
Therefore, the application of CDA in educational evaluation provides an opportunity to improve understanding and more appropriate teaching for each individual student, creating a more optimised learning experience that is based on accurate Thus, it is important for educators to integrate CDA as a tool in academic evaluation and lesson planning.
Through in-depth analysis of CDA results, teachers can determine the appropriate treatment for each student, facilitating more inclusive and individualised learning.
This also calls for training teachers in understanding and interpreting the results of CDA analysis, so that they can maximise the potential of this technology in education.
Bibliography