Qomaruddin M, Rahman A.A, Iahad N.A. (2014). User Acceptance Test of
Computer-Assisted Problem-Based Learning Assessment Tool (CAPBLAT).
Journal of Education and Learning. Vol.8 (1) 71-77.

User Acceptance Test of Computer-Assisted ProblemBased Learning Assessment Tool (CAPBLAT)
Muhammad Qomaruddin*
Universitas Islam Sultan Agung, Indonesia

Azizah Abdul Rahman**
Universiti Teknologi Malaysia, Malaysia

Noorminshah A.Iahad***
Universiti Teknologi Malaysia, Malaysia

Abstrak
Pembelajaran Berbasis masalah (PBL) merupakan metode desain instruksional yang digunakan oleh banyak
dosen untuk membuat pengalaman belajar lebih efisien dan bermakna. PBL menekankan dosen untuk
memfasilitasi pembelajaran kolaboratif. Mereka secara konvensional memainkan peran kecil dalam proses
penilaian secara formal. Peer-assessment dan self-assessment adalah metode yang paling sering digunakan oleh
dosen dalam penilaian di PBL. Makalah ini menyajikan penerimaan pengujian terhadap alat berbasis komputer
untuk peer-assessment dan self-assessment dalam metode pendekatan PBL, yang disebut Alat bantu Komputer
untuk Pembelajaran Berbasis Masalah (CAPBLAT). Dimana alat ini dirancang untuk membantu dosen
melakukan pengajaran dengan metode PBL dan membantu menilai mahasiswa dalam proses belajarnya.
Sebanyak empat puluh mahasiswa berpartisipasi dalam uji alat ini. Selama penelitian, kuesioner diberikan
kepada mahasiswa. Hasil mengenai penerimaan alat penilaian menunjukkan bahwa menggunakan CAPBLAT
dalam proses penilaian PBL mendapat penerimaan yang lebih baik dari dosen maupun dari mahasiswa.
Kata kunci: peer-assessment, self-assessment, Assessment, Problem-Based Learning

Abstract
Problem-based Learning (PBL) is an instructional design method that is used by many lecturers to create more
efficient and meaningful learning experiences. PBL places an emphasis on assisting the lecturers in facilitating
collaborative learning. They conventionally play a small role in the formal assessment process. Peer-assessment
and self-assessment are the most frequent methods of assessment employed by lecturers when conducting PBL.
This paper presents a user acceptance test of a computer-based tool for peer-assessment and self-assessment in
PBL, known as Computer-Assisted Problem-Based Learning Assessment Tool (CAPBLAT). The tool was
designed to assist lecturers in conducting the PBL teaching method and assess students‘ learning progress. A
total of forty students participated in the testing of the tool. During the study, questionnaires were administered
to students. The results concerning the assessment tool acceptance demonstrate that students showed positive
feedback and accepted the usage of CAPBLAT for the PBL assessment process.
`
Keywords: peer-assessment, self-assessment, Assessment, Problem-Based Learning

*

Muhammad Qomaruddin, Department of Informatics Engineering, Faculty of Industrial Technology,
Universitas Sultan Agung (unissula), Semarang – Indonesia.
E-mail: mqomaruddin@unissula.ac.id
**

Azizah Abdul Rahman, Department of Information Systems, Faculty of Computing, Universiti
Teknologi Malaysia, Malaysia
E-mail: azizahar@utm.my
***

Noorminshah A.Iahad, Department of Information Systems, Faculty of Computing, Universiti
Teknologi Malaysia, Malaysia
E-mail: minshah@utm.my

Introduction
Teaching and learning in the PBL approach differ from the traditional approaches. PBL
encourages students to be, specifically: active learners, self-directed learners and work together in a
group. It enables various methods of assessing students, including, for example: assessing the outcomes
of the PBL (such as a group project report) and assessing the performance of an individual student. The
most frequently used assessments in PBL are self-assessment and peer-assessment.
There are many advantages to be gained by utilizing self-assessment and peer-assessment.
Through self-assessment and peer-assessment, students are able to make judgments about how well they
have learned; rather than merely how much they have learned (Macdonald & Savin-Baden, 2004). Peerassessment helps to develop the acquisition of self-directed learning skills as students participate in the
assessment experience (Ballantyne, Hughes, & Mylonas, 2002). In addition, self-assessment and peerassessment provide valuable insights into the performance of students. It has been demonstrated that
peer ratings are a good predictor of future performance, and provide information regarding student
performance that is not measured by other traditional evaluation methods (Foreman, 2007; Sullivan,
Hitchcock, & Dunnington, 1999).
Strong support for self-assessment and peer-assessment is proven throughout literature studies;
however, difficulties have also been reported. Both types of assessments are time-consuming and thus
are regarded negatively by students. Students need to undertake the peer assessment more anonymously
and it is very difficult to assess peers within the confines of the PBL room (Papinczak, Young, &
Groves, 2006).
The use of computers in assessment in general is known as a definition of ―Computer-Assisted
Assessment (CAA)‖. Through this, CAA can provide advantages for both lecturers and students to
provide them with detailed formative feedback for their learning achievements compared to
conventional assessment. It also can reduce the somewhat tedious load of lectures by automating parts
of the task of marking students‘ work. Accordingly, the student receives an instant and objective score
together with specific and timely feedback. This is due to the detailed scoring data already being
digitized and recorded to central repositories in the students' records system (Bull & Danson, 2004).
This paper proposes an automated tool for PBL assessment by which to accomplish these
difficulties by utilising computers. In today‘s high technology environment, computers can play an
enormous role in the assessment process; in this case especially for peer-assessment and self-assessment
respectively. The paper will also discuss the acceptance test of the tool as received from the students.

CAPBLAT as Assessment Learning Tool
CAPBLAT (abbreviation for Computer-Assisted Problem-Based Learning Assessment Tool) is
a computer-based assessment tool designed for assessing students‘ progress in a Problem-Based
Learning (PBL) approach.
CAPBLAT serves the purpose of helping lecturers assess their students‘ learning achievement
in the PBL approach. Moreover, CAPBLAT can also assist students to, specifically: assess their peers‘
contributions, assess their own performances and monitor their individual learning achievement.
The CAPBLAT method was developed based on the framework of PBL assessment, named
PBLAsF (Problem-Based Learning Assessment Framework). This consists of seven instruments of
assessment methods, namely: 1) Peer-Assessment, 2) Self-Assessment
3) Individual-Activity
Assessment, 4) Group-Report Assessment, 5) Group-Presentation Assessment, 6) Pre-test and 7) Posttest/Final-test.
During the PBL session, all students are required to enter the login information to the tool. At
this point, they should insert a valid e-mail address and secret password for their sign in to the
CAPBLAT. Figure 1 shows the CAPBLAT login page screenshots.
The CAPBLAT performs numerous learning functions, including: providing learning material,
assessment rubrics, displaying results and feedback, facilitating the creation of a PBL session, a
publishing problem, generating group members, activating assessment and sending notification to the
students.
Among those features, this paper has only concentrated on reporting the utilization of
CAPBLAT for peer-assessment and self-assessment. Accordingly, during the research, students were
asked to use only two features of CAPBLAT, namely: peer-assessment tool and self-assessment tool.
In order to achieve the outcome (the course objectives and PBL objectives), identification of
criteria for each PBL objective was required (Montemayor, 2004). The research defines process skills
and scope of criteria for each of the objectives as outlined in Table 1. The four process skills,
specifically: ‗Knowledge building, knowledge work, communication, as well as critical analysis and

72

User Acceptance Test of Computer-Assisted Problem-Based Learning Assessment Tool
(CAPBLAT)

problem solving‘ have been determined based on the process skills commonly used by the PBL
practitioners who founded the literature studies.
The CAPBLAT also provided these process skills as described in Figure 1.
Table 1. The defined proces skills and scope of criteria.
No
1

Process Skills to be assessed
Knowledge building

2

Collaborative work

3

Communication skills

4

Critical analysis and Problem
solving

Scope of Criteria
- Application of Knowledge Base
- Self-directed learning (self-study)
- Demonstration of breadth and depth of knowledge
- Self-contribution (Team member contribution)
- Cooperation among team members
- Responsibility to team members
- Punctuality and Positive Influence
- Responsibility and commitment in all the team‘s tasks
- Attitude during discussion and professionalism
- Ability to communicate ideas clearly
- Being able to accept feedback with openness
- Positive reaction to feedback and criticism
- Critical analysis, reasoning and decision-making skills
- Provision of input which focuses on and is relevant to the case
- Self-awareness: Ability to accept and respond to criticism
gracefully

Figure 1. CAPBLAT Login Screenshots.

Research Method
The research was carried out in classrooms at the Universiti Teknologi Malaysia. A total of 40
students participated in this study, and they were divided into two classes (Class A for Theory of
Motivation and Class B for Behavior Disorder) and several groups. Each group consisted of four or five
members. All students involved were registered in the second semester, and this activity was their first
attempt in PBL learning and using a computer-based assessment tool for their peer-assessment and selfassessment. Quantitative analysis was then used to evaluate student technology (assessment tool)
acceptance.
Procedure
Before the commencement of the PBL class, an introduction of PBL was given to the students.
We also hosted a presentation of the use of CAPBLAT for the students, by means of a demonstration in
the classroom.
Qomaruddin M, Rahman A.A., Iahad NA. (2014). Journal of Education and Learning.
Vol.8 (1) pp. 71-77.

73

Figure 2. Student page designed to view problems, assessments and group member screenshots.

During the PBL session of the semester, students presented three problems as a trigger for their
work tasks in their respective groups (please refer to Figure 2 for examples of the problems). Each
group member was encouraged to share and discuss their work on the dedicated forum that has been
provided during the program. At the end of each problem, students were required to assess the online
form for their peers and their own performance. It is anticipated that these assessments would contribute
5% of the overall percentage of the course assessment. The peer-assessment and self-assessment form
consists of four process skills as depicted in Figures 3 and 4.
Each process skill consists of three or four criteria that have been adapted from Montemayor
(Montemayor, 2004), and Uden & Beamount (Uden & Beaumont, 2006). For each criteria of question,
students are required to assess their peers based on the four rating scale. This ranges from, namely: 1Poor, 2-Fair, 3-Good and 4-Excellent. Students have been reminded to make a fair judgment of their
peers based on each of the peer contributions.
Data Collection
A quantitative method was used to collect the data, which had been gathered via a
questionnaire. The questionnaire was administered to students for the purposes of examining technology
acceptance based on Technology Acceptance Model (TAM) by Davis (Davis, Bagozzi, & Warshaw,
1989).

74

User Acceptance Test of Computer-Assisted Problem-Based Learning Assessment Tool
(CAPBLAT)

Figure 3. Peer-Assessment Rubric Screenshots.

Figure 4. Peer-Assessment Screenshots.

Qomaruddin M, Rahman A.A., Iahad NA. (2014). Journal of Education and Learning.
Vol.8 (1) pp. 71-77.

75

Questions posed in the questionnaire were divided into two parts. In Part A, the questions were
determined based on three factors, namely: Perceived Usefulness (PU), Perceived Ease of Use (PEU),
and Attitude Towards Usage (ATU) with a total of 11 questions, as listed in Table 2. The 5-point Likert
scale from 5 for ―strongly agree‖ to 1 for ―strongly disagree‖ was used for the questions in Part A.
Meanwhile, Part B consists of open-ended questions in which students are invited to provide comments
and suggestions concerning the tool.

Result and Discussion
The result (Table 2) shows that scores for each question of PU, PEU and ATU are all (mean)
above 3,9 respectively. They range between ―Agree‖ and ―Strongly Agree‖. The students generally
agreed that CAPBLAT is useful, easy to use, and they have a positive attitude toward using it.

Table 2. The Questionnaire Questions
Factor
Question
Perceived Usefulness (PU)
Q1
The tool provides a comprehensive framework for PBL assessment which can be used effectively to assess
students‘ performance.
Q2
The tool is effective in assisting lecturers to assess their students by the PBL Method.
Q3
The tool helps lecturers save time in marking.
Q4
The tool is able to overcome the complexity and tediousness of the PBL assessment.
Q5
The tool effectively monitors student progress through more frequent assessments
Q6
The tool provides quick and specific feedback to students during problem sessions.
Perceived Ease Of Use (PEU)
Q7
Q8
Q9

The tool is easy to use.
It's easy to understand the contents of the tool.
The whole tool is well-developed & user friendly.

Attitude Towards Usage (ATU)
Q10
Q11

I like to use the application because it is easy and it is also quick to find and access information concerning
assessment.
I hope the application can provide more features for assessment and learning, so that other learning methods
can also use it.

Table 3. Statistics of Questionnaire Questions
Question
PU
Q1
Q2
Q3
Q4
Q5
Q6
PEU
Q7
Q8
Q9
ATU
Q10
Q11

Mean

Percentage (%)
SD

DA

N

A

SA

0
0
0
0
0
0

0
0
5
2,5
2,5
2,5

5
7,5
10
20
20
20

57,5
42,5
32,5
55
55
55

37,5
50
52,5
22,5
22,5
22,5

4,33
4,33
4,33
3,98
4,35
4,45

2,5
0
0

0
2,5
0

0
10
10

60
55
40

37,5
32,5
50

4,30
4,18
4,40

0
0

2,5
2,5

15
20

45
55

37,5
22,5

4,18
4,20

1=SD: Strongly disagree, 2=DA: Disagree, 3=N: Neutral, 4=A: Agree, 5=SA:
Strongly agree

76

User Acceptance Test of Computer-Assisted Problem-Based Learning Assessment Tool
(CAPBLAT)

Responses to question Q1 indicated that 95 % of students were in agreement that the tool
provides a comprehensive framework for PBL assessment by which to efficiently assess students‘
performances; while just 5% of them indicated a neutral position to this question. Responses to Q2
indicated that 92. 5% of students showed their agreement that the tool is effective in assisting lecturers
to assess their students by the PBL Method; while only 7.5% of them were shown as having a neutral
attitude to this question. Regarding responses from other questions in factor PU (Q3-Q6), more than 75
% of students showed their agreement, while the remainder chose neutral and disagreement stances.
According to the results of open-ended questions, it can be seen that most of the students gave
positive comments on their experiences with using the tool. Some of the comments included: ―the
system is good and accessible. It is user-friendly and gives students continuous feedback on their
progress throughout the course. Assessment is more reliable because each student was assessed by the
lecturer, peers and even themselves‖.

Conclusion
In this study, a computer-based tool for PBL assessment (CAPBLAT) was developed. The tool
has been designed to assist the lecturer conduct the PBL teaching process and assess their students‘
learning progress accordingly. The study involved students using the CAPBLAT for the purpose of,
specifically: reading the problem trigger, checking their progress, and assessing the online form for their
peers and their own performance respectively.
Reflecting upon the case study, the answers to the questionnaire as provided by the students
showed very positive feedback on the use of the tool. The students were in agreement that CAPBLAT is
useful, easy to use, and they displayed a positive attitude towards using it.

References
Ballantyne, R., Hughes, K., & Mylonas, A. (2002). Developing Procedures for Implementing Peer
Assessment in Large Classes Using an Action Research Process Assessment & Evaluation in
Higher Education: Routledge.
Bull, J., & Danson, M. (2004). A Briefing on Computer-assisted Assessment. Series 14,
Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989). User Acceptance of Computer Technology: A
Comparison of Two Theoretical Models. Management Science, 35, No. 8, 982-1003.
Foreman, M. (2007). Peer Assessment of Problem Based Learning—Fostering Reflective Practice in
SocialWork Students. Retrieved from www.aishe.org/readings/2007-1/
Macdonald, R., & Savin-Baden, M. (2004). A Briefing on Assessment in Problem-based Learning
Assessment Series No. 13. York Science Park, Heslington York YO10 5ZF: Learning and
Teaching Support Network (LTSN).
Montemayor, L. L. E. (2004). Formative and Summative Assessment of the Problem-Based Learning
Tutorial Session Using a Criterion-Referenced System. The International Association of
Medical Science Educators (IAMSE), 14.
Papinczak, T., Young, L., & Groves, M. (2006). Peer Assessment in Problem-Based Learning: A
Qualitative Study. Advances in Health Sciences Education, 12(2), 169-186.
Sullivan, M. E., Hitchcock, M. A., & Dunnington, G. L. (1999). Peer and Self Assessment during
Problem-based Tutorials. The American Journal of Surgery, 177(3 March 1999), 266-269.
Uden, L., & Beaumont, C. (2006). Technology and Problem-Based Learning: Hershey, PA :
Information Science Publishing.

Qomaruddin M, Rahman A.A., Iahad NA. (2014). Journal of Education and Learning.
Vol.8 (1) pp. 71-77.

77