International Journal of Language Education Volume 7.
Number 2, 2023, pp.
ISSN: 2548-8457 (Prin.
2548-8465 (Onlin.
Doi: https://doi.
org/10.
26858/ijole.
Enhancing StudentsAo Learning Outcomes through MALL in TOEFL Preparation Class for Railway Mechanical Technology Damar Isti Pratiwi Politeknik Perkeretaapian Indonesia Madiun.
Indonesia Email: damar@ppi.
Rengganis Siwi Amumpuni Universitas PGRI Madiun.
Indonesia Email: rengganissiwi@unipma.
Ainun Fikria Politeknik Perkeretaapian Indonesia Madiun.
Indonesia Email: ainun@ppi.
Riana Eka Budiastuti Universitas Muhammadiyah Semarang.
Indonesia Email: riana@unimus.
Received: 7 August 2021 Reviewed: 21 August 2022-December 2022 Accepted: 5 December 2022 Published: 30 June 2023 Abstract This study attempts to investigate studentsAo learning outcomes in TOEFL preparation class through Mobile Assisted Language Learning (MALL) for Railway Mechanical Technology.
The program was designed as a TOEFL preparation class that utilized Google Form.
Quizizz, and Kahoot! As game-based learning (MALL platfor.
It was a case study on the 3rd year students of the Indonesian Railway Polytechnic (N=.
The observation was conducted during 20 meetings .
The data were collected through pre-test and post-test, then analyzed using descriptive statistics, paired t-test.
Pearson r correlation, and Cohen d effect size.
The passing grade score determined by the college .
was also an additional consideration analysis.
The results indicate significant impacts and a large effect on studentsAo learning outcomes after joining the English Intensive Program through MALL in TOEFL preparation class.
Nonetheless, the program implementing MALL had a moderate correlation and was less effective in enhancing studentsAo learning outcomes.
These findings shed light on the benefits and constraints of implementing MALL in the English Intensive Program.
Moreover, some recommendations are given for teachers/instructors in TOEFL preparation class.
Keywords: Educational technology.
English intensive program.
ESP.
MALL.
learning outcomes Vol.
No.
2, 2023
International Journal of Language Education Introduction Based on Railway Act No 23/2007.
Indonesian Railway Polytechnic is a vocational college under the Ministry of Transportation of the Republic of Indonesia, which was established to create human resources who are competent, qualified, and professional in railway transportation.
One of the competencies is mastering English as an international language which has to be proven by taking TOEFL in the last semester.
The head of Human Resource Development on Transportation Agency instructed that all students in the college under the Ministry of Transportation must achieve passing grade scores as a graduation requirement.
In Indonesian Railway Polytechnic, the passing grade score was 480 based on the stakeholdersAo requirement to hire their employees.
Therefore, to achieve that passing grade score, the language unit of Indonesian Railway Polytechnic holds English Intensive Program (EIP) as a TOEFL preparation class to prepare the studentsAo skills in doing TOEFL.
Generally, every college has EIP to give intensive courses with specific purposes for It needs students to be full-time studying in this program (Szasz, 2.
In Indonesian Railway Polytechnic.
EIP is held for the third-year students during the 1st-semester break to provide full-time studying.
The program lasted for 40 hours which was conducted in a week.
an intensive course, students are required to attend the class 3 meetings per day .
meeting = 2 This program is started by pre-test and ended with a post-test to measure studentsAo achievement before and after joining the EIP.
After finishing the program, students have to take TOEFL as their graduation requirement.
Since March 2020, the Indonesian government was urgently asked all students to learn from home to reduce the spread of the novel coronavirus.
COVID-19.
All schools have to respond immediately by implementing fully online classes and utilizing e-learning platforms (Bailey & Lee, 2.
Specifically, in language learning, language instruction through Mobile-Assisted Language Learning (MALL) has proven to engage studentsAo interaction (Zain & Bowles, 2.
This model could be one best choice in conducting online classes.
Moreover, some previous research highlighted the positive impacts of implementing MALL for language learners (Chen & Lin, 2018.
Pratiwi & Waluyo, 2022.
Sato & Burden, 2.
and empowers teachers in developing materials (Boonmoh et al.
, 2021.
Khoshnevisan, 2.
Indonesian Railway Polytechnic responds to online learning needs by providing an internal e-learning tool called Learning Management System (LMS).
This platform offers room for teachers to upload the syllabus, materials, task, or quiz.
The students could also submit their assignments or quiz on this platform.
A discussion column is also available, which could be used to have interaction between teachers and students.
For having online classes, teachers usually utilize zoom meetings.
These digital technologies are also implemented in EIP.
Furthermore.
EIP also uses a game-based learning model to engage students during the teaching and learning process and improve their motivation (Akour et al.
, 2020.
Kariadi & Pratiwi, 2022.
Plass et al.
, 2.
Research questions The main objective of this study is to explore studentsAo learning outcomes after studying an English Intensive Program for Railway Mechanical Technology implementing MALL at Indonesian Railway Polytechnic.
Hence, the study addressed the following research questions:
How do students' learning outcomes in TOEFL preparation class for Railway Mechanical Technology differ before and after joining the program? How are the studentsAo improvements in TOEFL preparation class? Vol.
No.
2, 2023
Pratiwi.
Amumpuni.
Fikria & Budiastuti Literature review Mobile-Assisted Language Learning (MALL) Mobile-Assisted Language Learning is the technology of language learning with the aforementioned mobile devices' implication (Wagner et al.
, 2.
Mobile devices have two main characteristics: portability and connectivity (Miangah & Nezarat, 2.
Portability enables learners to learn every time and everywhere.
Those provide learners to learn in a non-classroom environment anytime and can be taken to different places due to their small size and weight.
Connectivity helps learners to be connected and communicate with the learning website to assess learning materials.
This could also connect with other devices to assess data and customize it for individual learners.
MALL is harnessing mobile technologies such as mobile phones, tablets, or laptops due to their portability and capability to accommodate spontaneous and personal modes of language learning (Hussein & Barzani, 2.
It includes several traits deemed to facilitate meaningful learning activities (Zain & Bowles, 2.
, bringing a huge transformation in foreign language learnersAo attitudes towards language learning.
Pratiwi and Ubaedillah .
mentioned that mobile technology greatly influenced the implementation of some techniques and methods of foreign language teaching.
Several previous studies have pointed out MALL's contributions in enhancing studentsAo learning outcomes and English skills (Miangah & Nezarat, 2012.
Wagner et , 2016.
Yudhiantara & Nasir, 2.
Yudhiantara & Nasir .
found that students had a positive perception and attitude toward mobile phones and MALL to support classroom activities.
Students could use mobile phones to read e-books, play audio and video files to visualize, and operate a dictionary to solve vocabulary difficulties.
Another research that used the game-based learning method .
ne of the MALL method.
showed that mobile learning activities connected the real-world knowledge and the gamesAo visual world (Gholami & Azarmi, 2.
The materials were designed to integrate aspects of the learnersAo needs, accessed inside or outside the classroom.
Furthermore.
MALL led to a new quality and reflected modern tendencies in education to the fullest by providing constant access to study resources at any time and in any place (Wagner et al.
, 2.
Besides, it activated all types of verbal activities and additional opportunities for all language learners.
Mobile learning was also potentially beneficial to enhance learnersAo professional language and digital competencies necessary in the professional world (Casay-Pitarch & Candel-Mora, 2.
It also promoted learnersAo language development, reduced anxiety, and increased motivation towards learning (Kemal, 2.
Based on studentsAo interaction, this learning model provided learners with the opportunity to have close interaction and conversation (Miangah & Nezarat, 2.
Many empirical studies have shown MALL advantages in language learning.
However, this model had constraints, such as a small screen that made reading activities difficult (Miangah & Nezarat, 2.
Teachers had to be aware of those limitations regarding the learners' tasks when carrying out MALL.
For teachers, creating materials to fit in the small screen was another constraint, besides general problems such as network connection problems and slow processing time due to the limited power of storage (Ahmed et al.
, 2020.
Ubaedillah & Pratiwi, 2.
Moreover, creating highly efficient MALL materials would fall on the teachers to supply students with media in and out of class.
Based on the advantages, challenges, and constraints regarding implementing MALL in language learning, this study implements Quizizz and Kahoot! as a digital platform.
Quizizz received positive comments as game-based learning to improve studentsAo learning outcomes Vol.
No.
2, 2023
International Journal of Language Education (Izzah, 2021.
Pratiwi et al.
, 2.
Kahoot! was regarded as interactive game-based learning, which received positive feedback from the learners (Basuki & Hidayati, 2019.
Chiang, 2020.
Pratiwi et al.
, 2.
Those two online tools could also be set into online game-learning in class or as interactive homework.
The learners can access anytime and anywhere outside class using their mobile devices.
So, learning activities ran more flexibly.
TOEFL preparation course as English intensive program English intensive program refers to a course in which students participate in a higher number of classes in a shorter period (Mukundan et al.
, 2.
Some studies showed that this program was a matter of controversy as an effective strategy in achieving studentsAo learning outcomes (Abouzeid, 2.
Kops .
found that an English intensive course effectively enhanced studentsAo achievement and reported positive feedback.
On the other hand.
Nasiri & Shokrpour .
described that students in the intensive program might not digest compressed materials and condensed knowledge due to the time limitation.
Thus, they would not perform their best compared with the regular course.
In the EFL of Lebanese context, the English Intensive Program effectively improved studentsAo English language proficiency by improving their CEFR level (Abouzeid, 2.
This study highlighted that intensive program could be successful by optimizing prioritized learning Ae focusing on a task or project in an intensive program.
However, when it dealt with the TOEFL preparation program, the intensive program was only effective for enhancing studentsAo scores, yet their language interaction was low (Alderson & Hamp-Lyons, 1.
Students who caused this only took the TOEFL course, not the real English course where English was a means of The focus was not on studentsAo need to communicate but also on the content (Khattak et al.
, 2.
A previous study conducted in the Indonesian university context (Masfufah, 2.
revealed that an English intensive program on TOEFL preparation class motivated the students in learning English and assisted the students in achieving the passing grade score .
Yet, some challenges were found during the program.
In that program, time limitations and studentsAo low basic skills were the main challenges.
The teacher had to give additional time to every meeting and provided group work and homework to practice more at home.
The TOEFL preparation class faced other challenges in different schools, such as studentsAo lack of vocabulary mastery, fatigue, passive attitude, and different language proficiency (Akmal et al.
, 2.
Therefore, it was suggested to provide a regular class for the TOEFL preparation program, not in an intensive program.
English Intensive Program designed for this study was for TOEFL Paper-Based Test (PBT) preparation class.
There are three parts of TOEFL PBT: .
listening comprehension, .
structure and written expression, and .
reading comprehension.
As a time-shortened course, a program lasted for six days .
meetings per da.
, and 1 TOEFL PBT skill was discussed in 2 days .
While learning in an online class, the teacher utilized MALL to practice TOEFL with the students in an interactive game-learning.
After class, students were also given links in Quizizz and Kahoot! for individual practice outside the class in which they could use their mobile devices.
Research method This is a case study applying quantitative analysis to find out studentsAo learning outcomes, the effectiveness of the implemented strategy, and its effect size.
The research is designed as a classroom research-based utilizing one group pre-test and post-tests design.
The participants of this study were 3rd-year students of the Railway Mechanical Technology study program at Vol.
No.
2, 2023
Pratiwi.
Amumpuni.
Fikria & Budiastuti Indonesian Railway Polytechnic who joined the English Intensive Program in 2021 (N=.
There were 44 males and 2 females with their ages ranged from 20 Ae 21 years old.
The following research schedule .
was an English intensive program design used in this study.
There were 20 meetings in total .
meeting = 2 hours.
1 day = 3 meeting.
The program lasted for eight days, including a pre-test on the first day and a post-test on the last day.
Each meeting implemented specific game-based learning as MALL used in the teaching and learning process in class and outside class .
The pre-test and post-test were in Google Form to make it easier to be administered.
Meeting Day Table 1.
Research schedule Activity Pre-test
Listening Comprehension Listening Comprehension Listening Comprehension Listening Comprehension Listening Comprehension Listening Comprehension Structure and Written Expression
Structure and Written Expression
Structure and Written Expression
Structure and Written Expression
Structure and Written Expression
Structure and Written Expression
Reading Comprehension Reading Comprehension Reading Comprehension Reading Comprehension Reading Comprehension Reading Comprehension Post-test
MALL
Google Form Quizizz Quizizz Quizizz Kahoot! Kahoot! Kahoot! Quizizz Quizizz Quizizz Kahoot! Kahoot! Kahoot! Quizizz Quizizz Quizizz Kahoot! Kahoot! Kahoot! Google Form The pre-test and post-test as the primary instruments were taken from the Official Guide to the TOEFL Test fifth edition published by McGraw Hill Education 2017.
All items were valid and reliable as standardized tests were taken from English Testing Services (ETS).
The result of pretest and post-test would be analyzed using Normalized-gain scores to compare the average score of pre-test and post-test results.
Pearson correlation coefficient was used to know the relationship between pre-test and post-test results.
In contrast, the Cohen d correlation coefficient determined the effect size of pre-test and post-test results.
HakeAos theory on normalized-gain (N-gai.
score could be counted using the formula below and interpreted based on table 2 (Coletta & Steinert, 2.
("#$%&)(("#)*)
N gain score = (" ,-)(("#)*) Vol.
No.
2, 2023
Note:
Spost Spre Smax International Journal of Language Education = average post-test score = average pre-test score = maximum score Table 2.
N-Gain score category N Gain Score Category 76 Ae 1.
Very Effective 56 Ae 0.
Effective 41 Ae 0.
Less Effective 01 Ae 0.
Least Effective Not Effective at All Then descriptive analysis through SPSS was conducted to know Skewness Kurtosis point to determine the data distribution.
If the data were normally distributed, the paired t-test could be calculated using the formula below (York, 2.
(Oc D)/N (Oc D) *Oc D Oe ( N ) (N Oe .
(N) Note:
Oc ya = sum of the differences N = total number of samples Pearson correlation coefficient was calculated using the formula below and interpreted based on table 3 (Muijs, 2.
Oc01(/ .
/ Oe x.
/ Oe y.
Oe .
SD2 SD3 Note:
x = pre-test score y = post-test score x0 = mean of pre-test yc0 = mean of port-test SD2 = standard deviation of pre-test SD3 = standard deviation post-test Table 3.
Effect size of pearson correlation coefficient Pearson Coefficient Pearson Correlation Interpretation 0 O r Ou 0.
Very strong Strong 8 > r Ou 0.
Moderate 5 > r Ou 0.
Vol.
No.
2, 2023
Pratiwi.
Amumpuni.
Fikria & Budiastuti 3 > r Ou 0.
1 > r Ou 0.
Modest Weak Pearson r coefficients vary between -1 and 1, with 1 indicating a perfect positive relationship and -1 a perfect negative relationship, and 0 = no relationship.
Cohen effect size was denoted as the following formula and interpreted based on table 4 below (Cohen et al.
, 2.
ycc= ycu 000/ Oe ycu yc Note:
ycu/ = mean of post-test ycu.
= mean of pre-test s = standard deviation Table 4.
Effect size of cohen correlation coefficient Cohen Coefficient Cohen Correlation Interpretation Strong d > 0.
Moderate 8 Ou d > 0.
Modest 5 Ou d > 0.
Weak 2 > d > 0.
After calculating the statistical formula, the last step of calculation was categorization whether the post-test results had passed the passing grade score .
, which was determined by the college.
Results Descriptive statistics were conducted on studentsAo scores in the pre-test and post-test.
The data were normally distributed based on values of Skewness and Kurtosis between -2 and 2 (Skewness = -.
71 and Kurtosis = .
so paired t-test calculation could be counted.
The normalizedgain score was 0.
45, which meant that using MALL in English Intensive Program for Railway Mechanical Technology was less effective.
Table 5.
Descriptive statistics Test Mean Min Max Skewness Kurtosis N-Gain Score PrePost- Paired t-test was conducted to compare the means of studentsAo scores before and after joining the English Intensive Program for Railway Mechanical Technology at Indonesian Railway Polytechnic.
In total, the means of studentsAo score (N=.
indicated an increase from 402.
76 (SD
= 44.
54 (SD = 38.
, t.
= 9.
356, p <0.
This meant that the English Intensive Program through MALL significantly impacts studentsAo achievement.
Vol.
No.
2, 2023
International Journal of Language Education Mean PosttestPretest Table 6.
Paired t-test Paired Differences 95% Confidence Std Std Interval of the Error Deviation Difference Mean Lower Upper Sig .
-taile.
Pearson correlation coefficient was calculated to find out the correlation between pre-test and post-test scores.
Table 7 indicated the results of the Pearson coefficient, which showed 0.
This meant that the correlation between pre-test and post-test results was moderate.
Post-test Pre-test Table 7.
Pre- and post-test correlation Post-test Pearson Correlation Sig .
-taile.
Pearson Correlation Sig .
-taile.
Pre-test The effect size of the pre-test and the post-test score was calculated through the Cohen d Table 8 revealed that the Cohen d coefficient was 1.
38, which meant that the effect size of implementing MALL in the English Intensive Program for Railway Mechanical Technology Program was strong.
Test PrePost- Table 8.
Pre- and post-test effect size Mean Cohen d Passing calculation, the results were shown in table 9.
There were 28 students .
who did not pass the passing grade score and 18 students .
13%) who passed the passing grade score for TOEFL categorization based on Indonesian Railway Polytechnic standard.
Score
<480
480 and up Table 9.
Passing grade categorization Number of Students Percentage Category Failed Passed Discussion The findings from this study revealed some essential information regarding the English Intensive program in TOEFL preparation class through MALL.
The normalized-gain score showed Vol.
No.
2, 2023
Pratiwi.
Amumpuni.
Fikria & Budiastuti that this was less effective in improving studentsAo learning outcomes.
The correlation of pre-test and post-test results was also moderate, shown by the Pearson correlation coefficient.
However, paired t-test calculation described that TOEFL preparation course through MALL had significant impacts on studentsAo achievement, supported by Cohen d results.
The Cohen effect size implied that the program had a strong effect size on studentsAo scores.
Yet, the significant impacts and strong effects still left more than half of the students could not pass the passing score determined by the These findings support the previous study (Sato & Burden, 2.
that investigated the impacts of MALL on learners.
The study revealed that learning L2 materials through MALL had a significant effect size on learners.
However, the effect of the visual aids was relatively shortterm, while that of the verbal aids appeared to continue longer.
It was suggested that teachers had to manipulate advanced technology to develop online learning contingency plans to mitigate any anticipated educational challenges (Bailey & Lee, 2.
The teacher should actively integrate online classes, provide further explanations and reply to online discussion posts that were important in answering studentsAo questions (Puspitasari & Weng, 2020.
Ubaedillah et al.
, 2.
In this study, the teacher tried to acquaint the learners with the ongoing activities and give them another chance if their first attempt fails.
Quizizz and Kahoot! activities created for homework .
fter class activitie.
allowed students to repeat the activities as many as needed.
Chuang .
stated that those strategies contained the provision of adequate information and gave more practice time for the students.
Furthermore, the teacher created group work to create peer support which could promote learnersAo participation.
This was functional to satisfy both the learnersAo learning and psychological needs.
Leaners' anxiety management was also controlled by creating a comfortable classroom environment and instructional procedure, which encouraged studentsAo involvement.
Zain and Bowles .
described some benefits of MALL, such as language performance development, positive attitude towards the learning process, motivation skills, improvement of studentsAo retention, collaborative learning models, and extensive learning opportunities.
This study also revealed that learning through MALL improved studentsAo performance, shown by studentsAo learning outcome improvement.
The learners also demonstrated a positive attitude and high motivation during the teaching and learning process.
During 20 meetings of the English Intensive Program, none of the students was absent nor came late more than 10 minutes.
Regarding studentsAo retention, this study did not explore more due to the time limitation of the program.
Collaborative learning had created between learners when they started to discuss in group work.
This model could promote a reduction in learnersAo anxiety levels to optimize the learning interaction (Wu.
Extensive learning opportunities led to autonomous learning through the accessibility and portability of mobile devices (Ngui et al.
, 2.
Mobile devices accommodated an engaging and encouraging language learning setting which could be done anytime and anywhere that triggered studentsAo initiative to learn English.
In this study, the extensive learning opportunity was still limited on resources given by the teacher.
The learners have not had any initiative to explore more on TOEFL preparation materials.
On the other hand.
English Intensive Program through MALL acknowledged some The tight schedule made some students feel frustrated and tired to complete the assignment, especially the homework .
utside clas.
, due to the time-consuming assignmentsAo This could be shown in table 10, which uncovered learnersAo motivation in exploring the materials outside class.
The data described that learnersAo anxiety increased so that their Vol.
No.
2, 2023
International Journal of Language Education motivation to practice the materials got lower day by day.
The total number of studentsAo works submissions decreased significantly from the first day to the last day of the program.
Platform Quizizz Kahoot! Table 10.
Game-based learning report Number of Students Practiced on MALL Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Other challenges were technical limitations such as insufficient mobile devices and internet Mobile devices needed to be charged so the learnersAo participation could not be maintained well during the English Intensive Program.
The insufficient mobile devices also led to the restriction on audio-visual materials and reading contents, which could displease students.
This affected the process since the teacher could not maximize the facilities to create an effective learning environment (Manan et al.
, 2.
The teacher was highly encouraged to design various mobile-assisted tasks to cater to different learnersAo needs and preferences (Chen & Lin, 2.
Internet connection always became an irritating issue during the online teaching and learning This led to the failure of instructional purposes as inaccessibility of the materials (Bahari & Salimi, 2.
Learning from the results of the study, researchers would suggest that TOEFL instructors should not only focus on the technological applications, but also the strategies and activities which utilize the applications (Pratiwi et al.
, 2.
MALL is a range of technology that can have particular effect on particular parts of the English syllabus (Wyse & Jones, 2.
Thus, the instructors should see MALL as useful tools which, with the appropriate pedagogy, can enhance teaching and learning (Pratiwi et al.
, 2022.
Ubaedillah et al.
, 2.
Certain concern regarding instructorsAo competencies in integrating MALL and the course syllabus may arise, so the instructors need to learn continuously to create appropriate atmosphere that meets studentsAo needs.
Furthers, the online discussions in small groups have to be more intense among students through peer correction feedback with instructorsAo supervision and guidance in order to boost studentsAo learning outcome.
The last but not least, schedule arrangement of the TOEFL preparation class has to be taken into account in order not to create tiredness and boredom which would lead into learnersAo anxiety and demotivation.
Conclusion This study focused on MALL implementation in English Intensive Program for Railway Mechanical Technology students.
The program was designed for the TOEFL preparation class before the students took TOEFL as their graduation requirement.
The MALL platform used in this study were Google form .
or pre-test and post-tes.
Quizizz, and Kahoot! .
or treatmen.
The test analysis revealed that English Intensive Program through MALL had significant impacts and a strong effect size with moderate correlation on pre-test and post-test.
Yet, it was less effective in enhancing studentsAo learning outcomes and achieving the passing grade score .
Nonetheless, this study did not explore whether it was caused by the program or the implementation of MALL since both were entity variables in this study.
Furthermore, those findings led to the benefits and constraints of implementing MALL in the English Intensive Program.
Although learnersAo anxiety arose due to the tight schedule and technical limitations, the findings exposed more benefits: language performance development, positive attitude towards Vol.
No.
2, 2023
Pratiwi.
Amumpuni.
Fikria & Budiastuti learning process, motivation skills, and collaborative learning model.
It was strongly suggested that the teacher develop a didactic strategy to reduce the constraints (Khoshnevisan, 2.
Moreover, as much as this study intends to offer, it has some limitations to be acknowledged.
this was a case study in a vocational college, the generalization could not be made for all language There needs to be more research on larger participants from general universities or those with different passing grade standards for TOEFL scores.
The different statistical analyses might be added to get more in-depth insight regarding the results.
Future studies are suggested to implement experimental research designs to add more discussion around this area of research.
Yet, both research questions were successfully addressed by the statistical analysis presented in the previous section of this study.
Declaration of conflicting interest There is no conflict of interest in this paper.
Funding acknowledgment No funding was received for this paper.
References