Journal of English Language Teaching Innovations and Materials Nur Arifah Drajati.
Anis Handayani Article received Article accepted Published : 31st July 2024 : 14th September 2024 : 5th Month Year (Jelti.
Vol.
October Trust in Automated Writing Evaluation (AWE): Multiple case study on feedback engagement and literacy Nur Arifah Drajati 1*.
Anis Handayani 2 1 Department of English Education.
Universitas Sebelas Maret.
Indonesia nurarifah_drajati@staff.
* Corresponding Author 2 Department of English Education.
Universitas Negeri Yogyakarta.
Indonesia anishandayani@uny.
DOI: http://dx.
org/10.
26418/jeltim.
Abstract Artificial Intelligence (AI) for educational purposes has recently been a research Specifically for academic writing.
AI-based feedback, often called automated writing evaluation (AWE), is an essential aspect to support studentsAo However, despite its popularity, the trust in AWE, which potentially contributes to the studentsAo writing improvement, has been explored on a limited Hence, this multiple case study attempts to examine the trust in AWE and how this trust influences feedback engagement and literacy.
Of 42 students joining a writing class, a total of 4 undergraduate students were involved to explore this issue.
Considering the Indonesian government's policy to conduct online learning amidst the COVID-19 pandemic era, classroom observation, stimulated recalls, and semi-structured interviews were conducted virtually via Zoom meetings to gather the data.
Results show that the students' trust in AWE varied, indicated by their varied responses to AWE.
Furthermore, their trust in AWE only affects the behavioral aspect, excluding the cognitive and affective Meanwhile, in feedback literacy, their trust only affected the aspect of appreciating the feedback.
These findings imply that educators should pay interest to students' trust in AWE since it, in some ways, influences improvement of their feedback engagement and literacy.
Keywords: academic writing.
automated writing evaluation (AWE).
feedback literacy.
feedback trust How to cite this paper: Drajati.
, & Handayani.
Trust in Automated Writing Evaluation (AWE): Multiple case study on feedback engagement and literacy.
Journal of English Language Teaching Innovations and Materials (Jelti.
, 6.
, 125-144.
http://dx.
org/10.
26418/jeltim.
Journal of English Language Teaching Innovations and Materials (Jelti.
, 6.
, 125-144 Copyright A 2024 by Author, e-ISSN 2657-1617 Nur Arifah Drajati.
Anis Handayani To link to this article: http://dx.
org/10.
26418/jeltim.
This work is licensed under a Creative Commons Attribution NonCommercial-ShareAlike 4.
0 International License.
Technology development has influenced educational sectors, including studentsAo learning process.
As with the impact of prior COVID-19 pandemic, teaching and learning has adapted to technologically-based learning processes, as well as in writing class.
Technologies supporting academic writing have been studied for Technological tools such as Grammarly (Ranalli, 2.
Criterion (Ranalli et al.
, 2.
, and Pigai (Bai & Hu, 2.
have been proven to support academic As academic writing is closely related to writing feedback, those tools offer an essential feature to support the writing process by providing automated feedback or automated writing evaluation (AWE) which is expected to ease the studentsAo writing process and support writing improvement independently.
Prior studies demonstrated that AWE supports studentsAo learning and writing improvement (Hojeij, 2022.
Ranalli, 2.
This kind of feedback has been proven to have some potential for language learning such as noticing inaccurate writing as well as improving the rhetorical writing quality (Ranalli, 2021.
Wilken.
Furthermore, studentsAo willingness to make use of the feedback improved when they received explicit feedback such as vocabulary enhancement feedback from Grammarly (Ranalli, 2.
However, despite their beneficial use of AWE, some students doubt the accuracy of this automated feedback (Koltovskaia, 2020.
Zhang, 2020.
Zhang & Hyland, 2.
which results in ignoring the AWE feedback (Ranalli, 2.
Li et al.
revealed that students rely more on teachersAo feedback than AWE feedback.
It may happen since AWE is a tool that does not cover human-human trust.
Also, high expectations of the tool are associated with high trust in the tool (Lyons & Guznov, 2.
In contrast, one may abandon using the tool when it does not meet their expectation (Ranalli, 2.
It seems that the tools' performance and the students' initial expectations have an impact on their trust and engagement with the tools.
Engagement in written feedback has been studied by researchers (Gravett & Winstone, 2019.
Koltovskaia, 2020.
Ranalli, 2021.
Zhang, 2020.
Zhang & Hyland, 2.
Feedback engagement, which can be interpreted as studentsAo responses to such feedback (Ranalli, 2.
, has become a critical key to achieving successful feedback.
Furthermore, feedback engagement can be influenced by studentsAo individual characteristics and contextual factors (Zhang, 2.
Liu & Yu .
reported that students are more engaged in indirect feedback than direct feedback, but the percentage of their revision on direct feedback is higher than in indirect feedback.
Furthermore, they suggested that AWE needs the teacher's assistance to confirm the feedback accuracy, also found by Saricaoglu .
Jiang & Yu .
strengthen the claim that the impact of AWE on students' learning and writing quality is highly affected by the teacher's willingness to scaffold and provide the contextual correction.
Journal of English Language Teaching Innovations and Materials (Jelti.
, 6.
, 125-144 Copyright A 2024 by Author, e-ISSN 2657-1617 Nur Arifah Drajati.
Anis Handayani StudentsAo trust and engagement in feedback is highly believed to affect feedback literacy, which is one of predictors in the success of feedback utilization.
Carless & Boud .
defines feedback literacy as the need for "understanding, capacities, and dispositions" .
to make sense of the feedback and use it Furthermore, they show that feedback literacy has four features namely .
appreciating feedbackAeunderstanding and appreciating the feedback to improve writing quality, .
making judgmentsAedeveloping the capability to give academic judgments to their own and other's writings, .
managing affectAe maintaining positive imposition when receiving critical feedback, and .
taking actionAeaware of the obligation to revise the writing after receiving the feedback.
Students need to be involved in the frequent activity of generating and receiving feedback to develop their feedback literacy (Carless & Winstone, 2.
It is indicated that their activities in giving feedback to their peers and receiving it from other sources, such as AWE, help them foster their feedback literacy.
Also, studentsAo trust in AWE potentially affects their feedback literacy, as it determines how they perceive the feedback (Han, 2019.
Yu & Liu, 2.
Zhang .
reported that studentsAo engagement with AWE enhanced their feedback awareness, understanding, and acceptance.
StudentsAo feedback literacy can further be observed from their behaviors toward feedback given by writing assistant applications.
As a new platform in AI-based applications.
Scribo takes a role in supporting academic writing with its features to facilitate the teaching and learning process.
This website-based application developed by Literatu allows teachers to create classes that can be joined by their students, similar to the features of Google classroom.
However, unlike Google classroom.
Scribo also has a feature for checking students' writing accuracy by providing immediate and automated feedback covering many aspects of writing: usage, spelling, grammar, passive voice, cohesion, vocabulary, sentence, and readability.
Furthermore.
Scribo interestingly has more detailed feedback than those provided by other grammar-based applications.
It also has features in which users may take a test and get a grade from the application which can be used to measure their writing Focusing on the feedback feature, it has the potential to support the academic writing process.
However, as many studies reported.
AI-based applications, including Scribo, potentially deal with usersAo issues of trust that affect their feedback Chapelle et al.
proved that distrust of AWE can arise due to inaccurate feedback given by the tool.
It indicates that studentsAo trust is one key factor in successful writing feedback.
Besides, studentsAo engagement with the feedback has been proven to influence studentsAo success in using the feedback, specifically for those with high English proficiency (Zhang & Hyland, 2.
Furthermore.
Zhang .
reported that individual differences and contextual factors influence studentsAo feedback engagement.
Feedback engagement is one of the factors influencing feedback literacy (Wood, 2.
AeAuan ability to read.
Journal of English Language Teaching Innovations and Materials (Jelti.
, 6.
, 125-144 Copyright A 2024 by Author, e-ISSN 2657-1617 Nur Arifah Drajati.
Anis Handayani interpret, and use the written feedbackAy (Sutton, 2012, p.
This somehow indicates that the more engaged the students are in feedback, the more gain they will receive in their feedback literacy.
Reviewing this prior literature, researchers studied the trust issues with AWE (Chapelle et al.
, 2015.
Ranalli, 2.
, feedback engagement (Zhang, 2.
, and feedback literacy (Wood, 2.
concerning the success of writing feedback However, studies on how trust issues influence feedback engagement and literacy, which is closely related to successful feedback uptake and writing improvement, have remained unexplored, specifically in the EFL context.
Responding to this gap, this study attempts to explore the trust issues with AWE provided by Scribo for EFL students and its influence on their feedback engagement and literacy.
Our research explicitly seeks to discern the ways students respond to automated writing evaluation (AWE), in what ways issues of trust influence their feedback engagement, and in what ways issues of trust influence their feedback literacy.
METHOD
Research context and participants We adopted a multiple case study as each participant constitutes the unit of analysis and each case was bounded by the individual and the duration of his or her participation in the research.
This study was conducted in an Academic Writing class for English language education in a large university in Indonesia.
Six students were selected based on their level of English proficiency, computer self-efficacy, and prior experience using the Scribo application.
To represent the level of English proficiency, we selected two students with higher scores in their previous writing class (Mei and Ald.
and another two with lower scores (Dedi and Ich.
This participant arrangement was then chosen to explore the relation of English proficiency, trust issues of feedback, and their response to the feedback as further analysis of Zhang & Hyland's .
finding that students with high English proficiency benefit more from AWE than those with low English In obtaining studentsAo computer self-efficacy, an online survey were given to them, adopting the ICT self-efficacy scale framework of Li et al.
Table 1.
The ParticipantsAo Demography.
Participants Gender Age Academic ICT selfwriting score efficacy High Prior experience in using Scribo Mei Yes Aldi Intermediate Dedi High Icha Low Yes Journal of English Language Teaching Innovations and Materials (Jelti.
, 6.
, 125-144 Copyright A 2024 by Author, e-ISSN 2657-1617 Nur Arifah Drajati.
Anis Handayani In approximately three months .
ight meeting.
, the students were given a premium account for accessing Scribo.
Their ages were in the range of 18-23 years The scores presented in Table 1 are from their previous writing classes in the third semester.
Table 1 presents detailed information about the participants.
All names are pseudonyms.
The students in this class were expected to learn how to write a summary of academic articles from reputable journals.
This class consisted of eight meetings and was conducted virtually, supported by the Scribo and Zoom meeting applications.
Table 2 shows the studentsAo activities during those eight Table 2.
The students' activities in the Academic Writing class Meeting Activity Learning how to operate Scribo briefly and how to make a summary of academic articles from the lecturerAos explanation Searching for some articles from reputable journals based on the student's research interest Writing, checking, and revising their summary of the first chosen research article in Scribo Outside class activities: Exchanging their writings with peers and giving feedback on their peersAo writings Presenting and discussing the results of the peer feedback in zoom Lecturer giving feedback on the studentsAo writings Writing, checking, and revising their summary of the second chosen article from a reputable journal in Scribo Outside class activities: Exchanging their writings with peers and giving feedback on their peersAo writings Presenting and discussing the results of the peer feedback in zoom Lecturer giving feedback on the studentsAo writings Data collection Amidst the Covid-19 pandemic, we collected the data virtually, facilitated by Zoom meetings.
Before we began our data collection, we asked for participantsAo consent by sending emails informing them of our research aims, the kinds of data we needed, and how the data were collected, kept, and analyzed.
After they gave their consent, we observed their classroom activities in the Academic Writing class for eight meetings.
The observation focused on .
time spent in checking texts in Scribo, .
contents of the feedback they chose, and .
their decision to accept or ignore the feedback given.
As the observation finished, we then scheduled a Zoom meeting for six chosen participants according to their available time.
The screen recording videos dealt with how they responded to Journal of English Language Teaching Innovations and Materials (Jelti.
, 6.
, 125-144 Copyright A 2024 by Author, e-ISSN 2657-1617 Nur Arifah Drajati.
Anis Handayani the automated feedback given by Scribo.
The texts used were the summary of the first chosen article from a reputable journal they wrote in the third meeting, consisting of 100 Ae 200 words.
Each participant was observed separately in different Zoom meetings and time allocations.
While the participants were checking their texts in Scribo, we asked several questions about why they accepted, ignored, or skipped the feedback.
This observation and stimulated recalls lasted for about 30-40 minutes for each participant.
After the observation and stimulated recalls finished, they were then, interviewed regarding their feedback engagement and feedback literacy.
adapted the concept of feedback engagement from Ellis .
and feedback literacy from Carless & Boud .
to construct the questions.
These semistructured interviews lasted for 20-30 minutes for each participant.
We recorded the Zoom meetings to avoid data loss and ease the transcription and data analysis Data coding and analysis The screen recording data were coded manually by two researchers to avoid research bias and analyzed using thematic analysis.
For the data coding, we followed the feedback categories of Scribo, which consisted of grammar and spelling, vocabulary, sentences, topic coverage, and cohesiveness.
However, since we did not control the participants in choosing which feedback they wanted to have, we only used the most chosen feedback categories, namely .
grammar and spelling .
enceforth GS), .
vocabulary (V), and .
cohesive (C) as our themes for analyzing the data.
We two researchers, separately analyzed the screen recording videos by counting the number of feedback messages received by the participants and in which ones they accepted, ignored, or skipped the feedback, and analyzed their reasons for doing so.
Since the responses of AuignoreAy and AuskipAy have the same action meaning that they did not employ the given feedback, we used the term AuignoreAy to simplify our interpretation.
Thus, we used the term "accept" when the participants accepted and used the feedback and the term "ignore" when they ignored the feedback.
We then, discussed the results and wrote the report in agreement.
Figure 1 is an example of the GS category received by participants.
Audios of stimulated recalls and semi-structured interviews were transcribed using the Sonix application.
To ensure the accuracy of the transcription, we reviewed the transcripts by comparing the transcripts and the As the accuracy was guaranteed, a thematic analysis was adopted to analyze the data.
We first coded the interview transcript under themes of feedback trust .
enceforth FT), feedback engagement (Ellis, 2.
: cognitive (CE), affective (AE), and behavioral engagement (BE), and feedback literacy (Carless & Boud, 2.
: appreciating the feedback (AF), making judgments (MJ), managing Journal of English Language Teaching Innovations and Materials (Jelti.
, 6.
, 125-144 Copyright A 2024 by Author, e-ISSN 2657-1617 Nur Arifah Drajati.
Anis Handayani Figure 1.
The example of GS category feedback affect (MA), and taking action (TA).
We two researchers, separately coded and analyzed the data.
We then discussed the results to assimilate the results.
After an agreement was achieved, we concluded the results.
FINDINGS
EFL studentsAo response and trust in automated writing evaluation (AWE) After the participants learned how to make a summary of academic articles from reputable journals, they wrote and checked their summaries in the AI-based application Scribo.
In our study, we used their second summary in which they learned and became familiar with the application before analyzing their responses to this automated writing evaluation (AWE).
In checking their texts in the application, they were given the freedom in choosing the feedback they wanted to receive.
Due to various reasons, they mostly checked their texts regarding grammar and spelling, vocabulary, and cohesiveness.
Table 3 summarizes the participantsAo frequency of the received feedback.
Table 3.
The frequency of the feedback received Categories Grammar & Spelling Vocabulary Cohesive Low proficiency Dedi Icha High proficiency Mei Aldi In responding to this AWE, they have various responses and reasons behind their decision to accept or ignore the feedback.
Since each student may have a Journal of English Language Teaching Innovations and Materials (Jelti.
, 6.
, 125-144 Copyright A 2024 by Author, e-ISSN 2657-1617 Nur Arifah Drajati.
Anis Handayani different trust level and various reasons, we decided to explore each participantAos trust in AWE and his/her reasons behind his/her decision to accept or ignore the AWE.
Dedi In responding to AWE.
Dedi only accepted some feedback that he thought was correct.
He revealed that he always checked the context first before deciding to accept or ignore the feedback.
Also, he felt that some feedback may be incorrect because AWE is automatically given by an AI which probably does not consider the context as seen in the excerpt Dedi-FT as follows:
AuI only accept the feedback when I think it is correct as I experienced that sometimes this application gives contextually incorrect feedback.
Ay Dedi-FT Despite his low trust in this AWE, he accepted numerous AWE feedback messages .
%) from this application.
He reported that this application is mostly helpful in correcting his grammar and spelling, but not in other aspects.
Figure 2 shows DediAos response toward AWE from the application.
Accept Grammar Vocabulary Cohesive & Spelling Figure 2.
DediAos response to AWE Icha Icha only checked her text regarding grammar and spelling (GS) and cohesiveness (C).
Furthermore, in responding to the feedback, she mostly accepted the suggestions .
ee Figure .
Here, she revealed that the application helped her improve her writing quality by giving corrective feedback She realized that her writing skill is low, so she mostly accepted the feedback .
%) without further thinking, specifically in the cohesiveness Journal of English Language Teaching Innovations and Materials (Jelti.
, 6.
, 125-144 Copyright A 2024 by Author, e-ISSN 2657-1617 Nur Arifah Drajati.
Anis Handayani Ignore Accept Grammar Vocabulary Cohesive & Spelling Figure 3.
IchaAos response to AWE As a result of her rather strong trust in this AWE, she only ignored one feedback message, which was the suggestion to change the passive voice into an active voice.
She said that changing the passive voice to active would change the meaning of the sentence.
Thus, she decided to ignore the feedback.
It was revealed in the excerpt Icha-FT.
AuThe feedback is to change the sentence into I conclude or we conclude, but this article is not mine, so I can't use this feedback.
It will make the sentence carry a different meaning.
Ay Icha-FT Mei As a student with high proficiency in English, she checked all chosen categories of the feedback given by the application.
However.
Mei tends to believe in her own ability rather than a mere website-based application.
She revealed that this AI-based application can probably be wrong in giving feedback, so she only accepted a few feedback messages .
%) when she highly believed that they were correct, specifically in the context.
Consequently, she ignored the feedback more frequently than she accepted it .
ee Figure .
Ignore Accept Grammar & Vocabulary Cohesive Spelling Figure 4.
MeiAos response to AWE As an example, she ignored the feedback to change the word "and" in the text to avoid repetition.
She decided to ignore it since she believed that her use of Journal of English Language Teaching Innovations and Materials (Jelti.
, 6.
, 125-144 Copyright A 2024 by Author, e-ISSN 2657-1617 Nur Arifah Drajati.
Anis Handayani AuandAy was correct and did not need any change.
Similarly, she also ignored the cohesiveness feedback since she did not trust AWE feedback after experiencing incorrect feedback from a similar application.
The excerpt Mei-FT shows her change of attitude toward AWE due to her previous experiences.
AuThe use .
and in my text is already contextually correct, why should I need to change it? Also, in cohesive.
I often ignored the feedback because previously I received obviously incorrect feedback from a similar application.
It seems like I lost my trust in AI-based feedback.
Ay Mei-FT Aldi Aldi is typically a student who merges what he knows and the feedback In checking his text, he accepted the feedback to change some passive voices into active voices since he remembered his lecturer told him to use active Similar to other participants, he always tried to comprehend and match the context before accepting feedback.
It was revealed in the excerpt AldiFT as follows:
AuI accept it .
o change some passive sentences into active one.
since my lecturer told me to use active sentences, specifically in writing the summary.
Ay Aldi-FT However, he only checked his text regarding the grammar and spelling He accepted all the feedback received since he knew that the feedback was contextually correct and he made mistakes in grammar and spelling .
ee Figure .
Ignore Accept Grammar & Vocabulary Spelling Cohesive Figure 5.
AldiAos response to AWE Considering their results of the stimulated recalls indicating that they frequently check the accuracy of the feedback before accepting it, we may conclude the levels of their trust according to their acceptance frequency of the feedback given.
Accordingly.
Aldi is found to have the highest trust in AWE while Mei has the lowest trust.
Figure 6 illustrates the participantsAo trust level of AWE.
Journal of English Language Teaching Innovations and Materials (Jelti.
, 6.
, 125-144 Copyright A 2024 by Author, e-ISSN 2657-1617 Nur Arifah Drajati.
Anis Handayani Mei Dedi Icha Aldi Figure 6.
The level of participants' trust Ways the trust level of AWE influences feedback engagement As we aimed to identify the time spent by students in engaging in the feedback, the students had flexible time to check their writing with the Scribo writing checker until they found their writing AoappropriateAo.
Table 4 summarizes the time spent by students in checking their summaries consisting of 100-200 Table 4.
Time spent in engaging with the feedback Students Aldi Icha Dedi Mei Time spent in the first checking 11 minutes and 15 seconds 10 minutes and 7 seconds 12 minutes and 21 seconds 7 minutes and 18 seconds As presented in Table 4, their trust in AWE did not actually determine the amount of time spent in checking the writing.
For instance.
Dedi spent more time than Icha, who has higher trust.
Also, we can identify a notable impact of trust on Mei who had the lowest trust level and spent the least amount of time to check her writings.
Aside from the time spent, we can also break down feedback engagement into three types: cognitive, affective, and behavioral engagement.
Regarding cognitive engagement.
Icha who has a rather high trust in AWE among the participants, reported that she comprehended the feedback well since the application provided further explanation of the feedback in detail.
In checking her text, she read the feedback more than once to comprehend the feedback completely .
ee the excerpt Icha-CA).
AuI did not find any difficulty in understanding the feedback.
I mean all the explanation was there.
We can just read it several times if necessary to understand the feedback.
Ay Icha-CA Regarding affective engagement, despite their low or high trust in AWE, the participants were mostly glad in receiving critical or AubadAy feedback.
They revealed that receiving AubadAy feedback was not that bad since they felt that they Journal of English Language Teaching Innovations and Materials (Jelti.
, 6.
, 125-144 Copyright A 2024 by Author, e-ISSN 2657-1617 Nur Arifah Drajati.
Anis Handayani were novice writers who needed corrective and helpful feedback.
It was revealed by Dedi in the excerpt Dedi-AE.
AuI am glad .
n receiving any kind of feedback including the bad one.
Since this feedback is from a technology .
ot huma.
and for my better improvement.
I am not feeling down or sad in receiving the feedback.
Ay Dedi-AE Last, regarding behavioral engagement, low trust in AWE affected their motivation to revise their texts.
This phenomenon happened to Mei, the one with the least trust of AWE among the participants.
She reported that this feedback did not really motivate her to revise her text since she considered the feedback given by AI-based applications was often inaccurate .
ee excerpt Mei-BE).
AuHonestly, this AWE did not really motivate me to revise my text.
This kind of AI-based application often gives me inaccurate feedback, so I prefer peer or teacher feedback over AWE.
Ay Mei-BE Accordingly, high trust in AWE did motivate the receivers to revise their Aldi, one having the highest trust of AWE, was motivated to revise his text after receiving AWE.
Excerpt Aldi-BE revealed that this AI-based application helped him check his text automatically so that he could improve his text quality without help from other people.
AuI was quite helped in improving my text quality from receiving AWE from this When receiving the AWE.
I understand which parts are wrong, so I can fix them and make my writing improved.
This feedback gives me the motivation to revise and improve my writing quality.
Ay Aldi-BE In sum, the results of the interviews reveal that studentsAo trust on AWE did not affect their cognitive and affective aspects of feedback engagement, as they mostly show good comprehension of the feedback and positive emotion in receiving the feedback .
ee Table .
Furthermore, the only aspect affected by their trust in AWE is the behavioral aspect.
This indicates that as the students have higher trust levels, they potentially acquire high motivation to revise their work after receiving the feedback.
Table 5.
The summary of studentsAo feedback engagement Students Aldi Icha Dedi Mei Cognitive Yes Yes Yes Yes Feedback engagement Affective Yes Yes Yes Yes Behavioral Yes Yes Journal of English Language Teaching Innovations and Materials (Jelti.
, 6.
, 125-144 Copyright A 2024 by Author, e-ISSN 2657-1617 Nur Arifah Drajati.
Anis Handayani Ways the trust in AWE influences feedback literacy Adapting the theory of feedback literacy from Carless & Boud .
, we see the participantsAo feedback literacy from the four aspects: appreciating the feedback, making judgments, managing affect, and taking action .
ee Table .
The participants mostly have their feedback literacy improved, which can be seen from the observation of the sixth and seventh meetings of the class.
After receiving AWE, all the participants, except Mei, mostly consider AWE as one of the efforts in improving their writing quality.
They did not think that this AWE was there to judge their writings, but rather to help them check and improve the Aldi, with high trust in AWE, has a very positive thought of AWE as a helpful tool .
ee excerpt Aldi-AF).
Table 6.
The summary of studentsAo feedback literacy Students Aldi Icha Dedi Mei Appreciating the Yes Yes Yes Feedback literacy Making Managing Yes Yes Yes Yes Yes Yes Yes Yes Taking Yes Yes Yes Yes AuYes, this AWE is very helpful.
It does help in checking and improving my writing quality.
Ay Aldi-AF As predicted.
Mei, who has the lowest trust in AWE, had a completely different thought.
In excerpt Mei-AF, she revealed that AWE did not really help her improve her writing since she had all the knowledge to improve her writing from her previous class, even before receiving this AWE.
However, she agreed that this AWE helped her enrich her vocabulary.
AuThis AWE did not really help me in improving my writing since I got the theory of improving my writing from my previous class .
efore checking with this AIbased applicatio.
Ay Mei-AF Surprisingly a different case happened from the making judgments aspect.
Despite their trust in AWE, all participants felt that their confidence and insights into how to make judgments of texts improved after receiving AWE from this Furthermore, the AWE from this application covers many aspects of language such as grammar, spelling, vocabulary, sentences, cohesive, and even topic coverage.
Excerpt Dedi-MJ revealed his confidence in making judgments after receiving AWE feedback.
Journal of English Language Teaching Innovations and Materials (Jelti.
, 6.
, 125-144 Copyright A 2024 by Author, e-ISSN 2657-1617 Nur Arifah Drajati.
Anis Handayani AuYes, since this application gives feedback just like a mentor showing our mistakes .
n the writing.
Here we can take an example from this feedback giving, for example, when we have to give a peer review.
So.
I think this AWE enriches my knowledge on how to give feedback to others.
Ay Dedi-MJ Their improved feedback judgment was demonstrated in the second peer feedback activity in the sixth and seventh meetings.
While students only focused on the grammar aspect in the first peer feedback activity, they make a different judgment of their peers' writings in the second peer feedback activity.
They pointed out the vocabulary, cohesiveness, and readability of their peersAo writings.
Similarly, for managing affect, their trust in AWE did not influence their ability to manage their emotion in receiving feedback, even for AubadAy feedback.
After experiencing writing and checking their texts with this application, they considered AubadAy feedback as an aid to help them improve their writing skills.
They were not sad or AudownAy to receive such feedback since they knew it was for their writing improvement.
Furthermore, they were rather motivated to improve their writing quality after receiving this kind of feedback.
Excerpt Dedi-MA revealed that Dedi was motivated to revise and do better in his writing after receiving AWE feedback.
AuThe feedback I got from Scribo .
ven the bad one.
motivates me to do better in my writings.
Ay Dedi-MA For taking action aspects, all participants agreed that AWE made them revise their text right after receiving the feedback.
They said that since AWE was typically corrective feedback, they directly revised their text after receiving the For instance, it happens to Aldi in which he frequently revised his writing directly after receiving the AWE feedback .
ee excerpt Aldi-TA).
AuI directly revise my text, but I will recheck my text to make sure that the feedback is appropriate.
Ay Aldi-TA
DISCUSSION
To put it briefly, our results show that students have varied responses to AWE feedback.
This result is probably expected since individuals have differences in mindset as well as the level of trust in the AI-based application.
Furthermore, despite their English proficiency, the students were somehow selective in accepting or ignoring the feedback, supporting Bai & Hu's .
and Tian & Zhou's .
They comprehend the feedback first, suit it with the context, and accept it if they think the feedback is correct.
This finding also strengthens the claims of Chapelle et al.
that students do not take AWE Journal of English Language Teaching Innovations and Materials (Jelti.
, 6.
, 125-144 Copyright A 2024 by Author, e-ISSN 2657-1617 Nur Arifah Drajati.
Anis Handayani feedback blindly.
Considering the studentsAo responses to AWE, we can note that their proficiency in English does not really affect their trust in AWE.
This result is somehow in contrast with Huang & Renandya's .
conclusion that low proficiency students think highly of AWE.
Furthermore, our results interestingly show that their previous experience with AWE influences their trust more than their English proficiency.
This case happened to Mei who has previous experience with AWE from a similar After receiving several inaccurate feedback messages, she has low trust in AWE.
She believes that automation cannot be trusted 100% since it is not superior to human intelligence.
This finding somehow supports Cope et al.
that AI cannot surpass human intelligence.
Also, it strengthens Chapelle et al.
's .
conclusion that distrust may appear from inaccurate feedback given by an application, which indicates that negative experience from an application potentially decreases their trust in the application.
Regarding feedback engagement, students here have similar feedback engagement despite their English proficiency.
This finding is somehow in contrast with the previous studies (Koltovskaia, 2020.
Zhang, 2.
, which reported that higher proficiency students have higher feedback engagement.
Furthermore, rather than their English proficiency, their beliefs and trust in AWE contributed more to the feedback engagement.
For instance.
Icha, who has high trust in AWE, finds it easy to comprehend the feedback given since she decided to read the feedback and revise her text several times with AWE support, spending a longer time to engage with the feedback given.
Furthermore.
Ranalli .
revealed that studentsAo trust in AWE may decrease due to the limited capacity of the application to provide expected feedback.
Our study also gets similar results that the students here do not trust the feedback since this AI-based application is restricted in giving contextually accurate feedback which may be provided by peer or teacher feedback.
Consequently, the students sometimes get confused with the provided feedback.
Furthermore, the fact that they cannot ask for any further information from the tools does not help them comprehend the feedback (Ranalli, 2.
However, still, at some points, this feedback helps them improve their writing quality.
Lee .
also reported that AWE stimulated students to enrich their writing quality both in content and language.
A notable finding was spotted in relation to the amount of time spent in using Scribo.
Mei with the lowest trust spent the least amount of time to check her writings.
This means that studentsAo trust indirectly affects studentsAo feedback engagement, which supports Zhai & Ma's .
belief that studentsAo trust influences their judgment on the usefulness of AWE feedback, potentially leading to shorter time spent in using the application.
Application developers should put particular attention on the feedback experiences brought by the application as it highly affects the studentsAo success in utilizing the given feedback, as reported by Zhang & Hyland .
Journal of English Language Teaching Innovations and Materials (Jelti.
, 6.
, 125-144 Copyright A 2024 by Author, e-ISSN 2657-1617 Nur Arifah Drajati.
Anis Handayani Our results also show that studentsAo trust in AWE, in some ways, influenced their feedback literacy.
However, this phenomenon only occurred in one aspect of feedback literacy: appreciating the feedback.
This may imply that students with high trust in AWE appreciate the feedback more than the ones with low trust in AWE.
As they appreciate the feedback, they will be more motivated to revise their writing, which may result in long-term accuracy of their writing (Link et al.
, 2.
Furthermore.
Han & Xu .
reported that feedback literacy is multifaceted and that it is highly influenced by individual differences.
It seems to be the most possible cause for the students to have varied feedback literacy after receiving AWE.
As Jiang & Yu's .
research revealed the role of teachersAo assistance in utilizing AWE, it suggests that the same teachersAo assistance is needed here to optimize the impact of AWE, specifically in their feedback Our finding also revealed that past experiences on feedback does not necessarily affect studentsAo feedback literacy.
All students in this study fulfilled their criteria of having good feedback literacy.
This finding is somehow in contrast with Malecka et al.
that past experiences give different impacts on studentsAo behavior in utilizing feedback.
These different results likely occurred due different kinds of past experiences between this studyAos participants and Malecka et al.
's .
While the participants have positive and supportive experiences with feedback in MaleckaAos study, the participants here have negative and unsatisfied experiences with feedback, so that it leads to more negative attitudes and trust during later feedback experiences.
This particularly supports Walker et al.
's .
finding that negative past experiences demotivate students in revising their works in future events.
Viewed from our results, we can see that trust in AWE partly influences the studentsAo feedback engagement and feedback literacy.
This result somehow sheds light on ideas for educators to increase their studentsAo trust in AWE since it influences studentsAo engagement and literacy of feedback as well as their writing improvement.
CONCLUSION AND SUGGESTIONS
We may summarize that studentsAo trust in AWE may be varied depending on their experiences, personality, and beliefs.
Also, it comes to light that trust in AWE did influence studentsAo feedback engagement and feedback literacy, although only in some aspects such as behavioral engagement and appreciating the feedback aspects of feedback literacy.
Furthermore, our finding also indicates that whereas AI gives feedback more quickly and automatically, teacher guidance and facilitation are still needed to make use of this AI more efficiently and accurately.
However, since we analyze the data qualitatively, we cannot Journal of English Language Teaching Innovations and Materials (Jelti.
, 6.
, 125-144 Copyright A 2024 by Author, e-ISSN 2657-1617 Nur Arifah Drajati.
Anis Handayani prove this proposition statistically accurate.
To have a more accurate result on this issue, further analysis is warranted.
This study only used qualitative data and small data sources.
Thus, to achieve more accurate and generalizable results, further studies may add quantitative data and analysis to complete the results.
Also, studying similar topics with different research methods may be worthwhile to compare the Furthermore, this study examining the studentsAo trust in feedback may be further researched by studying their trust in other types of feedback such as peer or teacher feedback.
Last, since the studentsAo trust in AWE feedback partially affects feedback engagement and literacy, further research on the factors affecting their trust may be interestingly needed to complete the literature on feedback and automation.
Despite our research limitations, our study implies that trust issues in AWE somehow contribute to the studentsAo feedback engagement and feedback literacy.
It somehow shed light for teachers on one factor influencing the studentsAo feedback engagement and feedback literacy regarding AWE.
It also indicates that studentsAo trust in such feedback is essential in improving their writing quality.
Thus, teachers and educational practitioners should then find such a way to improve their studentsAo trust.
Furthermore, since studentsAo trust is influenced by individual factors such as studentsAo beliefs and previous experiences, teachers should consider these factors before implementing such feedback for their academic writing class.
Last, we expected our study to be initial research leading to other studies related to AWE for other applications, concerning how to make use of this AWE effectively, specifically in Indonesia.
REFERENCES