Nurse Media Journal of Nursing
e-ISSN: 2406-8799, p-ISSN: 2087-7811
https://ejournal.undip.ac.id/index.php/medianers
11(2):255-267, August 2021
https://doi.org/10.14710/nmjn.v11i2 34258
ORIGINAL RESEARCH

The Content Validity and Interrater Reliability of
Nursing Outcome Classification Self-Care for
Toileting among Children with Physical
Disabilities
Yogi Hasna Meisyarah1, Sri Hartini2, Fitri Haryanti2
1School of Nursing, Faculty of Medicine, Public Health, and Nursing, Universitas Gadjah Mada, Indonesia

2Department of Pediatric and Maternity Nursing, Faculty of Medicine, Public Health, and Nursing, Universitas

Gadjah Mada, Indonesia

Article Info

Abstract

Article History:
Received: 16 August 2020
Revised: 18 August 2021
Accepted: 19 August 2021
Online: 27 August 2021
Keywords:
Inter-rater reliability; NOC;
physical disabilities; validity;
toileting self-care
Corresponding Author:
Sri Hartini
Department of Pediatric and
Maternity Nursing, Faculty of
Medicine, Public Health, and
Nursing, Universitas Gadjah
Mada, Indonesia
Email: srihartini.psik@ugm.ac.id

Background: Nursing Outcome Classification (NOC) Self-care for Toileting is an
instrument for measuring the abilities of toileting self-care among children with
physical disabilities. However, there has been no cultural adaptation in the
Indonesian context, especially among children with physical disabilities, so that the
validity and inter-rater reliability of the instrument has not been reported. Testing
the content validity and interrater reliability of this instrument is necessary so that
it could be used to measure the ability of toileting self-care.
Purpose: This study aimed to examine the content validity and inter-rater
reliability of NOC Self-care for Toileting among children with physical disabilities.
Methods: This was a descriptive quantitative non-experimental research with a
cross-sectional design. In the content validity test, this study employed seven
experts teaching in nursing academies with a minimum degree of S2/specialist and
had the experiences in basic nursing to assess the validity of NOC Self-care for
Toileting. As for the inter-rater reliability test, this study involved two raters of
nursing students who have passed block 2.4 lecture on “Growth and Development”
and 36 children with physical disabilities at a Special School in Bantul, Yogyakarta.
The validity test was analyzed using CVR, CVI, and Aiken’s V indexes, while the
inter-rater reliability test was analyzed using Kappa and percent agreement.
Results: The validity test showed the CVR value range of 0.71-1.00, the CVI value
of 0.91, and the V coefficient range of 0.86-1.00. In the inter-rater reliability test,
the Kappa value obtained was 0.958, and the percent agreement value was 97.22%
for all indicators of NOC Self-care for Toileting.
Conclusion: NOC Self-care for Toileting is a valid and reliable instrument for
measuring the abilities of toileting self-care among children aged 6-18 with mild to
moderate degrees of physical disabilities. Further research can be conducted by
involving different respondents and cultures with more than two raters if possible.

How to cite: Meisyarah, Y. H., Hartini, S., & Haryanti, F. (2021). The content validity and interrater reliability of
Nursing Outcome Classification Self-Care for Toileting among children with physical disabilities. Nurse Media
Journal of Nursing, 11(2), 255-267. https://doi.org/10.14710/nmjn.v11i2 32267

1.

Introduction
Based on the 2010 population and prevalence estimates from the World Health Survey and
Global Burden of Disease, there were around over a billion children (or about 15% of the world’s
population) that were estimated to be living with disability; Global Burden of Disease estimates
childhood disability prevalence to be 95 million (5.1%) children, of whom 13 million (0.7%) have
a severe disability (World Health Organization and World Bank, 2011). The proportion of children
with disabilities in Indonesia is 2.5% for children of 5-9, 3.5% for children of 10-14, and 4.2% for
children of 15-17 years old (Ministry of Health Republic of Indonesia, 2018). The 2017/2018
report from the Department of Education, Youth, and Sports (Disdikpora) of the Special Region
of Yogyakarta (DIY) revealed that the number of physical disability students in Special Schools
(SLB) in the Special Region of Yogyakarta was 231; the majority of these students lived in Bantul
Regency with a total of 107 children (Department of Education, Youth, and Sports, 2018).
According to Noble (2014), children with physical disabilities often experience difficulties in
toileting activities. These difficulties include moving and positioning in the toilet, so that a special
seat is needed for them to move. They also have difficulties taking off their bottoms, resulting in
Copyright © 2021, NMJN, e-ISSN 2406-8799, p-ISSN 2087-7811

Nurse Media Journal of Nursing, 11(2), 2021, 256
incontinence, cleaning themselves after toileting, and closing zippers. According to the American
Academy of Pediatrics (2012), the number of physical disabilities and illnesses can hinder a child’s
ability to become fully toilet-trained or easily adjust to bathroom use; children with visual
disabilities and those with sight deficiencies experience a disadvantage at several stages of toilet
training, and children who are deaf or have difficulty hearing may or may not find toilet training
challenging, depending on their ability to communicate. Children with cerebral palsy not only
tend to be slow in developing bladder control but may not have enough bladder awareness to
begin toilet training at the age of two or three; limited physical activity, undeveloped muscle tone,
or medications tend to cause constipation for children with cerebral palsy. Children and teens
with disabilities may have urinary and/or fecal incontinence for a wide variety of reasons,
including weak muscles and/or interruptions or delays in the signals to the brain that a person
needs to void (Macias et al., 2006). For students with disabilities, this can include a lack of
awareness that they need to use the restroom until it is too late (National Association for
Continence, 2013).
Toileting self-care abilities can be measured using Nursing Outcome Classification (NOC)
Self-care for Toileting. NOC is an instrument to measure client status before and after the nursing
intervention. The outcome helps evaluate the effectiveness of an intervention given independently
or in collaboration with nurses (Moorhead et al., 2013). Each outcome has a definition, a
measurement scale, a list of indicators related to the concept and supporting references. NOC
Self-care for Toileting is defined as a person’s actions to go to the toilet independently with or
without the aid of a device measured using 13 indicators with a 5-point Likert scale from severely
compromised to not compromised (Moorhead et al., 2013).
It is important to consider the validity and reliability of data collection tools (instruments)
when either conducting or critiquing research (Heale & Twycros, 2015). The higher the value of
the validity and the reliability of an instrument, the more accurate the data obtained from a study
would be (Hayati & Lailatussaadah, 2016). The NOC Self-care for Toileting instrument has been
proven to be valid and reliable in the United States and Korea (Keenan et al., 2003; Lee, 2007).
In Indonesia, this instrument has been tested for its inter-rater reliability in the population of
stroke patients in the hospitals with a kappa value of 0.9347 and a percentage agreement value of
96.3% (Khasanah et al., 2018). If an instrument is developed worldwide in many languages, the
translation and adaptation of the culture to new languages and countries are very relevant
(Ljungberg et al., 2015). Furthermore, da Silva et al. (2015) stated that NOC was created in
America and written in English. Therefore, cultural adaptation is needed before the indicators
and results of nursing are used in overseas health services and education. In this regard, the
researchers claim that there has been no cultural adaptation of NOC Self-care for Toileting in the
Indonesian context. Until now, a tool to measure toileting abilities, namely NOC Self-care for
Toileting, has not been proven to be used and has not been reported for the validity and interrater reliability, especially among children with physical disabilities. Accordingly, this study
aimed to examine the validity and inter-rater reliability of NOC Self-care for Toileting among
children with physical disabilities.
2. Methods
2.1 Research design
This was a descriptive quantitative non-experimental research with a cross-sectional design.
2.2 Setting and samples
This study was conducted in the Special School in Bantul, the Special Region of Yogyakarta,
Indonesia, from October to November 2018. The population was students with physical
disabilities. The samples were 36 children with physical disabilities who met the inclusion criteria,
including: (1) in the age of 6-18, (2) not menstruating, (3) not having diarrhea or constipation,
and (4) not taking laxative medications. The exclusion criteria were children who were not willing
to be the respondents and those with multiple disabilities (having more than one disability, for
example, deaf-speech impairment, deaf-blind, and physical disabilities-blind).
The content validity of the NOC Self-care for Toileting was validated by seven experts. The
researchers recruited the experts who taught in nursing academies with a minimum degree of
S2/Specialist and had experiences in basic nursing. The researchers gave the content validity
assessment guide sheet, the NOC Toileting Self Care rubric, and the content validity assessment
Copyright © 2021, NMJN, e-ISSN 2406-8799, p-ISSN 2087-7811

Nurse Media Journal of Nursing, 11(2), 2021, 257
sheet to the experts simultaneously to fill out the content validity assessment of the NOC Self-care
for Toileting. The inter-rater reliability test was conducted by two raters (assessors) of nursing
students who had passed block 2.4 lecture on “Growth and Development.” Two raters directly
observed the respondents’ self-care ability according to the rubric on the indicators of NOC Selfcare for Toileting.
2.3 Measurement and data collection
The NOC Self-care for Toileting was used to measure toileting self-care. This instrument
consisted of 13 indicators with 5 levels of assessment criteria, namely: (1) severely compromised,
(2) substantially compromised, (3) moderately compromised, (4) mildly compromised, and (5)
not compromised. In each assessment criterion (see Appendix 1), there was an operational
definition written in the rubric of the instrument adapted from Khasanah et al. (2018).
The cultural adaptation of NOC Self-Care for Toileting was done based on the translation
model for cross-cultural adaptation of instruments (Jones et al., 2001; Lee et al., 2009; Sousa &
Rojjanasrirat, 2011). First, NOC Self-care for Toileting instrument was translated from the
original language (English) into Indonesian by two different translating institutions. NOC Selfcare for Toileting translated by the first translating institution was called translation 1 (T1), and
the other one by the second translating institution was called translation 2 (T2). After the
instrument was translated, the researcher asked for inputs from the Indonesian literary experts
and the advisor concerning the results of the translation of the instrument to obtain the final
version of the translation (T12). After the final version of the translation (T12) was agreed upon,
a back-translation was done by a professional translating institution.
The next step was to test the content validity of the NOC Self-care for Toileting to seven
experts. Meanwhile, the inter-rater reliability test was conducted by two raters (assessors). Before
the data were collected, the researcher conducted training and formal meetings with the raters to
increase the agreement. The data collection began with explaining and asking about the
willingness to be research respondents represented by the respondents’ parents. If the children,
represented by the parents, were willing to serve as respondents, the parents would be asked to
sign the informed consent and fill in the respondents’ data. During the data collection, two raters
directly observed respondents’ self-care ability according to the rubric on the indicators of NOC
Self-care for Toileting. Then the two raters conducted the assessment independently at the same
time on the sheet provided by the researcher according to each rater’s perception.
2.4 Data analysis
The content validity was analyzed using the CVR (Content Validity Ratio), CVI (Content
Validity Index), and Aiken’s V indexes. The CVR is an item statistic useful for rejection or
retention of individual items and is internationally recognized as the method for establishing
content validity (Wilson et al., 2012). CVI can be used to rate each instrument item in terms of its
relevance to the construct. The CVI is CVR mean for all retained items (Hadzaman et al., 2018).
When all panelists say that the tested knowledge or skill is ‘‘essential,’’ or when none say that it is
‘‘essential,’’ we are confident to include or delete the item. It is when there is no consensus that
item issues arise. Two assumptions are made, each of which is consistent with the established
psychophysical principles. The CVR and CVI values obtained were analyzed based on Lawshe
(1975). The CVR values are in the range of -1 to 0 (not good), 0 (good), and 0 to 1 (very good)
(Lawshe, 1975). The values of V coefficient were interpreted based on Guilford and Fruchter
(1978) as follows: 0.80-1.00 (very high), 0.60-0.80 (high), 0.40-0.60 (medium), 0.20-0.40 (low),
0.00-0.20 (very low), and <0.00 (invalid). The interrater reliability was analyzed using Kappa
and percent agreement for two raters (Cohen, 1960; Shweta et al., 2015). The degree of agreement
of the Kappa value was analyzed based on the Altman benchmark scale as the following: <0.20
(poor), 0.21-0.40 (fair), 0.41-0.60 (moderate), 0.61-0.80 (good), and 0.81-1.00 (very good)
(Gwet, 2014). The Kappa value is accepted if >0.41 (McHugh, 2012). The values of percent
agreement value were also interpreted based on McHugh (2012) that the percent agreement can
be accepted if the value is more than 80%.
2.5 Ethical considerations
The ethical eligibility of this study was obtained from the Ethics Committee, the Faculty of
Medicine, Public Health, and Nursing, Universitas Gadjah Mada (Ref: KE/FK/0760/EC/2018).
Copyright © 2021, NMJN, e-ISSN 2406-8799, p-ISSN 2087-7811

Nurse Media Journal of Nursing, 11(2), 2021, 258
In the data collection process, the researcher explained the aims and objectives of the research to
the parents and asked the children’s willingness to be the respondents. Parents who agreed to
participate in the study completed the informed consent as a sign of consent to participate in the
study.
3. Results
3.1 Characteristics of respondents
As presented in Table 1, the total number of respondents in this study was 36 with ages
ranging from 6 to 18 years. Most children were aged in the group of 10-13 (44.4%) and 14-18
(44.4%) and experienced a moderate degree of disability (86.1%). Based on the gender, the
number of boys and girls was similar (50% each).
Table 1. The frequency distribution of respondent characteristics (n=36)
Demographic
Gender
Male
Female
Age (years old)
6-9
10-13
14-18
Degree of disability
Mild
Moderate
Severe

f

%

18
18

50
50

4
16
16

11.1
44.4
44.4

5
31
0

13.9
86.1
0

3.2 Translation and cultural adaptation process of NOC Self-care for Toileting
The translation process of the NOC Self-care for Toileting adopted the theory of cross-cultural
adaptation of Brislin (Jones, 2001; Lee et al., 2009; Sousa & Rojjanasrirat, 2011). The translation
from the original language (English) to Indonesian was carried out by two professional translation
agencies. The results of the Indonesian translation from the two professional translation
institutions were then discussed by the Indonesian language and literature experts and research
supervisors as input providers. The input providers compared the translation results from the two
language institutions and made necessary adjustments to the Indonesian language and cultures.
After receiving input from the Indonesian language and literature experts and research
supervisors, a final version of the instrument translation (translated synthesis) was obtained. One
final version of the Indonesian language instrument translation that had been agreed upon was
then further processed for back-translation in a different professional translation agency. After
the back translation was carried out, the researchers with the English language and literature
experts compared the back-translated instrument with the original English instrument.
3.3 The results of the validity test of NOC Self-care for Toileting
In the validity test of each NOC Self-care for Toileting indicator, the CVR values were in the
range of 0.71-1.00. According to Lawshe (1975), the values fall within the very good category. The
CVI value was 0.91, so it showed a very good validity value (Lawshe, 1975). Based on the Aiken’s
V approach, the values of V coefficient were in the range of 0.86 to 0.96 which fall within the very
high category (Guilford & Fruchter, 1978). The assessment results of the content validity of the
CVR and Aiken’s V can be seen in Table 2.
3.4 The results of the interrater reliability test of NOC Self-care for Toileting
In the inter-rater reliability test of all NOC Self-care for Toileting indicators, the Kappa value
obtained was 0.958. This value showed a very good degree of agreement (Gwet, 2014). The Kappa
value was accepted since it exceeds the minimum limit (0.41) (McHugh, 2012). The NOC Self-care
for Toileting produced the percent agreement value of 97.22%, so NOC Self-care for Toileting
could be accepted as it exceeded the minimum limit of percent agreement value of 80% (McHugh,
2012). In the inter-rater reliability test, each indicator of NOC Self-care for Toileting produced the
Kappa value in the range 0.86-1.00. The Kappa values of the 13 indicators showed a very good
Copyright © 2021, NMJN, e-ISSN 2406-8799, p-ISSN 2087-7811

Nurse Media Journal of Nursing, 11(2), 2021, 259
degree of agreement (Gwet, 2014) and were accepted since the values exceeded the Kappa value
limit of 0.41 (McHugh, 2012). The percent agreement value for each NOC Self-care for Toileting
indicator was in the range of 91.67-100%. The value was accepted, for it had exceeded the
minimum limit of percentage agreement value of 80% (McHugh, 2012). The results of the interrater reliability test of NOC Self-care for Toileting can be seen in Table 3.
Table 2. The assessment results of the content validity of the CVR and Aiken’s V (expert=7)
No
Indicator
1 Responds to full bladder in timely manner
2 Responds to urge to have a bowel
movement in timely manner
3 Gets in and out of bathroom
4 Removes clothing
5 Positions self on toilet or commode
6 Gets to toilet between urge and passage of
urine
7 Gets to toilet between urge and evacuation
of stool
8 Empties bladder
9 Empties bowel
10 Wipes self after urinating
11 Wipes self after bowel movement
12 Gets up from toilet or commode
13 Adjust clothing after toileting

CVR
1.00
1.00

Interpretation
Very Good
Very Good

V
0.93
0.93

Interpretation
Very High
Very High

0.71
1.00
1.00
0.71

Very Good
Very Good
Very Good
Very Good

0.86
0.96
0.96
0.93

Very High
Very High
Very High
Very High

0.71

Very Good

0.93

Very High

1.00
1.00
1.00
1.00
0.71
1.00

Very Good
Very Good
Very Good
Very Good
Very Good
Very Good

0.90
0.90
1.00
1.00
0.86
0.96

Very High
Very High
Very High
Very High
Very High
Very High

Table 3. The results of the inter-rater reliability test (n=36)
No

Indicator

1 Responds to full bladder in timely
manner
2 Responds to urge to have a bowel
movement in timely manner
3 Gets in and out of bathroom
4 Removes clothing
5 Positions self on toilet or
commode
6 Gets to toilet between urge and
passage of urine
7 Gets to toilet between urge and
evacuation of stool
8 Empties bladder
9 Empties bowel
10 Wipes self after urinating
11 Wipes self after bowel movement
12 Gets up from toilet or commode
13 Adjust clothing after toileting

Percent
Interpretation
Agreement

Kappa

Degree of
Kappa
Agreement Interpretation

94.44%

Accepted

0.910

Very Good

Accepted

100%

Accepted

1.00

Very Good

Accepted

97.22%
100%
94.44%

Accepted
Accepted
Accepted

0.955
1.00
0.909

Very Good
Very Good
Very Good

Accepted
Accepted
Accepted

94.44%

Accepted

0.907

Very Good

Accepted

100%

Accepted

1.00

Very Good

Accepted

100%
100%
91.67
100%
91.67
100%

Accepted
Accepted
Accepted
Accepted
Accepted
Accepted

1.00
1.00
0.881
1.00
0.862
1.00

Very Good
Very Good
Very Good
Very Good
Very Good
Very Good

Accepted
Accepted
Accepted
Accepted
Accepted
Accepted

4. Discussion
This study aimed to examine the validity and inter-rater reliability of the NOC Self-care for
Toileting among children with physical disabilities. The validity test showed the CVR value range
of 0.71-1.00, the CVI value of 0.91, and the V coefficient range of 0.86-1.00. In the inter-rater
reliability test, the Kappa value was 0.958, and the percent agreement was 97.22% for all
indicators of the instrument. On the basis of the data from this study and other similar studies
(Khasanah et al., 2018), this translation and cultural adaptation process results in a clearer and

Copyright © 2021, NMJN, e-ISSN 2406-8799, p-ISSN 2087-7811

Nurse Media Journal of Nursing, 11(2), 2021, 260
more comprehensible wording of the indicators than that obtained by directly translating it from
English into the Indonesia language without expert assessment.
Content validity can be defined as the ability of the selected items to reflect the variables of
the construct in the measure (Zamanzadeh et al., 2015). Sireci and Faulkner-Bond (2014) defines
content validity as the degree to which the content of a test is congruent with testing purposes.
The results of this study indicated that the values of the content validity of NOC Self-care for
Toileting among children with physical disabilities were very good and very high with the CVR
values in the range of 0.71-1.00, the CVI value of 0.91, and the values of the coefficient V in the
range of 0.86-0.96. Based on the explanation, it can be concluded that NOC Self-care for Toileting
has been proven to be valid for measuring the abilities of toileting self-care among children with
physical disabilities in Indonesia (Meisyarah et al., 2019). This confirms a research finding by
Mulyani and Nurjannah (2019) about the content validity test of the NOC on Toileting Self-care
among stroke patients, which produced an S-CVI value of 0.96-0.98, indicating that NOC Selfcare for Toileting is valid to measure the abilities of NOC Self-care for Toileting among stroke
patients in Indonesia.
A study in Korea showed that the mean Fehring ratio of the NOC Self-care for Toileting was
0.82, indicating that the tool had acceptable nursing sensitivity in Korea (Lee, 2007). A study
conducted by Head et al. (2003) also revealed a similar finding regarding the outcome content
validity of NOC Self-care for Activities of Daily Living (ADL) by the community health nurses,
resulting in an outcome content validity score of 0.8429 and outcome sensitivity validity score of
0.7082. Thus, the results supported the content validity and nursing sensitivity of the study
outcomes. Almeida et al. (2010) reported that NOC Self-care for Activities of Daily Living (ADL)
was validated by experts. The study regarding toileting as indicator of NOC self-care for activities
of daily living (ADL) showed a weighted average (0.76).
Based on Asaad (2004), several factors that can affect the validity include the internal factors
of items (unclear direction of items, unstructured vocabulary and sentences, item length, and
ambiguity of items), administrative factors and item scores, and nature and group criteria. In this
regard, the researchers have considered these factors in this study. Therefore, all NOC Self-care
for Toileting indicators produce very good validity values with the CVR values in the range of 0.711.00, the CVI value of 0.91, and the coefficient V values in the range of 0.86-1.00.
The inter-rater reliability test showed that the Kappa value and percent agreement fall within
the very good category with the Kappa value of 0.958 and the percent agreement of 97.22%. Based
on this explanation, the NOC Self-care for Toileting has proven to be reliable to measure the
abilities of Toileting Self-care among children with physical disabilities in Indonesia (Meisyarah
et al., 2019). A study by Khasanah et al. (2018) concerning the inter-rater reliability test for NOC
Self-care for Toileting among stroke patients also showed a Kappa value of 0.934 and a percentage
agreement of 96.3%.
A research carried out by Keenan et al. (2003) on the inter-rater reliability test for NOC Selfcare for Toileting in a home care setting in the United States also revealed similar results with
percent close to the agreement value of 95%. A study by McHugh (2012) revealed that perfect
Kappa value is rarely achieved; however, in this study, several indicators could produce perfect
Kappa and percent agreement values (k=1.00 and percent agreement=100%). Of the 13 Toileting
Self-care indicators, seven indicators have perfect Kappa and percent agreement values (k=1.00
and percent agreement=100%). The perfect Kappa value (k=1) and the percent agreement value
(100%) were shown in the indicators of (2) responds to urge to have a bowel, (4) removes clothing,
(7) gets to toilet between urge and evacuation of stool, (8) empties bladder, (9) empties bowel,
(11) wipes self after bowel movement, and (13) adjust clothing after toileting. The high value of
kappa and percent agreement in this study shows the suitability. Several factors can influence the
high Kappa and percent agreement values to obtain good inter-rater reliability values.
Several factors must be considered by instrument designers to increase the potential of an
agreement (Burns, 2014; Graham et al., 2012; Kane et al, 2015; Mueller, 2018; Nurjannah et al.,
2017; Ragupathi & Lee, 2020). The first factor is rater training. According to Graham et al. (2012),
the duration of rater training is also important, and effective rater training requires more than
one or two hours. In this study, the researcher conducted training for two raters for four hours in
two sessions. In the first session, the researchers explained the data collection procedures and
NOC Self-care for Toileting observation sheet to be used and directed perceptions between the
Copyright © 2021, NMJN, e-ISSN 2406-8799, p-ISSN 2087-7811

Nurse Media Journal of Nursing, 11(2), 2021, 261
raters regarding the NOC Self-care for Toileting observation sheet for two hours. In the second
session, the researcher with two raters conducted a simulation of data collection for two hours.
The second factor is rater selection. According to Graham et al. (2012), the expertise and
experience of raters that suit the instruments can increase agreement. Based on a study carried
out by Kane et al. (2015), raters’ professional background can influence inter-rater reliability. This
study employed two raters who have the same educational background, experience, and expertise.
Both of them have passed block 2.4 lecture on “Growth and Development” so that it is easy to
increase the inter-rater agreement.
The third factor is the accountability of raters’ assessment. Based on Graham et al. (2012),
rater training does not guarantee the consistency of the assessment conducted by raters, so that
re-training is needed. In this study, the researcher always monitored the performance of the two
raters and asked them if there were things still not understood. The researcher also reminded
them to assess independently so that the results obtained are truly reliable and not biased.
The fourth factor is the instrument rubric design. Graham et al. (2012) stated that the
consistency of raters’ assessment can be influenced by the instrument rubric design. According to
Ragupathi and Lee (2020), rubrics offer the possibility of objective, consistent evaluation
minimizing the difference in grades even when multiple raters are involved in the evaluation. In
this study, the rubric used was an adaptation of the NOC Self-care: Toileting rubric by Khasanah
et al (2018) with modifications and adjustments regarding the samples. This instrument is
accompanied by an operational definition of each indicator and has been given a detailed
assessment rubric for each indicator scale from severely compromised to not compromised. In
addition, this rubric has been validated by seven experts in their field of expertise and produced
very good CVR and CVI values and very high V coefficient values.
The fifth factor is the type of scale in the instrument. The best rubrics have three to five
descriptive levels to allow for discrimination in the evaluation (Mueller, 2018). This research used
NOC Self-care for Toileting observation sheet with 5 scales, namely severely compromised,
substantially compromised, moderately compromised, mildly compromised, and not
compromised, so that it has more varied scales.
The sixth factor is to consider fatigue when scheduling inter-rater reliability test time. Raters’
fatigue may increase errors (Burns, 2014). According to Aslett (2006), rater fatigue can affect the
results of instrument reliability. Nurjannah et al. (2017) revealed that a rater who conducts an
assessment outside of working hours until exhaustion might affect the stability of the assessment.
Researchers scheduled the interrater reliability test time in the morning when they were still fit,
and data collection was carried out one to two times a week with a duration of 10-15 minutes for
each respondent so that the fatigue factor did not really have an impact on this study.
5.

Implication and limitation
The NOC Self-care for Toileting instrument, according to the results of the present study, is
a valid and reliable instrument for measuring the abilities of toileting self-care among children
aged 6-18 with mild-moderate degrees of physical disabilities; therefore, the evaluated NOC
indicators ensure good reproducibility in clinical practice. Nurses can use this instrument to
measure the ability of toileting self-care among children with physical disabilities in clinical
practice so that nurses can provide good nursing care. The NOC Self-care for Toileting instrument
can also be used in Indonesia because the cultural adaptation in Indonesia has been carried out
in this study.
The limitations of this study include data collection techniques for four indicators of
defecation in NOC Self-care and the back-translation process that only used one professional
translation agency. Despite these limitations, this study offers seven experts who validated the
NOC Self-Care for Toileting instrument and conducted direct observations to assess interrater
reliability using two raters. The results of this study can help nurses or caregivers in assessing
toileting abilities in the community and clinically using the NOC Self-Care Toileting instrument
appropriately.
6. Conclusion
The overall content validity values of NOC Self-care for Toileting among children with
physical disabilities are included in the very good and valid category, while the overall inter-rater
reliability values are included in the very good and reliable category. The use of NOC Self-care for
Copyright © 2021, NMJN, e-ISSN 2406-8799, p-ISSN 2087-7811

Nurse Media Journal of Nursing, 11(2), 2021, 262
Toileting in this study has been adapted to the Indonesian culture. Thus, the Indonesian version
of NOC Self-care for Toileting can be used to measure the toileting self-care among children with
physical disabilities in Indonesia. Further research can be carried out by involving more than two
raters if possible and conducting research on respondents and different cultures.
Acknowledgment
The researchers would like to thank all respondents of children with physical disabilities at
the Special School in Bantul, Yogyakarta, who have helped a lot in this study.
Author contribution
YHM: Study conception and design, data collection, data analysis and interpretation, drafting
article; SH: Study conception and design, data collection, data analysis and interpretation,
drafting article; FH: Study conception and design, and drafting article
Conflict of interest
The authors declare that there was no conflict of interest in this study.
References
Almeida, M. d. A., Seganfredo, D. H., & Unicovsky, M. R. (2010). Nursing outcome indicator
validation for patients with orthopedic problems. Revista da Escola de Enfermagem da USP,
44(4), 1059-1064. https://doi.org/10.1590/S0080-62342010000400029
American Academy of Pediatrics. (2012). Toilet training children with special needs.
https://www.healthychildren.org/English/ages-stages/toddler/toilet-training/Pages/
Toilet-Training-Children-with-Special-Needs.aspx
Asaad, A. S. (2004). Measurement and evaluation: Concepts and principles. FEX Book Store.
Aslett, H. J. (2006). Reducing variability, increasing reliability: Exploring the psychology of
intra- and inter-rater reliability. Investigation on University Teaching and Learning,
4(I), 86-91. http://repository.londonmet.ac.uk/id/eprint/222
Burns, M. K. (2014). How to establish interrater reliability. Nursing, 44(10), 56–58.
https://doi.org/10.1097/01.NURSE.0000453705.41413.c6
Cohen, J. (1960) A coefficient of agreement for nominal scales. Educational and Psychological
Measurement, 20, 37-46. https://doi.org/10.1177/001316446002000104
da Silva, N. C. M., de Souza Oliveira, A. R., & de Carvalho, E. C. (2015). Knowledge produced from
the outcomes of the “Nursing Outcomes Classification--NOC”: Integrative review. Revista
Gaúcha de Enfermagem/EENFUFRGS, 36(4), 104–111. https://doi.org/10.1590/19831447.2015.04.53339
Department of Education, Youth, and Sports. (2018). Data individu sekolah dan data siswa
tingkat pendidikan dasar TK-SMA sekolah luar biasa tahun 2017-2018. [Individual school
data and student data for Kindergarten-High School special education levels 2017-2018].
Department of Education, Youth, and Sports Yogyakarta.
Graham, M., Milanowski, A., & Miller, J. (2012). Measuring and promoting of teacher and
principal (Issue February). US Department of Education: Center for Educator Compensation
Reform.
Guilford, J. P., & Fruchter, B. (1978). Fundamental statistics in psychology and education. Mc
Graw-Hill Kogakusha.
Gwet, K. L. (2014). Handbook of interrater reliability (4th Ed). Advanced Analytics.
Hadzaman, N. A. H., Takim, R., Nawawi, A. H., & Mohamad Yusuwan, N. (2018). Content validity
of governing in Building Information Modelling (BIM) implementation assessment
instrument. IOP Conference Series: Earth and Environmental Science, 140(1).
https://doi.org/10.1088/1755-1315/140/1/012105
Hayati, S., & Lailatussaadah. (2016). Instrumen validity and reliability. [Validity and reliability of
instrument]. Didaktika, 16(2), 169–179.
Head, B. J., Maas, M., & Johnson, M. (2003). Validity and community-health-nursing sensitivity
of six outcomes for community health nursing with older clients. Public Health Nursing,
20(5), 385–398. https://doi.org/10.1046/j.1525-1446.2003.20507.x
Heale, R., & Twycross, A. (2015). Validity and reliability in quantitative studies. EvidenceBased Nursing, 18(3), 66–67. https://doi.org/10.1136/eb-2015-102129
Copyright © 2021, NMJN, e-ISSN 2406-8799, p-ISSN 2087-7811

Nurse Media Journal of Nursing, 11(2), 2021, 263
Jones, P. S., Lee, J. W., Philips, L. R., Zhang, X. E., & Jaceldo K. B. (2001). An adaptation of
Brislin’s Translation Model for cross-cultural research. Nursing Research, 50(5):303-304.
https://doi.org/10.1097/00006199-200109000-00008.
Kane, A. E., Hilmer, S. N., Huizer-Pajkos, A., Mach, J., Nines, D., Boyer, D., Gavin, K., Mitchell,
S. J., & De Cabo, R. (2015). Factors that impact on interrater reliability of the mouse clinical
frailty index. Journals of Gerontology - Series A Biological Sciences and Medical Sciences,
70(6), 694–695. https://doi.org/10.1093/gerona/glv032
Keenan, G., Stocker, J., Barkauskas, V., Johnson, M., Maas, M., Moorhead, S., & Reed, D. (2003).
Assessing the reliability, validity, and sensitivity of nursing outcomes classification in home
care
settings.
Journal
of
Nursing
Measurement,
11(2),
135–155.
https://doi.org/10.1891/jnum.11.2.135.57285
Khasanah, U., Nurjannah, I., & Harjanto, T. (2018). Interrater reliability dari nursing outcome
classification perawatan diri eliminasi pada pasien stroke dengan diagnosis keperawatan
defisit perawatan diri eliminasi. [Interrater reliability of nursing outcome classification for
elimination self-care in stroke patients with a nursing diagnosis of elimination self-care
deficit] (Bachelor’s Thesis). Universitas Gadjah Mada, Yogyakarta, Indonesia.
Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 561–
580. https://doi.org/10.1111/j.1744-6570.1975.tb01393.x
Lee, B. (2007). Identifying outcomes from the nursing outcomes classification as indicators of
quality of care in Korea: A modified delphi study. International Journal of Nursing Studies,
44, 1021-1028. https://doi.org/10.1016/j.ijnurstu.2006.03.016
Lee, C. C., Li, D., Arai, S., & Puntillo, K. (2009). Ensuring cross-cultural equivalence in translation
of research consents and clinical documents: A systematic process for translating English to
Chinese. Journal of Transcultural Nursing, 20(1), 77–82. doi: 10.1177/1043659608325852
Ljungberg, A. K., Fossum, B., Fürst, C. J., & Hagelin, C. L. (2015). Translation and cultural
adaptation of research instruments-guidelines and challenges: An example in FAMCARE-2
for use in Sweden. Informatics for Health and Social Care, 40(1), 67–78.
https://doi.org/10.3109/17538157.2013.872111
Macias, M. M., Roberts, K. M., Saylor, C. F., & Fussell, J. J. (2006). Toileting concerns, parenting
stress, and behavior problems in children with special health care needs. Clinical Pediatrics,
12, 415–422. https://doi.org/10.1177/0009922806289 616
McHugh, M. L. (2012). Lessons in biostatistics interrater reliability: The Kappa statistic.
Biochemica Medica, 22(3), 276–282. https://hrcak.srce.hr/89395
Meisyarah, Y. H., Hartini, S., & Haryanti, F. (2019). Validitas dan interrater reliability dari
Nursing Outcome Classification (NOC) perawatan diri toileting pada anak disabilitas fisik
di SLB Negeri 1 Bantul. [Validity and interrater reliability of Nursing Outcome
Classification (NOC) toileting self-care for children with physical disabilities at SLB Negeri
1 Bantul]. (Bachelor’s Thesis). Universitas Gadjah Mada, Yogyakarta, Indonesia.
Ministry of Health Republic of Indonesia. (2018). Hasil utama Riskesdas 2018 [Main results of
Riskesdas 2018]. http://kesmas.kemkes.go.id/assets/upload/dir_519d41d8cd98f00/files/
Hasil riskesdas-2018_1274.pdf
Moorhead, S., Johnson, M., Maas, M. L., & Swanson, E. (2013). Nursing Outcome Classification
(NOC) (5th ed.). Elsevier Mosby.
Mueller, J. (2018). Authentic assessment toolbox. http://jfmueller.faculty.noctrl.edu/toolbox/
rubrics.htm
Mulyani, S., & Nurjannah, I. (2019). Validation of nursing outcomes’ indicators of nursing
outcomes classification of self-care for patients with stroke. Belitung Nursing Journal, 5(3).
https://doi.org/10.33546/bnj.768
National
Association
for
Continence.
(2013).
What
is
incontinence?.
http://www.nafc.org/bladder-bowelhealth/
Noble, E. (2014). Using prompts to improve toilet training for children with physical disabilities.
https://cdn.rifton.com/-/media/files/rifton/white-papers/toilet-training-children-physical
-disabilities.pdf
Nurjannah, I., Nurmasari, P., Nugraha, I. T., Katan, D. Y., & Hariyadi, K. (2017). Interrater
reliability of client categorization system as a psychiatric status rating scale in measuring
psychiatric patients’ health status. International Journal of Research in Medical Sciences,
5(5), 2193-2201. https://doi.org/10.18203/2320-6012.ijrms20171868
Copyright © 2021, NMJN, e-ISSN 2406-8799, p-ISSN 2087-7811

Nurse Media Journal of Nursing, 11(2), 2021, 264
Ragupathi, K., & Lee, A. (2020). Beyond fairness and consistency in grading: The role of rubrics
in higher education. In C. S. Sanger and N. W. Gleason (Eds.), Diversity and Inclusion in
Global Higher Education, pp 73-95. Springer Nature Singapore.
Shweta, Bajpai, R. C., & Chaturvedi, H. K. (2015). Evaluation of inter-rater agreement and interrater reliability for observational data: An overview of concepts and methods. Journal of the
Indian Academy of Applied Psychology, 41 (Special Issue 3), 20–27.
Sireci, S., & Faulkner-Bond, M. (2014). Validity evidence based on test content. Psicothema, 26(1),
100–107. https://doi.org/10.7334/psicothema2013.256
Sousa, V. D., & Rojjanasrirat, W. (2011). Translation, adaptation and validation of instruments or
scales for use in cross-cultural health care research: A clear and user-friendly guideline.
Journal of Evaluation in Clinical Practice, 17, 268-274. https://doi.org/10.1111/j.13652753.2010.01434.x
Wilson, F., Pan, W., & Schumsky, D. (2012). Recalculation of the critical values for Lawshe’s
content validity ratio. Measurement and Evaluation in Counseling and Development, 45(3),
197-210. https://doi.org/10. 1177/0748175612440286.
World Health Organization and World Bank. (2011). World report on disability.

https://www.unicef.org/protection/World_report_on_disability_eng.pdf

Zamanzadeh, V., Ghahramanian, A., Rassouli, M., Abbaszadeh, A., Alavi-Majd, H., & Nikanfar,
A.-R. (2015). Design and implementation content validity study: Development of an
instrument for measuring patient-centered communication. Journal of Caring Sciences,
4(2), 165–178. https://doi.org/ 10.15171/jcs.2015.017

Copyright © 2021 NMJN. This article is an open-access article distributed under the terms and conditions of the
Creative
Commons
Attribution-Share
Alike
4.0
(CC
BY-SA)
International
License
(https://creativecommons.org/licenses/by-sa/4.0).

Copyright © 2021, NMJN, e-ISSN 2406-8799, p-ISSN 2087-7811

Nurse Media Journal of Nursing, 11(2), 2021, 265
Appendix 1
The adaptation and modification of NOC Self-care for Toileting instrument (Khasanah et al.,
2018)
Code
031001

Indicator
Responds to full
bladder in timely
manner

Indicator Operational Definition
1: Unable to respond to the urge to urinate characterized by inappropriate
urination
2: Able to respond the urge to urinate but cannot reach the bathroom
3: Able to respond to the urge to urinate but cannot reach the toilet
4: Able to respond to the urge to urinate and can reach the toilet but has
not been able to position the body in the toilet when urinating
5: Able to respond to the urge to urinate by urinating in the toilet

030302

Responds to urge
to have a bowel

1: Able to respond to the urge to defecate in time
2: Able to respond the urge to defecate but cannot reach the bathroom
3: Able to respond to the urge to defecate but cannot reach the toilet
4: Able to respond to the urge to defecate and can reach the toilet but has
not been able to position the body in the toilet when defecating
5: Able to respond to the urge to defecate in the toilet

031013

Gets in and out of
bathroom

1: Unable to perform all indicators: a) walking to the bathroom, b)
opening the bathroom door, c) closing the bathroom door, d) walking
from the bathroom
2: Able to perform 1 of 4 indicators: a) walking to the bathroom, b)
opening the bathroom door, c) closing the bathroom door, d) walking
from the bathroom
3: Able to perform 2 of 4 indicators: a) walking to the bathroom, b)
opening the bathroom door, c) closing the bathroom door, d) walking
from the bathroom
4: Able to perform 3 of 4 indicators: a) walking to the bathroom, b)
opening the bathroom door, c) closing the bathroom door, d) walking
from the bathroom
5: Able to perform all indicators: a) walking to the bathroom, b) opening
the bathroom door, c) closing the bathroom door, d) walking from the
bathroom

031004

Removes clothing

1: Unable to perform all indicators: a) opening buttons or zippers, b)
taking off the skirt/outer pants, c) taking off the underwear, d) adjusting
the clothes before urinating/defecating
2: Able to perform 1 of 4 indicators: a) opening buttons or zippers, b)
taking off the skirt/outer pants, c) taking off the underwear, d) adjusting
the clothes before urinating/defecating
3: Able to perform 2 of 4 indicators: a) opening buttons or zippers, b)
taking off the skirt/outer pants, c) taking off the underwear, d) adjusting
the clothes before urinating/defecating
4: Able to perform 3 of 4 indicators: a) opening buttons or zippers, b)
taking off the skirt/outer pants, c) taking off the underwear, d) adjusting
the clothes before urinating/defecating
5: Able to perform all indicators: a) opening buttons or zippers, b) taking
off the skirt/outer pants, c) taking off the underwear, d) adjusting the
clothes before urinating/defecating

031005

Positions self on
toilet or commode

1: Unable to position themselves on the toilet or elimination aids
2: Able to position themselves on the toilet or elimination aids with the
assistance of others
3: Able to position themselves on the toilet or elimination aids with
assistive devices (for example: hand rail)
4: Able to position themselves on the toilet or elimination aids
independently without any assistive devices and the assistance of others
but need direction or supervision
5: Able to position themselves on the toilet or elimination aids
independently without any directions or supervisions, assistive devices,
and the assistance of others

Copyright © 2021, NMJN, e-ISSN 2406-8799, p-ISSN 2087-7811

Nurse Media Journal of Nursing, 11(2), 2021, 266
Code
031014

Indicator
Gets to toilet
between urge and
passage of urine

Indicator Operational Definition
1: Unable to reach the toilet between the urge to urinate and before the
release of urine with the assistance of others and a wheelchair
2: Able to reach the toilet between the urge to urinate and the release of
urine with a wheelchair
3: Able to reach the toilet between the urge to urinate and the release of
urine independently with the assistance of others
4: Able to reach the toilet between the urge to urinate and the release of
urine independently with assistive devices (for example, walking aids,
such as crutches or walker)
5: Able to reach the toilet between the urge to urinate and the release of
urine independently

031006

Empties bladder

1: Unable to release urine
2: Able to release urine with medication
3: Able to release urine with stimulation
4: Able to release urine with directions (for example, direction to push)
5: Able to release urine independently (without any directions and
stimulation)

031011

Empties bowel

1: Unable to evacuate stool
2: Able to evacuate stool with medication
3: Able to evacuate stool with stimulation
4: Able to evacuate stool with directions (for example, direction to push)
5: Able to evacuate stool independently (without any directions and
stimulation)

031007

Wipes self after
urinating

1: Unable to perform all indicators:
- Male: a) taking water with a water dipper, b) washing the testicular area,
c) washing the scrotal area, d) putting the water dipper back
- Female: a) taking water with a water dipper, b) washing the inner part
of the perineum, c) washing the outer part of the perineum, d) putting the
water dipper back
2: Able to perform 1 of 4 indicators:
- Male: a) taking water with a water dipper, b) washing the testicular area,
c) washing the scrotal area, d)putting the water dipper back
- Female: a) taking water with a water dipper, b) washing the inner part
of the perineum, c) washing the outer part of the perineum, d) putting the
water dipper back
3: Able to perform 2 of 4 indicators:
- Male: a) taking water with a water dipper, b) washing the testicular area,
c) washing the scrotal area, d) putting the water dipper back
- Female: a) taking water with a water dipper, b) washing the inner part
of the perineum, c) washing the outer part of the perineum, d) putting the
water dipper back
4: Able to perform 3 of 4 indicators:
- Male: a) taking water with a water dipper, b) washing the testicular area,
c) washing the scrotal area, d) putting the water dipper back
- Female: a) taking water with a water dipper, b) washing the inner part
of the perineum, c) washing the outer part of the perineum, d) putting the
water dipper back
5: Able to perform all indicators:
- Male: a) taking water with a water dipper, b) washing the testicular area,
c) washing the scrotal area, d) putting the water dipper back
- Female: a) taking water with a water dipper, b) washing the inner part
of the perineum, c) washing the outer part of the perineum, d) putting the
water dipper back

031012

Wipes self after
bowel movement

1: Unable to perform all indicators: a) taking water with a water dipper,
b) washing the anal area, c) washing the genital area from the front to
back, d) putting the water dipper back
2: Able to perform 1 of 4 indicators: a) taking water with a water dipper,
b) washing the anal area, c) washing the genital area from the front to
back, d) putting the water dipper back

Copyright © 2021, NMJN, e-ISSN 2406-8799, p-ISSN 2087-7811

Nurse Media Journal of Nursing, 11(2), 2021, 267
Code

Indicator

Indicator Operational Definition
3: Able to perform 2 of 4 indicators: a) taking water with a water dipper,
b) washing the anal area, c) washing the genital area from the front to
back, d) putting the water dipper back
4: Able to perform 3 of 4 indicators: a) taking water with a water dipper,
b) washing the anal area, c) washing the genital area from the front to
back, d) putting the water dipper back
5: Able to perform all indicators: a) taking water with a water dipper, b)
washing the anal area, c) washing the genital area from the front to back,
d) putting the water dipper back

031008

Gets up from toilet 1: Unable to move from the toilet or elimination aids
or commode
2: Able to move from the toilet or elimination aids with the assistance of
others
3: Able to move from the toilet or elimination aids with assistive devices
(for example: hand rail)
4: Able to move from the toilet or elimination aids independently without
any assistive devices and the assistance of others but need direction or
supervision
5: Able to move from the toilet or elimination aids independently without
any directions or supervisions, assistive devices, and the assistance of
others

031009

Adjust clothing
after toileting

1: Unable to perform all indicators: a) taking the clothes after
urinating/defecating, b) putting on underwear, c) putting on skirt/pants,
d) closing zipper or buttoning up
2: Able to perform 1 of 4 indicators: a) taking the clothes after
urinating/defecating, b) putting on underwear, c) putting on skirt/pants,
d) closing zipper or buttoning up
3: Able to perform 2 of 4 indicators: a) taking the clothes after
urinating/defecating, b) putting on underwear, c) putting on skirt/pants,
d) closing zipper or buttoning up
4: Able to perform 3 of 4 indicators: a) taking the clothes after
urinating/defecating, b) putting on underwear, c) putting on skirt/pants,
d) closing zipper or buttoning up
5: Able to perform all indicators: a) taking the clothes after
urinating/defecating, b) putting on underwear, c) putting on skirt/pants,
d) closing zipper or buttoning up

Copyright © 2021, NMJN, e-ISSN 2406-8799, p-ISSN 2087-7811