doi:10. 21831/cp. Cakrawala Pendidikan. Vol. No. October 2019 LEARNING GOAL-FREE PROBLEMS: COLLABORATIVELY OR INDIVIDUALLY? Sugiman1*. Endah Retnowati2. Paul Ayres3, & Murdanu4 1,2,4 Universitas Negeri Yogyakarta. Indonesia University of New South Wales. Australia *e-mail: sugiman@uny. Abstract: Goal-free is a learning strategy to present a problem without specific questions, in contrary to a goal-given problem. This research examined the goal-free effects during mathematics collaborative learning measured by cognitive load ratings and transfer performance. An experiment was conducted in authentic mathematics classrooms employing a factorial design with 2 problem presentations . oal-free goal-given problem. A 2 learning environments . ollaboratively vs. using a Geometry This consisted of four consecutive phases: introductory, acquisition, near and far transfer tests. seventh graders (Average: 12. 8 y. ) who were novices, participated voluntary. The findings showed that students who learned by goal-free problems had significantly higher far-transfer scores than when the goal was given. Interestingly, in the acquisition phase, the students in goal-free problems experienced significantly higher cognitive load than their counterparts. Further, it was found that overall, individuals scored significantly higher in a far-transfer test than those who learned collaboratively. however, during the acquisition phase individuals experienced significantly higher cognitive load than collaborative No pattern of interaction effects was indicated. Overall, it was concluded that goal-free problems may be learned either collaborativelly or individually. Nevertheless, the goal-free problems stimulated higher cognitive load during learning, which seemed to have a positive influence. Keywords: goal-free problems, cognitive load, collaborative, mathematics PEMBELAJARAN GOAL-FREE PROBLEM: SECARA KOLABORATIF ATAU INDIVIDUAL? Abstrak: Goal-free adalah sebuah strategi menyajikan masalah tanpa pertanyaan khusus, kebalikan dengan masalah goal-given. Penelitian ini bertujuan menguji dampak pembelajaran matematika kolaboratif menggunakan masalah goal-free diukur dari tingkat muatan kognitif dan kemampuan Eksperimen dalam penelitian ini menggunakan kelas matematika autentik dan desain faktorial dua penyajian masalah . oal-free vs goal-give. x dua proses penyelesaian masalah . olaboratif vs individ. dalam pembelajaran topik geometri. Penelitian ini terdiri atas empat fase berurutan, yaitu pendahuluan, pembelajaran, tes near-transfer, dan tes far transfer. Sebanyak 111 siswa kelas tujuh . ata-rata: 12. 8 tahu. yang masih pembelajar awam berpartisipasi secara sukarela. Hasil penelitian menunjukkan bahwa siswa yang belajar menggunakan goal-free mencapai skor far-transfer yang signifikan lebih tinggi. Temuan ini menarik karena di fase pembelajaran, siswa goal-free mengalami tingkat muatan kognitif yang signifikan lebih tinggi dibandingkan siswa goal-given. Selanjutnya, belajar secara individu menghasilkan skor far-transfer yang signifikan lebih tinggi daripada secara kolaboratif, meskipun selama fase pembelajaran belajar individu mengakibatkan tingkat muatan kognitif yang signifikan lebih tinggi dibandingkan kolaboratif. Tidak ada pola efek interaksi yang ditunjukkan. Sehingga dapat dinyatakan bahwa penyajian masalah goal-free dapat dipelajari secara kolaboratif atau Mengabaikan bahwa penyajian goal-free menstimulasi muatan kognitif lebih tinggi selama pembelajaran, hasilnya dapat lebih baik daripada penyajian goal-given. Kata Kunci: goal-free problems, muatan kognitif, kolaboratif, matematika INTRODUCTION Problem based learning (PBL) in mathematics is very popular, however little research is focused on how the problem is In PBL, students are are expected to be able to construct new knowledge by solving the given problems (Kirschner. Sweller, & Clark, 2. or by posing problems based on a given solution (Retnowati. Fathoni, & Chen. This might be done alone though it is more common that teachers allocate students into small groups to complete the problem solutions together. During this method of instruction, students receive minimum guidance from the teacher. This means that the teacher does not provide explicit information regarding the problem solving. However, research on effective and efficient problem solving design in accordance with studentsAo thinking capacity is still very much needed because PBL research rarely focuses on the design of the problem solving itself. It is fundamental for the teacher as a learning facilitator to design learning according to studentsAo thinking capacity so that an effective and efficient learning process is realized. This is because learning is a series of activities that involve cognitive mental processes to select, process and organize information to construct it into structured knowledge (Mayer, 1. This process involves cognitive structures that facilitate students to think in constructing These cognitive structures have characteristics that need to be considered in designing learning (Sweller, 2. Sweller. Merrienboer, & Paas . Paas. Renkl, & Sweller . Sweller . Sweller. Ayres, & Kalyuga . describe Cognitive Load Theory as an instructional theory . that is based on student thinking capacities and can be very useful in guiding learning and teaching. The theory states that the learning process is carried out most effectively in a state that is in harmony with our human cognitive architecture. Of great importance to the theory is that working memory, which has the role of facilitating studentsAo thinking processes, has limited capacity and duration in receiving and processing complex or new information (Miller, 1956. Cowan, 2. Especially for novice learners who do not have enough background knowledge to recognize Learning Goal-Free Problems: Collaboratively or Individually? and process new or complex information, the ability of their working memory to organize new knowledge is oftern inadequate. Considering the limited capacity of working memory, students who learn new or complex material should be facilitated with a learning design that minimizes cognitive load in working memory. Cognitive load is considered to be the cognitive demands made on working memory when learning and/or solving problems. Sweller . states that cognitive load memory can be caused by two sources, namely: . the complexity of the elements to be learnt . ntrinsic cognitive loa. from the presentation of teaching materials . xtraneous cognitive loa. Both of these sources are accumulative in working memory. Intrinsic cognitive load actually cannot be changed because it is related to the complexity of the interrelationship of elements in the learning material naturally (Sweller & Chandler. However, the material has high or low intrinsic cognitive load dependent upon the prior knowledge of the learner. For example, for grade three elementary school students, the fraction addition material has a high intrinsic cognitive load, but for students majoring in mathematics, this material has very low intrinsic cognitive While extraneous cognitive load can be manipulated because it depends on how the material is presented, complex material when presented in complex problem solving will be difficult to learn by students and result in high extraneous cognitive load. If presented more simply, because there are worked examples or systematic guidance, complex material can be more easily learned because the extraneous cognitive load is low. If the accumulation of intrinsic and extraneous cognitive load is minimized, then working memory will have the capacity for germane cognitive load, namely the capacity to understand the material, process it and construct new structured knowledge. Most problems provided to students have a fixed goal. For example in learning mathematics, problems often have a final goal: e. Aufind the value of x in an algebraic problemAy. The value of x in this case is referred to as the specified goal. According to Ayres & Sweller . , in solving such problems students often use a general problem-solver such as means-ends analysis and work backwards from the goal . ee also Youssef-Shalala. Ayres. Schubert, & Sweller. For many students this creates a high cognitive load and reduces learning. Therefore, problem-solving strategies that use fixed goals can be ineffective because students do not build logical or rational arguments to solving the problems from the given source information. counteract this situation, an effective learning strategy has been developed that removes the final goal. This strategy uses goal-free problems instead of goal-specific problems, and is known as the goal-free strategy . ee Ayres, 1. The present study uses goal-free problems as will be explained later. Equally important, learning experiences need to consider the culture that develops in the student environment. Collaborative learning always needs to be developed because collaboration is one of the noble values AUAUof itAos culture which is also a character of a nation. The ability to collaborate is also important and is developed through classroom learning, and has been considered as beneficial (Bhowmick. Chandra. Harper, , & Sweetin, 2. and even needed to reach the studentAos zone of proximal development as suggested by the VygootskyAos theory (Daniels, 2001. Johnson & Johnsjon. Therefore, ongoing research into PBL and collaborative learning needs to continue, especially how problems are presented, which is the main aim of this study. Although, many researchers argue that collaborative learning is superior to individual learning, some studies do not support this assumption . ee Retnowati. Ayres, & Sweller. For example. Retnowati. Ayres, & Sweller . found no significant difference between collaborative and individual learning when students used a PBL learning approach with specified goals. This study showed that when students have low initial knowledge . he material learned is new materia. , learning it through in collaboration did not result in higher transfer scores than individual learning. In other words, this type of PBL created more extraneous cognitive load than studying individually. addition, in collaborative learning, students are asked to transact with other students (Hung, 2. If each group member has a high extraneous cognitive load due to a given Cakrawala Pendidikan. Vol. No. October 2019 problem solving, there will be an additional extraneous cognitive load caused by interacting with other students who are also experiencing high extraneous cognitive load. As an alternative to a conventional problem-solving approach. Retnowati, et. tested the effectiveness of goal-specific PBL by learning through worked exampleproblem solving pair. Students were provided worked examples to be studied first, so they build prior knowledge. Learning with a pair of worked examples and a similar problem to be solved has been found to be more effective than problem solving without given solutions, because worked examples can reduce extraneous cognitive load . ee Cooper & Sweller, 1. However, in collaborative settings compared to individual settings, there was no significant difference between a PBL approach and learning using working example-problem solving pairs (Retnowati, et. , 2. Arguably because worked examples are less able to encourage students . ho have low initial knowledg. to interact in groups, although worked examples can function as initial knowledge to AuborrowAy In the present study, problems were presented without a specific goal. It was predicted that a goal-free PBL approach would lower extraneous cognitive load because students would avoid unhelpful working-backwards strategies such as means-ends analysis. Also it was predicted that PBL would be more effective when applied in collaborative learning because goal-free problems would enable students to communicate with each other . about the various problem-solving strategies they use from the same sources of information, with more available working memory capacity. Hence learning would be enhanced using collaboration. The study tested these assumptions using transfer problems and cognitive load measures. METHOD Study Format and Participants The current study used a 2 problem presentation strategy . oal-free vs. goal-give. by 2 learning environment . ollaborative vs. factorial design. This therefore led to four experimental groups: . goal-free problems studied collaboratively, . goal-free problems studied individually. goal-given doi:10. 21831/cp. problems studied collaboratively, and . goalgiven problems studied individually. A goalfree problem presentation means that the tobe-learned problem was presented without a specified goal, and therefore encourages students to work forward from the given information to the closest unknowns. On the other hand, goalgiven problem provides students with a specific goal to solve. The participants of this study were year 7 students from two Indonesian Junior High Schools. Both schools followed the same national curriculum and instruction methods. Presumably the students had studied at a similar level of mathematics, with a similar study load, and the same learning strategies. It was stated in the Indonesian national curriculum and informed by the school that the students had used both individual, classical, and collaborative learning approaches in mathematics and other subjects. Collaborative learning in this study was similar to what they were used to. that is, it involved smallgroup discussions during the acquiring of the tobe-learned material. The to-be-learned material used in the experiment was in Geometry. angles that formed parallel lines and transversal lines, triangles and parallelograms, selected from the curriculum. The material, which the participants had not previously studied, was presented in a problem-solving format. Two regular classrooms from each school participated voluntary. The original number of students in each class was 32 students, who were then divided at random into 4 collaborative learning groups consisting of 4 students. individual learners. Then again using random assignment half of the collaborative learning groups were given goal-free problems and the other half goal-specific problems. The same problem division was made for individual Data from several students who did not complete all stages of the study were removed, thus the total number of the participants was 111 students . ge average 12. 8 years old. 52 boys and 72 girl. , which may be grouped based on the first variable: 57 students in goal-free problems. 54 students in goal-given problems, and the second variable: 61 students in collaborative 50 students in individual learning. Learning Goal-Free Problems: Collaboratively or Individually? Procedure The procedure was similar to the Retnowati, . study where four phases were followed: introductory, acquisition, retention . ear transfe. test, and a far transfer test. the introductory phase, all students were given the same instruction to study the pre-requisite learning material of nine angle theorems (Table To gain a better understanding of each theorem, the students were given a protractor to measure the angles and then state the relation of both angles to each other in order to gain a stronger visualisation of the theorem. A set of geometrical figures were provided for this phase, as well as a protractor and a pencil. The angle formations were simple in structure and consistent with those used in textbooks. Time allocation for this phase was 40 minutes. Table 1. The Nine to-be-learned Angle Theorems Theorem Name of theorem Adjacent angles in a right angle sum to 90o Adjacent angles in a straight line sum to 180o Vertically opposite angles made by straight lines are equal Angles formed by lines running to the same point sum to 360o Corresponding angles made by parallel lines are equal Alternate angles made by parallel lines are equal Co-interior angles between paralel lines sum to 180o The three angles in triangles sum to The four angles in a quadrilateral sum to 360o In the following acquisition phase, students were expected to learn the main material by solving problems that required the application of two or more of the angle theorems. The 10 problems with fairly complex configurationswere presented in a booklet with the instruction at the beginning of the page. Students in the goal-free condition were asked to Aufind as many unknown angles as possibleAy, while those in the goalgiven condition were asked to Aufind the measure of angle xAy. Alloted time was 40 minutes where students could write their answers in the figures. There was only one problem on each page of the Studnets could work at their pace. Tools such as protractor, ruler, or calculator were not An example of the two types of strategies are shown in Figures 1a and 1b. For the goalfree problem (Figure 1. , the two given angles can be used in forward-moving steps to find the other nine unknown angles. In contrast for the goal-specific problem (Figure 1. designated goal . dominates the solution strategy. Although, the shortest forward-working strategy to solve this problem is by applying theorem number 2, 5, and 8 in that order, students my try to work backwards from the goal creating more subgoals and raising cognitive load. Goal-free problem Goal given problem learning phase. This instrument was developed using a nine point Likert scale (Van Gog & Paas, 2. with a range of 9 = very-very difficult and 0 = very-very easy. The question for students in individual settings is AuHow easy or difficult are you in solving individual problem solving?Ay And collaborative AuHow easy or difficult are you in solving a given problem solving after collaborating?Ay This question is placed on the bottom page of each problem in the booklet. There were 10 questions in the acquisition phase, five in the near-transfer test and five in the far-transfer test. Score 1 was given to every correct angle size generated and score 0 for the incorrect. For the goal free problems in the acquisition phase, the maximum score is the number of unknown angles. For the goal given problem, the maximum score is the number of given x. Proportion between the correct and the maximum scores was calculated in each phase results for the analysis purpose. The near transfer test has problems that have a similar structure to five of the problems randomly selected from the student worksheet in the acquisition phase. In this test, problems are goal specific and students are asked to determine find X. Problems in the far transfer test have a different structure to the near-transfer problems and have a differentcontext . nfamiliar, more difficult, more complex, need more than three theorems to solv. so different solution strategies are required even though the basic concepts are the same as to what has been experienced The material for the far-transfer test was developed by considering representations that had little in common with the material in the previous phase. In this test, students are also asked to find X. Figure 2 below is an example of a far-transfer test problem. Figure 1. An Example of the Problem Solving in the Acquisition Phase Students in the goal-given approach are asked to find the specified X angle size, while students in the goal-free approach are asked to complete all unknown angular measurements. Students write their answers directly on the page. The cognitive load of students is measured by the rating-question given at the end of each Cakrawala Pendidikan. Vol. No. October 2019 Figure 2. Example of Far-transfer Test doi:10. 21831/cp. The two test instruments were prepared through discussion, consultation, expert judgment, and piloting. Alpha Cronbach score for near transfer . and far transfer were respectively . 859 and . 857, indicating a high degree of reliability. In this study, data analysis of transfer ability and cognitive load was carried out in stages applying univariate analysis of variance (ANOVA) to test the main effect of each independent variable and the interaction effect of the two independent variables. instructions was also revealed. , . = 8. MSE = 1. 91, p = . 005, p2 = . This showed that goal-free problems caused higher cognitive load than goal-given problems. However, an interaction effect between groupings and instructions was not found. F < 0, p > . RESULT AND DISCUSSION Result StudentsAo cognitive load was measured consecutively in each learning phase, namely the Acquisition Phase. Near Transfer Phase, and Far Transfer Phase. The measurements were carried out on groups of students who took the Goal Given (GG) and Goal Free (GF) both in individual settings (IS) and collaborative settings (CS). The measurement results are presented in Table 1. Mathematical performance was measured on the Near Transfer test and Far Transfer test only . eeTable . Diagram 1. Cognitive Load during Acquisition Phase Data Analisis of Retention in Near-Transfer Test A significant main effect of groupings on near-transfer score was not indicated. F < 0, p > . 05, indicating that there was no difference between studying individually or in collaborative A significant main effect of instructions was found. , . = 8. MSE = 1. 64, p = 004, p2 = . This confirmed the hypothesis that goal-free problems facilitated more effective Data Analisis of Cognitive Load during Acquisition Phase A significant main effect of groupings was found. , . = 11. MSE = 1. 91, p = . p2 = . 10, where students working individually experienced heavier cognitive load than those in collaborative groups. A significant main effect of Table 2. Average Score of Cognitive Load Based on Instruction and Setting Type Acquisition Phase Near Transfer Phase Far Transfer Phase Individual (N=. Goal-Given Goal-Free N=54 N=57 Goal- Given N=54 Goal-Free N=57 Goal-Given Goal-Free N=54 N=57 Collaborative (N=. Table 3. Score of Mathematics Based on Instruction and Setting Type Near Transfer Phase Far Transfer Phase Individual (N=. Goal-Given N=54 Goal-Free N=57 Goal-Given N=54 Goal-Free N=57 Collaborative (N=. Learning Goal-Free Problems: Collaboratively or Individually? learning than goal-specific problems. However, an interaction effect between groupings and instructions was not found. F < 0, p > . Data Analisis of Retention in Far-Transfer Test A significant main effect of groupings on far-transfer score was indicated. , . = 28. MSE = 1. 45, p < . 001, p2 = . 20, rejecting the hypothesis since the far-transfer score of individuals was considerably higher than those in collaborative groups. A significant main effect of instructions was also revealed. , . = 11. MSE = 1. 45, p = . 001, p2 = . This supports the hypothesis that goal-free problems improved learning more than goal-specific problems. close-to-significant interaction effect between groupings and instructions was found. , . = 3. MSE = 1. 45, p = . 070, p2 = . Diagram 2. Score of Retention Near-Transfer Test Data Analisis of Cognitive Load during NearTransfer Test Unlike the cognitive load during acquisition phase, no main effect of groupings on cognitive load wasfound. , . = 1. 25, p > . 05, meaning individual students experienced as much cognitive load as those in collaborative groups. A significant main effect of instructions was found. , . = 16. MSE = 2. 18, p = . 044, p2 = . Interestingly, this result showed that goal-free problems caused higher cognitive load than goal-specific problems during near transfer test. However, an interaction effect between groupings and instructions was not found. F < 0, p > . Diagram 4. Score of Far-Transfer Test Data Analisis of Cognitive Load during FarTransfer Test Non-significant main effect of groupings, instructions or the interactions were found for the cognitive load during far-transfer test. F = 72. F < 0. F = 2. 54 respectivelly with p > . for all of these. This indicated that there was no significant difference of cognitive load in either treatment, albeit it seems an extreme gap that the cognitive load of students using goalfree problems in individual and collaborative learning, where the collaborative students experienced the lowest cognitive load. Diagram 3. Cognitive Load of Near-Transfer Test Phase Cakrawala Pendidikan. Vol. No. October 2019 doi:10. 21831/cp. Diagram 5. Cognitive Load during FarTransfer Test Phase Discussion The goal-free effects during mathematics collaborative learning were investigated with respect to cognitive load ratings and transfer Two strategies of grouping students were allocated, namely collaborative learning and individual learning. and two approaches of problem based instructions were created using goal-free and goal-given problems. Students were randomly assigned to one of the four experimental groups to follow the acquisition and test instructions. Two independent variables, cognitive load and transfer ability were measured in every phase of the experiment, acquisition phase, near-transfer test and far-transfer test. Three hypotheses were formulated: . Goal-free problems facilitate learning better than goal-given problems. Collaborative learning facilitates better than individuals. Goalfree problems in collaborative learning facilitate better than in individuals. Table 3 summaries the data analysis results. The findings showed that students who learned by goal-free problems had significantly higher both near-transfer and far-transfer scores than students who learned by goalgiven problems. This result confirmed the first hypothesis and indeed, replicated previous findings by Ayres . and Youssef-Shalala . Ayres & Sweller . suggested that goal-free problems could enhance learning because it reduces heavy extraneous cognitive load caused by the use of means-ends strategies. In other words, using goal-free problems, students are able to manage their cognitive load and hence, their working memory capacity is focused on knowledge construction and automation (Sweller, et. , al. , 2. The current research measured cognitive load during acquisition and test phases. It was assumed that when cognitive load is low, then learning performance is improved . ee Sweller, , 2011. Sweller, 2010. van Gog & Paas. On the other words, if score on transfer test is high, then students should have experienced lighter cognitive load caused by the instructions, therefore they are able to use their working memory load for knowledge construction and automation as the base of transfer ability. the contrary, if score on transfer test is low, then students should have experienced heavier cognitive load caused by the instructions, therefore their working memory load is most likely been exceeded by unnecessary processes rather than for knowledge construction. This research have found the opposite results, that studentAos cognitive load on learning goal-free problems was significantly higher than those on goal-given problems, during acquisition and near-transfer test. Possible reasons are discussed Table 3. Summary of Data Analysis Results Effect for/by Groupings Instructions Groupings x Instructions Transfer ability Near-transfer Far-transfer NS. Sig. Sig. Sig. NS. NS. *** Cognitive load Acquisition Near-transfer Far-transfer Sig. NS. NS. Sig. Sig. NS. NS. NS. NS. *p < . **p < . ***p = . Learning Goal-Free Problems: Collaboratively or Individually? It may be said that the context of goal-free problems naturally direct students to process many answers (Ayres & Sweller, 1. For an example, a goal-free problem used in this experiment has 13 unknown angles and were asked to solve all of these unknown angles. From the given angles, students had to solve the measure of unknown angles step-by-step in forward manners. This means students allocated their cognitive resources for at least the nine geometry theorems, the two given angles, and the 13 unknown angles. As the consequence of this direct instruction, theiy were able to manage their cognitive capacity for learning. On the other hand, students who were given one angle to solve, like in the goal-given problem, might allocate less cognitive resource although they had to create sub-goals to solve the problem using means-ends analysis (Ayres & Sweller. Nevertheless, with the limited automated prior knowledge, students might not be able to perform alternative sub-goals after failing the first one. Hence, their learning was not successful after transfer test. In addition to this goal-free effect, a closeto significant interaction effects found on fartransfer tests revealed that goal-free problems could be learned individually or collaboratively. It is likely this result was obtained because of the strong effect of goal-free problems over the counterpart, and though no further empirical evidence of the interaction effect can be shown from the data analysis. Learning in a collaborative group itself was hypothesised resulting in higher performance during transfer than learning However, the findings suggested rejecting this hypothesis. No significant difference was found for near-transfer test, but in the far transfer test, where individual were outscored collaborative learning. Though during acquisition phase, individuals had significantly heavier cognitive load than those in collaborative groups. Similar results were showed in a worked-example approach . ee Retnowati, et. , 2. Collaborative will be very useful when applied to complex material because it requires specific abilities (Hesse. Care. Buder. Sassenberg, & Griffin, 2. , specific collaboration attitudes (Johnson & Johnson, 2. , and grouping arrangements (Retnowati. Cakrawala Pendidikan. Vol. No. October 2019 , 2. It was assumed that the goalfree problems would increase the interaction among students in collaborative learning. discussions become intense, it was suspected that students learn more from each other by giving help or receiving elaborations (Webb & Mastergeorge, 2. Particularly, it could be said however, the presentations of others during studying many unknown angle measures might inhibit learning. In addition, there could be other factors associated to motivation that is needed to overcome the high cognitive load (Paas. Tuovinen, van Merrienboer, & Darabi. There was lack of intervention in this aspect, indeed this was not the focus of the current study. Complementary with the results of cognitive load measures above, learning goalfree problems in collaborative groups caused heavy cognitive load, and hence inefficient for improving transfer ability. CONCLUSION The results suggest that the use of goal-free problems in the context of mathematics learning can be superior to the goal-specific problems, in terms of transfer scores. This finding leads to further implementation in the classroom however, more research on each step performance may be The goal-free problems enable students to create several forward moves, however it is yet known how students select the first move and so on. The results of the cognitive load measures also suggest further investigation on the level of cognitive load in each move since it is unanswered yet why students had heavy cognitive load. Such study may inform us why students in goal-free problems experienced higher cognitive load. Moreover, study on instructional designs for collaborative learning is needed to meet our common sense that to work together is better than alone. It may be argued that providing collaborative learners problems that facilitate multiple moves possibly to discuss in the group is not sufficient Undoubtedly, there could be other significant factors influencing the effectiveness of the collaborative learning simultaneously. Nevertheless, this study concludes that goal-free problems may be studied both collaboratively or doi:10. 21831/cp. REFERENCE