Assessment of ESP students’ learning outcomes in a digital learning environment

Размещено на http://www.allbest.ru/

Размещено на http://www.allbest.ru/

Assessment of ESP students' learning outcomes in a digital learning environment

Kira M. Inozemtseva, Elizaveta V. Morozova, Ilya M. Kolesnikov

Abstract

Problem statement. The description of a pedagogic research in the field of formative assessment theory and practice is presented. The goal of the study is to define pedagogic and methodological conditions for effective evaluation of ESP students' learning outcomes in a digital learning environment. Methodology. On the basis of key theoretical provisions of formative assessment and interdisciplinary approaches LSP&CLIL, a complete set of criteria-based ESP evaluation tasks and assessment schemes were developed for Business English course delivered in the groups of BMSTU masters students majoring in economics. The approach to grading that correlates with BMSTU point-rating system was substantiated. The comparative analysis of modern platforms allowed to select the domestic service pruff.me, which was used as a ground for a digital assessment space (DAS), a part of BMSTU digital learning environment needed for implementing formative assessment of ESP learning outcomes. Results. The developed evaluation materials integrated in the DAS were tested by ESP students in a distant format. Conclusion. The research showed that creating DAS as an essential component of a university digital learning environment can contribute to effective assessment of ESP students' competence-based learning outcomes provided that underlying methodology is taken into account and formative assessment technology is implemented in full volume.

Keywords: formative assessment, English for specific purposes, digital learning environment, digital assessment space

Оценивание результатов обучения иностранному языку для профессиональных целей в цифровой среде

К.М. Иноземцева, Е.В. Морозова, И.М. Колесников

Аннотация

Постановка проблемы. Представлено педагогическое исследование в области теории и практики формирующего оценивания, целью которого является определение методико-педагогических условий эффективного оценивания результатов обучения иностранному языку для профессиональных целей в цифровой среде. Методология. На основе анализа теоретических положений формирующего оценивания и междисциплинарных подходов LSP и CLIL разработан полный комплект критериальных контрольных заданий и оценочных схем для оценивания результатов обучения дисциплине «Деловой иностранный язык (английский)» магистрантов экономических специальностей в МГТУ имени Н.Э. Баумана. Приведено обоснование подхода к выставлению оценок, коррелирующее с принятой в МГТУ имени Н.Э. Баумана балльно-рейтинговой системой. Сравнительный анализ современных платформ, используемых в целях оценивания, позволил сделать выбор в пользу отечественной платформы pruff.me и создать на ее основе пространство цифрового оценивания (ПЦО) как компонент цифровой образовательной среды МГТУ имени Н.Э. Баумана, предназначенный для реализации формирующего оценивания результатов обучения иностранному языку для профессиональных целей. Результаты. Разработанные контрольные материалы, интегрированные в ПЦО, апробированы студентами в дистанционном формате. Заключение. Показано, что создание пространства цифрового оценивания как важного компонента цифровой образовательной среды университета может способствовать эффективному оцениванию ком- петентностных результатов обучения иностранному языку для профессиональных целей при условии учета рассматриваемых в исследовании методологических концепций, а также при включении всех обязательных компонентов формирующего оценивания.

Ключевые слова: формирующее оценивание, иностранный язык, профессиональные цели, цифровая образовательная среда, цифровое пространство оценивания

Introduction

Problem statement. The pandemic period associated with the transfer of educational processes to distant format aroused great interest of educationalists in the theory and practice of ESP teaching in a digital environment. However, the problem of `digital' assessment of ESP competence-based learning outcomes has not received sufficient scientific coverage. In the context of a new educational paradigm, which implies `digital transformation' of universities, the issue of `digital' assessment needs to be studied in details.

Over the last decade there have appeared a wide number of scientific papers on teaching methodology in a digital environment. M.E. Vaindorf-Sysoyeva et al. (2020) highlighted the issue of creating a virtual educational environment. Vaindorf-Sysoeva ME, Gryaznova NS, Shitova VA. Distance learning methodology: study guide for universities. Moscow: Yurait Publ.; 2020. (In Russ.) V.I. Blinov et al. (2019) put forward the concept of digital didactics as a scientific discipline on the organization of educational processes in a digital society [1]. In the context of linguistic education, the term `digital linguodidactics' is being increasingly used. This term means a scientifically based system for organizing learning in a digital environment. Professor S.V. Titova (2017, 2022) has made a significant contribution to the development of this branch of didactics, with some aspects of digital assessment being considered in her scientific works [2; 3].

At present, teaching ESP to non-linguistic students is regulated by leading methodological approaches Language for Specific Purposes (LSP) [4] and Content Language Integrated Learning (CLIL) [5]. The choice in favor of CLIL, which is often made by foreign language departments, is focused on the parallel acquisition of a subject-specific foreign language and disciplines of the professional cycle [3]. Despite the difference in goal setting, LSP and CLIL are both interdisciplinary, which implies indirect assessment of both components of subject-language integration. This, in turn, requires development of interdisciplinary evaluation tasks.

LSP and CLIL methodology involves a departure from the traditional `norm-referencing approach' to evaluation This type of assessment is also called `summative.' Testing is considered to be the main method of summative assessment., in which the objects of evaluation are compared with a certain standard (the norm), to formative assessment. An appeal to formative assessment based on the theory of constructivism (L. Vygostky, 1934, J. Piaget, 1969) and the concept of `Mastery learning' (B. Bloom, 1956, M. Scriven, 1967) is due to the desire to create conditions for students' activity- based learning and their conscious interpretation of cognitive experience [6]. The concepts that are mentioned above are in line with the provisions of competence-based approach, requiring adequate assessment of learners' competences as the expected learning outcomes. Assessment of competencies is a complex problem of educational methodology. With a huge number of scientific papers devoted to this issue, there is still no consensus on how to evaluate it. Nevertheless, it is obvious that with a competence-based approach, both knowledge and practical skills should be evaluated. Otherwise, it is impossible to talk about achieving the goal - formation of competences.

In competence-based education the expected learning outcome of ESP training is formation of foreign language professional communicative competence, which represents the ability to “use a foreign language for solving professional tasks” [7]. Undoubtedly, ESP course evaluation should combine knowledge and skill-based communicative tasks based on clearly defined criteria. However, the analysis of ESP assessment practices shows that even in offline format teachers tend to simplify assessment procedures, perceiving competence-based learning outcomes as an abstract concept and therefore making a final judgement on students' achievements based on tests. This approach is not consistent with either LSP and CLIL methodology, or the competence-based paradigm.

Being a fundamental indicator of the education quality, assessment should be a constant process of monitoring the educational and cognitive activities of students. Formative assessment meets this requirement, as it is based on the dynamic observation of each student' academic growth, does not prioritize comparison with the standard/norm and does not use repressive function. According to L.V. Vilkova (2017), formative assessment is a “step-by-step movement of each student to the best learning outcomes through active inclusion in the analysis (reflection) designed to identify difficulties and gaps in a course acquisition and effectively fill them” [8]. In the context of LSP and CLIL, any practical communicative tasks combining subject-oriented content and being performed through a foreign language In CLIL methodology subject content is delivered not in, but through a foreign language, which is called `vehicle language' [4]. (debates, problem discussion, case-study, presentation, laboratory experiment report, graph interpretation, diagram analysis, technical instruction, patent application, minutes, elevator pitch, essay, business letter, scenario, leaflet, blog, etc.) can serve as objects of formative assessment [5]. Criteria assessment rubrics are required to evaluate the above tasks.

L.V. Vilkova (2017) identifies essential components of formative assessment, which are accepted in this study as constitutive:

1) `objectives-results' dyad, which implies transfer of learning objectives to expected learning outcomes, which is traditional for competence-based approach;

2) collaborative work with students to define the evaluation criteria, which promotes students' better understanding of the requirements and creates sense of belonging to the assessment process;

3) self-assessment promoting the development of critical thinking;

4) peer assessment aimed at developing objectivity and mutual support;

5) feedback, providing students with recommendations and tips that make them think better. Following L.V. Vilkova we consider it appropriate to complement summative and formative assessment methods, as it allows to get a clearer picture of the program acquisition [8].

In a digital learning environment, it is advisable to implement formative assessment of ESP learning outcomes on the basis of `integrated evaluation principle', highlighted by the authors of digital didactics concept (Blinov et al., 2019). According to this principle, “the traditional assessment process is transformed into a continuous personalized diagnosis-forming assessment of academic success” [1, p. 52]. This requires a comprehensive analysis of new didactic digital tools capable of providing instant feedback, informing a teacher and a student about the course and results of task fulfillment, issuing personalized `troubleshooting' recommendation and adjusting the immediate goals and scenarios for further development [1, p. 52].

The purpose of this study is to determine conditions for effective assessment of ESP students' leaning outcomes in a digital learning environment. The study took place at Linguistics faculty of BMSTU with the participation of 58 master students of Engineering Business and Management (EBM) faculty studying `Business English'. The research materials included 1) didactic materials of Business English course delivered at BMSTU, including case-studies, supplementary audio and video materials Townend J, Allison J, Emmerson P. The business upper intermediate student's book. Macmillan Education; 2013. and tests developed and approved by Linguistics faculty; 2) 20 modern platforms used for evaluation purposes - Wizer.me, iSpring, Genially, Interacty, PruffMe, Exam, Яндекс.Формы, StudySmarter, Symbaloo, Google Формы, Quizizz, Pear Deck, Kahoot, Typeform, Lumio, Unio, Learning Apps, Online Test Pad, BrandQuiz, Buncee.

Methodology

evaluation education foreign language

The methodological basis of the research included key theoretical provisions of formative assessment [9-11], interdisciplinary approaches LSP and CLIL [4; 12], digital didactics [1], digital linguodidactics [2; 3], distant learning. Vaindorf-Sysoeva ME, Gryaznova NS, Shitova VA. Distance learning methodology: study guide for universities. Moscow: Yurait Publ.; 2020. (In Russ.) Based on the analysis of scientific literature and Russian educational standards (FGOS), the goals and values of formative assessment, as well as the requirements for ESP learning outcomes of master students majoring in economics were identified. Then, a comparative analysis of 20 platforms used for assessment purposes was conducted with a view to select the best one. The selection criteria included simplicity and convenience of the interface, availability of videoconference mode, chat and cloud storage, uploading audio and video files, creating tests, presentations, sharing files and monitoring results online, free access. As a result, the domestic service pruff.me was selected for creating a digital assessment space (DAS), a part of BMSTU digital learning environment needed for implementing formative assessment of ESP learning outcomes.

In the course of study, the analysis of didactic and evaluation materials used for teaching ESP master students majoring in economics at BMSTU was conducted. All the materials were adapted to the tasks of formative assessment. In compliance with the formative assessment methodology, evaluation criteria for oral communicative tasks (case-studies, problem discussions, presentations) were identified in collaboration with the participating students. Based on the identified criteria, assessment schemes for the above practical tasks, as well as all other evaluation materials, were developed and integrated into pruff.me. All the materials have been tested by EBM master students in the process of Business English distant learning. Based on the results of evaluation materials testing, methodological recommendations for ESP teachers have been compiled.

Results and discussion

Business English course for EBM master students at BMSTU is designed for 1 year and includes 6 modules. During the training, the students are engaged in in the development of ESP speaking, listening, reading and writing skills in the field of business communication. Within the framework of a course, the students extend their knowledge of international business practice, business education, sales and marketing strategies, quality management, financial control. The ESP training is conducted on the basis of communicative method, with a significant time being devoted to case analysis, discussions, role plays and profession-oriented written tasks. Monitoring and assessment of ESP learning outcomes is carried out on the basis of oral and written interdisciplinary assignments (case-studies, presentations, problem discussions, tests).

Critical analysis of ESP online assessment practice at BMSTU showed that despite the absence of explicit criticism of ESP control materials used, the process of evaluation seemed to be one-sided, with the main focus made on electronic tests and oral communicative tasks evaluated intuitively by the teachers. This approach allows to record the average level of ESP course acquisition, but doesn't contribute to students' conscious interpretation of their academic achievements [13]. The virtual classroom observation confirmed the importance of creating a digital assessment space for ESP teachers and students, that allows to monitor students' individual progress and make an adequate assessment of learning outcomes in a digital learning environment.

In the course of this research evaluation materials of Business English course were brought into line with the requirements of formative assessment and integrated into pruff.me. The description of these materials including criteria- based assessment schemes is given below.

One of the most popular competence-based techniques used for educational and evaluation purposes at universities is a case study, a description of a situation containing a problem or a contradiction and based on real facts. Case study is widely used in interdisciplinary foreign language teaching as it demonstrates students' ability to apply theory for solving practical tasks and ensures content acquisition through emotional involvement. On the platform pruff.me ESP students can collaboratively solve the case tasks, listen to the audio and discuss the results in videoconference mode. The sample of case study placement on the platform is shown in Figure 1.

Figure 1. Case study `IT Solutions in Business'

Available from: https://pruffme.eom/landing/u2696543 /tmp1654084672 #/26c59f270 a6b86468e4de69ae03015d0 (accessed: 15.06.2022)

In the process of collaborative development of case studies evaluation criteria, it was decided to move away from the traditional format of evaluation scale (rubric) in favor of a criteria-based assessment scheme. The scheme combines tasks fulfilment criteria and indicators in a single description of four levels that describes ESP students' ability to solve a profession-oriented communicative task and informs the participants about the assigned points. For the convenience, each level is assigned its own range of points correlating to the point-rating system adopted at Linguistics faculty and to the traditional five-point scale familiar to students. For instance, level 1 corresponds to a score range of 0-4 or `bad', level 2 - 5-6 points (`satisfactorily'), level 3 - 7-8 points (`good'), level 4 - 9-10 points (`excellent'). The sample of case assessment scheme is presented in Figure 2.

Figure 2. Case-study assessment scheme

Available from: https://upload.pruffme.com/download/?media= 963b78b3c4c8dee5419fc7492d1fa532 (accessed: 15.06.2022)

The final communicative task fulfilled by ESP students participating in this research in the end of each term is individual multimedia presentation. The public defence of a presentation allows the students to demonstrate so-called `soft skills' including an ability to work with the information sources, insight into the problem, an ability to communicate information in an accessible form, establish contact with the audience. On pruff.me, presentation topics are given in a list. The logic of designing a multimedia presentation assessment scheme is similar to the same of the case study (Figure 2). The assessment criteria for presentation are: 1) relevance of the topic; 2) language and communication; 3) logic and persuasiveness; 4) design; 5) contact with the audience.

One more oral communicative task used in this study for evaluation purposes is a problem discussion on the video. Problem discussion is a kind of an argument aimed at achieving the truth, with participants' argumentation being the most important characteristic of the task execution. In the course of this assignment, a subject- related video is offered for ESP students' viewing on the platform. Then the teacher asks questions on the content initiating a discussion. Problem discussion on the video implies an accurate understanding and correct interpretation of what was seen and heard. The assessment scheme of this task is shown in Figure 3.

Figure 3. Assessment scheme for the discussion on the video

Available from: https://upload.pruffme.com/download/?media= 5d9c676f88f4d1f6743199e0716e0889 (accessed: 15.06.2022)

When fulfilling all oral communicative tasks (cases, presentations, discussions), the students carry out peer assessment by rating their groupmates achievements on the collaboratively developed criteria (see assessment schemes in Figures 2, 3). From the grades received, the teacher deduces the average pointrating value in accordance with the results of each student.

According to the idea of formative and summative assessment tools' complementarity proposed by L.V. Vilkova, tests were also included in the set of our evaluation tools and placed on pruff.me. In case of misunderstanding or difficulties encountered when performing the test, students have an opportunity to communicate with the teacher in a chat, where every participant can attach a file, record/send an audio or a video reply. The teacher in turn can see an avatar and a name of every student, which makes it much easier to navigate on the platform. Points are awarded for the correct answers to the test questions and summed up with the points received for oral communication tasks. The resulted value is taken as a final student's score.

An opportunity to chat with a teacher on the platform allows students to provide feedback. When clicking on the `Message section' icon, the teacher can see a window with the list of students and their module assignments completed. The students' messages in the chat are tied to control activities, which greatly facilitates the teacher's checking work and allows to monitor every student's academic achievement in dynamics.

Figure 4. ESP student self-assessment check-list Available from: https://upload.pruffme.com/download/?media=a6eb11397d60d244d2343ffbd97941c1 (accessed: 15.06.2022)

Another important component of formative assessment is self-assessment. The sample of self-assessment checklist filled by each student in the end of the 1^st and 2^nd terms is presented in Figure 4. The proposed self-assessment check-list placed on pruff.me is designed on the principle of Likert scale, which allows the respondents to carry out a qualitative evaluation of the educational process, students' perception of the course and also, if necessary, transfer the obtained qualitative data to numerical form. Qualitative characteristics are given in the form of statements, evaluating participants' achievements during the period under review on the Likert scale from 1 to 5 (`Over the past semester I've learned how to the quality of the course and the teacher's work (see statements 1-3 in Figure 4). These statements characterize the learning outcomes from the students' perspective.

The developed formative assessment tools were offered to the students participating in the research and caused a positive response. The reviews written by the participants in a free form contained the following comments: `it was convenient to chat with the teacher, get explanations', `evaluating each other is a bit tedious, but useful, you pay attention to the details', `it was important to evaluate yourself at the end of the term, it becomes clear what worked and what didn't', `it's nice that the teacher is interested in my opinion about the course'.

The experience gained during the described research showed the need to compile methodological recommendations for ESP teachers willing to implement formative assessment technology in a digital learning environment. The recommendations represent a step-by-step description of ESP teachers' actions, taking into account the formative assessment logic and values [14] and technical characteristics of the platform.

Conclusion

Within the framework of the performed study, it was defined that `digital' assessment of ESP students' leaning outcomes requires creation of a digital assessment space, a part of a university digital learning environment, based on a thoroughly selected platform that allows to place all evaluation materials and implies various modes of teacher-students online interaction.

In order to implement ESP `digital' formative assessment effectively, it is necessary to fulfill the following conditions:

1) take into account theoretical provisions of ESP&CLIL underlying methodology, when developing evaluation materials;

2) consider formative assessment as a dynamic observation of every student's academic growth aimed at identifying difficulties and filling the gaps, not at simple measurement of average results;

3) perform the evaluation procedures in accordance with formative assessment essential components, not excluding any of them.

In conclusion, implementing formative assessment technology has had a positive effect on ESP students' motivation making them the center of evaluation process. The ESP teacher, in turn, has been informed about the problems and got an opportunity to correct the course and teaching methods. The prospects for the conducted research lie in the field of designing digital assessment space as an essential component of universities' digital learning environment and ESP teacher professional development.

References

[1] Blinov VI, Dulinov MV, Esenina EYu, Sergeev IS. Pedagogic concept of digital vocational education and training. Moscow: Pero Publ.; 2019. (In Russ.)

[2] Titova SV. Digital technologies in a foreign language teaching: theory and practice. Moscow: Editus Publ.; 2017. (In Russ.)

[3] Titova SV. Assessment of project tasks in subject-language integration course: stages, forms, online instruments. Higher Education in Russia. 2022;(2):94-106. (In Russ.) http://doi.org/10.31992/0869-3617-2022-31-2-94-106

[4] Coyle D, Hood P, Marsh D. CLIL: Content and Language Integrated Learning. Cambridge: Cambridge University Press; 2010.

[5] Bloom BS. Taxonomy of educational objectives. Boston: Allyn and Bacon; 1956.

[6] Scriwen M. The methodology of evaluation. Perspectives of curriculum evaluation. Chicago: Rand McNally; 1967. p. 39-83.

[7] Inozemtseva KM. Profile of professional and methodical competence of ESP teachers of technical universities. Sovremennyje Problemy Nauki i Obrazovaniya. 2017;(3):111. (In Russ.)

[8] Vilkova LV. Formative assessment in implementation of a personally oriented approach (scientific school of Professor M.A. Vikulina). Vestnik Permskogo Gosudar- stvennogo Gumanitarno-Pedagogicheskogo Universiteta. Seriya 1. Psykhologicheskiye i Pedagogicheskiye Nauki. 2017;(2):15-23. (In Russ.)

[9] Clark I. Formative assessment: policy, perspectives and practice. Journal of Educational Administration and Policy. 2011;4(2):158-180.

[10] Briggs D, Ruiz-Primo M, Furtak E, Shepard L. Meta-analytic methodology and inferences about the efficacy of formative assessment. Educational Measurement: Issues andPractice. 2012;31(4):13-17.

[11] Prilipko EV. Assessment activity of a teacher in the competence-based paradigm: development of conscious competence. Innovatsionnyje Project i Programmy Obra- zovanija. 2017;(1):30-35. (In Russ.)

[12] Dudley-Evans T, St John M-J. Developments in English for specific purposes: a multidisciplinary approach. Cambridge: Cambridge University Press; 1998.

[13] Vikulina MA, Vilkova LV. The elements of the methodology regulating formative assessment of university students' academic achievements in foreign language mastering. PNRPU Linguistics and Pedagogy Bulletin. 2019;(2): 131-139. (In Russ.) http://doi.org/10.15593/2224-9389/2019.2.12

[14] Prilipko EV, Inozemtseva KM. Educational values in assessment: a professional development course for tertiary English language teachers in Russia. Proceedings of the 10th International Conference on Education and New Learning Technologies (EDULEARN18). Mallorca; 2018. p. 11163-11170.

Список литературы

[1] Блинов В.И., Дулинов М.В., Есенина Е.Ю., Сергеев И.С. Проект дидактической концепции цифрового профессионального образования и обучения. М.: Перо, 2019. 112 с.

[2] Титова С.В. Цифровые технологии в языковом обучении: теория и практика. М.: Эдитус, 2017. 248 с.

[3] Титова С.В. Оценивание проектных заданий в предметно-языковом интегрированном курсе: этапы, формы, онлайн-инструменты // Высшее образование в России. 2022. Т. 31. № 2. С. 94-106. http://doi.org/10.31992/0869-3617-2022-31-2-94-106

[4] Coyle D., HoodP., Marsh D. CLIL: Content and Language Integrated Learning. Cambridge: Cambridge University Press, 2010. 184 p.

[5] Bloom B.S. Taxonomy of educational objectives. Boston: Allyn and Bacon, 1956.

[6] Scriwen M. The methodology of evaluation. Perspectives of curriculum evaluation. Chicago: Rand McNally, 1967. Pp. 39-83.

[7] Иноземцева К.М. Профиль профессионально-методической компетентности преподавателя иностранного языка технического вуза // Современные проблемы науки и образования. 2017. № 3. С. 111.

[8] Вилкова Л.В. Формирующее оценивание в реализации личностно-ориентированного подхода (научная школа профессора М.А. Викулиной) // Вестник Пермского государственного гуманитарно-педагогического университета. Серия 1. Психологические и педагогические науки. 2017. № 2-1. С. 15-23.

[9] Clark I. Formative assessment: policy, perspectives and practice // Journal of Educational Administration and Policy. 2011. Vol. 4. No. 2. Pp. 158-180.

[10] Briggs D., Ruiz-Primo M., Furtak E., Shepard L. Meta-analytic methodology and inferences about the efficacy of formative assessment // Educational Measurement: Issues and Practice. 2012. Vol. 31. No. 4. Pp. 13-17.

[11] Прилипко Е.В. Оценочная деятельность преподавателя в компетентностной образовательной парадигме: формирование осознанной компетентности // Инновационные проекты и программы в образовании. 2017. № 1. С. 30-35.

[12] Dudley-Evans T., St John M.-J. Developments in English for specific purposes: a multidisciplinary approach. Cambridge: Cambridge University Press, 1998.

[13] Викулина М.А., Вилкова Л.В. Элементы методики формирующего оценивания учебных достижений в овладении иностранным языком обучающимися вуза // Вестник Пермского национального исследовательского политехнического университета. Проблемы языкознания и педагогики. 2019. № 2. С. 131-139. http://doi.org/10.15593/2224-9389/2019.2.12

[14] Prilipko E.V., Inozemtseva K.M. Educational values in assessment: a professional development course for tertiary English language teachers in Russia // Proceedings of the 10th International Conference on Education and New Learning Technologies (EDULEARN18). Mallorca, 2018. Pp. 11163-11170.

Размещено на Allbest.ru