Experiences of conditing online exam proctoring in low-resource settings: a Sri Lankan case study

Размещено на http://www.allbest.ru/

University of Colombo School of Computing

EXPERIENCES OF CONDUCTING ONLINE EXAM PROCTORING IN LOW-RESOURCE SETTINGS: A SRI LANKAN CASE STUDY

D.N. Felsinger, Sc (Hons) in Computer Science (Reading)

T. Halloluwa, Doctor of Philosophy, Senior Lecturer,

Department of Information Systems Engineering

C.L. Ishani Fonseka, B.Sc (Hons) in Computer Science, Lecturer

Colombo

Abstract

sri lanka academic education

Online tests are becoming increasingly common as online learning gains popularity. Online assessments are more prone to academic dishonesty than traditional exams. Ad hoc online proctoring techniques have been used by institutions to prevent academic misconduct during online tests. In contexts with unreliable infrastructure, such as slow internet connections and a lack of compatible equipment, it is challenging to figure out how to employ technology to proctor online tests swiftly and efficiently. The goal of this study was to gain a better understanding of the context and requirements for scaling up online exams in Sri Lanka as well as to investigate what options for improving the subsequent deployment of online proctoring systems in low -resource settings exist in order to maintain the caliber of academic programs. In this work, an in-class assignment for a Software Engineering course at a higher education institution in Sri Lanka was manually proctored using video conferencing software in a completely remote context. A postexam feedback survey was used to collect information about participants' experiences with online proctoring. The survey was supposed to reveal the experiences about the devices used, internet connectivity, the factors that led to stress and perceptions about cheating during the exam. The data obtained from both the assignment and the survey was analyzed in order to make wellinformed decisions about the future deployment of online proctoring technologies. This paper presents evidence of the experiment that strongly recommends that online exam proctoring solutions be developed after a careful examination of the constraints such as internet connection speed, stability, and affordability, as well as sudden power outages of the environment in which they will be utilised, in order to unlock the true potential of e-learning technologies.

Keywords: Online Proctoring; Online Education; Online Exams.

Анотація

ДОСВІД ДИСТАНЦІЙНОГО КОНТРОЛЮ ЗА ПРОВЕДЕННЯМ ОНЛАЙН-ІСПИТІВ В УМОВАХ ОБМЕЖЕНИХ РЕСУРСІВ: ТЕМАТИЧНЕ ДОСЛІДЖЕННЯ НА ШРІ-ЛАНЦІ

Ділки Н. Фелсінгер бакалавр комп'ютерних наук (з відзнакою) (читання) Школа обчислювальної техніки Університету Коломбо, Коломбо, Шрі-Ланка

Тіліна Халлува доктор філософських наук, старший викладач кафедри інженерії інформаційних систем Школа обчислювальної техніки Університету Коломбо, Коломбо, Шрі-Ланка

К. Л. Ішані Фонсека бакалавр комп'ютерних наук (з відзнакою), викладач кафедри інженерії інформаційних систем Школа обчислювальної техніки Університету Коломбо, Коломбо, Шрі-Ланка

Оскільки онлайн-навчання набуває популярності, онлайн тести стають все більш поширеними. Водночас при онлайн-оцінюванні знань студентів рівень академічної недоброчесності значно вищий ніж під час проведення іспитів традиційним способом. Для запобігання академічних порушень під час онлайн -тестів навчальні заклади використовували спеціальні методи онлайн-прокторінгу. Зважаючи на низький рівень оснащення інфраструктури, як-от: повільне підключення до Інтернету та відсутність сумісності обладнання - складно зрозуміти, у який спосіб краще застосувати технологію для швидкого та ефективного контролю онлайн-тестів. Мета цього дослідження - визначити необхідні ресурси і вимоги для поширення онлайн-іспитів у навчальних закладах Шрі-Ланки, можливі щляхи щодо покращення і поширення систем онлайн-прокторінгу в умовах обмежених ресурсів відповідно до вимог академічних програм. Було досліджено проведення прокторінгу аудиторного завдання для курсу «Програмної інженерії» у вищому навчальному закладі на Шрі-Ланці, який здійснювався за допомогою програмного забезпечення для відеоконференцій у повністю віддаленому доступі. Після іспиту було проведено опитування студентів щодо отриманого досвіду з онлайн - прокторінгу. Опитування мало виявити досвід щодо пристроїв, які були використані, підключення до Інтернету, факторів, які призвели до стресу та усвідомлення, що можливе виявлення списування під час іспиту. Для прийняття обґрунтованого рішення щодо подальшого поширення технологій онлайнпрокторінгу були проаналізовані дані, отримані як під час виконання завдання, так і під час опитування. У статті наведені отримані результати проведеного експерименту, які вказують на необхідність подальшого поширення онлайн - прокторінгу іспитів. Визначено, що для цього є необхідною умовою ретельне вивчення і врахування таких існуючих обмежень, як швидкість, стабільність і доступність Інтернетз'єднання, раптові відключення електроенергії в середовищі, у якому вони будуть використовуватися. Такий підхід допоможе розкрити справжній потенціал технологій електронного навчання.

Ключові слова: онлайн-прокторінг; онлайн-освіта; онлайн-іспити.

Introduction

Global adoption of online learning has increased rapidly in the recent past [1]. Access to quality learning material from world-renowned universities (e.g [2], [3]) and the ability to learn at one's own pace by engaging in Massive Open Online Courses (MOOC) are some of the benefits of online learning. Despite its advantages, online learning is facing challenges in conducting online exams while ensuring academic integrity and protecting the credibility of the programs [1], [4]. Academic dishonesty is more common in online tests than in physical exams since all the material that needs to be accessible is available with a single click or a voice command [5]. Siddhpura & Siddhpura [6] have identified cheating, collusion, impersonation, contract cheating and plagiarism as the possible types of academic misconduct in online exams. Gazing away from the screen, talking, using unauthorized materials, positioning oneself out of the view of the webcam, constant change of active window can be identified as behaviour associated with academic misconduct in online exams [7]. While using a false identity and using outside sources are two of the most frequent cheating strategies, candidates nevertheless come up with new strategies on a regular basis. There are methods to abuse internet access or navigation privileges despite careful monitoring [8].

There are several proctoring services with specialized proctoring software that helps instructors to conduct online examinations [9]. Examination candidates are required to have a webcam and an internet connection to access the proctoring platform to connect with the welltrained proctors. These services offer the highest reliability at prices calculated per session, which is often very costly if the requirement is to conduct a mass-scale examination.

There have been three different remote proctoring systems documented [10]. One such method is live proctoring, which is very similar to traditional in-person proctored exams but necessitates a proctor with higher technological proficiency. Live proctoring is an exam that is taking place with a real-time human proctor monitoring the process remotely (for example, by watching the students live online via Zoom). The second type is a video recording of the test taker that is later watched and judged by a human proctor. The third kind of proctored examinations involves automation, in which the technology records the exam and recognizes and alerts instances of potential student misconduct by introducing very complex systems that combine low-level features into high-level features using artificial intelligence, image processing algorithms, biometric technologies, and cloud technologies [11], [12]. Many of the studies have elaborated that an Automated Online Exam Proctoring system contains two main components, namely, Exam Candidate Authentication and Detection of Academic Misconduct [12], [13]. In most of the studies conducted to address the problems in online proctoring, a priority has been given to user authentication and successful methods have been implemented to correctly identify the test taker using technologies such as biometrics, continuous authentication etc. [14], [15]. To detect academic misconducts, previous work has utilized gaze estimations, head pose detections, keystroke metrics, mouse movement data etc. [16], [17].

Authors in [16] proposed a Visual Analytics system that consists of two major modules, namely, the suspected case detection engine and the visualization. The suspected case detection engine analyses head poses and mouse movements to determine the abnormal behaviour of an exam candidate. The focus of [16] has been improving the reliability and convenience of exam reviewers by effectively visualizing the results of the case detection engine in three levels of detail. The reviewers can use Student List View, Question List View and Behaviour view along with playback facility to catch the wrongdoers. However, the experiments of the study have been conducted on a device with an Nvidia Titan Xp GPU which is an expensive computational device. Therefore, it is a question whether the proposed system can be utilized in low-resource environments even by rebuilding it to be used on a CPU.

Authors in [17] have proposed a method to create a summary of pre-recorded videos from sequences of abnormal behaviour of exam candidates modelled using a two-state hidden Markov model, with observations defined as head poses. The video summaries generated using the skim generation technique aid with the post-examination review. The invigilators save time by reviewing the summaries, rather than videos that are hours long. Although the scalability issue of e-proctoring in large-scale exams is addressed by these proposed systems, the operational needs of their solutions have not been explicitly stated, therefore it is uncertain whether they can be applied in low-resource settings.

Muzaffar et al. [18] have reported that developing countries contribute more to online exam research than developed countries. Despite the higher contribution, developing countries have failed to implement their solutions often due to unstable infrastructure, so the proposals have been only limited to research publications. Proctoring solutions that are operational in real environments [12, 13] are from developed countries like the USA and Spain with stable infrastructure and high internet speed. Muzaffar et al. [18] further reported that economic situations and existing e-learning infrastructure are highly important for the adoption of online exams in a country and recommend considering network infrastructure, hardware requirements, implementation complexity and training requirements when designing feasible online exam solutions.

Institutions have tried to implement affordable ad-hoc online proctoring methods to deter academic misconducts in online exams. The study conducted in [19] has used Youtube to livestream the footage taken by two web cameras to proctor a student during an examination. They discourage using generic video streaming services to proctor online exams since it heavily relies on a third party and due to privacy concerns [19]. Many universities have adopted remote exam proctoring through video-conferencing software [20], [21]. This method is convenient and affordable because it does not require additional hardware and software. However, in cases where thousands of students take an online test simultaneously, monitoring the student's physical learning environment as well as everything the student performs during the examination to detect fraudulent activities manually using videoconferencing software can be a tedious task [22].

Table 1

Summary of related work

Requirements of the proctoring method of online exams

In onsite exams, students usually send signals or exchange chits as they look at each other. As a result, examiners in the face-to-face classroom had to walk around, carefully observe the students, and notify them if they engaged in any suspicious behavior. Remote online exams must meet certain requirements to be comparable to onsite exams in terms of ensuring academic integrity. There are two main types of evidence that should be obtained at online exams, to ensure the academic integrity is not violated.

Is this the right candidate for the exam?

A big challenge in remote online exams is making sure that there is no room for contract cheating or impersonation. Institutes must rightly identify the exam candidate, must check if the candidate is a student of the institution and make sure that the correct candidate answers the exam throughout the exam. Therefore, there should be evidence of continuous authentication from the start to the end of the exam.

What does the exam candidate do during the examination?

Unlike in onsite exams, there are many more opportunities for exam candidates in remote online exams to engage in academic misconducts. The activities of an exam candidate during an online exam are of two sorts. The first is the activities he or she performs in their physical learning place during the examination period such as using a mobile phone to search for answers or seeking help from someone in the room. The second is the activities that happen on device, such as accessing any other computer apps, including the internet browser, functions like copying, pasting, or printing, use of virtual machines which could lead to academic misconducts. Video evidence recorded using a webcam and evidence of system activity during the examination should be obtained from the start to the end of the exam.

There are challenges to overcome in designing a proctoring solution suitable for usage in Sri Lankan universities. The class size of a university is often between 100 - 200 and the exams are conducted at a scheduled time for the whole class. The universities do not possess enough staff members to offer live proctoring when the exams go online. Public funded universities are not financially strong enough to get services of third-party commercial proctoring solutions. The proctoring solution should only use the common, widely available devices without imposing a financial burden for the student. Speed, stability and affordability of the internet connections and sudden power outages are the other problems students face in adopting to the online exams.

Proctored online assessment

We used Zoom, a video conferencing software, to proctor a compulsory assignment given to second year computer science undergraduates following the software engineering module at the University of Colombo School of Computing, Sri Lanka. The exam consisted of a case study, where participants were allowed to use only the module textbook during the exam. Participants had to write their answers on paper within one hour, scan and upload it to Moodle course page. There were no live proctors present, instead the recorded video of the Zoom call had to be submitted with the answer scripts. This practice is in keeping with the recommendations made by authors of [23], who advised using a uniform online exam policy where a camera recording each student's computer screen and room is a requirement to deter cheating in online exams. The primary camera, secondary camera and the screen recording are the three main components, and their setup is shown in Figure 1. The hardware and software setup of the proctoring method, the procedure including the instructions given and the observations are discussed below.

Figure 1 Proctoring Apparatus Setup

2.2Л. Setup

The requirements exam candidates had to meet were a webcam (either built-in or a peripheral device), a smartphone, access to Zoom video conferencing software and an internet connection with a standard quality. The setup of the proctoring method and the purpose of each component are as follows:

^ Primary Camera

In this setup, a built-in webcam on exam candidate's laptop or a USB webcam is used as the primary camera to capture the face of the candidate. It provides the evidence on candidate authentication predominantly. This camera does not provide enough information on what the student does in his/her physical exam space.

^ Secondary Camera

Since a built-in camera has a narrow viewpoint, a smartphone camera was used to obtain evidence on what student does during the exam, his/her workstation as well as the surroundings. We instructed the participants to set up the smartphone at a distance where the workstation is clearly visible and provides a wider view on the surroundings.

ЃY Screen Recording

We needed to obtain evidence of the activities that happen on the participant's screen to fulfill all our requirements. Specifically, we wanted to make sure that the participant only uses the module textbook during the exam. The screen sharing feature of Zoom is used to record the activities on the screen of a candidate's device.

Upon setting all the hardware, we instructed the participants to create a Zoom meeting and to keep both the devices joined to the meeting and to share the screen. The video recording should ideally have two camera streams and a screen recording, to be counted as successful proctoring evidence. No live proctor was present, the submitted videos were reviewed later.

Procedure

The procedure of the exam proctoring is of three stages. The instructions the candidates must follow are detailed below.

ЃY Before the exam

-Setup the laptop/desktop computer with a webcam to provide a clear view of the face, in a place with good lighting conditions.

- Join both the computer and the smartphone to the Zoom call with the video on and microphone unmuted only on either of the two devices.

- Provide a 360-degree view of their workstation and surrounding using the smartphone.

- Place the smartphone at an appropriate distance to provide a clear view of the surroundings.

- Verify the student identity by showing the University ID card in front of the primary camera.

-Show the papers that you use to write on, using the primary camera to confirm that they are empty papers.

-Start screen sharing on Zoom. Use the `Screen' option to record all the windows open, not only a specific window.

- Access the case study through Moodle.

^ During the exam

- Write the answers on the piece of paper which was showed earlier.

- Do not stop the videos from either of the devices or the screen recording until the end of the exam.

/ After the exam

-Show the answer scripts to the webcam, scan and upload them to the Moodle course page.

- End the Zoom meeting, save the recording to your device and upload it as well.

Moreover, we instructed the participants to provide information about the device they used for the exam, and the details on network data consumption during the exam. A data usage monitoring software called Glasswire was used to measure the data usage of the session. We instructed the participants to select the time window of the exam on Glasswire and report the data usage which occurred in the said period. A screenshot of the system information was requested to obtain the device details such as the processor type and main memory capacity. The purpose of collecting that information was to get an understanding on common device types available with the students, which may affect the development of a holistic examination system upon gathering requirements and constraints.

2.3. Post-exam feedback survey

A survey consisting of 16 questions was used to collect data about the participant's experience about the proctored exam. We sent the survey to all the participants regardless of their consent to use the recorded videos for research purposes because the participants who did not provide consent to use the videos due to privacy concerns preferred to answer an anonymous survey regarding their experience. The survey consisted of 13 default questions and 3 conditional questions based on the answers for the previous question. The survey was anonymous, voluntary, and contained both close and open-ended questions. The questions were supposed to reveal the experiences about the devices used, internet connectivity, the factors that led to stress and perceptions about cheating during the exam. From the total of 197 participants, 147 (74%) had responded to the survey. The data required minimal cleaning, since all the participants had genuinely answered the questionnaire. Hence, only once record had to be removed due to duplication. Because of the exploratory nature of this study, the quantitative results are presented using summary statistics. The quantitative data was downloaded into a spreadsheet for analysis and the creation of data visualizations. The qualitative data from the surveys was downloaded into a spreadsheet and manually coded depending on the questions asked using deductive reasoning. Because the data was aligned with survey questions, coding was relatively simple.

The results

Proctored Online Assessment

In our online exam, we were able to collect 126 videos from participants who provided consent for research purposes, from a total of 197 participants. The time length of a video varies from 45 minutes to 90 minutes, as participants were allowed to conduct early submission if they had finished all the questions. The videos consist of the participants' behaviors, the events happening in their surrounding and the materials they refer to during the exam.

3.LL Network Data

A total of 126 records on internet data usage including upload and download bandwidth were collected. Some of the records had to be discarded due to inaccurate reporting. The statistics of time spent, upload bandwidth and download bandwidth are given in Table 1 examining 67 records. Most of the entries were inaccurate or did not provide the required information. Since two devices are connected to the zoom call, data consumption happens in both devices.

Table 2

Summary statistics of network data usage