Investigating learners’ perceptions of completion and certification in MOOCs

Understanding learners’ perceptions of their own learning outcomes is critical for accurately interpreting those outcomes and planning interventions to help improve them. Past research in massive open online courses (MOOCs) shows that many learners enroll in courses they do not finish, and much research has been conducted investigating the patterns and trends driving this. This paper uses a qualitative methodology to understand how learners perceive course completion and certification, and why they do or do not meet their learning goals. Data were analyzed from fifteen interviews with learners who had enrolled in at least one MOOC. The data suggests that learners have a complex understanding of completion that varies depending on their own goals and access to the material. It also shows that they see certification as distinct from completion, and will only be willing to pursue certification under certain circumstances.


Introduction
Understanding student's perceptions of their own educational experiences is crucial towards interpreting their behavior, and designing materials that best meet their needs. Massive Open Online Courses (MOOCs) pose a particular challenge towards understanding learner's experiences, as they are asynchronous, online courses designed to accommodate a large scale of learners with minimal instructor feedback. Online platforms have the ability to capture user's actions in the course, allowing researchers to study behavioral patterns and trends in and across courses. Although this data is very rich, it does not give insights into the learners perceptions of their own behavior and reasons motivation this behavior. Research has shown that MOOCs have a high dropout rate, with many learners enrolling in courses that they don't complete (Khalil et al, 2014). Others have sought to characterize completion and dropouts in MOOCs (Anderson, 2013;Kizilcec et al, 2013;Zhong et al, 2017), and predict what types of learner engagement will lead to completion (Al-Shabandar et al, 2018;Kloft et al, 2014). However, defining completion in MOOCs is complicated by the various options facing learners enrolling in courses. Typically, learners have the option to audit most of the course for free for a limited time period, or to pay a small fee for the opportunity to access the full course and the opportunity to earn a certificate of completion. Much of the recent research investigating completion rates in MOOCs focuses on learners who have paid for the course, assuming that the act of paying signifies their motivation to complete the course (Reich & Ruiperez-Valiente, 2019;Roy et al, 2022). However, many studies have shown that learners have complex motivations for enrolling in courses that may not always necessitate completing the entire courses as designed (Kzilcec & Shnieder, 2015;Milligan & Littlejohn, 2017;Salmon et al, 2017). This project seeks to shed light on learner's definitions of completion and certification, and the underlying reasons surrounding their course outcomes.

Related Work
Previous research has investigated learners conceptions of course completion and the reasons behind when they do not complete. Early work from Fini (2009) in a MOOC offered in 2008 showed that learners had varying definitions of course completion, and there was not a single standard that was uniformly shared by all learners. Researchers have since sought to unpack the reasons behind these variations. Tanner (2013) found that learners often dropped out of courses because they did not meet their expectations. Others pointed to a range of factors such as conflicts arising in learners home and work lives, their perception of the course design or pedagogy, the perceived usefulness of the material, or the learner's ability to manage time effectively (Eriksson et al, 2017;Hood et al, 2015;Wang & Baker, 2018). Several studies have found that the possession of certain personal characteristics such as goal orientation, grit, and time management had an impact on completion rates (Doo et al, 2021;Fellman et al, 2020;Gupta, 2021;Kroll & Reed, 2017). This work has given important insight into the broad trends of learner behavior in MOOCs, though questions still remain about how the learners describe their own course behavior and outcomes, particularly regarding when they choose to pay or audit the course for free, earn a certificate or not, and how they define completion. Drawing on the previous literature, this research sought to understand how learners interpret their own completion of a course, when and why they choose to purchase certificates, and what are the factors that they note for not completing a course or earning a certificate.

Methodology
This project utilized qualitative research methods to address the research questions, as the key interest is in the learner's own interpretation of their learning experience. The study population of interest were adult learners (18+ years old) who have enrolled in at least one MOOC. The study used a structured sample (Shah & Gorbatai, 2015) which identified learners based on pertinent variables to the study: gender, course completion, and verified status. A sample of 60 learners was drawn from 5 courses on edX covering the topics: history, political science, research methodology, mathematics and music. 15 learners consented to participate in the study, which are described in Table 1. Semi-structured interviews were conducted for data collection, and grounded theory was used for data analysis (Strauss & Corbin, 1997). The interviews were approximately ~30 minutes to 1 hour long, and participants were invited to speak about all of the MOOCs they had taken, not only the one they had been sampled on. The interviews were recorded, and transcripts were created. These transcripts were analyzed in nVivo, and coded twice. The first codes were derived from the text itself, and a codebook was created based on these condensed codes, then the data was analyzed again using the codebook and synethesized into the findings presented in this study.

Findings
Learners described a range of outcomes and reasons for those outcomes, based on their motivations for taking the course, the quality of the course, and their own circumstances. While definitions differed slightly, overall participants defined "completion" as having completed the parts of the course relevant to their interests and motivations. As one learner noted, "completion is to be exposed to the most relevant points of importance to me." The outcomes presented use this interpretation, assuming learners were able to complete the course according to their own determination. The learners' ability to complete the course is additionally hampered by their willingness to pay for gated-content, with audit students having only partial access to the material and verified students having full access. Paying for verified status also gives learners the opportunity to earn a certificate of completion if they obtain a high enough grade on the course. Participants described the reasons why they might or might not pay for verified status, and the circumstances under which they would try to earn the certificate. Table 2 presents these various outcomes for learners, and learner's reasons for these outcomes. Table 2. Outcomes for learners, and learner's reasons for these outcomes.

Completed Did not Complete
Audit -"Well-designed, engaging course" -"Complete as much as I can access for free" "Well-designed, engaging course" -"Need the certificate for credibility" -"Sense of accomplishment" -Unexpected conflict -"Did not need too All varieties of learners reported outcomes of completion, regardless of verification or certification status. They all also reported the quality of course design, and a positive learning experience in the course as a reason for their completion. Participants noted that "[a faculty member] who is really engaged, and seems to know their subject matter and loves teaching" as a factor in their decision to complete. Another felt that their level of learning was an important factor, noting "there was a lot of new information to learn, and I was enjoying the course actually, so that's a huge part of it." Others noted the course design itself, including the length of the videos and whether there was an opportunity to apply what they learned.
Many participants described being uninterested in paying for certificates, generally because they did not have a professional use for it. As one learner explained, "For work purposes, certificates are still important and in Brazil specially still important. But for personal things, I don't think this is an important thing, I mean it's cool to show friends and family but, in the end, it doesn't affect much." Others noted that the money was a factor, saying "I audit the courses and I don't pay, and I am actually a bit furious that they push [the price], and more and more to push for the certificate." Still, learners reported completing as much of the course as they had access too, saying "if I don't pay for it, I do all of [the course] that I can." Some learners reported paying for verified status and completing the course, but not earning a certificate. As one learner described, "I think for me, the problem is not the certificate, I want to have the full access to all the content, that is why I pay the fee." These learners may choose to complete the course on their own schedule, once they have access to all the materials. For learners who paid for the verified status and did earn the certificate, the reason given was either out of a personal sense of accomplishment, or to use the certificate as a marker of credibility for professional or personal reasons. One participant said, "I think if one has a certificate, it also gives a sense of achievement, although maybe it won't matter much at my level, I still felt it like happy, okay I've done this." Another reported "I took those certificates because I wanted to be proud of my [skills] and I put it down on my Facebook so that all my friends could see." Some noted that they earned a certificate because it may help them professionally in the future, "I don't really have a use for the certificate, but in long term I can see that my interest is in [this field] and I definitely feel like in the future it will probably come in handy." Many reported needing the certificate for professional reasons, noting the "credibility", or the importance of certificates in their work culture, "in the Indian system they're always looking for certificates whenever we are promoted, so you have to get a certificate for everything." Learners also reported many reasons for not completing a course. No matter whether the learner paid for verified status or not, all learners reported unexpected conflicts as a barrier to completing the course. As one participant said "I want to complete them most of the time, but I'll be honest, I might have enrolled in three or four other courses which I have not been able to finish because of others things, you know I have my work assignments, I have family to look after, and I have all those requirements and then also do courses." Other participants noted that they missed important deadlines due to a conflict that caused them not to be able to finish the course. One said, "I actually a failed a course because I didn't make it to the final exam, because the three days it was available I was sick." Many participants described exploring a course before deciding to pay, to ensure that the course was worth taking. They described many reasons for dropping out of courses due to it being the "wrong course" for them. This could be because the course is different than they expected, "you realize that it's not what is on the label, you know you get in it's just something else." It could also be that the course is inappropriately challenging, "they were way too easy, some of them are just so easy and silly", or it is above their level, "sometimes the content is too much". Others felt that the faculty teaching the course was "boring", causing them to lose interest in completing the course.
Some participants who paid for verified status were not able to complete the course or earn a certificate due to the course being "too hard". As one participant said, "You have to apply yourself a bit more but yeah, the ones that are hard you try to read up if you can, and if you don't get it you don't' get it. Some of them are really hard. I see the comments from the other students that they are having the same challenges." Others took personal responsibility for not finishing, saying "I was kind of slacking, yeah I was not watching not so often". Another blamed their own lack of grit, saying "actually some of that has to do with self-discipline, I'll be honest not finishing courses is probably my thing". Participants also reported earning a certificate, but not actually completing the entire course. As one participant explained, "I wasn't able to submit the essay on time, because I was working on the policy paper [for work] and they had the same deadlines, but I had already received a passing grade. You know, maybe I will still write that essay though, and try to publish it somewhere else."

Discussion
These findings show that learners have a dynamic understanding of completion in MOOCs, depending on their interests in the material, and their willingness to pay. Learners reported feeling a sense of completion when they met their own goals for themselves, regardless of whether they had earned a certificate or technically completed all sections of the course. Indeed, learners are aware of the limitations in their ability to technically complete the course without paying to earn a certificate. As one learner said, "Usually I do everything. If I pay for it, I will do all the exams of course. And if I don't pay for it, I do all the exams that I can." Or as another learner explained, "If you are not interested in the certificate, you can just skip. I prefer it if I don't see the tests, I just don't need that I would like to keep going with the classes." Learner's benchmark for completion is dependent on fulfilling their own learning goals, their intention to earn a certificate, and their willingness to pay for full access to the content. Indeed, some learners paid for verified status for full access to the content, and did not attempt to earn the certificate but nonetheless felt that they had completed it to their satisfaction. This shows that learners view completion as a distinct outcome from earning a certificate. This is further illustrated by learners who reported feeling that they had not completed the course, despite earning the certificate. Interestingly, learners noted that the act of earning the certificate disincentivized them from completing the course "because [they] did not need too".
This sheds light on previous research on retention in MOOCs which uses certification rates as a proxy for completion (Reich and Ruiperez-Valiente, 2019), suggesting that the certification rate in a course might be poorly aligned with the learners' sense of completion. Despite this, the findings show alignment with much of the literature investigating retention in MOOCs. For instance, learners reported a lack of self-discipline and poor time management as a cause for not completing a MOOC (Doo et al, 2021;Kroll & Reed, 2017;Fellman et al, 2020;Gupta, 2021;Hood et al, 2015;Cohen & Magen-Nager, 2016;Wang & Baker, 2018). A common reason cited for dropping out of course was that they did not find it engaging, either due to the course design, the instructors, or the level of the course (Hone & El Said, 2016;Goopio & Cheung, 2021;Chen et al, 2020). They also described having to drop out of a course because it was too hard, and they didn't feel they had access to the support that they needed (Greene et al, 2015;Aldowah et al, 2020). Finally, many learners did not complete courses due to external conflicts arising from their professional and personal lives (Eriksson et al, 2017).
While learners' sense of completion varied based on the circumstances, their understanding of certificates was fairly uniform. Most learners reported choosing to earn a certificate because they had a use for it. As one learner explained that they did not pursue certificates in MOOCs because, "I don't need the certificates for anything". Others noted that they chose to earn a certificate because they planned to put it on their social media profiles to boost their credibility and demonstrate their skills. As others have found, pursuing certificates could be used for short term professional goals, or to support long-term career trajectories (Littlejohnet al, 2016;Dillahunt et al, 2016). For some, however, a sense of accomplishment was enough of a reason to earn a certificate. One learner said, "You spend enough time reading, putting in all the hours, so why don't you get a certificate? I have to get the certificate." Another reported collecting almost 50 certificates in a binder as keepsakes to demonstrate her learning journey. Learners decisions about when and why to earn certificates depends on whether they had an internal or external use or need for the certificate.

Conclusion
This study sought to characterize the ways in which learners perceive completion and certification in MOOCs by conducting interviews with MOOC learners. The data suggests that learner's definition of completion is complex and relative to their own personal goals and perception of the material. It also shows that learner's perceptions of completion are distinct from the process of earning a certificate, which may or may not be in alignment depending on the circumstances. Finally, the data shows that learners typically only choose to earn a certificate if they feel they have a use for it, whether personal or professional. Futher research could shed light on how pervasive these variations are across the population of learners, as well as point to interventions that could be useful for some learners to help them accomplish their goals.