How Insurance Works

The Joy of Quality Assurance Training

surprised to see how many people had positive reactions to their training. So what I ended up doing was having these major categories: Positive, which I’ll show here — which I just mentioned 96% had a positive outcome for their Quality Assurance training. One person was not sure, and I’ll talk about that. And there were a few criticisms. And jumping ahead, the criticisms were not about the Quality Assurance training. When I broke down the categories of what people were most happy with, I ended up with six and I have listed them here. So course organization, alignment, accessibility, student benefits, the use of other rubrics, and then special mention of the eSpecialist. And what I want to do in the following slides is go through each one of these categories and talk a little bit about it. I think we have 20-25 minutes, I don’t want to spend too much time here. I mentioned this before, 96% stated their QA training was most helpful. Most faculty were taking QM training. Very few people were taking QOLT. There there was also another survey that was given CSU-wide last year, asking if their QA training effected their teaching effectiveness? 74. 1% stated it was very useful and 24. 5% stated it was somewhat useful, so you add toes together and 96% found it was useful or somewhat useful. So the results came out fairly close to the whole California State University System which has 23 campuses. Okay, in terms of what faculty found most useful was, in their Quality Assurance training, was course organization. And what what they ended up saying is — I didn’t write everything down they ended up saying, but in terms of what they like most, they gave a list and the things that kept coming up were creating videos, in terms of introductory videos. So faculty introductions to students. Creating a start here area. Faculty really like that. Help with making their syllabi better in terms of course organization because your syllabi is reflected in your course. And, of course, having better navigation throughout the course. And other research has certainly shown similar examples. You know, we know that course design is important to students success and it certainly takes a lot of time to do this, but it seems like the Quality Assurance training they are receiving is really paying off in terms of people really liking this. Alignment was the second most common category mentioned in terms of what they were happy with in terms of their training. This is interesting because just as we heard this morning, what Melanie was talking about, alignment is not an easy thing to do or understand, but it’s one of those things that I think makes a massive amount of difference when you finally train faculty in terms of what is alignment. So you are creating good course learning outcomes and module learning outcomes and everything in the course is connected. And, you know, I have mentioned this before, other research has shown this to be the case, where faculty are surveyed about, you know, what’s the most important thing about their own Quality Assurance training? And in some studies, alignment has been the main benefit, especially using Quality Matters, the Quality Matters rubric. It’s also interesting, too, if you read through the research, there was a study done in 2018 and it also said learning alignment was very important, but it also said it was one of the most controversial parts in using the QM Rubric. It’s controversial in that training faculty to understand alignment really revealed that a lot of faculty, or some faculty, didn’t realize it was connected to student learning. So I thought that was really interesting. And, you know, as we — if you’re a Peer Reviewer or Master Reviewer, you know that you have to spend quite a bit of time now making sure the learning outcomes are all connected to the module learning outcomes, to the assessments and so on. So I was pretty happy to see that. In terms of the third category, it was accessibility. This I was really happy to see. Again, if you have gone through, you know, being a Peer Reviewer or Master Reviewer, you know that accessibility seems to be one of those difficult categories for faculty to work on. And I think it’s the one where they probably get the most frustrated with as well. You know, we know as we are putting our courses together that, you know, these days they can’t be totally text-based or video-based without some mechanism to include all students in this. And I think Quality Matters does a really good job in terms of explaining accessibility. And the most recent addition of the rubric really makes great strides, I think, in terms of making every aspect of the course accessible. And, of course, we know that, you know, it’s much easier to know about accessibility first and how to incorporate that into a course before you get started because if you have a course and you have to make it accessible afterwards, it’s much more difficult and much more time consuming. The fourth category that ended up coming up was student benefits. So 15% of the faculty had mentioned that. And, again, here what I have done is listed a few of their comments. So in Terms of Using their Quality Assurance training, their courses become more engaging. So the students are engaged more within the course material, with the faculty, with their peers. Students, too, a few faculty had mentioned specifically that students noticed their courses are more organized. And this will come up later when I talk about closing the loop, aspects of what can be done. But they are noticing more organization. It would be interesting to do a study comparing non-QM courses to QM courses to see if students actually are noticing this organization or not. Now the next section that came up which was interesting, 8% of faculty mentioned the use of other rubrics. Now what they ended up not doing, which was too bad, they didn’t provide any details why they thought the use of other rubrics. But, you know, I guess to me it sort of made sense that they are using the rubric to put their course together, they notice how good the rubric is, therefore, they realize they can use other rubrics in other parts of their courses. So that seems to be a common thing that faculty seem to be doing here. The — another category that came up was — and I just titled it here special mention of eSpecialist, 8% of the faculty had specifically mentioned their eSpecialist or the Instructional Designer that is are helping them go through the QM process. I do want to mention here specifically that this overwhelming response, positive response to the Quality Assurance training they received is directly related to the especialist, so this whole project is sort of sitting on those people’s shoulders who work with these faculty. And, you know, the result that is came out, you know, is 96% of faculty that answered the survey were extremely happy with the training they received from these eSpecialist. And I have a quote from the CHLOE report, institutions that require Instructional Design input or rely on course design teams see greater student engagement with faculty and other students. And I think we are seeing this in other research as well. Faculty also mentioned it was nice to have the course reviewed before going through the entire QM peer review process. So they really liked that. I do want to mention here, too, since we are talking about the eSpecialist, at East Bay there are numerous funding opportunities for faculty to go through the QM review process. And they all get assigned to an eSpecialist, so this is where this result is coming from. Okay. So 96% found their training very helpful. 4%, really one person, wasn’t sure. And I thought this was interesting. I wasn’t quite sure if I wanted to put this into the data, but since it was part of it, it needed to be in there. So they answered they weren’t sure if it was helpful because they had not taught online yet. And I did put a little quote here, they haven’t taught online yet, but they quote on line/hybrid courses are terrible teaching experiences. They haven’t taught online yet, but this is what they feel. I’m really hoping — I didn’t collect any personal data from this particular study, but I’m hoping this person who wrote that will have a very different view once they actually start designing their course and teaching online. I found, too people that don’t want to teach online have this view that teaching online is sort of this terrible teaching experience, which, of course, isn’t really the case. Okay, criticisms, so what I found is people had some complaints within my survey, but they were unrelated to the QA training they received. I thought that was a good thing. What some faculty noted is that student evaluations were the same or slightly lower in the courses. And I think what they were expecting was that they go through the whole QM process, making their courses better, and they were hoping that students — the student evaluations would reflect that. And that, a couple of people noted, didn’t happen for them. So part of the reason for that is other faculty had mentioned, the courses seemed more difficult. So when they compare the course before it went through the QM process and compared it afterwards, it seemed more difficult for students. And that corresponds really interestingly to some research that was done at our University in, let’s see, it must have been 2017-2018. I was partly involved with some of there where we looked at the grades of students who went through a class that was — where the faculty had Quality Assurance training and compared it to online classes where the faculty did not. Now what we ended up seeing there, and I think and I think right now Roger Wrenn and Erick Kong were talking about this in another room, they saw the number of A grades actually went down in the QM courses, while the number of B’s went up. And I didn’t write this down here, I probably should have, the number of D’s, the number of F’s, and the number of W’s, so unauthorized withdraw, also went down in QM courses. So that was a really good trend. So maybe these courses really are more difficult because there are many more things people have to concentrate on in terms of designing the courses and students are picking up on that in their evaluations. But in general, I think it seems to be working in terms of reducing the number of students who are actually failing. Okay, so in terms of what I think should come out of this, so I’m just calling this “closing the loop.” Recommendations based on this research. I think what needs to happen is more Quality Assurance training needs to be offered. And this should be both lectures and tenure/tenure-track faculty. At East Bay we have training for all of the tenure/tenure-track people, but we need to work more on getting the lectures in, because as most of us know, lectures are teaching most of our classes, so they need to be offered to all faculty. The second point is start offering course development training to in-class faculty. And then introduce them to these QM or the QLT rubric or whatever rubric University are using because this has been shown in other research that faculty are happening with the QM training they are receiving. So I think it would be useful to introduce this to all faculty, maybe during faculty review week or at the very beginning of the semester. I don’t know if this is going on, but I think there should be a rubric, like QM or QLT for in-class courses. And this, you know, and some of these are sort of tied together, where I think faculty are really enjoying their Quality Assurance training and it’s working in their classes. It seems like a natural progression to move into in-class courses with something like this. Number four, I think there should be more workshops on creating alignment maps and making courses accessible, both for online and in-class faculty. I know we’ve got a few Instructional Designers at our University and they are working like mad, you know, because of what we are all going through, but I think with alignment being an extremely important part of course design, it seems like it would be useful for everyone, both online and in-class faculty to have some type of training on that. Number five, now this one sort of comes out of my own experience. So I think that we should be making course design Quality Assurance training mandatory for MA/MS/Ph.D. students who plan on teaching. Now that’s probably most students. [Laughter] What I have been doing in our MA program, especially we have four options for MA students, one is teaching, anybody who comes through me as an advisor in the teaching option must go through sort of this sort of reduced QM Rubric process. And they have to design part of their teaching course online or all online, I’m happy if they want to do that. But I think this is really important to start people off really early in their careers, their teaching careers, on some type of Quality Assurance training. Six, so a couple of comments that came out of the research is that they felt like students should be introduced to the QM Rubric. So make the QM Rubric available to students so they can see what faculty are doing in terms of creating the courses. And I tend to mention in my courses, like, I have used this rubric, I’ve had this training, and here is how the course is designed based on that. I think students probably have a more positive experience in the class if they know you actually spent some time designing your class, and spent quite a bit of time doing that. Now all this comes down to number seven, we need to hire more eLearning specialist. This has been pretty clear, especially now where, like I said, our University we’ve got a few and they are now trying to train all faculty on doing something online to keep the classes going. But I think looking at this research, I know our eLearning specialist are working like crazy and I think if we just had more of them to spread the work out, I think that would be a really good thing. Number eight is sort of a wish of something I’d like to do at some point, I mentioned this in the beginning, doing some kind of evaluations in online courses with QA-TRANDZ faculty versus non-QA-trained faculty to see if students are noticing the differences. Now I notice there are differences with student evaluations and what goes into all of these, but I would really like to know what students are feeling as they go through these classes. And let me check the time. Hold on one second. I see I have gone over a little bit.>>NANCY RAGIAS: We have a few minutes for questions, Kevin. We’re good.>>KEVIN KAATZ: Good good.>>NANCY RAGIAS: We’re scheduled until 12:20. Yeah, 12:20, so you have a couple minutes. Go ahead. Do you want me to read the questions to you on Q&A?>>KEVIN KAATZ: Sure, sure.>>NANCY RAGIAS: This one was at the very beginning. She said, are they trained to do so? And I’m not sure what that was in reference to. It was at the very beginning of the session. Zacharia, if you can tell us what you meant by that. It would be great if they could share the actual survey questions.>>KEVIN KAATZ: I did that at the ending. I only sent out one question. It’s probably the second or third slide. Yeah, it’s up there.>>NANCY RAGIAS: Okay. Are student evaluation forms different for online course are versus face-to-face courses?>>KEVIN KAATZ: No, they are exactly the same at our University.>>NANCY RAGIAS: What do you think about applying QM Rubrics to face-to-face courses, and training faculty that way?>>KEVIN KAATZ: In my “closing the loop” I feel it would be extremely helpful. I sent this out, when we went to all online three weeks ago when faculty had no clue of what online teaching was, I sent them a version of the QM Rubric and told them, you are being thrown into this, be careful, but here is a useful tool to design your courses better. I hope what they end up doing when we go back to face-to-face is they use some of those things they are learning from the rubric.>>NANCY RAGIAS: Okay. That looks like it in the questions. Thank you, everyone, for joining us. I have placed the link to the evaluation in the chat for you, so please, we encourage you to share your feedback about the session there so that we can continue to offer some wonderful sessions for you. Have a good lunch! And we’ll see you all this afternoon.>>KEVIN KAATZ: All right, thanks, everybody! All right, thanks, Nancy.>>NANCY RAGIAS: Thank you, Kevin. Great.>>KEVIN KAATZ: Talk to you soon.>>NANCY RAGIAS: Okay. Bye-bye.>>KEVIN KAATZ: Bye. [Session concluded] CART Disclaimer: This rough edit transcript, which may contain missing, misspelled or paraphrased words, is only provided for your immediate review and is not certified as verbatim and is not to be cited in any

Leave a Reply

Your email address will not be published. Required fields are marked *