5.3: Formative and Summative Evaluation

In the first week of August 2010 we wrapped up our 4th annual BrainBlast conference. It was quite a success. After 4 years, things tend to run a lot smoother at BrainBlast than they used to. According to most of the participants, the training was great, the keynote speakers were outstanding, and the food from Sandy’s was excellent. We did a few things differently this year. One big change is that we’re making a considerable push toward Moodle in our district, having already made it available to our teachers as an online classroom management system for the past 3 years. For BrainBlast 2010, we made sure almost every secondary teacher was enrolled in a Moodle session by providing enough sessions teaching this great tool. We’re also upgrading to Windows 7 in our district this year, and provided Windows 7 training to teachers in the first phase of the transition.

General Statistics

218 people participated in the survey. This number comprises 63% of the attendees, specifically 210 (70%) of the teachers, and 8 (18%) of the administrators. Table 1 indicates the highest-ranking courses, with the average scores for each on a scale of 1 to 4.

Table 1: Course Rankings and Approval

Mark Kastleman was a huge hit both in his first-day keynote session and the two follow-up sessions he held with the administrators. Barely outranked only by Nick Harris’ PowerPoint session, he delivered quite a dynamic keynote presentation about the relationship between the brain and learning. We were fortunate and appreciative that he and Mike King were able to share their insights at BrainBlast, and that Mark was able to provide follow-up sessions for the administrators.

This year we made sure to collect data on what people didn’t like about the conference. That was a major focus of the survey results, and we intentionally gathered negative feedback, which constituted most of the comments returned. We want to know what we’re not doing right. Constructive criticism will be one of the most helpful things to guide future BrainBlasts. The following information is not intended to be a “slam” against BrainBlast or anyone involved in organizing this excellent training event. Instead, I hope the comments prove helpful in optimizing and enhancing the learning experience for all BrainBlast participants. Some of the most common themes that popped up in the survey results are highlighted below.

Irrelevancy of the Classes

“I would have liked to actually been in the classes that I registered for. There were only two classes on my schedule that I wanted to be in. The others I did not register for, which was kind of annoying.”

“The hardest thing is when you are signed up for classes that you are not interested in at all. I have no desire to do Moodle and I was enrolled in the class for a second year in a row. Last year was fine, but this year I had no desire to listen. Is there a way we could let you know what classes we don’t want to take along with the ones we do?”

“Design [the] number of classes to accommodate number of students asking for them so you get the choices you want.”

“I felt that I was already pretty versed in [MyStudent] and didn’t learn anything new. I was not happy with having to take it over courses like Blogs!”

Many participants commented that they already knew the material in the classes they were selected for, or that they didn’t get any of the classes they wanted and that the material wasn’t relevant to them at all. This was an especially common concern in the Lulu class, as well as a few others. “It was not made clear how we would use the websites offered in [the Lulu] course,” one participant noted. “The online books seem too expensive to use in the classroom.” Another stated that Lulu “did not really apply to my curriculum.” “I didn’t really see the practical application of how I could use [Lulu] in class,” agreed yet another teacher. The common concern was that Lulu books tend to be too expensive for most parents.

A couple factors are directly related to the participants having been placed into classes not relevant to them:

1. We did not collect much information at registration about teachers’ existing skill sets, let alone use this information to place teachers in classes that would be suitable for them.
2. We did not use the wishlisted courses to determine how many sessions to offer. Last year, the average participant got 68% of the courses they wishlisted. This year, the participants only got 16%. The reason is this: When participants initially registered, they specified up to 6 courses they would like to take. They did this last year, too, except this year we did not delineate these courses by their track (elementary, secondary, or administrator). As a result, a lot of participants were wishlisting courses in tracks that would not be available to them. Elementary teachers were wishlisting courses in the secondary track, secondary teachers were wishlisting courses in the elementary track, and all teachers were wishlisting courses in the administrator track.

The bottom line is, we may as well have not even bothered with a wishlist, since it hardly determined which classes were assigned to whom.

Table 2 indicates the preferred courses per track, as well as whether or not the course was even available in the track. The second column in each table shows the number of times the course was wishlisted during the initial registration, and the third column shows how many slots we actually provided for the course for the entire conference.

Table 2: Preferred Courses vs. Available Seats (click to enlarge)

Bling Your Blog! was one of the most requested classes (246 participants requested it), yet we only made room for 2 sessions (40 participants, or 16% of the requesters), and only in the elementary track. “I would have liked to attend a course on blogs,” remarked one teacher. “That was the one I wanted the most, but didn’t get to go to it.” “Please have a course on creating blogs for beginners,” begged another, “and have enough sessions [so] all who want it can have it.” Conversely, Lulu was one of the most unrequested classes. Only 76 requested it, yet we provided room for 5 sessions (100 participants). And we provided 6 sessions (120 participants) of Keeping in Touch with Google for elementary teachers, yet only 21 in the elementary track actually requested it! It’s hardly a surprise that these two ended up as some of the lowest ranking courses in the survey results, with so many disinterested people forced into them.

Recommendations for Improvement

Even if it means we need to open up BrainBlast registration earlier to allow for more planning time, we should consider the wishlisted classes and provide a suitable number of sessions to accommodate them. However, there is still value to a strictly randomized scheduling process. Teachers may attend classes they would not have normally requested due to an unfamiliarity with the material, but once they participate in the class they acquire new knowledge and skills directly relevant to their subject, classroom, and instructional strategies. At least, that’s one factor that has driven the design and development of BrainBlast. Every class should be relevant to everyone. Clearly, we did not accomplish that this year.

We should not discount the role a teacher’s general disinterest plays. If they simply don’t care about the subject material when they walk into the class, this preconception will carry all the way through, and stifle any engagement and learning they may experience. We need to find a way to identify which courses may be most relevant to a teacher, even if they don’t know it themselves. At the very least, a middle ground should be found between encouraging teachers to explore new technologies, and supplying the correct number of classes they specifically choose and feel are most relevant to them. As one teacher helpfully recommended, “Make sure that the courses are relevant to the participant . . . the majority of the courses were not very applicable to me. Plus, I only received 1 out of 6 courses that I had selected. The ones I had chosen would have been extremely relevant!”

Separation of Beginner and Advanced Classes

“It is frustrating to be in a course and want to learn more than you already know and it is based around the beginner.”

“Within the first part of the class I had things figured out, but we couldn’t move into more depth because there were other people in there that struggled.”

“Please, please, please have a beginners track and an advanced track. It is SO frustrating to not be able to learn anything new because the instructor runs out of time due to the fact there are people in the class that can’t even open the program they are talking about. ‘Wait! Slow down! How did you open Windows Movie Maker?!’ Very frustrating!”

“Some of the presenters tried to teach too much and I was way behind and couldn’t keep up.”

“Make a track that separates those who have little / no experience and those who have more experience and want to go deeper.”

The suggestion to divide classes by skill sets was very common this year. “Organize the classes by beginner, intermediate, advanced as well as by subject,” a teacher recommended. “It gets very frustrating to be in a class where the teacher is trying to explain a more complex subject and some of the students can’t even remember how to login or what the Internet icon looks like.” “I really feel that classes need to start being identified as beginner classes and classes for more advanced learners,” commented another. “A couple of the classes attended were not as helpful because they are skills I have learned at previous BrainBlasts.”

This is not a bad suggestion, and there’s no denying that teachers attending BrainBlast have a wide range of abilities. Some come to the conference as technology experts, others as beginners, and this creates problems and negative feelings from different people in the classroom. “Have the instructors not wait for the one or two that don’t get it or are behind to catch up with the class,” recommended one teacher. “It really wastes time for those (the bulk of the class) that are ready to move to the next step.” The in-room techs should be able to help teachers catch up so the teacher doesn’t have to stop and wait, and perhaps we need to make that role more clear to them. However, it is rather difficult to create classes that appeal to everyone, when a few people are holding things back. It also creates a poor learning experience if some participants are constantly behind and relying on the school techs to assist them, just to keep up with the teacher.

Recommendations for Improvement

There is a need to organize classes into beginner and advanced, even if these are subdivisions within the elementary and secondary tracks. Our teachers have widely varied skill sets, and if we’re going to be randomly assigning them to classes, we shouldn’t operate under the assumption that “one class fits all.” Evaluating different technical skills is tricky, but well-crafted skill assessments should occur at registration time. At the very least, the random assignments should occur within subsets of designated beginner and advanced classes.

No More Shirts!

“Please get rid of the shirts and use that money for prizes.”

“If you must keep the shirt, make it a t-shirt. They’re cheaper.”

“Use the money spent on shirts to go towards equipment to be used in the classroom.”

“Give us more useful tech stuff. I would rather have had a tripod than a shirt.”

“No more shirts, they are atrocious and they are a waste of money.”

The general disdain for shirts was yet another common manifestation. I believe there is value to making BrainBlast shirts available to all who want it, but perhaps for next year participants could opt out of receiving a shirt when they register.

Other Ideas: Workshops

“I would like to have the option of being able to take the full two days to have a working workshop in Moodle,” suggested one teacher, “so that I can come away with tangible material that I can use in the classroom.” Day-long workshops are not uncommon to technology conferences, and some will provide this option, typically on an optional last day. Other times this is done during the regular conference schedule, so the participant enrolls in day-long workshops in lieu of a diversity of classes. Generally, the idea for BrainBlast has been to provide a good range of classes to expose teachers to multiple technologies, and to set aside workshop-oriented instruction for E-volve and iLead classes. However, many participants mentioned that the class time was too short to reasonably cover the material:

“Most of the classes went by too fast, especially on wikis. I had never even heard of one and there was just not enough time to really understand.”

“Make more ‘beginning level’ classes available and expand time on some of the sessions so we have more time to ‘create’ or to produce a finished product.”

“Some classes need to spend more time on basics. Go deep instead of broad.”

“Got Media & Stuff was the weakest because it contained so much info and not enough time to explain about each site.”

“Student Created Video would have been better if there was enough time. The time was cut short due to the general session going over a bit.”

“Could have used more time in Bling My Blog….and Google…”

Whether or not longer, focused workshops would incorporate well into BrainBlast remains to be determined. With our training efforts, it would probably be more suitable to divide workshop-based courses into other training venues, such as E-volve or iLead.

Other Ideas: Virtual BrainBlast

“It would be nice to have this more than once a year,” suggested another teacher. I’m reminded of FETC, which is one of the largest annual educational technology conferences in the U.S. Most educators are probably familiar with it, but many do not know that they also host Virtual FETC conferences a couple times throughout the year. The online version of FETC is portrayed in a virtual web-based environment with INXPO’s Virtual Events & Conferences. Participants can travel around the virtual conference setting, viewing products and communicating with vendors in real-time, collecting virtual memorabilia, taking notes, and interacting with other educators online. Sessions are broadcast at set times, typically chosen from archived recordings of the last FETC, though live streams are possible, too. During the sessions, participants can, and are even encouraged to chat amongst each other about the topic being presented. Some have claimed that the best learning during a conference happens before and after the sessions, when everyone networks together to talk about what was learned, explore how they can apply the new information, and share their own insights to a group of eager-to-listen professionals in their field. Collaboration is a hugely beneficial aspect of any conference, and virtual conferences are ideal for this form of collaboration. Many participants can gather together in the same room, more than would be possible in a physical setting, and have the opportunity to collaborate with the group in a way that might not be as easily accomplished at a large meeting of educators.

Virtual conferences are not unusual. Second Life is another popular platform for delivering virtual conferences such as Virtual Worlds: Best Practices in Education (WBPE), SLanguages, UNC: Teaching and Learning with Technology, SLACTIONS, and others. While I wouldn’t advocate opening up Second Life to all teachers inside the district (not at this point, at least), this could be a viable platform for a virtual BrainBlast. But on the other hand, a virtual BrainBlast wouldn’t need to be so sophisticated. Just gathering teachers in a basic web-based environment where they can communicate and participate in great training has tremendous benefits. And it would be a considerably more cost-effective supplement to BrainBlast than the summer event at Weber High School.

Evaluating Learning

One obvious question we may be overlooking is, “How well do we assess the learning of the participants?” BrainBlast is designed to be a highly constructionist, hands-on learning experience. Every classroom is equipped with a computer lab, and during the sessions the participants are expected to produce concrete artifacts and gain both conceptual and procedural knowledge of specific technology tools. Yet no official learning evaluation process is in place that I’m aware of. At the very least, the instructors and school techs stationed in each room should take notes during or after the training sessions of the overall learning experience, common problems that arose, and answer questions regarding the effectiveness of the session. The direct observation would prove invaluable in determining which instructional strategies do and do not work.

There are three particular aspects we should measure in any evaluation: efficiency, effectiveness, and impact. BrainBlast is undeniably efficient: all participants are given a wide range of training within a 12-hour timeframe over 2 days. The effectiveness is a little more debatable. Do all participants leave the sessions knowing the material they were supposed to learn? Are there always clear objectives in each class that should be accomplished by each participant? While we don’t want participants to take a grueling multiple choice quiz at the end of each session to determine if they actually learned anything, we can assess their learning by viewing their completed artifacts.

However, does BrainBlast actually have an impact? After a teacher takes a blogging class at BrainBlast, do they become an active blogger throughout the school year? If a teacher receives training on using Flip Video cameras, do they start using this in their class, and finding ways to engage students with video creation? 140 secondary teachers received training in Moodle. And now, a few months later, we still only have about 30-40 teachers actively using our Moodle system. This is about the same number of teachers that were using Moodle last year. Did BrainBlast actually have an impact on Moodle usage?

Granted, we shouldn’t expect teachers to use all the educational technology tools they have learned about in their daily instruction, but currently we do not formally measure if teachers are using what they learned a month, three months, or six months after the event. From past experience, and having received the most basic questions on how to blog, upload WeberTube videos, use a document camera, etc. from teachers that knowingly have participated in BrainBlast’s related training venues, the effectiveness and impact of BrainBlast needs to be determined.

Where is Social Bookmarking?

I commented on this topic 2 years ago after BrainBlast 2008. Classes on social bookmarking have been strangely absent from BrainBlast’s course offerings. Since 2007, Delicious, the most popular social bookmarking tool, has been consistently ranked as the top #1 and #2 most useful tools for learning by the Centre for Learning & Performance Technologies, which gathers data and feedback from educators all over the world. Why is social bookmarking such an important tool for teachers? There are numerous educational resources on the web, and social bookmarking is the most popular and effective way for teachers to share them amongst each other. It’s not just ordinary things like math manipulatives or quiz sheets that are shared, but links to videos, podcasts, desktop applications, helpful teacher blogs, educational trends, and bleeding-edge teaching ideas. The worldwide teacher network is opened up through social bookmarks, and our teachers can tap into this learning network by taking their bookmarks online.

Saving social bookmarks is as easy as installing the Delicious add-on in your browser and clicking Save. It’s no more complicated than clicking “Add to Favorites” in Internet Explorer. Plus, you can access your bookmarks from any computer, so you don’t have to panic when you realize that cool link to the video you wanted to show your class was bookmarked on your home computer, but not your work computer. Social bookmarking is such an invaluable asset. No longer do you have to hunt for your own resources for your classroom lessons, but you can tap into the research that thousands of other teachers have done and find existing links that others have shared. New links can be emailed to you daily, or you can subscribe to RSS feeds of bookmarks shared by different users, or bookmark groups that are created around topics. 99% of Weber School District’s Links of the Day come from Delicious and Diigo (another popular social bookmarking service which has some great extra features like highlighting and annotating pages). And about 99.9% of the links that get sent to my inbox are never put up on the Links of the Day page, so it would be beneficial if teachers have access to these other links themselves.

The bottom line is this: Our teachers are going to a technology conference, to learn about tools for learning and teaching. They are sitting in computer labs and learning about great web sites. But what do they do when they want to save these links? They whip out their notebooks and pens and write them down. These could easily be saved with a simple click and the teacher could access it later from any computer. The final step in the process is absent. All the BrainBlast links could even be merged into a shared bookmark group that anyone, anywhere can access for future reference. It’s rather strange that one of the top tools for learning has never even been mentioned at BrainBlast.

Conclusion

Overall, participants were quite happy with how the conference played out. There is clearly some room for improvement, particularly in ensuring the classes are relevant to everyone, and taking unique skillsets and aptitudes into account when the participants’ classes are assigned. We must improve our evaluation procedures as well, and pay careful attention to collecting good, reliable data if we are to determine how best to improve this event. There may be some value to including an online venue to supplement BrainBlast, and making sure that all the materials are easily accessible at a later date.

And finally, the best suggestion we received from the survey:

“Free massages would be nice.”