Monday, September 11, 2017

How do you measure engagement? What we learned during Summer Reading 2017

Starting with this year, my role in summer reading is a specific one: I work at developing and implementing our measurement tools and strategies for capturing program outcomes and other evaluative data. To that end, the reading clubs committee's meeting that matters most to my work is the one where they confirm the program goals for the year. Not the theme, not the design of the program--the program goals. Once I've got a grasp of what the goals are, and why they've been chosen, I start thinking about how we can collect data to help us gauge our success.

For Summer Reading 2017, which here at Skokie was called Camp Imagine, our reading clubs committee identified two main goals of our program:

Goal 1: Increase the completion rate by 10%. This goal is a change in outputs--the proportion of registrants who went on to finish the program this summer as compared to the proportion from last summer. You need two data points to get your completion rate: the number of people who signed up for the program, and the number of people who completed the program. This is pretty standard data to collect for reading programs across the country.

Goal 2: Increase engagement. This goal is both a behavior in itself, which is an output, and a change in behavior, or an outcome. In other words, the reading clubs committee was designing a program that would hopefully result in more people participating across the whole summer as well as engaging with both the program and the library at greater rates than they might have otherwise.

(Also, a note here: we didn't attempt to measure engagement last summer, which means we didn't have a baseline to compare this summer's numbers to. As a result, we were actually measuring our success toward the goal of facilitating engagement, as opposed to increasing engagement year-over-year. We can track that next year, using our 2017 numbers as our baseline for comparison.)

Now, you may be thinking to yourself: but how do you measure engagement? That, my friends, is a really, really good question. And there's not necessarily a single best answer. You see, measuring engagement starts by first identifying what engagement means to you with regard to your specific program or service.

So pretty much immediately after I got word that these were the goals selected by the reading clubs committee, I went to them with some questions. Specifically, what did they have in mind when they envisioned patron engagement? That's when I learned that they envisioned Camp Imagine participants sticking with the program throughout the summer; doing the program with some specific participatory behaviors; using the library in new ways; and interacting with one another and the community around books and reading. That may still seem somewhat amorphous and abstract, but this definition gave me plenty to work with--especially since the committee was designing the program so that participants could choose specific activities and challenges that tied into the engagement goal.

That length-of-participation type of engagement--sticking with the program throughout the summer--was measurable by seeing how many days, on average, Camp Imagine participants were active in the program. We tallied the number of days between registration and final visit to the Camp Imagine desk for 20% of our participants (a solid sample, considering we had over 3700 participants). This was a straight output numbers analysis.

Because of the way we implement our summer reading program and the volume of participants involved, we knew we'd stick to a participant survey to capture most of the rest of the data to get at whether we were actually facilitating engagement. With the goals and definitions in mind, I got down to creating our survey questions to measure engagement.

The reading clubs committee was defining engagement as participating in a few direct ways with the program. That's not the same as just participating in the program. Rather, a specific engagement activity option was available at every level of Camp Imagine: to submit something to the Camp Mailbox. It might be a book review; a picture or drawing of a friend; a favorite recipe; a map of places traveled, or hoped to travel; and more sharing prompts like these. We had a physical Camp Mailbox just next to our Camp Imagine desk, where all registration and badge completion interactions took place. The program was designed to give participants the option of engaging directly with the program and program themes by submitting to the Camp Mailbox, and this type of participation was incentivized when we featured Camp Mailbox submissions on a display next to the camp desk and on our Facebook page. And so, when it came time to think about quantifying this type of engagement, we asked two specific questions on the summer reading survey:

1) While participating in Camp Imagine, I submitted an item to the Camp Mailbox. (Options: Yes / No / Not Sure)

2) While participating in Camp Imagine, I looked at items submitted to the Camp Mailbox on display in the library or on the library's Facebook page. (Options: Yes / No / Not Sure)

The next definition of engagement was using the library in new ways. Once again, this type of engagement was built into the design of the program as an option. At every level of the program, participants were invited to find relevant information in a library database or resource; try reading in a new genre; set up an account with and try a downloadable resource like Hoopla or RBDigital; and more. To get at whether participants were in fact using the library in new ways, we asked one specific question on the summer reading survey:

3) While participating in Camp Imagine, I tried a library resource I'd never used before. (Options: Yes / No / Not Sure)

Finally, the last definition of engagement was that participants would ideally interact with one another and the larger community around books and/or reading. Again, this type of behavior was incentivized, with many levels of the program including the activity of sharing a book or other media recommendation with another person--be it friend, family member, librarian, or anyone else. This type of engagement is also where my initial research into measuring engagement--specifically during summer reading--paid off. I had explored write-ups and descriptions of how California libraries have been tracking summer reading outcomes since 2011, and that's where we landed on this concept of engagement with reading programs as resulting in participants considering their communities as places valuing reading. With the combination of our program-incentivized sharing and the concepts learned from California's outcomes iterations, we asked two specific questions on the summer reading survey:

4) While participating in Camp Imagine, I shared a book recommendation with a family member, friend, or other community member. (Options: Yes / No / Not Sure)

5) As a result of participating in Camp Imagine, I feel that Skokie is a community of readers. (Options: Strongly Agree / Agree / Neither Agree nor Disagree / Disagree / Strongly Disagree)

The survey was rounded out by questions from the PLA Project Outcomes summer reading surveys--we feel it's important that we contribute to the project, which provides its aggregate data for library advocacy across the country, whenever it fits into our own strategies and plans for capturing program outcomes.

Our final survey was one half-sheet of paper, front and back, and we got a response rate that allowed us a pretty high degree of confidence in the data we collected.

Curious what we found?

Our graphic designer Vanessa Rosenbaum created a gorgeous 4-page final report for Camp Imagine. Here's the page about engagement. I put together the data analysis and narrative, and the sidebar includes items from the Camp Mailbox and anecdotes.

From analyzing the length of time that Camp Imagine registrants engaged in the program (which was 73 days long in total), we found that:
  • Youth who earned at least one badge (i.e., didn't just register and never return) participated on average 34 days, or about five weeks.
  • Teens who earned at least one badge participated on average 29 days, or about four weeks.
When we looked at the top 50th and 25th percentiles of youth and teen participants, these numbers were even higher. This type of sustained engagement has implications for combatting the summer slide.

From our survey questions, here's what we learned about program engagement:
  • 84% of participants submitted something to the Camp Mailbox.
  • 61% of participants viewed the Camp Mail.
  • 65% of participants tried a library resource they'd never used before.
  • 75% of participants recommended a book to a family member, friend, or other community member.
  • 94% of participants feel Skokie is a community of readers.
Now, considering participants had the option, at every single level of the program, to complete each level in the traditional summer reading way--by reading alone--we think these numbers are pretty remarkable. In every metric, over half of program participants opted to participate in our engagement activities alongside the traditional reading activities. And the fact the 94% of participants feel our community is one of readers? Well, that makes a librarian's heart happy. And these data all provide solid baselines for continuing to measure engagement over the course of subsequent reading clubs.

So that's how we did it: that's how we measured engagement during Summer Reading 2017.

Saturday, July 8, 2017

Program Kits for Summer Bookmobile Pop-Ups

Two summers ago, I wrote about our forays into pop-up programming for the summer. I mentioned in that post that I'd be assessing this volunteer-staffed, in-the-library, weekly activity model for the following summer, and my youth program team and I definitely considered all aspects of the program. What we ended up with last summer was an in-library program schedule with a much higher volume of activities--which meant we could focus our pop-up energies elsewhere. And so we did: to the bookmobile. What we discovered last summer is that we can pretty simply pop-up with activities for bookmobile patrons, but we have to provide staffing for every pop-up; our bookmobile staff see such a high volume of patrons at each stop that they can't lend one of their regular staffers to lead an activity.

So for this summer, we considered what we'd learned, then iterated again. For 2017, we're sending activities out to the bookmobile once again. Since we know providing staffing is vital, and scheduling staff can be difficult during the summer, we needed to be really flexible to ensure that any staff member could confidently and competently lead a pop-up program with the bookmobile, little advance training necessary. And so we created Bookmobile Program Kits, each with video instructions that we can play at the pop-up.

Ribbon Cutting for New Bookmobile by Skokie Public Library, CC BY-NC-SA 2.0

See, we got a magnificent new bookmobile a little over a year ago. This gorgeous library on wheels boasts an exterior awning, under which we can set up a folding table and lead activities. Also under the awning, beneath a panel on the exterior wall of the bookmobile, there is a flat-screen television. Which means we can play videos while we're set up under the awning leading activities. Thus our idea of how-to videos for bookmobile programming was born.

This summer we've got five different bookmobile program kits available, each with a bin full of supplies and a how-to video featuring a library staff member of community volunteer who has experience leading that same activity in a formal library program or space. Our kits, all with a STEAM connection:

We've only made a few pop-up appearances with the bookmobile so far this summer, but word is they're going quite well. The short videos help to reinforce to bookmobile patrons that they can participate in the activity even if they only have five minutes--we've found most patrons plan enough time for browsing and checking out books, but don't plan to stay for a longer activity. Staff leading the activities have shared that the kits work well logistically: all the supplies are easily accessible, the activities don't require a huge amount of space for patrons to engage, and the video allows ease of participation even when there's a larger size group of eager participants. And no one forgets how to do an activity, because you have the how-to on a loop.

At this point in the summer, we're considering these Bookmobile Program Kits a success. And a bonus: we didn't specifically brand the how-to videos as pertaining to the bookmobile pop-ups, so we can reuse them in so many potential programs and spaces in the future.

Tuesday, March 21, 2017

Actually, She Did That: The Civic Lab for Women's History Month

The team of folks here at my library who curate the Civic Lab were having a meeting a few weeks ago where we were discussing potential topics for future Civic Lab pop-ups. Sometimes we tie our pop-ups to formal programs on our calendar, sometimes to topics in the news, sometimes to installations in the library, and sometimes to specific days or months of import or conversation. We were brainstorming what topic to focus on for Women's History Month, and we had plenty to choose from--there's a lot going on right now affecting women, have you noticed? You might be surprised, then, to hear that the person who came up in conversation was Kanye.

Or maybe you're not too surprised, because he came up in the context of one particularly annoying and eye-roll-inducing line from Famous: "I made that bitch famous," said in reference to Taylor Swift. As if he, a man, made her, a huge pop star who is a woman, famous because he physically took the stage and microphone away from her while she was winning an award. Gross.

And so we had our topic for the Civic Lab for Women's History Month: women who have accomplished something, but who do not get their deserved credit (often it goes to a man or group of men), or they are better known for something irrelevant to their accomplishments.

We called it "Actually, She Did That"--taking the mansplainer's favorite opening word of "actually" and shedding light on some excellent women throughout history whom many do not know and whose accomplishments have been snatched from them.

The central activity in "Actually, She Did That" was a game of sorts. On a column constructed out of our multipurpose crates, we affixed large images of 11 different women who fit our criteria stated above. (As one of the mother/daughter participant pairs said, these 11 are only the tip of the iceberg when it comes to women not getting the credit due to them.) Each image included the woman's name and date of birth (as well as death, where relevant). On the table next to the column, we had 11 slips of paper. Each slip noted the accomplishment of one of these women, with a parenthetical about how or why she hasn't gotten credit for that accomplishment. The goal was to try to match the woman to her accomplishment, learning more about these 11 fantastic women along the way.

Our 11 featured women were:
  • Nellie Bly (1864-1922) - Bly was a brilliant, pioneering journalist, despite popular opinion that she couldn't be a good journalist because she was a woman. Bly was an early undercover investigative journalist, checking herself into a mental asylum and writing articles exposing the despicable treatment of (mostly female) patients in these facilities.
  • Selma Burke (1900-1995) - A sculptor, Burke was the artist behind the FDR profile that was used on the dime. Yet the (male) engraver typically gets credit for the design, rather than Burke.
  • Laverne Cox (1984- ) - Cox is the first transgender actress to be nominated for an Emmy in an acting category. Yet despite her talent and prowess as an actress, much media coverage of Cox returns to questions about her gender assigned at birth--regardless of its lack of relevance to her career.
  • Rosalind Franklin (1920-1958) - Franklin's research led to her discovery of the double helix structure of DNA. Her male lab partner stole her findings and gave them to Crick and Watson, who went on to win the Nobel Prize for DNA discoveries.
  • Katherine Johnson (1918- ) - One of NASA's "human computers" whose supreme math skills allowed early astronauts to safely start to explore space, Johnson and her colleagues have only recently started to get recognition due to the book and film Hidden Figures.
  • Regina Jonas (1902-1944) - The first female rabbi, Jonas was refused ordination for years despite having gone through the same training as her male colleagues. She was finally ordained before being sent to a concentration camp. She died in Auschwitz.
  • Hedy Lamarr (1914-2000) - Lamarr was a brilliant inventor, developing spread spectrum communication and frequency hopping technology which are now the basis for cell phones and wi-fi. Yet she is often known only for being a beautiful actress.
  • Ada Lovelace (1815-1852) - She wrote the first computer program, although her male friend Charles Babbage is usually credited as the first computer programmer. Lovelace is usually first credited as daughter of Lord Byron. So not only does she not get credit for what she did, but she's defined in relation to her male relative.
  • Wilma Mankiller (1945-2010) - Mankiller was the first female chief of the Cherokee Nation. Many American history texts ignore her leadership and maintain there has never been a female head of state in the U.S.
  • Arati Prabhakar (1959- ) - Prabhakar was the head of DARPA, the Defense Advanced Research Projects Agency, from 2012 until January of this year. Research and developments under her watch have included huge strides in biomedical technology like prosthetics. Credit is typically given to the presidential administration at the time of the invention.
  • Chien-Shiung Wu (1912-1997) - Called the "First Lady of Physics," Wu worked on the Manhattan Project. Her work in nuclear physics won a Nobel Prize for her male colleagues, but she was not recognized. Even though the winning experiment was called the "Wu Experiment."

We had some really wonderful conversations with patrons as they engaged in this activity. Many recognized a few names or pictures, but couldn't place their finger on where they'd seen or heard of these women before. We share biographical facts with participants, many of them shaking their heads in frustration at just how common this type of credit-stealing is. One teen girl, participating with a friend, remarked after hearing the stories of several of the women, "Why do they keep giving away credit?" We talked about how it wasn't a question of these accomplished women giving away credit, but rather them having credit taken from them or given to someone else. These teens got mad. They demand better, for the world to see them and their friends and other women. As it should be.

Alongside this activity of matching women to their accomplishments, we also had a few other elements available for Civic Lab participants. We had a number of great titles on offer for folks interested in learning about more women and their accomplishments, including:
  • 50 Unbelievable Women and Their Fascinating (And True!) Stories by Saundra Mitchell, illustrated by Cara Petrus
  • Bad Girls Throughout History: 100 Remarkable Women Who Changed the World by Ann Shen
  • The Book of Heroines: Tales of History's Gutsiest Gals by Stephanie Warren Drimmer
  • Dead Feminists: Historic Heroines in Living Color by Chandler O'Leary & Jessica Spring
  • Rad American Women A-Z by Kate Schatz, illustrated by Miriam Klein Stahl
  • Wonder Women: 25 Innovators, Inventors, and Trailblazers Who Changed History by Sam Maggs, illustrated by Sophia Foster-Dimino

We also put together a handout with resources for hearing more women's stories through an email newsletter, podcasts, and online videos. (See the handout here.)

The handout also includes three questions to get folks considering the stories of women in their own lives, as well as how they can make space to hear and share the stories of women:
  1. What have women in your life accomplished? Have they gotten credit for these accomplishments?
  2. What would you say to them in acknowledgement of what they have accomplished?
  3. How can you help to share the stories of women and their work?

We intentionally posed that first question on one of our crates, and we provided sticky notes and pencils for participants to weigh in. During the two hours a coworker and I facilitated "Actually, She Did That," however, no one wrote a response to the question. We don't think it was from lack of interest, but rather from the greater appeal of learning about the women whose images were front and center in the installation. We're hopeful that the public question, as well as the handout, provided fodder for reflecting on the women in participants' lives.

Monday was appearance number one for "Actually, She Did That." We'll be popping up again this Friday, and we're eager to see what types of interactions are prompted this time around. From there, we want to think about how to continue this idea of making clear space for women and women's stories beyond just Women's History Month.

Saturday, January 28, 2017

Together at the Table: A Civic Lab Program for MLK Day

My library aims to observe Martin Luther King, Jr. Day each year with multigenerational programming that engages our community on issues from Dr. King's work and themes in his speeches and writing. When it came to planning for MLK Day 2017, the new Civic Lab here at the library felt like a completely natural connection. We wanted to intentionally connect our goals for civic engagement with Dr. King's vision for a truly united country, starting with our library community. And so our 2017 event was created: Together at the Table.

The title Together at the Table is lifted from Dr. King's "I Have a Dream" speech from the 1963 March on Washington for Jobs and Freedom:
I have a dream that one day on the red hills of Georgia the sons of former slaves and the sons of former slave owners will be able to sit down together at the table of brotherhood.
Our Together at the Table program was, at its core, a drop-in community gathering meant to foster feelings of empathy and belonging among all the many people who live in Skokie and who use the library. From noon until 8 p.m. on that Monday, Together at the Table took place in our larger multi-purpose programming space which is adjoined with a kitchen. We pushed together four of our tables, end to end, to create a long banquet-style table with chairs. Spread across the tabletops were tent talkers to get conversations started; we chose 6 questions from "The 36 Questions That Lead to Love." We purchased many dozens of muffins (all kosher) and set out hot coffee, hot water and tea, and ice water as well. And so the stage was set.

A snapshot from Together at the Table,
by adult program librarian Mimosa Shah.
Throughout that day, four excellent high school volunteers helped facilitate the program alongside library staff. Those high schoolers greeted patrons at the door--and we indeed saw patrons of every age. After encouraging (and in some cases helping) patrons to make a name tag, the volunteers led them to a seat. New arrivals were seated next to people already in the program, and so the volunteer would make introductions--nobody in the room was a stranger, nobody without someone to talk with. The volunteers would then go to the kitchen to get a muffin for each participant. More often than not, before the volunteer could even come back with the muffin, the patrons were diving into conversations with their tablemates.

As I pitched this program to my fellow programming team members and staff, I kept coming back to the central idea that participants would be able to break bread with one another and get to know fellow community members. I had the support of supervisors throughout the library that these conversations be open to staff at every level as well--and we did have about two dozen staff drop in to meet and talk with patrons throughout the day. Breaking bread is such a strong human interaction that seems to span all cultures; friendships and family are made over a shared meal. We wanted to facilitate that feeling of goodwill and community in our library--in particular at a moment, and at the beginning of the week of the inauguration, when many in our community were feeling unsettled, isolated, and worried.

Was the day successful? To be honest, I would have counted Together at the Table a success if even a single conversation took place in that space. I wanted to foster a space for meaningful interaction--something deeper than a rote "How are you?" / "Fine, and you?" back and forth or a conversation about the weather. And with that modest goal, I was blown away by the interactions that took place and the relationships that were formed.

Some snippets of the interactions, almost all between people who came to the shared table as strangers:

  • "We discovered that we have friends in common!" A patron who participated without knowing anyone struck up a conversation with someone seated near them, and they discovered that they share mutual friends. They remarked about feeling like the community is a close-knit, interconnected one.
  • A girl scout leader in the program with her troop connected with our deputy director, who leads a troop in a neighboring town. The two groups have made plans to get together.
  • An older elementary boy, who spends many of his out-of-school hours at the library, expressed delight and excitement at not only learning the library director is, in his words, "Korean like me!" but in being able to have a conversation with him as well.
  • Two young men who entered the program separately and were seated next to one another became friends in the course of their conversation, trading numbers before leaving so they could meet up again.
  • One woman responded to the prompt about something she's dreamed of doing with her desire to open a female-friendly auto repair garage. After talking about this dream with her seatmates, one identified himself as a person in real estate and he shared with the woman information on how to find a location for her business. They exchanged numbers and made plans to continue their conversation.
  • A 21-year-old man shared that the day of the program marked his twentieth day since coming to the U.S. from Afghanistan. In the course of his conversations with his tablemates and one of the teen volunteers, he became curious about the library's volunteer opportunities as a way to meet people and improve his English. He ended up talking with our volunteer coordinator, who had come into the room.

These are just a handful of the interactions that played out across the shared table. Our community embraced the ethos of Together at the Table wholeheartedly. I was struck by how open-hearted and honest the conversations I had, and the conversations I heard about, were. By how willing our community is to forge bonds with one another.

We had several participants ask if we plan to do this sort of program again, and many of us on staff and in our administration want to. If a root of civic engagement is feeling invested in one's community, Together at the Table certainly showed we have strong foundation in Skokie for this work.