The Ultimate Guide to Conducting Community Member Research Surveys
Survey research is fast, nearly free, and broadly accessible. But surveys should never dictate your community strategy. Expert community builders know that survey research is valuable - but only to a point.
For instance, experienced community builders will never directly ask members what they want or what their community platform should be. When we ask for anyone to self-report what they want or desire, we are setting ourselves up to receive unreliable data inputs that could steer you in the wrong direction.
When asked, people will say they want you to create all kinds of community activities, events, and programs that, once launched, they will never attend. In addition, asking people about their needs is a valuable thing to do on its own - it makes people feel heard and seen. But if you then cannot deliver on those needs? You have asked someone to risk vulnerably sharing for no return whatsoever. There is no faster way to chip away at trust.
However, community surveys are an excellent fit for 4 crucial goals:
Establishing demographics and benchmarks of community health
Rapidly inspiring new content and program ideas
Ranking top member goals and discussion topics
Gathering a list of people who want to get more involved
One survey can serve all of these needs or you may focus on one of these goals. Once you know what your goals for survey research are, you can break those goals down into questions that help you gain insights you need.
Setting Goals for Community Member Surveys
Survey Goal 1
Establish demographic
and health benchmarks
If you do not yet understand the common identities shared by your community members, a survey can help you clarify and focus your efforts. Demographics give you at least a surface-level understanding of potential member identities. Demographic information can tell you where you may be falling short (for example, in terms of gender or age representation). You may also ask about psychographic information (personalities, interests, opinions) to gauge similarities and differences among members. (Shout out to Robert Yang for this idea and evolution of my thinking!)
It can be smart to include a "Decline to state" option here and to include open responses for any of these, ensuring more inclusivity in your survey design.
In addition, you may want to establish community health benchmarks. For instance, you can ask a Net Promoter Score (NPS) question and ask for qualitative insights about why members give the score they do. Those qualitative insights allow you to immediately address any low-hanging fruit. You may also ask how connected members feel to your organization and to one another, establishing a connection benchmark you can then measure your future work against.
Pro Tip: Be especially sensitive to yourself when reading open-ended community health score responses. They can be hard to read and not take personally, but I assure you that the responses are not a reflection on you as a human being or even on your individual work! So give yourself lots of space and rest before reading and then creating follow-up tasks.
As an example, one of my clients once sat down with a glass of wine one weekend, and she read every single qualitative response to a benchmark survey, following up personally with everyone who gave harsh but legitimate criticism. We may not all be as brave as Diane, but we can aspire to be.
Survey Goal 2
Inspire new content and program ideas
When members make no progress, they disengage. Surveys are great tools for understanding *the progress your members want to make*. Notice that is different from trying to understand what your members *want*. They may not know they want a specific program unless you can connect it to the progress it can help them make. When you understand these deeper desires, you can then design programs that naturally sparking engagement.
Start by creating a list of potential goals that members may share (e.g. getting feedback, learning a specific skill), and include an open option for a write-in.
With the answers to this question, you can develop a list of the top challenges (which you can also rank, see Survey Goal 3). When you know challenges and have a clear community purpose and values, program creation becomes a natural process rather than forced and rigid.
For instance, if you learn from your survey that the top challenge members face is "scaling their company" and your values are personalization, authenticity, and diversity, you may brainstorm for several minutes all of the ways you might gather people to address this challenge. You could, for instance, create a custom accountability group connecting people with different business backgrounds. Or you might create a member advisory board of diverse experts who can offer personalized mentoring to members while also giving them special access to your team. Let the ideas flow.
Pro Tip: If your survey software allows it, pipe in the answers from this question into a ranking question, as we will discuss for Goal 3.
Survey Goal 3
Rank Top Goals & Topics
While people's self-reported "ideas" may be unreliable, having people rank their goals and topics can encourage them to be more picky and thoughtful about their responses. So while you might have a long list of all the goals that members want to achieve together based on the questions for Survey Goal 2, you may not know how strongly members prioritize each. You especially may not understand how these goals vary across different demographics, like job titles, age, or industry. Running a cross-tabulation analysis will allow you to break down goals by demographics and psychographics and also see which goals are universal.
Therefore, it is always a good idea to ask members to rank their goals and challenges, who they want to meet, and the topics they most want to discuss. Equipped with this information, you will have a sense of the urgency of each goal as well as an idea of urgent topics for discussion (this information often helps you decide how to develop channels, subgroups, event themes, or discussion categories).
Survey Goal 4
Gather a List of People
to Get More Involved
Surveys are not the end goal. Relationships are.
Per the advice of Erika Hall, a user researcher and author of Just Enough Research (shout out to Sarah Judd Welch for bringing this book into my life!), I recommend that community builders use surveys as screeners for further discussion.
Always end the survey by asking if people would be willing to talk further about their answers, and then ask for their email or other contact information. It is unlikely you will be able to talk to everyone who indicates they would like to talk further, so be sure to:
Follow up with everyone who said they would be willing to talk and let them know (1) you appreciate their contribution and openness, and (2) you will follow up with them if they are selected
2. Take your time deciding who you would most like to speak to from the survey. Whenever possible, get a diverse array of voices and really focus on your advocates and detractors.
3. Schedule interviews in whatever format is most comfortable for them: some people like Zoom or a phone call whereas others prefer email, IM, or even text. What you lose in candid responses from live interviews, you make up for in the ability to consider answers, ask follow-up questions, and gain intentional and considered answers via email.
You will likely want to interview a handful of people who did not complete the survey as well, but this initial group will give you a good sample of those who are committed enough to share their opinions. And in community, commitment is key.
Pro Tip: If you have promised anonymity in your survey, be sure to indicate that, for those who provide their contact information here, their answers will be connected to their identity.
The Steps to Survey Research
1. Establish criteria for inclusion in the survey
Not everyone's opinions will matter. In fact, you want to be crystal clear about who you want to hear from before you send out surveys. There are a few common criteria to consider:
Do you want to hear from your entire community, or only a part of it (the most engaged, the least engaged)
Ask yourself:
Who is it you need to build deeper relationships with?
What voices are missing in your community today — and in your resulting strategy?
What is your clear goal for this survey, and what target groups will be most likely to help us reach that goal?
From there, establish clearly defined criteria and send the survey only to those members.
2. Formulate a few hypotheses.
In rigorous social science research, you need a theory-backed hypothesis for your research to be considered rigorous. Now, you don't work in that environment (most likely), but that does not mean you should be lazy and simply look for random patterns in data. That practice is known as p-hacking, and is deemed unethical in academic circles.
Sit and think about your goals and what patterns you think will emerge from the survey findings. As you go through step 3, allow this to be a process in which you re-word questions, refine them, and question your assumptions.
Don't forget to write down your hypotheses! It is easy to delude yourself into thinking that you guessed correctly when you have no written proof of your hypotheses. The distance between your hypothesis and the results often also becomes a great story or aha! moment to share with others.
3. Write the survey
Now it's time to write the survey. Keep the survey as short as you can, ideally less than 10 questions. See below for tips I wish I had known when I began doing this work years ago.
4. Send out the survey, diligently.
I recommend a window of no more than 5 business days to complete the survey, with 1-2 reminders in that timeframe. Ideally leave it open over a weekend.
5. Analyze the results.
If you don't have statistical chops (most community managers don't, it's OK!), see if you can partner with a BI analyst in your company or hire a statistical analyst on a site like UpWork. Check references first, your mileage may vary. These people can help explain the concepts of statistical significance to you, help you understand where and when it is appropriate to segment data, and find patterns you may otherwise miss.
6. Present the insights.
Don't keep the insights to yourself! Share your methods and findings broadly in a deck that can be distributed around the organization.
Top Tips for Writing Member Research Surveys
1. Trying to re-engage disengaged members is a pain in the butt and hardly ever yields meaningful results. There are, however, a few exceptions:
If you have redesigned the community and taken into consideration new value propositions, user journeys, and more.
If your members do not even know about your community offering (they were invited in a haphazard way at one point), you can use the survey as a reminder
2. You can use surveys as a crafty way to remind people what the benefits of your community are.
If you are doing research to figure out why people are not engaging, throwing in a question such as, "Which of our programs have you benefited from most recently?" and including a multiple-select choice of options alongside the option of, "I did not know about any of these," can help. I have seen clients gain hundreds of new participants after reminding them that the community has specific feature sets.
3. Only ask one question at a time.
For example, don't ask: "What are your thoughts and feelings about our monthly events program?" Thoughts and feelings are closely related, so you can see how easy it would be to dash off this question in a survey. However, most participants will answer one or the other question or give you otherwise untargeted, unhelpful information.
4. Don't create false binaries when asking a question.
For example, don't ask: "Are you joining the community because you love animals or because you are bored?" Neither answer may be true. A better way to rewrite that question would be a Y/N "Are you joining the community because you love animals?" or — far better — an open-ended or multiple-choice option with an "other" and fill-in selection.
5. If you want to hear from multiple types of users and be able to segment their responses, don't segment your data to soon! Unless you have a background in stats, you are going to want to lean on the survey software's (Qualtrics, Survey Monkey, Typeform, etc.) analysis tools as much as possible.
The easiest way to do this is to have respondents self-select the group they fall into or to duplicate the survey and segment who the survey goes out to. If you have email software like Convertkit or Mailchimp, you could also segment by sending the survey to the groups in separate emails and creating skip logic according to the questions you want to ask to the specific groups.
Wrapping Up
If you ever feel overwhelmed when creating your survey, always come back to this: Focus. Focus ruthlessly. The more questions you ask, the less likely the data are to be reliable as people get fatigued.
Surveys may not be a panacea, but they can help you meet many important goals. Just make sure to clearly establish those goals at the outset. Then let the insights spark new ideas. You got this.