Creating a research culture at Twickle
Interviewing streamers at TwitchCon 2018
We decided early on in Twickle’s conception that we’d be a research driven company. I was responsible for most of our research activities. My cofounder, who is traditionally a software developer, was especially welcome to research and developed into an excellent researcher himself (lucky for me).
When we started out, we made tools for musicians on Twitch. We pivoted a few times to land on helping esports teams manage their fans, analytics, and sponsorships. Research informed every step of our journey. We’ve held over 400+ research sessions with our users, utilizing a few different flavors of research: generative research, evaluative research, and constant feedback.
Interviewing a streamer at TwitchCon 2018
How we use generative research
We use generative research to understand how we can serve our users. Each of our products has been born out of generative sessions with users. It serves an essential strategic role in picking out opportunities our competitors haven’t noticed.
A good example of how we use generative methods was illustrated in our streamer analytics project.
How streamers use analytics
We were in the midst of a product expansion and looking for unmet needs to design for. One of the biggest needs we heard from our streamers was that they weren’t sure how to improve their content or what metrics to be concerned about. Being lean startup disciples, we thought they should start with finding some leading metrics and focus on those while experimenting with different content. Before building something, we set out to discover how streamers used analytics in the first place.
Streamers have a lot of data about their content at their fingertips: post-stream analytics, native platform analytics, 3rd party analytics, etc. How do they make sense of this and what mental models do they have around analytics?
Our research project looked like this:
Goal:
-
Understand how Twitch streamers use analytics and how they think about analytics
Research questions:
-
What are the pre-existing mental models associated with social media platform data?
-
How (if at all) do they use Twitter, Discord, and Twitch data to inform their decisions?
Process: We held 10, 60 minute remote interviews, using a semi-structured protocol
-
Recruited streamers from our own user email list and posts on Reddit/Craigslist, using a screener survey that garnered 40 entries
-
Constructed a protocol based on our top research questions
-
Held remote video interviews over Discord / Google Hangouts
-
Walked through the questions and had participants share the artifacts they used
-
Transcribed interviews and coded for themes
-
Synthesize themes into a shareable document
-
Discuss findings and implications
Insights:
-
Having different analytics for each platform made progress and improvement difficult to measure-
Streamers use Discord, Twitter, Twitch, YouTube, and Instagram for their content. Using all of the native platform data analytics tools made it really difficult to glean any insight on whether they were improving or not.
-
It takes a lot of time to switch between each platform and learn each platform’s analytics UI nuances
-
They think that data is helpful and they should use it, they just aren’t sure how to.
-
-
Most streamers don’t use analytics to improve their content, but instead to reflect on after they stream or post something in the moment-
After a stream, it’s a reflective, cool down type moment for streamers. StreamLabs and Twitch send out an email detailing their stream analytics right after the stream, which helped them think about how the stream went
-
They try to discover when in their stream there was a huge spike in viewership and what was going on during that time
-
They can see where other viewers come from, so they can infer which other streamers to network with
-
A rare archetype of streamer will meticulously track their stats to improve, accomplished by using spreadsheets and notebooks, making a complex web of data
-
-
3rd party tools like SullyGnome are too overwhelming to glean anything from despite being industry standards-
Many reported the UI to be confusing, not knowing what to look at when there is so much functionality
-
They mainly use this to share with sponsors, albeit very rarely
-
“How are you supposed to act on those numbers? It’s hard to tell what’s doing well and why”
-
-
There isn’t a way to see data on how you’re Discord is performing-
Most streamers have a mental account of how well it’s doing, but beyond that it’s impossible to discern Discord engagement and growth
-
Outcomes: We were able to use this information to design our own analytics tool that made seeing progress and improvement over time really easy. We then conducted concept validation with another segment of users and were able to include an MVP in our next iteration.
Our tool aggregated all of their analytics across platforms in one place and made it easy to see how much you’ve improved compared to last week. This was a top feature in our premium plan, resulting in increased revenue and higher retention.
The released iteration of our analytics product.
How we use evaluative research
We use evaluative research as a means to understand how usable our product is and better understand how well it’s performing. Anytime a new feature was to come out, we’d use concept validation and usability testing on a prototype in Axure. Once a feature was released, we’d use MixPanel and HotJar to track our user’s actions and see how we can improve it from analyzing data from those platforms.
A good example of how we used evaluative methods was in usability testing our Twitch extension onboarding.
The layout of a Twitch stream page, with our Twickle extension being below the stream. NEAT!
Twitch extension onboarding
Twitch had just released their extensions platform, essentially interactive overlays and panels developed by 3rd parties that are present on a streamer’s page. This was a dream come true for us. We had previously thought of how cool it would be to have something interactive on the page beyond chat. Now, we had our chance.
We wanted an easy way for viewers to sign up for Twickle and for streamers to discover us on the extensions marketplace. The extension seemed like a perfect way to get traction.
The challenge: it was incredibly confusing to try and get an extension set up on your stream. It was a totally novel concept in the Twitch world and the design of extension discovery and set up was lacking.
We had to make sure our onboarding from the Twitch extension marketplace was a super smooth setup.
We had developed a prototype of our onboarding in Axure. We then decided to conduct usability tests to evaluate how people went through our flow and discover ways to improve it.
Prototypes to check out:
Goal:
-
Understand how people set up our extension and find improvements
Process: We held 10, 60 minute remote usability sessions, using a semi-structured protocol
-
Recruited streamers from our own user email list and posts on Reddit/Facebook posts/Craigslist, using a screener survey that garnered 120 entries
-
Constructed a task-based usability protocol
-
Held remote video interviews over Discord / Google Hangouts
-
Walked through different onboarding tasks and had participants answer questions as they completed tasks
-
Transcribed interviews and coded for themes
-
Synthesize themes into a shareable document
-
Rank different UX improvements in terms of importance and time to fix
-
Discuss findings, make changes to the prototype, rinse and repeat!
Insights:
-
9/10 streamers were able to fully setup the extension-
One user was unable to figure out how to “activate it” on their page
-
Iteration: We’ll need to add information on the last step in the Twitch dashboard to show what to do
-
-
When asking for permissions, users were unclear about what we do with their information-
Iteration: We ought to include some information for them about why we ask for those permissions and what we do with them
-
-
The “set up your points” page took a lot of thinking to get through-
“How many points should joining my Discord be worth? Hmm, I don’t know”-
Iteration: Remove this step, add smart defaults, and keep the editing of points optional
-
-
-
Having to sync all of your accounts at this point took a lot of time-
“Can I fill these in later or….?”-
Iteration: Make it clear that the sync your social media account page is optional
-
-
-
At the end of the onboarding, streamers wonder “is it already on my Twitch?”, “When am I done?”-
This is possibly due to Extension language use in the Twitch dashboard-
Iteration: Need to explain that they still need to add the image panel on their Twitch
-
Iteration: Add a clearer CTA on the completion page, consider a check mark + button closing the page to clearly imply completion
-
-
-
Streamers loved being offered to join our Discord-
This made them feel like we were there for them and ready to help
-
Outcomes: We were able to use this information to design a better extension onboarding flow. When we released the extension on Twitch, we were able to start way ahead of other companies releasing extensions.
The extension being listed on Twitch’s extension marketplace serves to be one of our best traction channels to date.
Streamer and Twickle advisor posing next to our original partner program poster, Twickle backdrop, and Twickle TinkyWinky
Getting constant feedback from our users
We wanted to make it effortless for our users to give us feedback. Here are some of the top ways we made that happen: a partner program, Discord channel, research panel, and Intercom.
Twickle Partner Program
We worked closely with our lead users. These were users who loved our product, used it daily, posted about it all the time, referred their friends, and had incredible vision for what they wanted Twickle to be. We set up meeting cadences every week to discuss how their stream was going and answer any research needs we were having at the time. We eventually formalized this process, calling these users “Twickle Partners”. Streamers could apply in a survey and we’d be in touch if they were a good fit.
We’d developed close relations with our partners, even hanging out with many of them at different TwitchCons. This program allowed us to generate product advocates, get incredible insight into the lives of streamers through weekly meetings, and better design Twickle.
Twickle Discord Channel
We created a Twickle Discord channel that anyone could join and chat directly with us. This allowed our users to freely voice their feedback, connect with other Twickle users, and chat with us directly in a venue they already felt comfortable in. It’s served as a crucial meeting place for our community and an incredible venue to get users talking about our product and how they use it.
A snapshot of our feature requests channel. Bumpin’!
Twickle Research Panel
While we had a Discord and Twickle partners, we wanted a larger population that would be freely available for research beyond our power users. We decided to create a research panel. We’d ask users about their demographics and behaviors in a survey link, sent to them 3 days after signing up to allow us the ability to reach out to them for any of our research needs. We have 100+ users signed up and have used it for countless usability tests and foundational interviews.
Intercom
We’ve utilized Intercom for our in-product feedback mechanism. This allowed users to ask questions and send feedback while using Twickle by using the Intercom widget in the lower right space of our product. Our users had easy access to us without having to send an email or join our Discord.
A Twickle partner and I at TwitchCon.
In conclusion
Through these research methods, we were able to create a research first culture at Twickle. This led to some really awesome features and insights that resulted in a deep understanding of the content creation and esports space.