To help creators connect with more fans and grow their membership community, we’re always experimenting and testing new ways for creators, like yourself, to grow their Patreon brand and membership. Much like creativity itself, testing new ideas takes time and repeated versions until we reach a final product. If you’re curious, here’s how we approach experimentation at Patreon.
What are experiments?
An experiment is when we release a product, feature or design to a subset of our community to see how it affects their experience. By limiting the experience to some, we can compare this experience to our wider community and measure impact, so that we can be sure that what we’re building will help creators grow.
Experiments are part of the core building process at Patreon. They are responsible for informing the data to make decisions when we develop products. We look across the funnel for areas where we can improve, starting with how creators can get more audience to their page, to getting more fans to successfully pledge, choosing the right tier pricing, helping members discover and activate their benefits as well as maintaining a healthy community.
We are always listening to feedback from our creators and experiments are a way for us to test our different approaches to many of the challenges that creators have shared with us. If you’d like to share specific feedback or a product request, you can learn how to do so here.
Who participates in experiments?
Participation in experiments depends on the nature of the experiment and product areas that we are testing. Typically, we randomly select participants to remove bias. We consider a variety of enrolment factors such as the size of the testing group, the time period the experiment is run and various other audience criteria.
On some occasions, creators will have the opportunity to sign up and participate in experiments that are in the early stages of product development. Creators will be notified of these opportunities either in-product or via email. Be sure to opt-in to receiving communications from Patreon, which you can check to see in your email notification settings here.
How we test
-
Start with principles. Principles serve as a guide for how we set up and conduct experiments, and as an anchor for what values we want to embody.
- Example: One of our guiding principles is always to be creator-first, and so our goals and hypotheses should ladder up.
-
Identify goals. In this case, we want to help turn more page visitors into patrons, and keep more patrons subscribing past their first month. Based on past research and data (including the feature request form, community conversations, support tickets, surveys, research studies and more), we form ideas for how we could achieve just that.
- Example: We’d like to help video creators get more fans to become members.
-
Form and test hypotheses. A hypothesis is an assumption that we can test.
- Example: If creators could tease fans with snippets of their video posts, more fans would become members after they preview that content.
- We then test the hypothesis “in the wild” for a portion of potential patrons. Throughout the test, we monitor both metrics we want to change (for example, whether there’s an increase in the percentage of visitors to your page who sign up) and related metrics to ensure the test doesn’t have unintended side effects.
-
Review results and observations. Once we’ve collected enough data, we analyse whether we can confidently attribute the results to the specific changes we tested, rather than pure chance. Sometimes the data confirms what we were thinking. Other times, it surprises us with an unexpected insight.
- Example: How effective are video previews at converting fans to members? How does that differ on-platform vs off-platform? How often do creators promote their video previews?
- Iterate based on what we’ve learned. If a test is inconclusive or unsuccessful, we’ll refine what we’re testing. We could use a different layout, wording or colour, or go back to the drawing board. If the results look promising, we could expand the test to a wider audience. Once we have confidence that what we’re testing will benefit the majority of creators, we’ll roll it out.
More FAQs
How long do experiments last?
It varies. Experiment durations depend on the size of the audience required to get unbiased results. We need to run experiments long enough to have confidence in the results and be sure that they are not due to chance. Depending on the product surface areas and audience eligibility, some experiments may take longer to gain traffic than others.
Can I opt out of experiments?
Typically, experiment participation is chosen at random and individuals will remain in the experiment to ensure unbiased results. In some scenarios, creators will be invited to participate in early stages of the experiment.
Will I be notified about experiments?
Because we run tests of all kinds every day to constantly iterate our product, not every experiment will notify creators ahead of time. In some cases, we may notify creators based on their participation eligibility to inform them of recommended actions to take ahead of the experiment. Most of the time, creators, fans and members won’t know that they’re part of an experiment to eliminate bias.
How will I know if an experiment is successful?
If the feedback and insights we receive on the experiment are positive, we’ll make the changes live beyond our testing groups. For any new products, designs or features, we will notify creators either in-product or via mobile or email notification. Be sure to opt in to Patreon’s product newsletter, and join our official Patreon Creator Community Discord server to follow our #patreon-announcements channel.