Frequentist-based Sequential Planning & Analysis Tool

This tool allows you to both plan for and analyze a frequentist-based sequentially-designed experiment. For more information on how to plan & analyze frequentist-sequential design tests, see this Q&A on the Analytics Toolkit written by Georgi Georgiev when Lucia Van den Brink chose to interview him to help explain the process.

Special thanks to Merritt Aho for his work creating the original and updated versions of this calculator.

I typically recommend starting with a simple Runtime calculation to see if you can run a test in less than 2 weeks with a standard setup. If your lift estimate is outrageous and/or you know you will need more than 2 weeks, a Sequential design might be right for you. Sequential testing is about planning to peek and ending your test early if you cross a decision boundary if you under (or over) perform to expectations. The article listed at the top of this page is a great educational primer on the topic I highly recommend, but the important key is to remember the importance of planning. This tool was designed to enable both planning and flexibility. During the analyses phase (checkpoints), it’s important to remember to change the number of the checkpoint so you don’t lose your previous checkpoint history! Make sure to make use of the shortcut feature so you don’t lose your work! And reach out if you want help on how to use the tool. I’m here and happy to help.