Skip to main content

Module 5 · Experimentation & Validation

5.1 Experimentation & the Scientific Method

Skills Track

How do I test my riskiest assumptions and gather evidence that my business will work?

Expert interview

Evidence Doesn't Care About Your Job TitleDr. Greg Watson · 3:28Chapters, transcript & audio →

Skills Track: Sections 5.1, 5.2, and 5.3 provide the complete academic foundation for every student. The concepts apply to business, employment, education, public service, healthcare, and community life. Optional Venture Track boxes offer tactical business-launch examples without replacing required instruction. Instructor-managed assignments are published separately.

Learning Objectives

By the end of this module, you should be able to explain how experimentation helps people make better decisions when they do not yet know whether an idea will work. You will learn to identify the assumptions hidden inside a proposed solution, turn an important assumption into a specific, testable hypothesis, and choose an appropriate way to gather evidence. You will also become more comfortable interpreting numbers and stories together, recognizing misleading comparisons, and distinguishing what a test actually establishes from what you hope it proves.

You will examine why ethical experimentation matters whenever real people are involved, including issues of consent, safety, privacy, accessibility, and fairness. Through workplace, historical, organizational, and entrepreneurial examples, you will practice judging whether evidence is strong enough to justify a decision—or whether the responsible next step is to learn more. You do not need to plan a startup or arrive with advanced statistical knowledge. The goal is to develop a transferable professional capability: the ability to test meaningful claims, learn from results that challenge your assumptions, and make thoughtful decisions under uncertainty.

Before We Begin: The Message That Everyone Liked

Imagine that Elena, the community-college student whose journey we began in Module 1, has reached an encouraging moment at the clinic where she works. In Module 2, she discovered that patients' difficulties with appointments were more varied than staff originally assumed. In Module 3, she learned that even a simple communication improvement depends on a system of people, resources, privacy protections, and reliable follow-through. During Module 4, she helped explore several options, including clearer reminders and a shorter preparation checklist. Patients and coworkers looking at the proposed materials seemed to understand them. The ideas felt promising, and the team had reason to be encouraged.

Now comes the question nobody can answer by looking at a prototype: Will the new approach actually improve the experience it is intended to improve? A patient might say a redesigned reminder is easier to read yet still miss an appointment because the bus is late. Someone might understand the checklist but never open the message containing it. The new wording might help people who read English confidently while creating difficulties for others. A prototype can reveal whether something seems understandable or usable in a particular setting. It does not automatically show whether an important outcome will change when the idea becomes part of daily life.

Elena's challenge is familiar even if you've never worked in healthcare. A supervisor introduces a new shift-handoff form and employees say they like it. A teacher develops a study guide and students describe it as useful. A community organization creates a new registration page and volunteers praise its appearance. A business founder shows people a product demonstration and hears that it looks exciting. In every case, the people involved may be sincere. In every case, the positive reaction is worth hearing. Yet the bigger question remains: Did anything meaningful improve, and how could we know?

This is where experimentation enters our journey. It does not require a laboratory coat, expensive software, or a personality that loves mathematics. It requires intellectual courage of a different kind: enough confidence to make a clear prediction, and enough humility to allow the world to disagree. When we experiment responsibly, we move from defending a good idea to learning what is true about it. That shift is as important for a first-time employee proposing a better process as it is for someone hoping to launch a company.

The clinic illustration is a fictional educational scenario, not a report of actual patient research. Any real test affecting clinical care, patient communication, or protected information would require authorization, privacy and accessibility review, and appropriate professional oversight. Throughout this chapter we will use imaginary data to demonstrate reasoning, never as findings about real patients.

5.1.1 Entrepreneurial Learning as Experimentation

5.1.1.1 Why a promising solution is still a collection of questions

Consider someone who has designed a better way to organize a restaurant's takeout orders. The new system uses clearer labels, a different shelf arrangement, and a short checklist before an order leaves the kitchen. During a demonstration, employees understand it immediately. The manager is excited because the old process has produced complaints. But three different claims are hiding inside this single moment: the system may be easy to understand, it may reduce incorrect orders, and it may be practical during a crowded Friday evening. These claims are related, but they are not identical. A demonstration might teach us something about understanding; a busy-shift trial might teach us about errors and workflow; a cost analysis might reveal whether the improvement is affordable. If we treat evidence about one claim as proof of the others, we may make a very confident decision for the wrong reason.

An assumption is something we are treating as true without sufficient evidence for the decision at hand. Assumptions are unavoidable. Every plan contains beliefs about people, technology, timing, costs, and consequences. The problem is not having assumptions; it is forgetting that they are assumptions. A hypothesis makes one of those beliefs explicit so it can be examined. An experiment is a planned method for gathering evidence that helps us evaluate a claim. We are not using the word to mean any activity that happens to generate feedback. We mean a process designed in advance to answer a particular question, with a reasonable explanation of what was observed and what remains uncertain.

Research supports teaching entrepreneurship as a process of disciplined learning rather than a string of inspired guesses. Shepherd and Gruber (2021) analyze the Lean Startup framework and identify validated learning, customer development, and minimum viable products as connected elements. Their argument is not that an experiment makes any venture successful; it is that the assumptions beneath a business model become more useful when entrepreneurs make them visible and investigate them. This principle transfers easily to a workplace improvement. When a team knows what it expects to change and why, it can examine the results instead of arguing about whose idea sounded best at the meeting.

5.1.1.2 The logic behind build–measure–learn

In Module 4, we learned that a prototype can be a sketch, storyboard, role-play, or early version of a product or service. The familiar build–measure–learn cycle connects such a prototype to evidence. First, we create or prepare something that exposes a specific assumption to observation. Next, we gather information about the behavior or outcome that matters. Finally, we compare what happened with what we expected and decide what we learned. The sequence repeats, but the point is not endless activity. The point is to replace an uncertain belief with a better-informed one while keeping the cost and risk of learning proportionate.

The word build can mislead beginners. It sometimes sounds as though the first step must be to create an app, purchase equipment, or build a finished product. Often the smallest useful test involves no construction at all. A supervisor might compare how two versions of a written instruction are understood by volunteers who agree to participate. A library might ask people to locate a service using a draft directory. A student exploring a consumer problem might observe how people currently accomplish a task. What matters is whether the action produces relevant, ethically collected evidence about a specific question, not whether it looks like a traditional product launch.

The intellectual model behind this approach is closely related to the scientific method. Scientists do not begin with a requirement that their favorite explanation must be confirmed. They begin with questions, propose explanations, gather evidence, and revise what they think when results do not fit. Coali, Novelli, Sirigiri, and Spina (2024) describe the scientific approach to entrepreneurial decision-making in terms of theory building—developing an explanation of how something should work—and evidence gathering—examining whether the predicted consequences appear. An introductory entrepreneurship student does not need to become a laboratory researcher to benefit from this discipline. You need to become someone who can say, “Here is what I believe, here is why, and here is what might change my mind.”

5.1.1.3 Learning is not the same as getting a favorable result

Imagine a community organization expecting that moving a workshop from Saturday morning to Wednesday evening will increase attendance. Staff conduct a small, appropriately authorized trial, but turnout remains low. Has the test failed? The proposed scheduling change may have failed to produce the expected improvement, but the organization may have learned that timing was not the primary barrier. Participants may need transportation, clearer communication, or a different workshop topic. The distinction between an unsuccessful intervention and an unsuccessful learning process is essential. A negative result can be useful when the question was meaningful, the method was responsible, and the team interprets the evidence carefully.

A favorable result deserves the same caution. If attendance rises the week the schedule changes, perhaps the new time helped. Perhaps a community partner promoted the event, the weather improved, or the previous week happened to include a holiday. The change and the improvement occurred together, but that does not automatically mean one caused the other. Correlation means two things vary together; causation means that one produced a change in the other. Experiments are valuable partly because good designs can help distinguish those possibilities, although not every educational or organizational test can establish causation with high confidence.

This difference between a desirable result and a trustworthy conclusion should change how we talk about progress. Instead of asking, “Did our idea win?” we can ask, “Did the evidence help us decide what to do next?” A team that admits it does not yet know something may be in a stronger position than a team that confidently claims success after one encouraging conversation. The first team can choose a better next step. The second may begin spending resources on a belief it has never seriously examined.

Entrepreneur-in-Action

Dropbox: An Early Demonstration Is Not a Finished Business

Before Dropbox became a familiar file-sharing service, cofounder Drew Houston encountered an ordinary frustration: he had forgotten a USB drive and could not easily access the files he needed. As the team developed a file-synchronization service, explaining the experience was itself difficult. An early demonstration video gave technically interested people a way to see what the product was intended to do. Contemporary reporting and later interviews with Houston describe how a video aimed at an early-adopter audience helped generate a substantial waiting list during the product's private-beta period (Houston, 2008; Schonfeld, 2011).

The tempting version of the story is simple: a founder made a video, people signed up, and the business was validated. The better lesson is more precise. The demonstration supplied evidence of attention and interest among a particular audience. It did not, by itself, establish whether those people would continue using the service, pay for it, recommend it beyond that community, or trust it with important files. Those were different questions requiring additional evidence. Later private-beta usage and the public launch were further stages in the learning process, not details automatically guaranteed by the popularity of the video.

For the aspiring founder, the story makes experimentation feel accessible: you may be able to learn something important before building every feature. For the student interested in a professional career, it illustrates the same logic. A short demonstration can reveal whether people understand a concept, but performance in daily operations must be studied separately. The lesson is not “make a video.” It is “match the evidence you gather to the claim you are trying to examine.”

Case sources: Houston (2008), official Dropbox launch account; Schonfeld (2011), a contemporaneous interview with Houston about the early demonstration and waitlist. Figures from early accounts differ by context and period; this case intentionally does not use a numerical signup claim as proof of product-market fit.

See all 24 entrepreneurship case studies →

5.1.2 Identifying Critical Assumptions

5.1.2.1 Every attractive idea rests on a chain of beliefs

Suppose a small college department proposes a digital appointment tool. The idea seems straightforward: people will book appointments online, staff will spend less time answering calls, and students will receive faster service. Yet several assumptions must hold for the improvement to occur. Students need reliable access to the tool and confidence using it. Appointment categories must match the reasons they seek help. Employees need accurate calendars. The system must protect information and provide accessible alternatives. And even if bookings become faster, the available staff time might remain the real bottleneck. A beautiful booking screen cannot create appointments that do not exist.

The business model work from Module 3 taught us to see an organization as a connected system. Experimentation asks which assumptions in that system are most consequential and most uncertain. A critical assumption is a belief whose failure would seriously undermine the proposed improvement or venture. A risky assumption is one that combines significant consequences with uncertainty. Sometimes those are assumptions about customer need or value; sometimes they concern technical feasibility, cost, permissions, staffing, distribution, or the ability to deliver what was promised. The exact list depends on the situation, and a well-funded organization and a student with no startup capital will face different constraints.

Leatherbee and Katila (2020) studied early-stage teams using Lean Startup methods and examined the connections among hypothesis formulation, probing, and convergence on business ideas. Their research suggests that purposeful hypothesis-based learning can be part of how teams develop ideas, while team characteristics influence how the method is used. This is a useful reminder that testing is not a mechanical recipe. It requires judgment about what matters, what resources exist, and how people respond when evidence challenges an established belief.

5.1.2.2 Value, demand, feasibility, delivery, and growth are different uncertainties

It is easy to describe every uncertainty as “Will people like it?” But different assumptions need different evidence. A value assumption concerns whether an offering creates a benefit people actually care about. A demand assumption concerns whether people will take meaningful action to obtain that benefit. A feasibility assumption concerns whether the proposed solution can be made or performed using available capabilities and constraints. A delivery assumption concerns whether the value can be provided reliably, at the promised quality and time. A financial assumption concerns whether costs, pricing, and resources can support the effort over time. A growth assumption concerns whether enough relevant users or customers can be reached and retained as the organization expands.

Imagine a mobile pet-grooming service. Pet owners might appreciate the convenience, so a value assumption could be plausible. But owners might not be willing to pay the proposed price, creating a demand problem. The vehicle may lack adequate water storage, creating an operational constraint. Appointments may take longer than expected, reducing daily capacity. Marketing may attract inquiries without producing repeat bookings. A single questionnaire asking, “Would you use this service?” cannot answer all of these questions. We must separate them before deciding what to test.

The same distinctions apply to noncommercial improvement. A school may find that students value evening advising, but the school may not have staff available to provide it. A clinic may discover that patients prefer a simpler message but cannot use the proposed communication channel. A nonprofit may pilot a service successfully with volunteer labor but struggle to sustain it. The purpose of distinguishing uncertainties is not to discourage action. It is to prevent a promising outcome in one dimension from hiding a serious weakness in another.

5.1.2.3 Finding the assumption that could change the decision

When many assumptions are uncertain, which should be examined first? The most visible assumption is not always the most important. A founder may want to test a logo because it is easy to show people, while the real threat is that a required license will be expensive or unavailable. A department might debate the layout of a new form while the main uncertainty is whether the information can legally be collected. Good experimental judgment considers decision importance, uncertainty, cost of learning, and potential harm. We begin with the question most capable of changing a consequential decision, provided the inquiry can be conducted safely and responsibly.

An assumption map can help. One axis represents uncertainty: how little do we really know? The other represents impact: how serious would it be if the assumption proved wrong? A high-impact, high-uncertainty belief deserves attention. A low-impact belief might wait, especially if resolving it would be costly. This is a thinking tool rather than a statistically precise ranking. A team might disagree about where an assumption belongs on the map. That disagreement is often productive because it exposes what each person believes and what evidence they are using.

Camuffo, Cordova, Gambardella, and Spina (2020) studied a scientific approach to entrepreneurial decision-making in a randomized field experiment. Entrepreneurs trained to formulate theories and test them made more discerning decisions and were more willing to pivot when warranted by evidence in that study. We should not generalize one study to every founder or industry, but the mechanism matters: clearer assumptions can help people distinguish an idea worth continuing from one that needs to change. We can be open-minded without becoming indecisive, and decisive without pretending that our first judgment is certain.

5.1.2.4 Avoid the comfort of testing what you already know

People often begin with easy questions because easy questions feel productive. A restaurant redesign team tests whether a new menu font is readable. A founder compares two shades of blue on a website. A manager asks whether employees prefer the new title of a process. These may be legitimate questions, but they can become a distraction if the central business or organizational assumption remains unexamined. It is possible to collect hundreds of data points and still avoid the question that could change your plan.

Imagine an entrepreneur who wants to sell healthy lunches near a college campus. She has designed attractive packaging and tested which photographs students prefer. Meanwhile, she has not investigated whether enough people are available during the proposed lunch period, whether they can pay the price, or whether food handling and location requirements can be met. Her experiments are not necessarily wrong; they are premature. The more serious uncertainty sits upstream of the details she is optimizing. Researchers call this a problem of test prioritization: inquiry is only valuable when it addresses a decision that matters.

A useful self-check is to finish this sentence: “If this assumption turned out to be false, I would have to ______.” If you cannot identify any meaningful consequence, the assumption may not deserve first place in your testing sequence. If the answer is “change the customer group,” “redesign the service,” “find different partners,” or “stop before spending more money,” you may have found a more consequential uncertainty. The challenge is not to collect the most feedback. It is to learn what will improve the next decision.

5.1.3 Constructing Testable Hypotheses

5.1.3.1 From a hopeful statement to a prediction we can examine

Listen to the difference between two statements. The first is, “Students will love our new information page.” The second is, “Among students who have agreed to participate in a usability review, at least eight of ten will locate the correct advising contact within two minutes using the revised page, without assistance.” The second statement is not guaranteed to be true, and ten participants would not establish universal performance. But it tells us who will be observed, what they will try to do, what counts as an outcome, and how we might decide whether the page deserves a larger evaluation. It is a testable hypothesis, rather than a hopeful description of enthusiasm.

A useful instructional structure is: For [defined participants], under [specified conditions], we predict [observable behavior or outcome] at [predefined level], because [the mechanism we believe will produce the result]. The mechanism is especially important. If we believe simpler directions will reduce confusion, we should explain why: fewer steps, clearer language, better navigation, or a more accessible format. Otherwise the prediction is merely a number we hope to hit. The goal is to connect what we think causes the problem to something we can reasonably observe.

Coali et al. (2024) emphasize theory building and evidence gathering as complementary pillars of scientific entrepreneurial decisions. In plain language, that means our hypothesis needs both a why and a what. The why explains the idea behind the prediction; the what tells us what should occur if the explanation is useful. This approach helps students avoid the common mistake of writing a target without a theory. “Get fifty signups” might be a marketing goal. “A clear explanation of the service will lead relevant visitors to request more information at a higher rate than the current explanation” is a claim that can guide a particular test.

5.1.3.2 Making a claim falsifiable without demanding certainty

A hypothesis is falsifiable when conceivable observations could count against it. “People like convenient services” is so broad that almost any result can be made to fit. “Among our initial target participants, at least half will complete the reservation task without help using this prototype” is much more vulnerable to disagreement from evidence. If only a few can finish, we would need to revise our confidence in the idea or in the way it was implemented. Being willing to make such a prediction can feel uncomfortable because it exposes our thinking to criticism. In an educational setting, that discomfort is part of learning.

However, rejecting or revising a hypothesis is not the same as proving its exact opposite. If only four of ten participants complete the task, perhaps the design is confusing. Perhaps the instructions were unclear, the equipment malfunctioned, or the participants did not represent the users who would normally encounter the service. A result can contradict a prediction without identifying precisely why it was wrong. That is one reason experiments must record context and limitations. Falsifiability makes a claim open to testing; it does not make interpretation automatic.

Researchers who examine the Lean Startup framework stress that a venture begins with many uncertain business-model assumptions, not a set of established truths (Shepherd & Gruber, 2021). A student can use the same logic when designing a professional improvement. Instead of asking whether the organization should keep the new system forever, ask whether one carefully bounded prediction holds under defined conditions. As the evidence accumulates, the organization can form more informed judgments about wider use. This staged approach reduces both intellectual overconfidence and unnecessary operational exposure.

5.1.3.3 Measures, denominators, and decision rules

A measure is what you observe or record; a metric is a defined quantitative measure used to understand performance. A denominator identifies the total number of relevant opportunities or participants underlying a rate. If six people completed a task, is that good? It depends on whether six people attempted it or six hundred. “Six completions” is a count. “Six completions out of ten attempts,” or 60%, is a rate with a meaningful denominator. Both can be useful, but the rate says more about how frequently the behavior occurred among those observed. A denominator does not automatically make a sample representative, so we must also say who those participants were and how they entered the study.

A decision rule states in advance how results will inform the next step. For example: “If fewer than seven of ten volunteers can locate the information, we will revise the navigation before testing with a larger group.” This is a preliminary learning threshold, not a universal scientific standard. It helps reduce the temptation to change the definition of success after seeing the result. The rule should also describe what happens when results are mixed or ambiguous, because real tests rarely deliver the simplicity of a classroom answer key.

Consider a hypothetical restaurant manager testing a revised order-checking routine. She predicts that fewer than five of every hundred eligible orders will contain a packing error during a short authorized pilot. She also defines a safety boundary: the new routine must not delay food handling in a way that creates health risks. Even if error rates improve, that second boundary matters. A measure of speed or accuracy cannot excuse harm to people. We should be explicit about both the benefit we hope to observe and the constraints that make the test acceptable.

5.1.3.4 Hypotheses about people require humility

When a prediction concerns human behavior, it's tempting to treat people as if they were switches on a machine. Change the wording, and the desired behavior should follow. Actual people bring habits, responsibilities, identities, accessibility needs, and circumstances that an experiment may not capture. A reminder that works during a calm weekday morning might be ignored by someone working two jobs. An ordering interface that suits frequent users might confuse someone using it for the first time. A training program that produces a good quiz score may not improve performance months later. A hypothesis should be specific, but the researcher should remain humble about the range of conditions in which it might apply.

This humility should affect both participant selection and interpretation. If an early study includes only classmates who already know the project, their willingness to help may be different from that of strangers or customers. If an experiment observes only people using the English-language website, its results say little about those who rely on other languages or access methods. Defining a population is not a bureaucratic detail. It is part of being truthful about whom the evidence represents.

The reward is greater clarity, not a guarantee. At the end of a good hypothesis-writing process, you should be able to explain your prediction to someone who does not share your enthusiasm and hear them say, “I understand what would count for it and what would count against it.” That moment is a form of intellectual progress. You have turned a private belief into a public question that can be investigated.

Venture TrackVT-5A · Optional

Companion to 5.1.3 Constructing Testable Hypotheses

Write a Testable Startup Hypothesis

Sam wants to offer mobile bicycle repairs to commuters who cannot easily leave their bicycles at a shop. His first statement sounds confident: “Commuters will love an on-site repair service.” But a founder cannot make a sound launch decision from a belief that broad. What does love mean? Which commuters? What would they have to do for the idea to deserve further investigation?

Watch Sam rewrite the belief into something he can examine: “Among twenty adult bicycle commuters recruited because they have experienced a repair disruption in the previous three months, at least six will request information about a clearly described, realistically priced mobile repair appointment within seven days.” The numbers are illustrative planning thresholds, not market forecasts. The six requests would be preliminary evidence of interest in that defined group, not proof of six paying customers.

A founder's practical hypothesis card

FieldSam's preliminary entry
Customer groupAdult commuters with recent repair disruptions
Claimed needRepair without transporting a disabled bicycle
Test behaviorVoluntary request for appointment information
ContextHonest explanation of proposed service and estimated price
Preliminary thresholdSix requests among twenty relevant participants
LimitationInterest requests are not booked, completed, or paid repairs

Sam should also ask whether he can safely and legally perform the work, obtain appropriate coverage, and meet appointment promises. Those are separate assumptions, not things a positive interest test can erase. The tactical skill is to give one belief a defined audience, observable behavior, honest conditions, and a realistic interpretation.

5.1.4 Experiment Types and Their Trade-offs

5.1.4.1 The right experiment depends on the question

Imagine that a college cafeteria is considering changing its lunch service. The team wants to know whether students understand the ordering options, whether waiting time can be reduced, and whether enough people would choose the new menu to justify the additional cost. These are three distinct questions. A short observation or usability session could reveal whether signs and menus are understandable. A timed operational pilot could show how the service performs during a selected period. Transaction records and carefully interpreted purchasing behavior could help evaluate demand. Conducting a survey might provide context, but it would not automatically establish how many customers would purchase once real prices and waiting times were involved.

Different methods produce different kinds of evidence. Interviews help us understand experiences and explanations; observation reveals behavior in context; surveys gather structured responses from a selected group; usability tests show how people interact with a design; landing-page tests may measure attention or stated interest; concierge tests allow a human to deliver a proposed service manually; pilots examine an offering on a limited scale; and controlled comparisons, including A/B tests, compare outcomes under different conditions. These are not rungs on a universal ladder from bad to good. The appropriate choice depends on the uncertainty, ethics, cost, access, and strength of conclusion required.

The methodological literature on online experimentation explains why controlled experiments can be powerful for measuring causal effects, but also why running and interpreting them creates operational and statistical challenges (Kohavi et al., 2009; Bojinov & Gupta, 2022). We should neither worship nor dismiss them. A carefully conducted interview can reveal an overlooked reason for a problem that a large A/B test would never explain. A controlled comparison can answer whether a defined change affects a measured outcome more convincingly than a collection of personal impressions. A mature decision often draws on both kinds of knowledge.

5.1.4.2 Observation, interviews, and surveys: learning what is happening

An interview invites a person to describe experiences, choices, and constraints. In Module 2 you learned to ask about what people actually did rather than steer them toward compliments about a proposed solution. The same principle matters when selecting an experiment. If the question is why students miss an evening service, a conversation about schedules and transportation may be more informative than a multiple-choice poll about favorite appointment times. Interviews offer depth, but a few interviews cannot tell us how common a pattern is across an entire community without additional evidence.

Observation can show us behaviors that people may struggle to describe. A supervisor might notice that employees skip a checklist not because they dislike it but because it is stored far from where the work occurs. Observing the sequence could reveal a design problem rather than a training problem. The limitations are important: the observer may misunderstand the context, people may behave differently while being watched, and participants' privacy and workplace rights must be respected. We should seek permission and avoid collecting unnecessary personal information, especially in sensitive settings.

Surveys can help summarize responses from more people, but they require careful wording and sampling. Imagine asking, “Don't you agree our new service is easier to use?” The question effectively announces the answer you want. A more neutral question asks what the participant experienced, or presents balanced response options. Even then, a survey completed mostly by people who love a program may give a different picture from one answered by those who stopped using it. The number of responses matters, but how respondents were recruited can matter just as much.

5.1.4.3 Mockups, landing pages, concierge tests, and pilots: testing different promises

A mockup is a representation of a design, not necessarily an operational product. It can reveal whether an explanation is clear or a process makes sense to intended users. A landing page is a web page designed around one action—such as requesting more information, joining a waitlist, or scheduling a conversation. It may reveal whether a defined message motivates visitors to take that action. It does not, by itself, prove willingness to pay or long-term satisfaction. A concierge test is a small-scale demonstration in which people manually provide the value that a later system might automate. It can reveal actual service challenges, but it may depend on expensive personal effort that would be hard to sustain.

A pilot introduces a process or service on a limited basis with safeguards and defined boundaries. Imagine an authorized library pilot that allows a small set of consenting patrons to try a redesigned pickup procedure. The pilot can reveal operational consequences under actual conditions: timing, staff workload, accessibility problems, and unexpected exceptions. But a successful one-week pilot may not represent end-of-semester demand or a crowded public event. The closer a test moves toward real operations, the more carefully the organization must think about risk, cost, participant information, and the effects on people who do not participate.

These methods are useful because they allow a team to match the cost of learning to the importance of the uncertainty. An inexpensive paper walkthrough may be appropriate before developing software. A small supervised operational trial may be appropriate before changing a department-wide process. A founder may need a genuine purchase or deposit signal to investigate willingness to pay—but must be truthful about what can be delivered, what any payment means, and whether refunds or consumer protections apply. We are seeking authentic evidence, not permission to deceive people into giving us data.

5.1.4.4 Controlled comparisons: when the alternative matters

Suppose a school changes how it explains scholarship eligibility and sees an increase in completed applications. The team may be pleased, but how much of the change came from the new explanation? Perhaps the school also increased outreach or the scholarship deadline was approaching. A comparison group helps answer this question by showing what happened under another condition. In an A/B test, eligible participants are assigned to two alternatives—for example, existing wording and revised wording—and the resulting behavior is compared. When assignment is genuinely random and other conditions are managed, the design can reduce some competing explanations for observed differences.

A comparison does not have to be a digital test. A manufacturing group might compare two approved work instructions under similar controlled conditions. A nonprofit might use a carefully designed staged implementation to study two communication approaches. Yet many real-world settings cannot fairly or safely withhold a service, assign people at random, or isolate all influences. Some outcomes also require much more time than a short trial permits. The correct response is not to ignore evidence, but to match the strength of the claim to what the design actually supports.

Kohavi et al. (2009) describe how online controlled experiments can reveal effects that expert intuition misses while requiring careful attention to setup, measurement, and analysis. Their technical research comes from digital environments where large numbers of interactions can sometimes be measured. An introductory class should understand the underlying reasoning without assuming every small business or college department can reproduce those conditions. A classroom comparison involving eight volunteers is not equivalent to a well-powered randomized trial involving thousands of users. Both can teach us something, but not the same thing.

5.1.5 Ethical and Responsible Testing

5.1.5.1 People are not just data points

Imagine that an employee believes a revised appointment message could improve clinic attendance. She wants evidence quickly, so she proposes sending different medical instructions to different patient groups and checking who responds. The idea may sound like a straightforward experiment. Yet the participants are real people with clinical needs, privacy rights, language needs, and reasons to trust the organization. A misleading message could cause missed care. A poorly protected spreadsheet could expose sensitive information. A decision to include only people who speak English could create unequal benefits. Before the team thinks about measurement, it must ask whether the proposed activity is permitted, fair, and safe.

Ethical experimentation involves designing learning activities that respect the people affected and limit avoidable harm. In human-subjects research, the Belmont Report articulates three widely influential principles: respect for persons, beneficence, and justice (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1979). Those principles do not automatically make every workplace usability test a federally regulated research study. They do, however, provide a strong ethical lens: people should be treated as persons with agency, potential benefits should be weighed against risks, and burdens and benefits should not be distributed unfairly.

In a healthcare, education, employment, or government setting, an idea may require formal approval or specialized review before any test begins. The absence of a requirement for a formal institutional review board in a particular setting does not remove other legal, professional, privacy, accessibility, or organizational obligations. Students should never assume that a classroom activity authorizes experimentation on patients, employees, minors, vulnerable populations, or people who depend on a service. Responsible entrepreneurial initiative includes knowing when to seek oversight and when to use a simulated case rather than conduct a live test.

Informed consent means that people receive enough understandable information about a proposed activity and its relevant risks to make a meaningful, voluntary decision about participating. Consent is not simply a box checked at the end of a long form. The person should know what will happen, what information will be collected, whether there are alternatives, and whom to contact with concerns. In some organizational settings consent takes different forms or formal requirements apply; an instructor should never substitute a textbook example for institutional rules.

Truthful representation is especially important in startup experiments. A landing page that clearly says “Join our interest list for a service being explored” is different from one that falsely promises a functioning product or invents testimonials. A mock booking interface may measure clicks, but it must not mislead participants into believing an urgent service has been secured when it has not. Some legitimate research protocols use carefully reviewed forms of incomplete disclosure, but that is not a general permission for student entrepreneurs to deceive customers. When uncertainty exists, choose the more transparent method.

Respect also means avoiding pressure. A student who interviews classmates should recognize that people may want to be helpful and may feel uncomfortable saying no. A manager testing a new employee procedure holds authority over the people affected, which complicates voluntariness. A healthcare worker should not presume that patients can freely decline activities introduced by the clinic. The investigator must consider these relationships, explain boundaries, and choose methods that do not exploit dependence or unequal power.

5.1.5.3 Privacy, accessibility, fairness, and stopping conditions

Data should be treated as a responsibility, not a souvenir. If a student needs only a count of how many people could locate a campus web page, they probably do not need names, phone numbers, medical histories, or personal stories attached to those counts. Data minimization means collecting and retaining only information genuinely necessary for the stated purpose. Even anonymized information can sometimes become identifiable when combined with other details, especially in small communities. Privacy planning includes where information is stored, who can access it, how long it is kept, and how findings are shared.

Accessibility and fairness also affect experimental validity. If a test is conducted only on the latest smartphone, the results may say little about people who use older devices, assistive technology, shared computers, or limited bandwidth. A procedure that helps one group may disadvantage another. Researchers should therefore ask whose experience is visible in the evidence and whose is absent. This is not an argument that every small exploratory test must represent the whole population; it is an argument that conclusions and implementation decisions must acknowledge that limitation.

Finally, responsible experiments need stopping conditions. If a process produces unexpected safety concerns, coercion, a significant privacy risk, or harm to service users, it should be paused and referred to appropriate authorities rather than continued merely to reach a sample-size target. Even without obvious harm, a test that is clearly failing may need to end early—although changing a planned experiment midstream complicates what its numerical results can establish. The priority is human well-being and honest learning, not protecting an experimental design at any cost.

5.1.5.4 The real ethical question is bigger than “Can we test it?”

A company may have the technical capacity to monitor customers' clicks in extraordinary detail. A college may have access to data about students' behavior. A manager may be able to assign different working conditions to teams. Yet having the means to collect data is not the same as having the authority or justification to do so. The ethical question is not only whether an experiment could produce a useful result. It is whether the knowledge is worth seeking in this way, with these people, under these conditions, and with adequate protection against foreseeable harms.

A useful exercise in judgment is to imagine explaining the proposed test to the person most likely to bear its risks. Would the explanation be clear and respectful? Could that person reasonably decline or ask questions? Are the benefits being claimed larger than the evidence can justify? Would the organization be willing to disclose the method and limitations of the results? These questions do not eliminate ethical dilemmas, but they can prevent a team from using the language of innovation to excuse practices it would otherwise recognize as unfair.

The best experiment is not the most dramatic demonstration or the largest spreadsheet. It is one that addresses a meaningful uncertainty, produces interpretable evidence, and treats the people involved with dignity. That is a standard worth carrying into any profession—and a standard worth establishing before the excitement of a new venture begins to influence judgment.

Emerging practices

AI Can Suggest a Test, but It Cannot Make the Evidence True

Generative AI tools can help brainstorm assumptions, draft neutral questions, propose outcome measures, and identify possible confounding factors. Digital analytics platforms can also collect behavioral data at a scale that small organizations once found difficult to imagine. These capabilities create opportunities, but they can also make weak experiments look impressively scientific. AI-generated participant responses are synthetic text, not real customer evidence. Automated dashboards may contain tracking errors, biased samples, or metrics that do not represent the outcome people care about. Before relying on any tool, ask where the observations originated, whether data collection was authorized, whether the measure is appropriate, and whether a human can inspect the reasoning. Faster analysis is valuable only when it supports more trustworthy decisions.

References

Sources cited in this section. The full Module 5 reference list closes section 5.3.

Bojinov, I., & Gupta, S. (2022). Online experimentation: Benefits, operational and methodological challenges, and scaling guide. Harvard Data Science Review, 4(3). https://doi.org/10.1162/99608f92.a579756e

Camuffo, A., Cordova, A., Gambardella, A., & Spina, C. (2020). A scientific approach to entrepreneurial decision making: Evidence from a randomized control trial. Management Science, 66(2), 564–586. https://doi.org/10.1287/mnsc.2018.3249

Coali, A., Novelli, E., Sirigiri, A., & Spina, C. (2024). A scientific approach to decision-making: Key tools and design principles. Journal of Business Venturing Design, 3, Article 100023. https://doi.org/10.1016/j.jbvd.2024.100023

Houston, D. (2008, September 11). Dropbox launches to the public! Dropbox Blog. https://blog.dropbox.com/topics/company/dropbox-launches-to-the-public

Kohavi, R., Longbotham, R., Sommerfield, D., & Henne, R. M. (2009). Controlled experiments on the web: Survey and practical guide. Data Mining and Knowledge Discovery, 18, 140–181. https://doi.org/10.1007/s10618-008-0114-1

Leatherbee, M., & Katila, R. (2020). The lean startup method: Early-stage teams and hypothesis-based probing of business ideas. Strategic Entrepreneurship Journal, 14(4), 570–593. https://doi.org/10.1002/sej.1373

National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. (1979). The Belmont report: Ethical principles and guidelines for the protection of human subjects of research. U.S. Department of Health and Human Services. https://www.hhs.gov/ohrp/regulations-and-policy/belmont-report/read-the-belmont-report/index.html

Schonfeld, E. (2011, November 1). (Founder stories) How Dropbox got its first 10 million users. TechCrunch. https://techcrunch.com/2011/11/01/founder-storie-how-dropbox-got-its-first-10-million-users/

Shepherd, D. A., & Gruber, M. (2021). The lean startup framework: Closing the academic–practitioner divide. Entrepreneurship Theory and Practice, 45(5), 967–998. https://doi.org/10.1177/1042258719899415