Conducting AB testing on Customers

The Ethical Way


We have come a long way since the days of horrific human experiments during World Wars, the Stanford prison experiment, the Guatemalan STD Study and many more. where inhumane treatments were all in the name of science. However, we still have much to learn, with incidents like the clinical trial disaster in France and Facebook’s emotional and psychological experiments of recent years violating the rights of persons and serving as a clear reminder to constantly keep our ethics sharply up front.

As data scientists, we are always experimenting – not only with our models or formulas but also with the responses from our customers. AB tests or randomized experiments may require human subjects, who are willing to undertake a trial or treatments such as seeing certain content when using a Web app, or undergoing a certain exercise regime.

What may initially seem like a harmless experiment, might cause harm or distress. For example, Facebook’s experiment of provoking negative emotions from some users and positive emotions from others could have serious consequences. If a user, who was experiencing emotional distress happened to have seen content which provoked negative feelings, it could possibly spur on a tragic event such as physical harm. Careful understanding of our experiments and our test subjects may prevent inappropriate testing required prior to implementing our research, or products and services. Consent is the best tool to assist data scientists working with data generated by people. Similarly, to guidelines for clinical trials, it’s informed consent specifically that is needed to avoid potential unintended consequences of experiments.

If an organization specializing in exercise science accepted participation from a person who has a high risk of heart failure, and did not ask for a medical examination before conducting the experiment, then the organization is potentially liable for the consequences.

Often a simple, harmless AB test might not be as simple and harmless as it looks. So how do we ensure we are not putting our human subjects’ well-being and safety in danger when we conduct our research and experiments?

A first port of call is using informed user consent. This doesn’t mean pages and pages of legal jargon on sign up or being vague in an email when reaching out for volunteers for your study. This could rather be a popup window or email that is clear on the purpose of the experiment and any warnings or potential risks the person needs to be aware of. Depending on how intense the treatment is, a medical or psychological examination is a good idea to ensure that the participant can cope with the given treatment. Being unaware of people’s vulnerabilities can lead to unintended consequences. This can be avoided through clearer warnings, or the next level up which may be online assessments, or even expert examinations.

The next step in ensuring your AB test or experiment runs smoothly and ethically is making sure you understand local and federal regulations around conducting research experiments on humans. In the US, these regulations have been outlined above. The regulations mainly look at:

  • Informed consent, with a full explanation of any potential risks to the subject.
  • Providing additional safeguards for vulnerable populations such as children, mentally disabled people, mentally ill people, economically disadvantaged people, pregnant women, and so on.
  • Government-funded experiments need the approval of Institutional Review Board or an independent ethics committee before conducting experiments.

During the AB test or experiment, it’s also a good idea to regularly check-in and see how your subjects are responding to the treatments, not only for the purpose of scientific research, but also to quickly solve any health or well-being issues. This could be in the form of a short popup survey or email to check if the user is safe and well, or face to face consulting. Also, having an opt-out option allows the subject to take control if they feel their health or well-being is at risk. Having some people opt-out might seem inconvenient for your study, but a serious or tragic incident as a result of a participant having to go through the full course of the treatment is a far worse outcome.

Observational studies might be a good alternative if the above steps are in no way feasible for your experiment. Observational studies are limited when making conclusions, and only real experiments allow you to make confident conclusions from the data. However, in some situations, it is not possible nor ethical to force treatments onto subjects. For example, it’s not ethical inject cancer cells into random subjects, but you can study cancer patients with the inherited attributes you are looking for to help with your research.

It is understood that there can be some overhead in carefully preparing, setting up and following ethical guidelines for an experiment or AB test. However, the serious consequences of not doing it properly, as well public distrust, will only lead to a reluctance to share data, hindering our ability to effectively do our work.

Data Science Blog

Authors

Rebecca Merrett

Writes technical blogs and other content for Wargaming Sydney/BigWorld Technology.

Rebecca MerrettLinkedIn
Raja Iqbal

Raja is the CEO and Chief Data Scientist at Data Science Dojo. He has worked at Microsoft Bing and Bing Ads in various research and development roles in data science and machine learning.

Raja IqbalLinkedIn

Follow us on: