Using Science to Imagine an Alternative Reality

In one of the reader comments, a woman asked an interesting question. She explained that she felt a lump in her breast at age 38 and decided to have a mammogram. The mammogram office had to call her insurance company to pre-approve the screening, which it eventually did. A few months later, she had been diagnosed with stage-1 breast cancer. Imagine what would have happened if … my insurance would have denied covering the mammogram? she asked. Presumably, she wouldn’t have had the mammogram, and her cancer would have been allowed to progress to a more dangerous, even deadly, state.

The reader’s question is known as a counterfactual—it imagines what could have resulted if the preceding facts were different. We all engage in this type of thinking, perhaps wondering how our life would be altered if we went to a different college or followed a different career path. Usually, these sorts of questions are imponderables, asked rhetorically or just for fun.

But counterfactuals also have practical application. Consider the mammogram example, only this time the patient’s insurance company didn’t approve the medically appropriate screening and she ended up dying from breast cancer. Had the scans happened, the cancer may have been found and the patient’s life saved. Her surviving spouse is now suing the insurance company and the court wants to know the counterfactual probability that her life would have been saved by the screenings.

That precise fact pattern lies at the heart of a new paper by Tuck professor Raghav Singal, an expert in analytics, decision-making, and modeling. In Counterfactual Analysis in Dynamic Latent State Models, Singal and coauthor Martin Haugh of Imperial College build a novel optimization-based framework to help us home in on answers to previously unanswerable questions.

Researchers started working on the counterfactual problem 30 years ago, in a paper by Alexander Balke and Judea Pearl (who won the Turing Prize in 2011). Balke and Pearl studied a system that was static and observable—the state of the system didn’t change, and the relevant data in the system was known. For example, a person has a headache and takes a Tylenol for relief. The headache persists, and the person wonders if the outcome would have been different if they had taken ibuprofen. Even under those relatively simple conditions, it’s still impossible to precisely define the counterfactual probability of the ibuprofen curing the headache. So Balke and Pearl pioneered a method that computed upper and lower bounds for the counterfactual probability of ibuprofen working, say, between 15 and 40 percent. This field of research has been active in the intervening decades.

The counterfactual that Singal and Haugh chose to study—the progression of breast cancer over time—is much more complicated than the initial system studied by Balke and Pearl. For one, the patient’s health is always changing; it’s dynamic. Moreover, you never really know the patient’s true health. Even if the patient gets a mammogram every six months, the results aren’t perfectly accurate. She might have cancer, but the scan didn’t detect it, or she might not have cancer, but the scan gave a false positive. The technical term for this state is latent, or hidden.

In this type of system, the precise counterfactual survival probability of the patient receiving regular mammograms (given she did not survive under a different screening policy) is unknown. There just isn’t enough information. Like Balke and Pearl, Singal and Haugh have to settle for computing upper and lower bounds of probability. They do this in the paper using tools such as Bayes’ Rule and optimization—two concepts Singal teaches in the core courses Analytics I and II.

What our framework does is use all the information we have and thinks of all the possibilities in terms of the causal mechanisms, and each possibility maps to an estimate of the probability, Singal explains. But there are infinite such possibilities, so we solve two optimization problems. The first one computes the maximum probability, and the second computes the minimum probability.

Calibrating their model using the open-source data from the University of Wisconsin Breast Cancer Simulator and the SEER database from the National Cancer Institute, Singal and Haugh show that their framework can possibly get the counterfactual answer into a fairly narrow range, such as 70 to 90 percent, which is desirable. However, if it does not, one can go a step beyond and make that range narrower by injecting domain-specific knowledge into the system. We show that such knowledge can easily be embedded as linear constraints in the optimization model, Singal explains, and doing so results in a much narrower range since one shrinks the set of possibilities. Not being medical professionals, Singal and Haugh don’t take a stand on what is the appropriate domain specific knowledge to use in this case. However, they do argue that as soon as you start embedding the right knowledge into the model, the range of probabilities will shrink.

For now, the paper is generating a lot of academic interest, and was accepted at the International Conference on Machine Learning (ICML). Singal and Haugh are preparing a version to submit to a peer-reviewed journal. Practically speaking, the application of the model remains to be seen. Perhaps it could be used in court to help victims of medical malpractice prove their case. As the first paper to bound a counterfactual in a dynamic latent-state model via a principled optimization-based framework, Singal is excited to make headway in the notoriously difficult intersection of causality and optimization. It’s a very challenging problem that we’ve been able to get a handle on, he says.