How should evidence be built?

Bruce Charlton has a post on “Evidence Based Medicine” (EBM), among other things, which often involves large randomized, controlled trials (RCTs), in “An Evidence Free World.”

I largely agree. The moniker is typically misleading, as EBM typically involves ignoring large amounts of evidence. Rather, a large RCT is one tool, which will be properly used in certain cases to investigate certain phenomena. Like any tool, a RCT may or may not be the best tool to use at a given time, and even when it is, may be constructed or implemented to varying degrees of effectiveness.

Similarly, large RCTs seem like something that isn’t the start of evidence, but typically a tool used later on.

Instead, the beginning is probably an individual noticing something unusual. Then, collecting anecdotes (similar experiences), either in one’s own experience or others’. Then, if it makes sense given the subject area, starting to construct a more formal probability framework to see if such experiences are inexplicable given current standardly posited cause-and-effect relationships. Something like a large RCT would probably most usefully come later, be carefully designed, and be just one more (although often important) line of evidence used to draw a conclusion about certain cause-and-effect relationships.

I.e., a large RCT is more the tail than the dog of evidence. They probably should come once we already have good reasons to believe there is a causal relationship, and where we understand the causal area sufficiently to not design a RCT that is misleading.

Science is something anyone can do (as noted above, most science starts with someone noticing something that isn’t quite right or is unusual). A modern notion is that science is the work of very expensive trials and experiments – but more often than not, these trials or experiments are not only not that effective at figuring out cause-and-effect relationships, but waste a huge amount of money (something Charlton suggests as well).

My suspicion is that a reliance on large RCTs is more the product of the professionalization and attempted bureaucratization of certain areas of inquiry, where an ‘amateur‘ mindset and motivational structure would be often better suited (and be more flexible in terms of discerning which tools are best suited for figuring out a given cause-and-effect relationship).

Leave a Reply

Your email address will not be published. Required fields are marked *