As it turns out there are others doing my job for me by teaching rationality. Julia Galef does so on her YouTube channel, podcast, and through her organization. Subjects include exposing sphexish thinking and introduction to Bayesianism.
One thing I have always done when finding new evidence to support a hypothesis of mine is to ask if the evidence was likely to exist anyways even if the hypothesis was false. It happens so automatically, that I don’t notice it. For example, suppose I have indigestion after drinking a lot of milk. Does it mean I’ve finally become lactose-intolerant? Probably not if I normally have indigestion even when I don’t drink milk. This is the type of thing Julia teaches.
Another concept I was always aware of but had no name to put to it was “orphan beliefs.” This is what happens when you adopt a belief based on certain evidence, but then fail to update your beliefs when the old evidence is invalidated. It has happened before that I have planned on making an egg sandwich, decided that a turkey sandwich would take less effort, put the bread in the toaster, and then stupidly dropped an egg in the frying pan anyways. It seems the brain does not work like a Bayesian net. However, a habit I have had for as long as I can remember is to sit and think about things I already “know” from different perspectives, describing them from basis to conclusion and back again down multiple branches. When encountering old ideas I no longer believe, I reevaluate the entire structure. Thus, I catch these “orphan beliefs” eventually.
Logic. Love. Liberty.