Defining and Assessing Good Judgment

My 2005 book, Expert Political Judgment: How Good Is It? How Can We Know?, traces the evolution of this project. It reports a series of relatively small scale forecasting tournaments that I started in 1984 and wound down by 2003. A total of 284 experts participated as forecasters at various points. They came from a variety of backgrounds, including government officials, professors, journalists, and others, and subscribed to a a variety of political-economic philosophies, from Marxists to libertarians.

Cumulatively they made 28,000 predictions bearing on a diverse array of geopolitical and economic outcomes.

The results were sobering. One widely reported finding was that forecasters were often only slightly more accurate than chance, and usually lost to simple extrapolation algorithms. Also, forecasters with the biggest news media profiles tended to lose to their lower profile colleagues, suggesting a rather perverse inverse relationship between fame and accuracy.

The expert political judgment project also compared the accuracy track records of "foxes" and "hedgehogs" (two personality types identified in Isaiah Berlin’s 1950 essay The Hedgehog and the Fox). The more theoretically single-minded hedgehogs performed less well, especially on long-term forecasts within the domain of their expertise, than the more eclectic foxes.

These findings received considerable media attention and came to the attention of the Intelligence Advanced Research Projects Activity (IARPA) inside the United States intelligence community—a fact that was partly responsible for the 2011 launch of a four-year geopolitical forecasting tournament that engaged tens of thousands of forecasters and drew over one million forecasts across roughly 500 questions of relevance to U.S. national security. From 2011 to 2015, Barbara Mellers and I served as co-principal investigators of the Good Judgment Project (GJP), a research collaborative that emerged as the wide-margin winner of the IARPA tournament.

The aim of the tournament was to improve geo-political and geo-economic forecasting. Illustrative questions included “What is the chance that a member will withdraw from the European Union by a target date?” or “What is the likelihood of naval clashes claiming over 10 lives in the East China Sea?” or “How likely is the head of state of Venezuela to resign by a target date?” The tournament challenged GJP and its competitors at other academic institutions to come up with innovative methods of recruiting gifted forecasters, methods of training forecasters in basic principles of probabilistic reasoning, methods of forming teams that are more than the sum of their individual parts and methods of developing aggregation algorithms that most effectively distill the wisdom of the crowd.

Among the more surprising findings from the tournament were:

1. the degree to which simple training exercises improved the accuracy of probabilistic judgments as measured by Brier scores;
2. the degree to which the best forecasters could learn to distinguish many degrees of uncertainty along the zero to 1.0 probability scale (many more distinctions than the traditional 7-point verbal scale used by the National Intelligence Council);
3. the consistency of the performance of the elite forecasters (superforecasters) across time and categories of questions;
4. the power of a log-odds extremizing aggregation algorithm to out-perform competitors; and
5. the apparent ability of GJP to generate probability estimates that were "reportedly 30% better than intelligence officers with access to actual classified information."

These and other findings are laid out in the 2015 book, “Superforecasting.” My co-author Dan Gardner and I stress that good forecasting does not require powerful computers or arcane methods. It involves gathering evidence from a variety of sources, thinking probabilistically, working in teams, keeping score, and being willing to admit error and change course. We also suggest that the public accountability of participants in the later IARPA tournament boosted performance. Apparently, “even the most opinionated hedgehogs become more circumspect” when they feel their accuracy will soon be compared to that of ideological rivals.

I see forecasting tournaments as a possible mechanism for helping intelligence agencies escape from blame-game (or accountability) ping-pong in which agencies find themselves whipsawed between clashing critiques that they were either too slow to issue warnings (false negatives such as 9/11) and too fast to issue warnings (false positives). Tournaments are ways of signaling that an organization is committed to playing a pure accuracy game –and generating probability estimates that are as accurate as possible (and not tilting estimates to avoid the most recent “mistake”).

The Good Judgment research program continues to recruit new forecasters for new forecasting tournaments at www.goodjudgmentproject.com.

Tetlock, P. E. & Gardner, D. (2015). Superforecasting: The Art and Science of Prediction. New York: Crown.

Tetlock, P.E. (2005). Expert political judgment: How good is it? How can we know? Princeton: Princeton University Press.

Tetlock, P.E., Mellers, B., Rohrbaugh, N., & Chen, E. (2014). Forecasting tournaments: Tools for increasing transparency and the quality of debate. Current Directions in Psychological Science, 23(4), 290-295

Mellers, B.A., Stone, E., Murray, T., Minster, A., Rohrbaugh, N., Bishop, M., Chen, E., Baker, J., Hou, Y., Horowitz, M., Ungar, L., & Tetlock., P.E. (in press). Identifying and Cultivating “Superforecasters” as a Method of Improving Probabilistic Predictions. Perspectives in Psychological Science.

Merkle, E., Steyvers, M., Mellers, B. & Tetlock, P.E. (in press). Item Response Models of Probability Judgments: Application to a Geopolitical Forecasting Tournament. Decision.

Tetlock, P. E. & Mellers, B. (2014), Judging political judgment. Proceedings of National Academy of Sciences, 111 (32), 11574-11575.

Suedfeld, P & Tetlock, P.E. (2014). Integrative complexity at Forty: Steps toward resolving the scoring dilemma. Political Psychology, 35(5), 597-601.

Satopaa, V.A., Baron, J., Foster, D.P., Mellers, B.A., Tetlock, P.E., & Ungar, L.H. (2014). Combining multiple probability predictions using a simple Logit model. International Journal of Forecasting, 30(2), 344-356.

Baron, J., Ungar, L. Mellers, B. & Tetlock, P.E. (2014). Two reasons to make aggregated probabilities more extreme. Decision Analysis 11(2), 133-145.

Satopaa, V., Jensen, S., Mellers, B.A., Tetlock, P.E. & Ungar, L. (2014). Probability aggregation in the time-series dynamic, hierarchical modeling of sparse expert beliefs. Annals of Applied Statistics, 8(2), 1256-1280.

Inchauspe, J., Atanasov, P., Mellers, B, Tetlock, P. E., & Ungar, L. (2014). A Behaviorally Informed Survey-Powered Market Agent. Prediction Markets.

Tetlock, P.E., Metz, S.E., Scott, S., & Suedfeld, P. (2014). Integrative complexity coding raises integratively complex issues. Political Psychology, 35(5), 625-634.

Mellers, B. A., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., Scott, S., Moore, D., Atanasov, P., Swift, S., Murray, T., & Tetlock, P. (2014). Psychological strategies for winning a geopolitical tournament. Psychological Science, 25(5), 1106-1115.

Mellers, B. A., Ungar, L., Fincher, K., Horowitz, M., Atanasov, P., Swift, S., Murray, T., & Tetlock, P. (2014). The Psychology of Intelligence Analysis: Drivers of Prediction Accuracy in World Politics. Journal of Experimental Psychology: Applied.

Tetlock, P.E., Horowitz, M., & Herrmann, R. (2012). Should systems thinkers accept the limits on political forecasting—or push the limits? Critical Review.

Horowitz, M. & Tetlock, P.E. (2012) Trending Upward: How the intelligence community can better see into the future.” ForeignPolicy.com. September 6, 2012:

Tetlock, P. E., & Mellers, B.A. (2011). Intelligent management of intelligence agencies: Beyond accountability ping-pong. American Psychologist, 66(6), 542-554.

Committee on Behavioral and Social Science Research to Improve Intelligence Analysis for National Security (2011). Intelligence analysis for tomorrow: Advances from the behavioral and social sciences. The National Academies Press. Washington D.C.

Tetlock, P. E. (2010). Second thoughts about expert political judgment. Reply to symposium on “Expert political judgment: How good is it? How can we know?” Critical Review, 22(4), 467-488. 

Tetlock, P.E. (November-December, 2010). Experts all the way down.  The National Interest, 76-86. 

Tetlock, P.E. (September-October, 2009). Playing tarot on K Street, The National Interest, 57-67. 

Goldgeier, J., & Tetlock, P.E. (2007). Psychological approaches complement – rather than contradict – international relations theories. In C. Reus-Smit & D. Snidal (Eds.) The Oxford handbook of international relations. New York: Oxford University Press. 

Tetlock, P.E. (2007). Psychology and politics: The challenges of integrating levels of analysis in social science. In E.T. Higgins & A. Kruglanski (Eds.), Social psychology: Handbook of basic principles. New York: Guilford. 

Parker, G., & Tetlock, P.E. (2006). Counterfactual history: Its advocates, its critics and its uses. In P.E. Tetlock, R.N. Lebow & G. Parker (Eds) (2006). Unmaking the West: What-if scenarios that rewrite world history. Ann Arbor, MI: University of Michigan Press. 

Tetlock, P.E., & Parker, G. (2006). Counterfactual thought experiments: Why we can’t live with them and how we must learn to live with them. In P.E. Tetlock, R.N. Lebow & G. Parker (Eds.) (2006). Unmaking the West: What-if scenarios that rewrite world history. Ann Arbor, MI: University of Michigan Press. 

Tetlock, P.E., & Henik, E. (2004). Theory-driven versus imagination-driven reasoning about what could have been: Are we fated to be prisoners of our preconceptions? In D. Mandel, D. Hilton, & P. Catellani (Eds), The psychology of counterfactual thinking. London: Routledge. 

Tetlock, P.E. (2003). Correspondence and coherence indicators of good judgment. In D. Hardman & L. Macchi (Eds.). Thinking: Psychological perspectives on reasoning, judgment and decision making. Cambridge University Press. 

Weber, S., & Tetlock, P.E. New Economy: The Pentagon’s plan for futures trading. New York Times, Business Section, August 11, 2003, C3. 

Tetlock, P.E. (2002). Cognitive biases in path-dependent systems: Theory driven reasoning about plausible pasts and probable futures in world politics. In T. Gilovich, D.W. Griffin, & D. Kahneman. (Eds.). Inferences, heuristics and biases: New directions in judgment under uncertainty. New York: Cambridge University Press.

Tetlock, P.E. (2002). Exploring empirical implications of deviant functionalist metaphors: People as intuitive politicians, prosecutors, and theologians. In T. Gilovich, D.W. Griffin, & D. Kahneman (2000). Inferences, heuristics and biases: New directions in judgment under uncertainty. New York: Cambridge University Press.

Tetlock, P.E. (2002). Social-functionalist frameworks for judgment and choice: The intuitive politician, theologian, and prosecutor. Psychological Review, 109, 451-472. 

Tetlock, P.E., & Mellers, B. (2002). The great rationality debate: The impact of the Kahneman and Tversky research program. Psychological Science, 13, 94-99. 

Suedfeld, P., & Tetlock, P.E. (2001). Individual differences in information processing. In A. Tesser & N. Schwartz (Eds.), Blackwell international handbook of social psychology: Intra-individual processes, (Vol. 1). London: Blackwell Publishers. 

Tetlock, P.E. (2001). The virtues of cognitive humility: For us as well as them. In R. Gowda & J. Fox (Eds.), Judgments, decisions, and public policy: Behavioral decision theoretic perspectives and applications. New York: Cambridge University Press. 

Tetlock, P.E., & Lebow, R.N. (2001). Poking counterfactual holes in covering laws: Cognitive styles and historical reasoning. American Political Science Review, 95, 829-843. 

Tetlock, P.E. (1999). Theory-driven reasoning about possible pasts and probable futures: Are we prisoners of our preconceptions? American Journal of Political Science, 43, 335-366. 

Tetlock, P.E. (1998). Close-call counterfactuals and belief system defenses: I was not almost wrong but I was almost right. Journal of Personality and Social Psychology, 75, 639-652. 

Tetlock, P.E. (1998). Social psychology and world politics. In S. Fiske, D. Gilbert, & G. Lindzey (Eds.), Handbook of social psychology (4th ed.). New York: McGraw-Hill. 

Tetlock, P.E. (1998). The what-if school of history. TIME, 151, 163. 

Tetlock, P.E., & Belkin, A. (1996). Counterfactual thought experiments in world politics: Logical, methodological, and psychological perspectives. In P.E. Tetlock & A. Belkin (Eds), Thought experiments in world politics. Princeton: Princeton University Press.

Tetlock, P.E., & Tyler, A. (1996). Winston Churchill’s cognitive and rhetorical style: The debates over Nazi intentions and self-government for India. Political Psychology, 17, 149-170.  

Tetlock, P.E., Peterson, R., & Berry, J. (1993). Flattering and unflattering personality portraits of integratively simple and complex managers. Journal of Personality and Social Psychology, 64, 500-511. 

Sniderman, P., Tetlock, P.E., Carmines, E.G., & Peterson, R. (1993). The politics of the American dilemma: Issue pluralism. In P. Sniderman, P.E. Tetlock, & E.G. Carmines (Eds.), Prejudice, politics and the American dilemma. Stanford: Stanford University Press.

Tetlock, P.E., McGuire, C., Peterson, R., Feld, P., & Chang, S. (1992). Assessing political group dynamics: A test of the groupthink model. Journal of Personality and Social Psychology, 63, 402-423. 

Suedfeld, P., Tetlock, P.E., & Streufert, S. (1992). Conceptual/integrative complexity. In C. Smith (Ed.), Handbook of thematic content analysis (pp. 393-401). New York: Cambridge University Press. 

Tetlock, P.E., McGuire, C., & Mitchell, P.G. (1991). Psychological perspectives on nuclear deterrence. Annual Review of Psychology, 42, 239-276. Palo Alto: Annual Reviews, Inc. 

Tetlock, P.E. (1991). Learning in U.S. and Soviet foreign policy: In search of an elusive concept. Introductory chapter in G. Breslauer & P.E. Tetlock (Eds.), Learning in U.S. and Soviet foreign policy. Boulder, CO: Westview. 

Tetlock, P.E. (1989). Methodological themes and variations. In  P.E. Tetlock, R. Jervis, C. Tilly, P. Stern, & J. Husbands (Eds.), Behavior, society, and nuclear war. (Vol. 1). New York: Oxford University Press.

Tetlock, P.E. (1989). Gorbachev: His thinking is complex. Washington Post, Outlook Section, December 17, 1989, B5.

Tetlock, P.E. (1988). Monitoring the integrative complexity of American and Soviet policy rhetoric: What can be learned? Journal of Social Issues, 44(2), 101-131. 

Tetlock, P.E. (1986). A value pluralism model of ideological reasoning. Journal of Personality and Social Psychology: Personality Processes and Individual Differences, 50, 819827. Reprinted in E. Aronson & A. Pratkanis (Eds.), (1991), International library of critical readings in psychology. London: Elgar Publishing Co. 

Tetlock, P.E. (1986). Psychological advice on foreign policy: What do we have to contribute? American Psychologist, 41, 557-567. 

Tetlock, P.E., & McGuire, C. (1986). Cognitive perspectives on foreign policy. In S. Long (Ed.), Political behavior annual. Boulder, CO: Westview Press. Reprinted in R. White (Ed.), Psychology and the prevention of nuclear war. New York: New York University Press (1986) and in N. Kressel (Ed.), Political psychology: Classic and contemporary readings. New York: Paragon House (1993) and in G.J. Ikenberry (Ed.), American foreign policy: Theoretical essays, 5/e. St. Cloud, FL: Longman Publishers (2005). 

Tetlock, P.E. (1985). Integrative complexity of American and Soviet foreign policy rhetoric: A time-series analysis. Journal of Personality and Social Psychology: Interpersonal Relations and Group Processes, 49, 1565-1585. 

Tetlock, P.E. (1983). Policy-makers’ images of international conflict. Journal of Social Issues, 39, 67-86. 

Tetlock, P.E. (1983). Psychological research on foreign policy: A methodological overview. In L. Wheeler (Ed.), Review of personality and social psychology.

Tetlock, P.E. (1979). Identifying victims of groupthink from public statements of decision makers. Journal of Personality and Social Psychology, 37, 1314-1324. 

Suedfeld, P., & Tetlock, P.E. (1977). Integrative complexity of communications in international crises. Journal of Conflict Resolution, 21, 169-184. 

Suedfeld, P., Tetlock, P.E., & Ramirez, C. (1977). War, peace and integrative complexity: United Nations speeches on the Middle East problem, 1974-1976. Journal of Conflict Resolution, 21, 427-442. 

Tetlock, P. E., Metz, S. E., Scott, S., Suedfeld, P. (2013). Integrative complexity coding raises integratively complex issues. Political Psychology.

Accountability and Attributions of Responsibility

I proposed in a 1985 essay that accountability is a key concept for linking the individual levels of analysis to the social-system levels of analysis. Accountability binds people to collectivities by specifying who must answer to whom, for what, and under what ground rules. Some forms of accountability can make humans more thoughtful and constructively self-critical (reducing the likelihood of biases or errors), whereas other forms of accountability can make us more rigid and defensive (mobilizing mental effort to defend previous positions and to criticize critics). In a follow-up 2009 essay, I noted how little we still know about how psychologically deep the effects of accountability run—for instance, whether it is or is not possible to check automatic or implicit association-based biases, a topic with legal implications for companies in employment discrimination class actions.

In addition, I have also explored the political dimensions of accountability. When, for instance, do liberals and conservatives diverge in the preferences for “process accountability” that holds people responsible for respecting rules versus “outcome accountability” that holds people accountable for bottom-line results? I call this line of work the “intuitive politician research program.”

Fincher, K. & Tetlock, P. E. (2015). Brutality Under Cover of Ambiguity: Activating, Perpetuating and De-Activating Covert Retributivism. Personality and Social Psychology Bulletin.

Patil. S. & Tetlock, P.E. (2014). Punctuated incongruity: A new approach to managing trade-offs between conformity and deviation. In B. Staw and A. Brief (eds), Research in organizational behavior, 34, 155-171. JAI Press, Greenwich, CT.

Tetlock, P.E. & Fincher, K. (2014). Social functionalism. In Gawronski, B., & Bodenhausen, G. V. (Eds.). (in press). Theory and explanation in social psychology. New York: Guilford Press.

Tetlock, P. E. Veieder, F., Patl, S., & Grant, A. (2013). Accountability and ideology: When left looks right and right looks left. Organizational Behavior and Human Decision Processes.

Patil, S. Vieider, F., & Tetlock (2012). Process and outcome accountability. Oxford Handbook of Public Accountability. New York: Oxford University Press.

Tetlock, P. E. (2011) Vying for rhetorical high ground in accountability debates: It is easy to look down on those who look soft on… Administration and Society, 43(6), 693-703.

Tetlock, P.E., Self, W.T., & Singh, R. (2010). The punitiveness paradox: When is external pressure exculpatory – And when a signal just to spread blame? Journal of Experimental Social Psychology, 46, 388-395.

Tetlock, P.E., Visser, P., Singh, R., Polifroni, M., Elson, B., Mazzocco, P., & Rescober, P. (2007). People as intuitive prosecutors: The impact of social control motives on attributions of responsibility. Journal of Experimental Social Psychology, 43, 195-209.

Moore, D., Tetlock, P.E., Tanlu, L., & Bazerman, M. (2006). Conflicts of interest and the case of auditor independence: Moral seduction and strategic issue cycling. Academy of Management Review, 31, 10-29. 

Tadmor, C., & Tetlock, P.E. (2006). Biculturalism: A model of the effects of second-culture exposure on integrative complexity. Journal of Cross-Cultural Psychology, 37, 173-190.

Tetlock, P.E. (2000). Cognitive biases and organizational correctives: Do both disease and cure depend on the ideological beholder? Administrative Science Quarterly, 45, 293-326. Reprinted in L. Thompson (Ed.), The social psychology of organizational behavior. (pp. 384-406). New York: Taylor and Francis Books. Reprinted in M. Bazerman (Ed.), The international library of critical writings in business and management. Cheltenham: Elgar.

Goldberg, J.H., Lerner, J.S., & Tetlock, P.E. (1999). Rage and reason: The psychology of the intuitive prosecutor. European Journal of Social Psychology, 29, 781-795.

Green, M., Visser, P., & Tetlock, P.E. (2000). Coping with accountability cross-pressures: Low-effort evasive tactics and high-effort quests for complex compromises. Personality and Social Psychology Bulletin, 26, 1380-1392.

Markman, K.D., & Tetlock, P.E. (2000). Accountability and close-call counterfactuals: The loser who almost won and the winner who almost lost. Personality and Social Psychology Bulletin, 26, 1213-1224.

Markman, K.D., & Tetlock, P.E. (2000). “I couldn’t have known”: Accountability, foreseeability and counterfactual denials of responsibility. British Journal of Social Psychology, 39, 313-325.

Lerner, J., & Tetlock, P.E. (1999). Accounting for the effects of accountability. Psychological Bulletin, 125, 255-275.

Tetlock, P.E. (1999). Accountability theory: Mixing properties of human agents with properties of social systems. In J. Levine, L. Thompson, & D. Messick (Eds.), Shared cognition in organizations: The management of knowledge. Erlbaum: Hillsdale, N.J.

Tetlock, P.E., & Lerner, J. (1999). The social contingency model: Identifying empirical and normative boundary conditions on the error-and-bias portrait of human nature. In S. Chaiken & Y. Trope (Eds.), Dual process models in social psychology. New York: Guilford Press.

Lerner, J., Goldberg, J., & Tetlock, P.E. (1998). Sober second thought: The effects of accountability, anger, and authoritarianism on attributions of responsibility. Personality and Social Psychology Bulletin, 24, 563-574.

Tetlock, P.E. (1998). Losing our religion: On the collapse of precise normative standards in complex accountability systems. In R. Kramer & M. Neale (Eds.), Influence processes in organizations: Emerging themes in theory and research. Thousand Oaks, CA: Sage.

Tetlock, P.E., Lerner, J., & Boettger, R. (1996). The dilution effect: Judgmental bias, conversational convention, or a bit of both? European Journal of Social Psychology, 26, 915-935.

Tetlock, P.E., & Boettger, R. (1994). Accountability amplifies the status quo effect when change creates victims. Journal of Behavioral Decision Making, 7, 1-23.

Tetlock, P.E. (1992). The impact of accountability on judgment and choice: Toward a social contingency model. In M. Zanna (Ed.), Advances in experimental social psychology (vol. 25) (pp. 331376). New York: Academic Press.

Tetlock, P.E. (1991). An alternative metaphor in the study of judgement and choice: People as politicians. Theory and Psychology, 1, 451-477. Reprinted in R. Hogarth & W. Goldstein (Eds.), Judgment and decision-making: An interdisciplinary reader. Cambridge: Cambridge University Press, 1996.

Tetlock, P.E. (1988). Structure and function in political belief systems. In A.G. Greenwald & A. Pratkanis (Eds.), Attitude structure and function. Hillsdale, NJ: Erlbaum.

Tetlock, P.E. (1985). Accountability: The neglected social context of judgment and choice. In B. Staw & L. Cummings (Eds.), Research in organizational behavior (Vol. 7, pp. 297-332). Greenwich, CT: JAI Press. 

Tetlock, P.E., Skitka, L., & Boettger, R. (1989). Social and cognitive strategies of coping with accountability: Conformity, complexity, and bolstering. Journal of Personality and Social Psychology: Interpersonal Relations and Group Dynamics, 57, 632-641.

Tetlock, P.E., & Kim, J.L. (1987). Accountability and judgment in a personality prediction task. Journal of Personality and Social Psychology: Attitudes and Social Cognition, 52, 700-709.

Tetlock, P.E., & Manstead, A.S.R. (1985). Impression management versus intrapsychic explanations in social psychology: A useful dichotomy? Psychological Review, 92, 59-77.

Tetlock, P.E. (1985). Accountability: A social check on the fundamental attribution error. Social Psychology Quarterly, 48, 227-236.

Tetlock, P.E. (1983). Accountability and perseverance of first impressions. Social Psychology Quarterly, 46, 285-292.

Tetlock, P.E. (1983). Accountability and complexity of thought. Journal of Personality and Social Psychology: Attitudes and Social Cognition, 45, 74-83.

Tetlock, P.E. (1981). Pre- to post-election shifts in presidential rhetoric: Impression management or cognitive adjustment? Journal of Personality and Social Psychology: Attitudes and Social Cognition, 41, 207-212.

Tetlock, P.E. (1980). Explaining teacher explanations of pupil performance: A self-presentation interpretation. Social Psychology Quarterly, 43, 283-290.

Levi, A., & Tetlock, P.E. (1980). A cognitive analysis of Japan’s 1941 decision for war. Journal of Conflict Resolution, 24, 195-211.

Taboo Cognition and Sacred Values

I use a different “functionalist metaphor” to describe my work on how people react to threats to sacred values—and on the pains they take to structure situations so as to avoid open or transparent trade-offs involving sacred values. Real-world implications of this claim are explored largely in peer-review outlets such as the Journal of Consumer Research, California Management Review, and Journal of Consumer Psychology. This research argues that most people recoil from the specter of relativism: the notion that the deepest moral-political values are arbitrary inventions of mere mortals struggling to infuse moral meaning into an otherwise meaningless universe. Rather, humans prefer to believe that they have sacred values that provide firm foundations for their moral-political opinions. People can become very punitive “intuitive prosecutors” when they feel sacred values have been seriously violated, going well beyond the range of socially acceptable forms of punishment when given chances to do so covertly.

McGraw, P., Schwartz, J. & Tetlock, P.E. (2012) From the Commercial to the Communal: Reframing Taboo Trade-Offs in Religious and Pharmaceutical Marketing. Journal of Consumer Research.

Schoemaker, P, & Tetlock, P.E. (2011). Taboo scenarios: How to think about the unthinkable. California Management Review, 54(2), 5-24.

Kray, L.J., George, L.G., Liljenquist, K.A., Galinsky, A.D., Tetlock, P.E., & Roese, N.J. (2010). From what might have been to what must have been: Counterfactual thinking creates meaning. Journal of Personality and Social Psychology, 98, 106-118.

McGraw, P., & Tetlock, P.E. (2005). Taboo trade-offs, relational framing, and the acceptability of exchanges. Journal of Consumer Psychology, 15(1), 35-38.

Tetlock, P.E., & McGraw, P. (2005). Theoretically framing relational framing. Journal of Consumer Psychology, 15(1), 2-16.

Tetlock, P.E., McGraw, A.P., & Kristel, O. (2004). Proscribed forms of social cognition: Taboo trade-offs, blocked exchanges, forbidden base rates, and heretical counterfactuals. In N. Haslam (Ed.), Relational models theory: A contemporary overview. Mahway, NJ: Erlbaum.

McGraw, P, Tetlock, P.E., & Kristel, O. (2003). The limits of fungibility: Relational schemata and the value of things. Journal of Consumer Research, 30, 219-229.

Tetlock, P.E. (2003). Thinking about the unthinkable: Coping with secular encroachments on sacred values. Trends in Cognitive Science, 7, 320-324.

Tetlock, P.E., Kristel, O., Elson, B., Green, M., & Lerner, J. (2000). The psychology of the unthinkable: Taboo trade-offs, forbidden base rates, and heretical counterfactuals. Journal of Personality and Social Psychology, 78, 853-870.

Fiske, A., & Tetlock, P.E. (1999). Taboo trade-offs: Constitutive prerequisites for social life. In S.A. Renshon & J. Duckitt (Eds.), Political psychology: Cultural and cross-cultural perspectives. London: MacMillan.

Tetlock, P.E. (2000). Coping with trade-offs: Psychological constraints and political implications. In S. Lupia, M. McCubbins, & S. Popkin (Eds.), Political reasoning and choice. Berkeley: University of California Press.

Fiske, A., & Tetlock, P.E. (1997). Taboo trade-offs: Reactions to transactions that transgress spheres of justice. Political Psychology, 18, 255-297. Reprinted in M. Bazerman (Ed.), Negotiation, decision making and conflict management. Blackwell.

Tetlock, P.E., Peterson, R., & Lerner, J. (1996). Revising the value pluralism model: Incorporating social content and context postulates. In C. Seligman, J. Olson, & M. Zanna (Eds.), Ontario symposium on social and personality psychology: Values. Hillsdale, NJ: Erlbaum.

Political versus Politicized Psychology

I have a long-standing interest in the tensions between political and politicized psychology, arguing that most political psychologists tacitly assume that, relative to political science, psychology is the more basic discipline in their hybrid field. Political actors—be they voters or national leaders—are human beings whose behavior should be subject to fundamental psychological laws that cut across cultures and historical periods. I also raise the contrarian possibility in numerous articles and chapters that reductionism can run in reverse—and that psychological research is often driven by ideological agenda (of which the psychologists often seem to be only partly conscious). I have also developed variants of this analysis in articles on the links between cognitive styles and ideology (the fine line between rigid and principled) as well as on the challenges of assessing value-charged concepts like symbolic racism and unconscious bias (is it possible to be a “Bayesian bigot”?). I have also co-authored papers on the value of ideological diversity in behavioral and social science research. One consequence of the lack of ideological diversity in high-stakes, soft-science fields is frequent failures of turnabout tests (scientific-debate hypocrisy detectors).

Oswald, F., Mitchell, G., Blanton, H., Jaccard, J., & Tetlock, P.E. (2015s), Revisiting the Predictive Validity of the Implicit Association Test. Journal of Personality and Social Psychology.

Tetlock, P.E. & Mitchell, G. (in press). Why so few conservatives and should we care? Society.

Mitchell, G., & Tetlock, P.E. (in press). Implicit attitude measures. In R.A. Scott & S.M. Kosslyn (Eds.), Emerging trends in the social and behavioral sciences. Thousand Oaks, CA: Sage Publications.

Duarte. J., Crawford, J., Jussim, L. Haidt, J,, Stern, C. & Tetlock, P.E. (2014). Ideological diversity will improve social psychological science. Behavioral and Brain Sciences.

Duarte. J., Crawford, J., Jussim, L. Haidt, J,, Stern, C. & Tetlock, P.E. (2014). A Reply to the Commentaries. Behavioral and Brain Sciences.

Blanton, H., Jaccard, J., Strants, E.. Mitchell, P.G., & Tetlock, P. E. (2014). Toward a meaningful metric of implicit prejudice. Journal of Applied Psychology.

Tetlock, P.E., Mitchell, P.G., & Anastasopoulos, J. (2013). Detecting and punishing unconscious bias. The Journal of Legal Studies, 42, 83-110.

Oswald, F. Mitchell, G., Blanton, H., Jaccard, J. & Tetlock, P. (2013). Predicting ethnic and racial discrimination: A meta-analysis of IAT research. Journal of Personality and Social Psychology.

Tetlock, P.E., Mitchell, P. G. &  Anastasopoulos, J. (2013). Detecting and punishing unconscious bias. The Journal of  Legal Studies.

Tetlock, P.E. (2012). Rational and irrational prejudices: How problematic is the ideological lopsidedness of social-personality psychology? Perspectives in Psychological Science, 7, 519-521. 

Tetlock, P.E., & Mitchell, G. (2009). Implicit bias and accountability systems: What must organizations do to prevent discrimination? In B.M. Staw & A. Brief (Eds.), Research in organizational behavior (vol. 29). New York: Elsevier. Pp. 3-38.

Tetlock, P.E., & Mitchell, G. (2009). A renewed plea for adversarial collaboration. In B.M. Staw & A. Brief (Eds.), Research in organizational behavior (vol. 29). New York: Elsevier. Pp. 71-72.

Tetlock, P.E., & Mitchell, G. (2009). Adversarial collaboration aborted, but our offer still stands. In B.M. Staw & A. Brief (Eds.), Research in organizational behavior (vol. 29). New York: Elsevier. Pp. 77-79.

Blanton, H., Jaccard, J., Klick, J., Mellers, B.A., Mitchell, G., & Tetlock, P.E. (2009). Strong claims and weak evidence: Reassessing the predictive validity of the race IAT. Journal of Applied Psychology, 29, 567-582.

Blanton, H., Jaccard, J., Klick, J., Mellers, B.A., Mitchell, G., & Tetlock, P.E. (2009).  Transparency should trump trust: Rejoinder to McConnell and Leibold (2009) and Ziegert and Hanges (2009). Journal of Applied Psychology, 29, 598-603.

Mitchell, G., & Tetlock, P.E. (2009). Facts do matter: A reply to Bagenstos. Hofstra Law Review, 37, 937-953.

Tetlock, P.E., & Mitchell, G. (2008). Calibrating prejudice in milliseconds. Social Psychology Quarterly, 71, 12-16.

Tetlock, P.E., Mitchell, G., & Murray, T.L. (2008). The challenge of debiasing personnel decisions: Avoiding both under- and overcorrection. Industrial and Organizational Psychology, 1, 439-443.

Tetlock, P.E., & Oppenheimer, M. (2008). The boundaries of the thinkable: Environmentalism in the early twenty-first century. Daedalus, 137(2), 59-70.

Tetlock, P.E. Perchance to scream. Review of Drew Westen’s “The political brain.” Times Literary Supplement, December 14, 2007, p. 23 (No. 5463).

Tetlock, P.E. (2007). Diversity paradoxes: Review of Scott Page’s “The difference: How the power of diversity creates better groups, firms, schools, and societies.” Science, 316, 984.

Mitchell, P.G., & Tetlock, P.E. (2006). Anti-discrimination law and the perils of mind reading. The Ohio State University Law Review, 67, 1023-1121.

Arkes, H., & Tetlock, P.E. (2004) Attributions of implicit prejudice, or “Would Jesse Jackson ‘fail’ the Implicit Association Test?” Psychological Inquiry, 15(4), 257-278.

Tetlock, P.E., & Arkes, H. (2004). The implicit-prejudice exchange: Islands of consensus in a sea of controversy. Psychological Inquiry, 15(4), 311-321.

Tetlock, P.E. (1994). Political psychology or politicized psychology: Is the road to scientific hell paved with good moral intentions? Political Psychology, 15, 509-530.

Tetlock, P.E. (1994). How politicized is political psychology and is there anything we should do about it? Political Psychology, 15, 567-577.

Tetlock, P.E., Armor, D., & Peterson, R. (1994). The slavery debate in antebellum America: Cognitive style, value conflict, and the limits of compromise. Journal of Personality and Social Psychology, 66, 115-126.

Suedfeld, P., & Tetlock, P.E. (1991). Psychologists as policy advocates: The roots of controversy. In P. Suedfeld and P.E. Tetlock (Eds.), Psychology and social policy. Washington, D.C.: Hemisphere.

Suedfeld, P., & Tetlock, P.E. (1991). Psychological advice about political decision making: Heuristics, biases, and cognitive defects. In P. Suedfeld & P.E. Tetlock, Psychology and social policy. Washington, DC: Hemisphere.

Sniderman, P.M., & Tetlock, P.E. (1986). Symbolic racism: Problems of motive attribution in political analysis: Journal of Social Issues, 42, 129-150.

Sniderman, P.M., & Tetlock, P.E. (1986). Reflections on American racism. Journal of Social Issues, 42, 173-188. 

Tetlock, P.E. (1984). Cognitive style and political belief systems in the British House of Commons. Journal of Personality and Social Psychology: Personality Processes and Individual Differences, 46, 365-375. Reprinted in 2003 by J. Jost & J. Sidanius (Eds), Political psychology: Key readings. New York: Taylor and Francis.)

Tetlock, P.E., Hannum, K., & Micheletti, P. (1984). Stability and change in senatorial debate: Testing the cognitive versus rhetorical style hypotheses. Journal of Personality and Social Psychology: Attitudes and Social Cognition, 46, 979-990.

Hypothetical Societies and Intuitions About Justice

In collaboration with Greg Mitchell and Linda Skitka, I have conducted research on hypothetical societies and intuitions about justice “experimental political philosophy”). The spotlight here is on a fundamental question in political theory: who should get what from whom, when, how, and why? In real-world debates over distributive justice, however, it is virtually impossible to disentangle the factual assumptions that people are making about human beings from the value judgments people are making about end-state goals, such as equality and efficiency. Hypothetical society studies make it possible for social scientists to disentangle these otherwise hopelessly confounded influences on public policy preferences.

Tetlock, P.E., & Mitchell, G. (2010). Situated social identities constrain morally-defensible choices: Commentary on Bennis, Medin, & Bartels (2010). Perspectives in Psychological Science. 5, 206-208.

Mitchell, G., & Tetlock, P.E. (2009). Disentangling reasons and rationalizations: Exploring perceived fairness in hypothetical societies. In J. Jost, A.C. Kay & H. Thorisdottir (Eds.), Social and psychological bases of ideology and system justification. New York: Oxford University Press. Pp. 126-157.

Mitchell, P.G., & Tetlock, P.E. (2006). An empirical inquiry into the relation of corrective justice to distributive justice. Journal of Empirical Legal Studies, 3, 421-466.

Mitchell, P.G., Tetlock, P.E., Newman, D., & Lerner, J. (2003). Experiments behind the veil: A hypothetical societies approach to the study of social justice. Political Psychology, 24, 519-547.

Tetlock, P.E. (1994). The market experience: The worst system except for all the others? Review of R. Lane, The market experience. Contemporary Psychology, 39, 589-591.

Mitchell, P. G., Tetlock, P.E., Mellers, B. A., & Ordonez, L. (1993). Judgments of social justice: Compromises between equality and efficiency. Journal of Personality and Social Psychology, 65, 629-639.

Skitka, L., & Tetlock, P.E. (1993). Providing public assistance: Cognitive and motivational processes underlying liberal and conservative policy preferences. Journal of Personality and Social Psychology, 65, 1205-1224.

Skitka, L., & Tetlock, P.E. (1993). Of ants and grasshoppers: The political psychology of allocating public assistance. In B. Mellers & J. Baron (Eds.), Psychological perspectives on justice. Cambridge: Cambridge University Press.

Tetlock, P.E., & Mitchell, P.G. (1993). Liberal and conservative approaches to justice: Conflicting psychological portraits. In B. Mellers & J. Baron (Eds.), Psychological perspectives on justice. Cambridge: Cambridge University Press.

Skitka, L., & Tetlock, P.E. (1992). Allocating scarce resources: A contingency model of distributive justice. Journal of Experimental Social Psychology, 28, 491-522. Sniderman, P., Piazza, T., Tetlock, P.E., & Kendrick, A. (1991). Racism and the American ethos. American Journal of Political Science, 35, 423-447.