alhdzsz
  • About
  • Blog
  • Publications
  • Projects
    • Water Integrity Risk Index (WIRI)
    • Data Viz Book
    • FIRSA

LLMs for Political Text with R

A tutorial for R and reticulate with Gemini and ChatGPT

Artificial Intelligence
Data Visualization
Text as Data
This tutorial shows how to connect to LLMs via APIs in R with the reticulate package.
Author

Alfredo Hernández Sánchez

Published

May 16, 2025

Introduction

LLMs are everywhere now! Increasingly they are being used in social science research and can help scholars and students annotate texts much faster and cheaper than humans (Laurer et al. 2024). However, connecting to these models via API natively from R is challenging. Here I show a basic framework for classifing text and getting structured output from two of the most popular LLMs, ChatGPT and Google’s Gemini. This tutorial assumes a working knowledge of R and some familiarity reticulate package for interfacing with Python APIs.

library(reticulate)
library(dplyr)
library(purrr)
library(tidyr)
library(stringr)
library(ggplot2)

To begin, I load the appropriate Python environment using reticulate::use_condaenv("gptenv", required = TRUE), which ensures that my code runs inside a Conda environment where the required Python packages are installed. The gemini_key is used to authenticate requests to Google’s Gemini API, while the openai_key is used for accessing OpenAI’s GPT models. Replace the text with your own keys. Each call costs money so make sure to never share/show them!

use_condaenv("gptenv", required = TRUE)

gemini_key <- "[YOUR-KEY-HERE]"

openai_key <- "[YOUR-KEY-HERE]"

Custom Functions

To run structured prompts through both OpenAI and Google models, I wrote two helper functions: gpt_get_structured_output() and gemini_get_structured_output(). Each function takes a string (in this case a sentence) and a shared prompt, sends them to the respective API using Python code embedded via reticulate::py_run_string(), and returns the model’s text output. I also use safely() wrappers (safe_gpt and safe_gemini) to ensure that if a particular sentence fails (due to API rate limits or formatting issues), the rest of the pipeline continues.

After collecting the model responses, I parse them into structured key–value pairs using parse_kv_block(), which extracts results like framing: justice into a clean named list.

# Sends a prompt and input text to OpenAI's GPT-3.5 and returns the structured output.
gpt_get_structured_output <- function(text, 
                                      prompt, 
                                      api_key) {
  
  # Concatenate the system prompt and user input
  ai_prompt <- prompt
  
  # Construct and run Python code that uses the OpenAI client API (v1.0+)
  py_run_string(paste0(
    "import openai\n",
    "client = openai.OpenAI(api_key='", api_key, "')\n",
    "sys_prompt = '''", ai_prompt, "'''\n",
    "user_prompt = '''", text, "'''\n",
    "response = client.chat.completions.create(\n",
    "  model = 'gpt-3.5-turbo',\n",
    "  temperature = 0,\n",  # deterministic output
    "  messages = [\n",
    "    { 'role': 'system', 'content': sys_prompt },\n",
    "    { 'role': 'user', 'content': user_prompt }\n",
    "  ]\n",
    ")\n",
    "result = response.choices[0].message.content\n"
  ))
  
  # Return the raw text output from GPT
  py$result
}

# Wrap the GPT function in a safely() wrapper to handle errors gracefully
safe_gpt <- safely(gpt_get_structured_output)

# Sends a prompt and text to Google’s Gemini 1.5 Pro model and returns the structured result.
gemini_get_structured_output <- function(text, 
                                         prompt, 
                                         api_key) {
  
  # Combine the prompt and user input into one string
  ai_prompt <- paste0(
    prompt,
    text
  )
  
  # Construct and run Python code that uses the Gemini client
  py_run_string(paste0(
    "import google.generativeai as genai\n",
    "genai.configure(api_key='", api_key, "')\n",
    "model = genai.GenerativeModel('models/gemini-1.5-pro-latest')\n",
    "response = model.generate_content(\"\"\"\n", ai_prompt, "\n\"\"\")\n",
    "result = response.text\n"
  ))
  
  # Return the raw text output from Gemini
  py$result
  
}

# Wrap the Gemini function in a safely() wrapper to handle errors gracefully
safe_gemini <- safely(gemini_get_structured_output)

# Parses key-value outputs into a named list (e.g. `multilateralism: true`)
parse_kv_block <- function(text_block) {
  
  # Split text into lines
  lines <- stringr::str_split(text_block, "\n")[[1]]
  
  # Extract key-value pairs using a regular expression
  kv <- stringr::str_match(lines, "^([a-zA-Z_]+):\\s*(.*)$")
  
  # Identify valid rows where both key and value are present
  valid <- which(!is.na(kv[,2]) & !is.na(kv[,3]))
  
  # Return as a named list
  out <- setNames(kv[valid, 3], kv[valid, 2])
  as.list(out)
  
}

Prompt Design

The prompt is designed to extract structured metadata from sentences drawn from UN General Assembly speeches (Baturo, Dasandi, and Mikhaylov 2017).1 Specifically, we ask the models to assess whether a sentence is about multilateralism or multipolarity, and if it uses a normative framing (justice, injustice) or not (neither, unknown). I explicitly tell the model not to explain itself—just to return these labels, one per line, exactly in the format I specify. This makes it easy to parse the output later using a regular expression (regex) in R.

The same prompt will be passed to both models. The output is parsed into structured key–value pairs for later comparison.

prompt <- paste0(
  "You are reading sentences from the UN General Assembly speeches. ",
  "Infer if the sentence is about systemic change in the context of multilateralism or multipolarity. ",
  "Return only the following key-value pairs, each on a new line:\n",
  "multilateralism: true or false\n",
  "multipolarity: true or false\n",
  "framing: justice, injustice, or neither\n",
  "If the sentence is not about international relations, set framing to 'unknown'. ",
  "Do not return any explanation or formatting. Only the two lines of key-value pairs.\n\n"
)

Examples

Let’s try out our custom functions and prompt with some example text:

gemini_get_structured_output("The BRICs countries gaining ground in global affairs.", prompt, gemini_key)
[1] "multilateralism: false\nmultipolarity: true\nframing: neither\n"
gpt_get_structured_output("The BRICs countries gaining ground in global affairs.", prompt, openai_key)
[1] "multilateralism: false\nmultipolarity: true\nframing: neither"

And with a nonsensical sentence to see what it does:

gemini_get_structured_output("Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam.", prompt, gemini_key)
[1] "multilateralism: false\nmultipolarity: false\nframing: unknown\n"
gpt_get_structured_output("Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam.", prompt, openai_key)
[1] "framing: unknown\nmultilateralism: false"
Raw Text Data
sentence_id sentence
MNG_71_2016_57 We urge multilateral institutions to take the lead in those important efforts.
SGP_77_2022_60 The United Nations-led multilateral system will be key in pulling everyone together to row in the same direction and not leave anyone behind.
JPN_17_1962_85 On the problem of trade, we should-remind ourselves that the General Agreement on Tariffs and Trade has been making, since its inception, major contributions to the furtherance of free trade on a multilateral, non-discriminatory basis.
KOR_68_2013_3 The Republic of Korea is pleased that the United Nations, in partnership with its Member States and under the stewardship of Secretary-General Ban Ki-moon, is strengthening multilateralism by successfully responding to the diverse challenges that the international community faces.
VNM_71_2016_23 We are seeing multilateral institutions expand, both in numbers and in strength.
TLS_72_2017_48 Only through dialogue, negotiation, multilateral cooperation and respect for democracy can we meet the ambitious goals we have established for 2030.
MYS_44_1989_107 The international economic front, very few encouraging steps have been taken to achieve the stated goals of multilateralism, interdependence and mutual co-operation.
TUV_75_2020_17 Concretizing the Multilateral response in a UN we need and a future we want.
NRU_75_2020_65 We need a global financial system that is more responsive to the urgent needs of developing countries, and I strongly urge you to take on this issue as part of your effort to reaffirm multilateralism.
NZL_33_1978_61 I recognize that the principles of an open multilateral trading system raise difficult domestic political problems, but economic development is about change and innovation and we need to look at today's problems with our eyes very firmly fixed on the longer term.
MNG_71_2016_156 We reaffirm our firm commitment to the multilateral system with the United Nations at its core.
VUT_76_2021_13 This demands a reaffirmation of faith in multilateralism since no national capabilities, however large, can on their own be adequate to effect meaningful change or transition.
KHM_74_2019_79 I would like to conclude by sharing our belief that in a multipolar world, multilateralism can succeed only if it rejects all forms of interference.
KIR_77_2022_43 However, the lack of solidarity even through multilateralism continues to be the stumbling block to addressing the global climate change emergency.
PLW_74_2019_38 We also look forward to the successful conclusion of negotiations and the adoption of a new treaty on high seas biodiversity that will set the multilateral framework for us to effectively protect the high seas and complement our national efforts.
SGP_63_2008_96 In fact, smaller countries can turn the emerging multipolarity to advantage if we combine our strengths in regional and international institutions.
MNG_45_1990_97 We have undertaken the work of bringing our national legislation into conformity with our international obligations and of withdrawing reservations made earlier in regard to some multilateral treaties and conventions.
LAO_36_1981_64 Bilateral or multilateral cooperation among the two groups of countries of IndoChina and ASEAN, and also other countries of the region with those outside the region, must in no case prejudice the security and interests of other countries of the area, nor should it be directed against any other country.
IDN_77_2022_74 It is why we need a renewed multilateralism that is fit for purpose and fit for its time.
IDN_49_1994_40 It is essential that serious and concerted efforts be undertaken in the Conference on Disarmament in conjunction with the broader multilateral endeavours in the Amendment Conference on the partial test-ban Treaty to ensure the conclusion of a universal and effectively verifiable treaty within a fixed time-frame.
NLD_43_1988_114 The spectacular emergence of new economic centres in various parts of the third world is only one indication that we are witnessing a development towards greater multipolarity.
FRA_74_2019_75 That is not to say that our multilateralism is worn out, that we no longer listen to ourselves or that we are no longer effective.
RUS_66_2011_19 Its goal is to enhance productive multilateral collaboration to address the urgent problems of the contemporary world.
ITA_62_2007_34 It is only through multilateralism, by marshalling everyone’s energies, that we can hope to do good.
TUR_32_1977_133 The principles enshrined in the Charter must always guide our actions in our bilateral and multilateral relations.
CYP_74_2019_8 For some, alternatives to multilateralism may seem attractive.
AUT_74_2019_61 At the outset, I shared with the Assembly my concern about the state of multilateralism.
HRV_72_2017_84 Croatia’s commitment to multilateralism is unwavering.
DEU_56_2001_121 The future belongs to responsible governance for one world, governance based not on hegemonic claims, but on cooperation, solidarity and multilateralism.
HUN_23_1968_105 Our co-operation with the Czechoslovak Socialist Republic, the consolidation and strengthening of our common international activity will create a new and more solid basis for bilateral and multilateral negotiations with a view to building up a real and peaceful system of security in Europe.
ROU_28_1973_106 Romania itself follows the process of developing and modernizing its national economy The Romanian people make consistent efforts for the building up of a multilaterally developed society through the more effective use of the country's human and material potential for the acceleration of the development and modernization of the productive forces, the improvement of the management and organization of the society, the deepening of democracy and the laying of new premises for raising the standards of life and civilization of the whole people.
AND_76_2021_1 Mr President, Mr Secretary General, Your Excellencies, Ladies and Gentlemen, I would like to begin by congratulating His Excellency Mr Abdulla Shahid on his election as President of this General Assembly and His Excellency Mr Antonio Guterres on his reappointment for a second term as Secretary-General of the United Nations (UN), this great assembly of all peoples and for all peoples, whose main objective is to build a system of values and a code of conduct based on multilateralism and cooperation.
LIE_65_2010_14 Past experience has shown time and again that multilateral action can be very effective when it is based on a broad political consensus, which is not the same as giving everyone a right to veto.
NLD_75_2020_32 Critical evaluation and continuous efforts to improve are key elements of multilateral cooperation.
AUT_64_2009_2 We have all witnessed something new and different: a genuine opportunity for a real renaissance of multilateralism.
PRT_42_1987_53 The climate of dialogue achieved during the conference allowed for a sound analysis of multilateral economic co-operation.
ITA_76_2021_57 The fight against climate change requires multilateral engagement and pragmatic cooperation among all major global players — both rich and emerging economies.
DEU_59_2004_43 To make this multilateral cooperation sustainable and capable, we need a courageous and comprehensive reform of the United Nations that faces up to the challenges we are facing.
AND_72_2017_85 That is why much of our foreign policy is focused on multilateral forums, as we showed four years ago during the Andorran presidency of the Council of Europe.
DEU_63_2008_63 My proposal to multilateralize the fuel cycle showed how those risks can at the very least be minimized.
CRI_62_2007_64 Neither the chronic pessimism of some nor the ungrateful egoism of others will check the slow but determined march forward of those of us who believe in multilateralism and in a future of greater shared welfare for all.
BRA_63_2008_46 Its distorted form of representation stands between us and the multilateral world to which we aspire.
PER_73_2018_41 We will continue to promote initiatives to help restore democracy in our sister country within the framework of the Organization of American States, the Lima Group and in other multilateral forums.
URY_35_1980_86 Now we must go from words to action and allocate the necessary funds, negotiate bilateral and multilateral financial aid and at the same time offer UNEP and the International Union for Conservation of Nature and Natural Resources the sustained support that will permit the practical and progressive adoption of their plans.
JAM_70_2015_31 The next major step for multilateralism will come in a few weeks when we hope to conclude negotiations in Paris on a global agreement on climate change.
BRA_64_2009_20 No one is yet clearly willing to confront serious distortions of the global economy in the multilateral arena.
LCA_77_2022_6 We have arrived at this watershed moment in history because, we, the Members of the United Nations, have not adhered to the rules and principles of the Organization that we created seventy-six years ago as a multilateral answer to the human propensity to use arms against our fellow humans, instead of joining with them to turn those weapons into tools for peace and development.
ARG_61_2006_21 We believe that, in order to face this criminal threat successfully, we must carry out a sustained multilateral and legitimate response.
JAM_49_1994_7 This is an exciting and daunting task, particularly for a small island State like Jamaica, whose faith and commitment to multilateralism and the United Nations remain unwavering.
GTM_64_2009_4 My presence highlights our commitment to multilateralism in general and to the United Nations in particular.
JAM_58_2003_16 These are real concerns that underscore the need to strengthen 2 multilateralism, to restore confidence in the United Nations system, to buttress its centrality in decisions that affect us all and to thereby enhance its capacity to enforce.
COL_76_2021_1 We meet again in this global forum, which has historically been a vital space for the development of multilateralism, peacebuilding and responses to threats to our shared home, against the backdrop of a cruel pandemic that is battering our health systems, our economies, our gains in equity and progress on fulfilling the 2030 Agenda for Sustainable Development.
KNA_49_1994_116 Above all, the concept of the vulnerability index must become part of the economic language and armamentarium of all multilateral and national development institutions.
CHL_77_2022_80 We need a modernized United Nations, where we all have the same goals, Based on multilateralism, justice and peace at all times and in all places, we must commit to taking the necessary actions, not just making statements, so as to stop Russia’s unjust war on Ukraine and put an end to all abuses by the powerful everywhere in the world.
SLV_25_1970_59 Moreover, the principles and norms the violation of which constitute aggression should be regarded as part of a system, and in this sense such a definition cannot be taken to mean the indication of limits for isolated cases; rather the definition must recognize the multilateral conditioning of the major international principles.
GTM_76_2021_76 I also draw the Assembly’s attention to the fact that Taiwan could provide its experience, capacities and knowledge to the strengthening of multilateralism, taking into account the challenges we face today.
CHL_70_2015_26 We are calling for an architecture of multilateral institutions that will provide backing for the national policies proposed under the 2030 Agenda.
PAN_44_1989_36 Persecution at the international level, not excluding multilateral agencies of which we are a member, is intended to put an end to national efforts to achieve economic recovery and meet the basic needs of the Panamanian people.
BRA_56_2001_51 In the field of trade, it is high time that multilateral negotiations were translated into greater access for goods from developing countries to the more prosperous markets.
DOM_52_1997_35 Also, together with other countries, it is formulating strategies to take advantage of funds from multilateral sources.
MLT_64_2009_63 In his report on the work of the Organization, the Secretary-General has laid emphasis on the need to embark on a multilateral effort of immense magnitude “that draws upon the strengths and contributions of all the countries of the world, as well as their citizens” (A/64/1, para.
DJI_70_2015_25 We will have the opportunity to return to this, but allow me now to address the implications of the macroeconomic fragility in the global economy, the many and varied challenges facing our country and the importance of establishing a credible and beneficial multilateral trading system for all.
DZA_73_2018_47 The rejection of policies based on force, with the dangers that they entail, requires that we constantly seek, through dialogue and consensus, to strengthen multilateral action.
BHR_71_2016_71 We still hear the same irresponsible sectarian discourse from Iran and have witnessed the damage done to our bilateral and multilateral relations.
MAR_43_1988_72 At the same time, we must take advantage of the forthcoming multilateral talks, especially the Uruguay Round, to improve the international economic environment in the spheres of the flow of .net resources, trade and the stabilization and rising of commodity prices.
DZA_32_1977_140 Real reform of the international monetary system: consideration of the legitimate interests of the developing countries in multilateral trade negotiations: the creation of a joint commodity fund; the adoption of a code of conduct on the transfer of technology; the solution of the problems of indebtedness of the developing countries; the promotion of the industrialization of the third world -this list is not exhaustive—are so many objectives which the international community must set for itself within a reasonable period of time.
MAR_74_2019_54 Multilateral action is not just an organizational structure of international relations; it is also a state of mind.
QAT_74_2019_74 Despite the different situations and circumstances among countries, we need international cooperation today more than ever and must ensure the credibility of multilateral action to counter the challenges of climate change.
DZA_64_2009_12 Multilateral institutions, also undermined by contradictions and a lack of coherence, are unable to overcome impasses in negotiations on vital questions that have a direct impact on our populations.
MLT_38_1983_36 Many countries have today recognized and expressed support for our status of neutrality, both at the bilateral level and in the multilateral forums of the Movement of Non-Aligned Countries, the Commonwealth and the Conference on Security and Co-operation in Europe.
MLT_76_2021_3 If anything, COVID only reaffirmed the priority of multilateralism, and that it is the key to a strong and sustainable future.
MAR_73_2018_66 The multilateral system, despite the criticism it faces, will remain essential to addressing the challenges and issues facing the international community.
MAR_76_2021_22 The COVID-19 pandemic has shown that there is an urgent need for pragmatic multilateral action whose legitimacy is rooted in the effective realization of the rights of citizens to security, health and development.
QAT_76_2021_60 Qatar’s bet on international institutions and multilateral cooperation is a strategic one.
BHR_61_2006_69 Today, all of us need to join in efforts to achieve the objective of a collective security system through multilateral diplomacy, for which the United Nations is the main arena.
KWT_48_1993_15 Kuwait welcomes the accession to United Nations membership by Eritrea, the Czech Republic, the Slovak Republic, Monaco, Andorra and The Former Yugoslav Republic of Macedonia and trusts that their accession to the Organization’s membership will consolidate further the efficacy and universality of multilateralism.
IRQ_68_2013_72 While the United Nations is our agreed forum for international cooperation to address the problems and challenges that we face through multilateral negotiations, we believe that the time has come to reform its bodies, in particular the Security Council, as the primary body responsible for international peace and security, in order to make them consistent with the aspirations of the peoples of the world in the twenty- first century and more representative, transparent and able to meet the challenges they face.
SAU_61_2006_7 The world today faces a host of global problems that can be successfully addressed only within the context of multilateral cooperation under the umbrella of the United Nations.
OMN_66_2011_2 We are confident that his diplomatic experience, especially in the field of multilateral diplomacy, will have a great impact on the success of this session.
MAR_76_2021_23 Mr President, The bid to deliver vaccines, our last line of defence against the pandemic, is in fact an opportunity to assert our will and demonstrate our ability to infuse multilateral action with a new impetus.
USA_41_1986_121 We applaud the success of the meeting of the General Agreement on Tariffs and Trade (GATT) trade ministers last week in Uruguay, where agreement was reached to launch a new round of multilateral trade negotiations covering a wide range of topics important to economic growth, with over 90 other countries members of GATT, the United States is working to maintain the free flow of international trade.
USA_15_1960_58 We should also look forward to appropriate and timely financial assistance from these two multilateral financial sources as the emerging countries qualify for their aid.
CAN_39_1984_49 We shall encourage super-Power and multilateral discussion on all outer space weapons and shall commission further studies on how a space-weapons ban might be verified.
CAN_74_2019_12 Strengthening multilateralism means involving all stakeholders.
USA_63_2008_34 Other multilateral organizations have spoken clearly as well.
CAN_39_1984_65 In his most recent report he asks this: "Why has there been a retreat from internationalism and multilateralism at a time when actual developments both in relation to world peace and to the world economy would seem to demand their strengthening?"
USA_34_1979_132 In addition, it could recommend how current institutions could be strengthened and whether new multilateral ones should be created.
CAN_74_2019_75 We are fully committed to bringing innovative ideas to prevent and respond to crises, and we firmly believe in the power of multilateral solutions and partnership to address the most intractable of global challenges.
CAN_76_2021_80 However, the benefits of multilateralism have not always been evenly or fairly distributed, and the potential for peace is yet to be realized in far too many places.
USA_63_2008_15 Multilateral organizations have responsibilities.
CAN_49_1994_35 As the leader of the multilateral Working Group on Refugees, we are tackling with determination that task entrusted to us by the international community.
CAN_40_1985_93 Moreover we shall work towards the conclusion of a multilateral agreement prohibiting the possession and use of radiological weapons.
CAN_40_1985_31 The challenges facing the multilateral system reach beyond this institution.
CAN_56_2001_19 We in Canada, as long- standing proponents of multilateralism and the United Nations system, have strongly welcomed the close collaboration evident between the United States Administration and the United Nations over these last two months.
USA_20_1965_212 We would be prepared to increase the amount of capital flowing through multilateral channels.
CAN_38_1983_194 They have not lost faith in the multilateral institutions we have so painstakingly constructed.
CAN_59_2004_32 It is always preferable to have multilateral authority for intervention in the affairs of a sovereign State.
CAN_40_1985_125 Bilateral relations between countries on the one hand, and the multilateral system on the other, ought to be mutually complementary and reinforcing.
USA_33_1978_43 The facilities of IMF have been expanded, and discussions are under way to expand the facilities of the multilateral development banks.
CAN_49_1994_102 Canada is ready to proceed with a comprehensive re- evaluation of its multilateral system as it applies to economic and social issues.
PAK_60_2005_108 It is in the same spirit of constructive engagement that Pakistan wishes to pursue cooperative multilateralism and to strengthen the United Nations — this unique forum that is indispensable, in our interdependent world, for all States, large or small, powerful or weak.
AFG_41_1986_34 The imperialist policy of using economic assistance as a means of exerting political pressure on the developing countries is no longer confined within the limits of bilateral relations; it has also been extensively employed to undermine the capabilities of multilateral institutions, with a view to curtailing or completely stopping the flow of international development assistance to developing countries, particularly those that dare refuse to submit to imperialist diktat.
PAK_21_1966_212 The purpose of the conference, among other things, would be to consider, first, how the security of States without nuclear weapons can best be assured, preferably through multilateral guarantees; secondly, how they may co-operate among themselves in preventing further proliferation of nuclear weapons; and thirdly, the development and use of nuclear energy for exclusively peaceful purposes, through mutual co-operation and for mutual benefit, including the carrying out of explosions of nuclear devices for peaceful purposes under appropriate international supervision.
LKA_32_1977_110 Existing national insurance agencies in developed countries could pool their risks through a reinsurance agency that might be set up on a multilateral basis.
BGD_49_1994_168 Bangladesh looks forward to consolidating our relations with South Africa in more concrete ways, bilaterally and in all multilateral forums.
PAK_27_1972_102 Pakistan attaches the highest priority to both bilateral and multilateral efforts for disarmament.
PAK_66_2011_23 It is the United Nations and multilateralism that will safeguard that future.
BTN_67_2012_56 As a small State, we have always attached the highest importance to multilateralism and the primacy of an effective United Nations that serves the interests of all its Member States.
AFG_64_2009_9 In our increasingly interdependent world and the multilaterally oriented international system, the United Nations must assume greater responsibility for finding collective solutions to our challenges.
LKA_47_1992_81 The successful conclusion of the convention on chemical weapons demonstrates the competence of the United Nations in multilateral negotiations.
LKA_77_2022_12 In multilateralism, we talk to each other, we develop a relationship of trust and confidence and, if something were to come up, you have the base to work from.
NPL_32_1977_77 Similarly, there has been little progress in the multilateral trade negotiations in GATT on the reduction of trade barriers of developed countries, which restrict the exports of primary and processed commodities from developing countries.
IND_36_1981_154 Total bilateralism at the expense of multilateral cooperation would run into alliances that may not be in the interest of harmonious relations and relaxation of tensions.
PAK_71_2016_4 Today, three decades after the end of the Cold War, our multipolar world is more free and vibrant, yet still chaotic and turbulent; more interdependent, but more unequal; more prosperous, yet still afflicted by poverty.
PAK_66_2011_13 Each September we return to this great city and this grand stage so that we can restate and reaffirm the principles and values of multilateralism.
PAK_59_2004_94 Today, there is welcome resurgence of support for multilateralism.
MDV_68_2013_63 SIDS need to be given full recognition within the global governance regimes and multilateral and financial institutions, and adequately integrated and institutionalized within the United Nations system.
NPL_37_1982_17 International economic co-operation and the system of multilateral co-operation continue to erode at an alarming rate.
BTN_38_1983_94 Recently there has been much concern expressed at the deterioration of the resource position of some multilateral development organizations, UNDP in particular.
PAK_53_1998_146 This suggests the need for concerted action by the international community, which should include, first, the 14 strengthening of the capacity of multilateral institutions to address the issues of trade, finance and development in an integrated and coherent manner, thereby ensuring the effective governance of globalization.
CMR_74_2019_81 This occasion gives me the opportunity to warmly acknowledge the contribution of bilateral and multilateral partners that have always reaffirmed Cameroon’s unity and territorial integrity.
CMR_39_1984_157 As a corollary, multilateralism is dangerously declining; there are clear signs of this, mainly the current decrease in the resources made available to institutions such as UNDP, the fact that many States, particularly the more powerful ones, do not hasten to resort to the machinery offered by the United Nations to resolve issues of world-wide interest and, to some extent, the difficulties experienced now by UNESCO.
ZWE_45_1990_61 Zimbabwe therefore wishes to appeal to the international community and to the various donors and multilateral economic institutions for support of our national efforts.
ZWE_52_1997_7 Africa has consistently underlined the need for reform of the United Nations and other multilateral bodies, to promote the democratization and effectiveness of the international decision-making process.
ZWE_73_2018_40 My country remains committed to strengthening multilateralism and the peaceful resolution of differences.
SWZ_63_2008_61 The Kingdom of eSwatini reaffirms its commitment to the purpose and preservation of the central role of the United Nations in multilateral affairs.
SOM_74_2019_6 In this globalized world, no nation, no matter how wealthy, strong or prepared, can individually stand alone against the tide of global challenges, which requires common action and coordinated multilateral responses.
ZMB_72_2017_5 The theme for this session, “Focusing on people: Striving for peace and a decent life for all on a sustainable planet”, provides us with an opportunity to evaluate the existing multilateral approaches to address the challenges affecting our peoples.
CPV_71_2016_10 As a small island State, Cabo Verde bases its foreign policy on the principles enshrined in the Charter of the United Nations and is certain that multilateralism is the most appropriate way to approach issues on the international agenda.
SEN_75_2020_44 Furthermore, in implementing the Declaration on the commemoration of the seventy-fifth anniversary of the United Nations, which Senegal fully endorses, we must renew our commitment to the agenda for UN reform in order to restore multilateralism to its rightful place with the United Nations at its centre.
LSO_59_2004_75 That must be done against the background of multilateralism, upon which international peace and security is premised.
GNQ_77_2022_7 Equatorial Guinea is today launching an appeal on the need and importance of enhancing multilateralism and international cooperation, which are so necessary to address those global challenges and to revive development and sustainable economic growth in our countries, especially developing countries.
BFA_76_2021_7 The strategic vision you presented for your second term confirms your commitment and determination to work towards strengthening multilateralism and the United Nations in solving the multiple problems facing our world, especially in these difficult times caused by the coronavirus disease (COVID-19) pandemic.
KEN_77_2022_104 It is time for multilateralism to reflect the voices of the farmers, represent the hopes of villagers, champion the aspirations of pastoralists, defend the rights of fisherfolk, express the dreams of traders, respect the wishes of workers and indeed protect the welfare of all the peoples of the global South.
GIN_50_1995_90 The United Nations and its specialized agencies, as the paramount instrument for cooperation and multilateral agreement, see their responsibilities increased today more than ever before.
GIN_28_1973_13 The problem of international peace and security, the defence of national sovereignties, decolonialization and the struggle against imperialism continue to be the focus of both bilateral and multilateral meetings.
GNB_75_2020_44 We cannot talk about multilateralism while allowing the economic and financial embargo to continue to be imposed on one of the most supportive members of our Organization — Cuba.
SLE_57_2002_57 Never since the end of the Second World War has multilateral cooperation become such a necessary means for resolving international disputes and for ensuring the well-being of people everywhere.
SLE_67_2012_70 We are confident that with sustained support from our bilateral and multilateral partners, we shall promote socioeconomic progress and provide a better standard of living for our people in an atmosphere of peace and security.
ZMB_44_1989_9 We meet today against a background of great hopes and expectations for the success of multilateralism, which is so vital for the well-being of mankind.


Apply Functions to Sentence Data Frame

In this step, I apply the GPT and Gemini models to each sentence in my dataset using purrr::map(). The safe_gpt and safe_gemini functions call the APIs with a shared prompt and safely handle any errors. I extract the results using map_chr(), replacing failures with "FAILED". Finally, I drop the intermediate result columns to keep the dataframe clean. Depending on the size of the corpus, this might take a long time!

classified_df <- df %>%
  mutate(
    gpt_result = map(sentence, ~ safe_gpt(.x, prompt, openai_key)),
    gpt_output = map_chr(gpt_result, ~ ifelse(is.null(.x$error), 
                                              .x$result, 
                                              "FAILED")),
    gemini_result = map(sentence, ~ safe_gemini(.x, prompt, gemini_key)),
    gemini_output = map_chr(gemini_result, ~ ifelse(is.null(.x$error), .
                                                    x$result, 
                                                    "FAILED"))
  ) %>% 
  select(-gpt_result, -gemini_result)

Parse the Output into Columns

Next, I parse the structured outputs from both models using parse_kv_block() and expand them into separate columns with unnest_wider(), adding _gpt and _gemini suffixes. I then remove the raw output columns and create a new variable, framing_overlap, which captures agreement between models on the framing label or marks it as "conflict" when they differ.

classified_df <- classified_df %>%
  mutate(
    parsed_gpt = map(gpt_output, parse_kv_block),
    parsed_gemini = map(gemini_output, parse_kv_block)
  ) %>%
  unnest_wider(parsed_gpt, names_sep = "_gpt") %>%
  unnest_wider(parsed_gemini, names_sep = "_gemini") %>% 
  select(-gpt_output, -gemini_output) %>%
  mutate(
    framing_overlap = case_when(
      parsed_gemini_geminiframing == parsed_gpt_gptframing ~ parsed_gemini_geminiframing,
      TRUE ~ "conflict"
    )
  )
Classified (Parsed) Data
sentence_id sentence parsed_gpt_gptmultilateralism parsed_gpt_gptmultipolarity parsed_gpt_gptframing parsed_gemini_geminimultilateralism parsed_gemini_geminimultipolarity parsed_gemini_geminiframing framing_overlap
MNG_71_2016_57 We urge multilateral institutions to take the lead in those important efforts. true false justice true false neither conflict
SGP_77_2022_60 The United Nations-led multilateral system will be key in pulling everyone together to row in the same direction and not leave anyone behind. true false justice true false justice justice
JPN_17_1962_85 On the problem of trade, we should-remind ourselves that the General Agreement on Tariffs and Trade has been making, since its inception, major contributions to the furtherance of free trade on a multilateral, non-discriminatory basis. true false justice true false neither conflict
KOR_68_2013_3 The Republic of Korea is pleased that the United Nations, in partnership with its Member States and under the stewardship of Secretary-General Ban Ki-moon, is strengthening multilateralism by successfully responding to the diverse challenges that the international community faces. true false justice true false neither conflict
VNM_71_2016_23 We are seeing multilateral institutions expand, both in numbers and in strength. true false neither true false neither neither
TLS_72_2017_48 Only through dialogue, negotiation, multilateral cooperation and respect for democracy can we meet the ambitious goals we have established for 2030. true false justice true false neither conflict
MYS_44_1989_107 The international economic front, very few encouraging steps have been taken to achieve the stated goals of multilateralism, interdependence and mutual co-operation. true false justice true false injustice conflict
TUV_75_2020_17 Concretizing the Multilateral response in a UN we need and a future we want. true false neither true false neither neither
NRU_75_2020_65 We need a global financial system that is more responsive to the urgent needs of developing countries, and I strongly urge you to take on this issue as part of your effort to reaffirm multilateralism. true false justice true false justice justice
NZL_33_1978_61 I recognize that the principles of an open multilateral trading system raise difficult domestic political problems, but economic development is about change and innovation and we need to look at today's problems with our eyes very firmly fixed on the longer term. true false justice true false neither conflict
MNG_71_2016_156 We reaffirm our firm commitment to the multilateral system with the United Nations at its core. true false neither true false neither neither
VUT_76_2021_13 This demands a reaffirmation of faith in multilateralism since no national capabilities, however large, can on their own be adequate to effect meaningful change or transition. true false justice true false neither conflict
KHM_74_2019_79 I would like to conclude by sharing our belief that in a multipolar world, multilateralism can succeed only if it rejects all forms of interference. true true justice true true justice justice
KIR_77_2022_43 However, the lack of solidarity even through multilateralism continues to be the stumbling block to addressing the global climate change emergency. true false justice true false injustice conflict
PLW_74_2019_38 We also look forward to the successful conclusion of negotiations and the adoption of a new treaty on high seas biodiversity that will set the multilateral framework for us to effectively protect the high seas and complement our national efforts. true false justice true false neither conflict
SGP_63_2008_96 In fact, smaller countries can turn the emerging multipolarity to advantage if we combine our strengths in regional and international institutions. false true neither true true neither neither
MNG_45_1990_97 We have undertaken the work of bringing our national legislation into conformity with our international obligations and of withdrawing reservations made earlier in regard to some multilateral treaties and conventions. true false justice true false justice justice
LAO_36_1981_64 Bilateral or multilateral cooperation among the two groups of countries of IndoChina and ASEAN, and also other countries of the region with those outside the region, must in no case prejudice the security and interests of other countries of the area, nor should it be directed against any other country. true false justice true false neither conflict
IDN_77_2022_74 It is why we need a renewed multilateralism that is fit for purpose and fit for its time. true false justice true false neither conflict
IDN_49_1994_40 It is essential that serious and concerted efforts be undertaken in the Conference on Disarmament in conjunction with the broader multilateral endeavours in the Amendment Conference on the partial test-ban Treaty to ensure the conclusion of a universal and effectively verifiable treaty within a fixed time-frame. true false justice true false justice justice
NLD_43_1988_114 The spectacular emergence of new economic centres in various parts of the third world is only one indication that we are witnessing a development towards greater multipolarity. false true neither false true neither neither
FRA_74_2019_75 That is not to say that our multilateralism is worn out, that we no longer listen to ourselves or that we are no longer effective. true false neither true false neither neither
RUS_66_2011_19 Its goal is to enhance productive multilateral collaboration to address the urgent problems of the contemporary world. true false justice true false neither conflict
ITA_62_2007_34 It is only through multilateralism, by marshalling everyone’s energies, that we can hope to do good. true false justice true false justice justice
TUR_32_1977_133 The principles enshrined in the Charter must always guide our actions in our bilateral and multilateral relations. true false justice true false neither conflict
CYP_74_2019_8 For some, alternatives to multilateralism may seem attractive. true false neither true false neither neither
AUT_74_2019_61 At the outset, I shared with the Assembly my concern about the state of multilateralism. true false neither true false neither neither
HRV_72_2017_84 Croatia’s commitment to multilateralism is unwavering. true false neither true false neither neither
DEU_56_2001_121 The future belongs to responsible governance for one world, governance based not on hegemonic claims, but on cooperation, solidarity and multilateralism. true false justice true true justice justice
HUN_23_1968_105 Our co-operation with the Czechoslovak Socialist Republic, the consolidation and strengthening of our common international activity will create a new and more solid basis for bilateral and multilateral negotiations with a view to building up a real and peaceful system of security in Europe. true false justice true false justice justice
ROU_28_1973_106 Romania itself follows the process of developing and modernizing its national economy The Romanian people make consistent efforts for the building up of a multilaterally developed society through the more effective use of the country's human and material potential for the acceleration of the development and modernization of the productive forces, the improvement of the management and organization of the society, the deepening of democracy and the laying of new premises for raising the standards of life and civilization of the whole people. true false justice true false neither conflict
AND_76_2021_1 Mr President, Mr Secretary General, Your Excellencies, Ladies and Gentlemen, I would like to begin by congratulating His Excellency Mr Abdulla Shahid on his election as President of this General Assembly and His Excellency Mr Antonio Guterres on his reappointment for a second term as Secretary-General of the United Nations (UN), this great assembly of all peoples and for all peoples, whose main objective is to build a system of values and a code of conduct based on multilateralism and cooperation. true false justice true false justice justice
LIE_65_2010_14 Past experience has shown time and again that multilateral action can be very effective when it is based on a broad political consensus, which is not the same as giving everyone a right to veto. true false justice true false neither conflict
NLD_75_2020_32 Critical evaluation and continuous efforts to improve are key elements of multilateral cooperation. true false justice true false neither conflict
AUT_64_2009_2 We have all witnessed something new and different: a genuine opportunity for a real renaissance of multilateralism. true false neither true false justice conflict
PRT_42_1987_53 The climate of dialogue achieved during the conference allowed for a sound analysis of multilateral economic co-operation. true false neither true false neither neither
ITA_76_2021_57 The fight against climate change requires multilateral engagement and pragmatic cooperation among all major global players — both rich and emerging economies. true false justice true true neither conflict
DEU_59_2004_43 To make this multilateral cooperation sustainable and capable, we need a courageous and comprehensive reform of the United Nations that faces up to the challenges we are facing. true false justice true false justice justice
AND_72_2017_85 That is why much of our foreign policy is focused on multilateral forums, as we showed four years ago during the Andorran presidency of the Council of Europe. true false neither true false neither neither
DEU_63_2008_63 My proposal to multilateralize the fuel cycle showed how those risks can at the very least be minimized. true false justice true false neither conflict
CRI_62_2007_64 Neither the chronic pessimism of some nor the ungrateful egoism of others will check the slow but determined march forward of those of us who believe in multilateralism and in a future of greater shared welfare for all. true false justice true false justice justice
BRA_63_2008_46 Its distorted form of representation stands between us and the multilateral world to which we aspire. true false justice true false justice justice
PER_73_2018_41 We will continue to promote initiatives to help restore democracy in our sister country within the framework of the Organization of American States, the Lima Group and in other multilateral forums. true false justice true false neither conflict
URY_35_1980_86 Now we must go from words to action and allocate the necessary funds, negotiate bilateral and multilateral financial aid and at the same time offer UNEP and the International Union for Conservation of Nature and Natural Resources the sustained support that will permit the practical and progressive adoption of their plans. true false neither true false neither neither
JAM_70_2015_31 The next major step for multilateralism will come in a few weeks when we hope to conclude negotiations in Paris on a global agreement on climate change. true false justice true false neither conflict
BRA_64_2009_20 No one is yet clearly willing to confront serious distortions of the global economy in the multilateral arena. true false injustice true false injustice injustice
LCA_77_2022_6 We have arrived at this watershed moment in history because, we, the Members of the United Nations, have not adhered to the rules and principles of the Organization that we created seventy-six years ago as a multilateral answer to the human propensity to use arms against our fellow humans, instead of joining with them to turn those weapons into tools for peace and development. true false justice true false injustice conflict
ARG_61_2006_21 We believe that, in order to face this criminal threat successfully, we must carry out a sustained multilateral and legitimate response. true false justice true false justice justice
JAM_49_1994_7 This is an exciting and daunting task, particularly for a small island State like Jamaica, whose faith and commitment to multilateralism and the United Nations remain unwavering. true false justice true false neither conflict
GTM_64_2009_4 My presence highlights our commitment to multilateralism in general and to the United Nations in particular. true false neither true false neither neither
JAM_58_2003_16 These are real concerns that underscore the need to strengthen 2 multilateralism, to restore confidence in the United Nations system, to buttress its centrality in decisions that affect us all and to thereby enhance its capacity to enforce. true false justice true false injustice conflict
COL_76_2021_1 We meet again in this global forum, which has historically been a vital space for the development of multilateralism, peacebuilding and responses to threats to our shared home, against the backdrop of a cruel pandemic that is battering our health systems, our economies, our gains in equity and progress on fulfilling the 2030 Agenda for Sustainable Development. true false justice true false neither conflict
KNA_49_1994_116 Above all, the concept of the vulnerability index must become part of the economic language and armamentarium of all multilateral and national development institutions. true false justice true false neither conflict
CHL_77_2022_80 We need a modernized United Nations, where we all have the same goals, Based on multilateralism, justice and peace at all times and in all places, we must commit to taking the necessary actions, not just making statements, so as to stop Russia’s unjust war on Ukraine and put an end to all abuses by the powerful everywhere in the world. true false justice true false justice justice
SLV_25_1970_59 Moreover, the principles and norms the violation of which constitute aggression should be regarded as part of a system, and in this sense such a definition cannot be taken to mean the indication of limits for isolated cases; rather the definition must recognize the multilateral conditioning of the major international principles. true false justice true false justice justice
GTM_76_2021_76 I also draw the Assembly’s attention to the fact that Taiwan could provide its experience, capacities and knowledge to the strengthening of multilateralism, taking into account the challenges we face today. true false justice true false neither conflict
CHL_70_2015_26 We are calling for an architecture of multilateral institutions that will provide backing for the national policies proposed under the 2030 Agenda. true false justice true false justice justice
PAN_44_1989_36 Persecution at the international level, not excluding multilateral agencies of which we are a member, is intended to put an end to national efforts to achieve economic recovery and meet the basic needs of the Panamanian people. true false injustice true false injustice injustice
BRA_56_2001_51 In the field of trade, it is high time that multilateral negotiations were translated into greater access for goods from developing countries to the more prosperous markets. true false justice true false justice justice
DOM_52_1997_35 Also, together with other countries, it is formulating strategies to take advantage of funds from multilateral sources. true false neither true false neither neither
MLT_64_2009_63 In his report on the work of the Organization, the Secretary-General has laid emphasis on the need to embark on a multilateral effort of immense magnitude “that draws upon the strengths and contributions of all the countries of the world, as well as their citizens” (A/64/1, para. true false justice true false neither conflict
DJI_70_2015_25 We will have the opportunity to return to this, but allow me now to address the implications of the macroeconomic fragility in the global economy, the many and varied challenges facing our country and the importance of establishing a credible and beneficial multilateral trading system for all. true false neither true false neither neither
DZA_73_2018_47 The rejection of policies based on force, with the dangers that they entail, requires that we constantly seek, through dialogue and consensus, to strengthen multilateral action. true false justice true false justice justice
BHR_71_2016_71 We still hear the same irresponsible sectarian discourse from Iran and have witnessed the damage done to our bilateral and multilateral relations. true false injustice true false injustice injustice
MAR_43_1988_72 At the same time, we must take advantage of the forthcoming multilateral talks, especially the Uruguay Round, to improve the international economic environment in the spheres of the flow of .net resources, trade and the stabilization and rising of commodity prices. true false justice true false neither conflict
DZA_32_1977_140 Real reform of the international monetary system: consideration of the legitimate interests of the developing countries in multilateral trade negotiations: the creation of a joint commodity fund; the adoption of a code of conduct on the transfer of technology; the solution of the problems of indebtedness of the developing countries; the promotion of the industrialization of the third world -this list is not exhaustive—are so many objectives which the international community must set for itself within a reasonable period of time. true false justice true false justice justice
MAR_74_2019_54 Multilateral action is not just an organizational structure of international relations; it is also a state of mind. true false justice true false neither conflict
QAT_74_2019_74 Despite the different situations and circumstances among countries, we need international cooperation today more than ever and must ensure the credibility of multilateral action to counter the challenges of climate change. true false justice true false justice justice
DZA_64_2009_12 Multilateral institutions, also undermined by contradictions and a lack of coherence, are unable to overcome impasses in negotiations on vital questions that have a direct impact on our populations. true false injustice true false injustice injustice
MLT_38_1983_36 Many countries have today recognized and expressed support for our status of neutrality, both at the bilateral level and in the multilateral forums of the Movement of Non-Aligned Countries, the Commonwealth and the Conference on Security and Co-operation in Europe. true false neither true false neither neither
MLT_76_2021_3 If anything, COVID only reaffirmed the priority of multilateralism, and that it is the key to a strong and sustainable future. true false justice true false justice justice
MAR_73_2018_66 The multilateral system, despite the criticism it faces, will remain essential to addressing the challenges and issues facing the international community. true false justice true false neither conflict
MAR_76_2021_22 The COVID-19 pandemic has shown that there is an urgent need for pragmatic multilateral action whose legitimacy is rooted in the effective realization of the rights of citizens to security, health and development. true false justice true false justice justice
QAT_76_2021_60 Qatar’s bet on international institutions and multilateral cooperation is a strategic one. true false neither true false neither neither
BHR_61_2006_69 Today, all of us need to join in efforts to achieve the objective of a collective security system through multilateral diplomacy, for which the United Nations is the main arena. true false justice true false justice justice
KWT_48_1993_15 Kuwait welcomes the accession to United Nations membership by Eritrea, the Czech Republic, the Slovak Republic, Monaco, Andorra and The Former Yugoslav Republic of Macedonia and trusts that their accession to the Organization’s membership will consolidate further the efficacy and universality of multilateralism. true false justice true false neither conflict
IRQ_68_2013_72 While the United Nations is our agreed forum for international cooperation to address the problems and challenges that we face through multilateral negotiations, we believe that the time has come to reform its bodies, in particular the Security Council, as the primary body responsible for international peace and security, in order to make them consistent with the aspirations of the peoples of the world in the twenty- first century and more representative, transparent and able to meet the challenges they face. true false justice true false justice justice
SAU_61_2006_7 The world today faces a host of global problems that can be successfully addressed only within the context of multilateral cooperation under the umbrella of the United Nations. true false justice true false neither conflict
OMN_66_2011_2 We are confident that his diplomatic experience, especially in the field of multilateral diplomacy, will have a great impact on the success of this session. true false neither true false neither neither
MAR_76_2021_23 Mr President, The bid to deliver vaccines, our last line of defence against the pandemic, is in fact an opportunity to assert our will and demonstrate our ability to infuse multilateral action with a new impetus. true false justice true false justice justice
USA_41_1986_121 We applaud the success of the meeting of the General Agreement on Tariffs and Trade (GATT) trade ministers last week in Uruguay, where agreement was reached to launch a new round of multilateral trade negotiations covering a wide range of topics important to economic growth, with over 90 other countries members of GATT, the United States is working to maintain the free flow of international trade. true false justice true false neither conflict
USA_15_1960_58 We should also look forward to appropriate and timely financial assistance from these two multilateral financial sources as the emerging countries qualify for their aid. true false justice true false neither conflict
CAN_39_1984_49 We shall encourage super-Power and multilateral discussion on all outer space weapons and shall commission further studies on how a space-weapons ban might be verified. true false justice true false neither conflict
CAN_74_2019_12 Strengthening multilateralism means involving all stakeholders. true false justice true false neither conflict
USA_63_2008_34 Other multilateral organizations have spoken clearly as well. true false neither true false neither neither
CAN_39_1984_65 In his most recent report he asks this: "Why has there been a retreat from internationalism and multilateralism at a time when actual developments both in relation to world peace and to the world economy would seem to demand their strengthening?" true false justice true false injustice conflict
USA_34_1979_132 In addition, it could recommend how current institutions could be strengthened and whether new multilateral ones should be created. true false neither true false neither neither
CAN_74_2019_75 We are fully committed to bringing innovative ideas to prevent and respond to crises, and we firmly believe in the power of multilateral solutions and partnership to address the most intractable of global challenges. true false justice true false neither conflict
CAN_76_2021_80 However, the benefits of multilateralism have not always been evenly or fairly distributed, and the potential for peace is yet to be realized in far too many places. true false injustice true false injustice injustice
USA_63_2008_15 Multilateral organizations have responsibilities. true false justice true false neither conflict
CAN_49_1994_35 As the leader of the multilateral Working Group on Refugees, we are tackling with determination that task entrusted to us by the international community. true false justice true false neither conflict
CAN_40_1985_93 Moreover we shall work towards the conclusion of a multilateral agreement prohibiting the possession and use of radiological weapons. true false justice true false neither conflict
CAN_40_1985_31 The challenges facing the multilateral system reach beyond this institution. true false neither true false neither neither
CAN_56_2001_19 We in Canada, as long- standing proponents of multilateralism and the United Nations system, have strongly welcomed the close collaboration evident between the United States Administration and the United Nations over these last two months. true false neither true false neither neither
USA_20_1965_212 We would be prepared to increase the amount of capital flowing through multilateral channels. true false neither true false neither neither
CAN_38_1983_194 They have not lost faith in the multilateral institutions we have so painstakingly constructed. true false neither true false justice conflict
CAN_59_2004_32 It is always preferable to have multilateral authority for intervention in the affairs of a sovereign State. true false justice true false neither conflict
CAN_40_1985_125 Bilateral relations between countries on the one hand, and the multilateral system on the other, ought to be mutually complementary and reinforcing. true false neither true false neither neither
USA_33_1978_43 The facilities of IMF have been expanded, and discussions are under way to expand the facilities of the multilateral development banks. true false neither true false neither neither
CAN_49_1994_102 Canada is ready to proceed with a comprehensive re- evaluation of its multilateral system as it applies to economic and social issues. true false justice true false neither conflict
PAK_60_2005_108 It is in the same spirit of constructive engagement that Pakistan wishes to pursue cooperative multilateralism and to strengthen the United Nations — this unique forum that is indispensable, in our interdependent world, for all States, large or small, powerful or weak. true false justice true false neither conflict
AFG_41_1986_34 The imperialist policy of using economic assistance as a means of exerting political pressure on the developing countries is no longer confined within the limits of bilateral relations; it has also been extensively employed to undermine the capabilities of multilateral institutions, with a view to curtailing or completely stopping the flow of international development assistance to developing countries, particularly those that dare refuse to submit to imperialist diktat. true false injustice true false injustice injustice
PAK_21_1966_212 The purpose of the conference, among other things, would be to consider, first, how the security of States without nuclear weapons can best be assured, preferably through multilateral guarantees; secondly, how they may co-operate among themselves in preventing further proliferation of nuclear weapons; and thirdly, the development and use of nuclear energy for exclusively peaceful purposes, through mutual co-operation and for mutual benefit, including the carrying out of explosions of nuclear devices for peaceful purposes under appropriate international supervision. true false justice true false neither conflict
LKA_32_1977_110 Existing national insurance agencies in developed countries could pool their risks through a reinsurance agency that might be set up on a multilateral basis. true false justice true false neither conflict
BGD_49_1994_168 Bangladesh looks forward to consolidating our relations with South Africa in more concrete ways, bilaterally and in all multilateral forums. true false neither true false neither neither
PAK_27_1972_102 Pakistan attaches the highest priority to both bilateral and multilateral efforts for disarmament. true false neither true false neither neither
PAK_66_2011_23 It is the United Nations and multilateralism that will safeguard that future. true false justice true false neither conflict
BTN_67_2012_56 As a small State, we have always attached the highest importance to multilateralism and the primacy of an effective United Nations that serves the interests of all its Member States. true false justice true false neither conflict
AFG_64_2009_9 In our increasingly interdependent world and the multilaterally oriented international system, the United Nations must assume greater responsibility for finding collective solutions to our challenges. true false justice true false neither conflict
LKA_47_1992_81 The successful conclusion of the convention on chemical weapons demonstrates the competence of the United Nations in multilateral negotiations. true false justice true false neither conflict
LKA_77_2022_12 In multilateralism, we talk to each other, we develop a relationship of trust and confidence and, if something were to come up, you have the base to work from. true false neither true false neither neither
NPL_32_1977_77 Similarly, there has been little progress in the multilateral trade negotiations in GATT on the reduction of trade barriers of developed countries, which restrict the exports of primary and processed commodities from developing countries. true false injustice true false injustice injustice
IND_36_1981_154 Total bilateralism at the expense of multilateral cooperation would run into alliances that may not be in the interest of harmonious relations and relaxation of tensions. true false justice true false justice justice
PAK_71_2016_4 Today, three decades after the end of the Cold War, our multipolar world is more free and vibrant, yet still chaotic and turbulent; more interdependent, but more unequal; more prosperous, yet still afflicted by poverty. false true injustice false true injustice injustice
PAK_66_2011_13 Each September we return to this great city and this grand stage so that we can restate and reaffirm the principles and values of multilateralism. true false justice true false neither conflict
PAK_59_2004_94 Today, there is welcome resurgence of support for multilateralism. true false neither true false neither neither
MDV_68_2013_63 SIDS need to be given full recognition within the global governance regimes and multilateral and financial institutions, and adequately integrated and institutionalized within the United Nations system. true false justice true false justice justice
NPL_37_1982_17 International economic co-operation and the system of multilateral co-operation continue to erode at an alarming rate. true false injustice true false injustice injustice
BTN_38_1983_94 Recently there has been much concern expressed at the deterioration of the resource position of some multilateral development organizations, UNDP in particular. true false neither true false neither neither
PAK_53_1998_146 This suggests the need for concerted action by the international community, which should include, first, the 14 strengthening of the capacity of multilateral institutions to address the issues of trade, finance and development in an integrated and coherent manner, thereby ensuring the effective governance of globalization. true false justice true false justice justice
CMR_74_2019_81 This occasion gives me the opportunity to warmly acknowledge the contribution of bilateral and multilateral partners that have always reaffirmed Cameroon’s unity and territorial integrity. true false neither true false neither neither
CMR_39_1984_157 As a corollary, multilateralism is dangerously declining; there are clear signs of this, mainly the current decrease in the resources made available to institutions such as UNDP, the fact that many States, particularly the more powerful ones, do not hasten to resort to the machinery offered by the United Nations to resolve issues of world-wide interest and, to some extent, the difficulties experienced now by UNESCO. true false injustice true false injustice injustice
ZWE_45_1990_61 Zimbabwe therefore wishes to appeal to the international community and to the various donors and multilateral economic institutions for support of our national efforts. true false justice true false justice justice
ZWE_52_1997_7 Africa has consistently underlined the need for reform of the United Nations and other multilateral bodies, to promote the democratization and effectiveness of the international decision-making process. true false justice true false justice justice
ZWE_73_2018_40 My country remains committed to strengthening multilateralism and the peaceful resolution of differences. true false justice true false neither conflict
SWZ_63_2008_61 The Kingdom of eSwatini reaffirms its commitment to the purpose and preservation of the central role of the United Nations in multilateral affairs. true false justice true false neither conflict
SOM_74_2019_6 In this globalized world, no nation, no matter how wealthy, strong or prepared, can individually stand alone against the tide of global challenges, which requires common action and coordinated multilateral responses. true false justice true false neither conflict
ZMB_72_2017_5 The theme for this session, “Focusing on people: Striving for peace and a decent life for all on a sustainable planet”, provides us with an opportunity to evaluate the existing multilateral approaches to address the challenges affecting our peoples. true false justice true false neither conflict
CPV_71_2016_10 As a small island State, Cabo Verde bases its foreign policy on the principles enshrined in the Charter of the United Nations and is certain that multilateralism is the most appropriate way to approach issues on the international agenda. true false justice true false neither conflict
SEN_75_2020_44 Furthermore, in implementing the Declaration on the commemoration of the seventy-fifth anniversary of the United Nations, which Senegal fully endorses, we must renew our commitment to the agenda for UN reform in order to restore multilateralism to its rightful place with the United Nations at its centre. true false justice true false justice justice
LSO_59_2004_75 That must be done against the background of multilateralism, upon which international peace and security is premised. true false justice true false justice justice
GNQ_77_2022_7 Equatorial Guinea is today launching an appeal on the need and importance of enhancing multilateralism and international cooperation, which are so necessary to address those global challenges and to revive development and sustainable economic growth in our countries, especially developing countries. true false justice true false justice justice
BFA_76_2021_7 The strategic vision you presented for your second term confirms your commitment and determination to work towards strengthening multilateralism and the United Nations in solving the multiple problems facing our world, especially in these difficult times caused by the coronavirus disease (COVID-19) pandemic. true false justice true false neither conflict
KEN_77_2022_104 It is time for multilateralism to reflect the voices of the farmers, represent the hopes of villagers, champion the aspirations of pastoralists, defend the rights of fisherfolk, express the dreams of traders, respect the wishes of workers and indeed protect the welfare of all the peoples of the global South. true false justice true false justice justice
GIN_50_1995_90 The United Nations and its specialized agencies, as the paramount instrument for cooperation and multilateral agreement, see their responsibilities increased today more than ever before. true false justice true false neither conflict
GIN_28_1973_13 The problem of international peace and security, the defence of national sovereignties, decolonialization and the struggle against imperialism continue to be the focus of both bilateral and multilateral meetings. true false justice true false justice justice
GNB_75_2020_44 We cannot talk about multilateralism while allowing the economic and financial embargo to continue to be imposed on one of the most supportive members of our Organization — Cuba. true false injustice true false injustice injustice
SLE_57_2002_57 Never since the end of the Second World War has multilateral cooperation become such a necessary means for resolving international disputes and for ensuring the well-being of people everywhere. true false justice true false justice justice
SLE_67_2012_70 We are confident that with sustained support from our bilateral and multilateral partners, we shall promote socioeconomic progress and provide a better standard of living for our people in an atmosphere of peace and security. true false justice true false neither conflict
ZMB_44_1989_9 We meet today against a background of great hopes and expectations for the success of multilateralism, which is so vital for the well-being of mankind. true false justice true false justice justice


Visualize

The plot below summarizes the outputs of both models and their overlap. I show the number of sentences each model assigned to the categories “justice”, “injustice”, and “neither”, with a separate bar for instances of agreement (“Overlap”). This side-by-side layout reveals the inconsistency across models, especially for ambiguous or borderline cases.

Important!

⚠️ The crucial insight for researchers considering LLMs for large-scale text analysis is the following: model choice matters, and the same prompt may yield divergent results across providers.

References

Baturo, Alexander, Niheer Dasandi, and Slava J. Mikhaylov. 2017. “Understanding State Preferences with Text as Data: Introducing the UN General Debate Corpus.” Research & Politics 4 (2): 19. https://doi.org/10.1177/2053168017712821.
Laurer, Moritz, Wouter van Atteveldt, Andreu Casas, and Kasper Welbers. 2024. “Less Annotating, More Classifying: Addressing the Data Scarcity Issue of Supervised Machine Learning with Deep Transfer Learning and BERT-NLI.” Political Analysis 32 (1): 116. https://doi.org/10.1017/pan.2022.15.

Footnotes

  1. The sample sentences used for this tutorial may be downloaded here.↩︎

  • CC BY-NC 4.0 License

Logo