Vol-3197/short1
Jump to navigation
Jump to search
Paper
| Paper | |
|---|---|
 edit
| |
| description | |
| id | Vol-3197/short1 |
| wikidataid | →Q117341496 |
| title | Asking Human Reasoners to Judge Postulates of Belief Change for Plausibility |
| pdfUrl | https://ceur-ws.org/Vol-3197/short1.pdf |
| dblpUrl | https://dblp.org/rec/conf/nmr/BakerM22 |
| volume | Vol-3197→Vol-3197 |
| session | → |
Asking Human Reasoners to Judge Postulates of Belief Change for Plausibility
Asking Human Reasoners to Judge Postulates of Belief
Change for Plausibility
(Extended Abstract)
Clayton K. Baker* , Thomas Meyer
University of Cape Town and Centre for Artificial Intelligence (CAIR), Cape Town, South Africa
Abstract
Empirical methods have been used to test whether human reasoning conforms to models of reasoning in logic-based artificial
intelligence. This work investigates through surveys whether postulates of belief revision and update are plausible with
human reasoners. The results show that participants’ reasoning tend to be consistent with the postulates of belief revision
and belief update when judging the premises and conclusion of the postulate separately.
Keywords
revision postulates, update postulates, human reasoning, survey
1. Introduction human reasoning is consistent with postulates, advanced
by Alchourrón, Gärdenfors and Makinson (AGM) [8], for
It has been shown that human reasoning displays non- belief revision. To enable comparison, we also hypothe-
monotonicity, but the methodologies that test this rela- sise that human reasoning is consistent with postulates,
tionship differ within the AI community. An example advanced by Katsuno and Mendelzon (KM) [9], for rea-
[1] of one approach is through surveys in which English soning with belief update. We investigate the hypotheses
translations of the postulates of defeasible reasoning were at the postulate level and at the system level. We use
judged for plausibility by human reasoners. As another the language of propositional logic in the formulation
example, a combined approach [2] was also used to inves- of our postulates to construct logically closed belief sets.
tigate the link between formal theories of non-monotonic Additionally, we note that once our hypotheses are tested
reasoning and the extent to which humans reason defea- using propositional logic as the underlying language, our
sibly. The combined approach involved a theoretical and results can be lifted to other forms of logic. This work
empirical analysis. In the theoretical analysis, the predic- extends previous work that investigated postulates of
tions of each system was compared using the Suppression defeasible reasoning [10] and belief change [11] with
Task [3], a logical experiment used in the psychology human reasoners via surveys. Furthermore, this work is
community in which subjects appear to retract valid logi- an extended abstracted of a paper that is currently under
cal inferences when subjects gain new information. In the review for a special issue of the Journal of Applied Logic.
empirical analysis, three experiments were used to test
the predictions of each system, as well as the inferences
of human reasoners, with strict and defeasible knowl- 2. Background
edge. While there are empirical studies that investigated
the relationship between non-monotonic reasoning with 2.1. Belief Revision
human reasoning, the relationship between belief change
The first form of belief change we investigated was revi-
and human reasoning has been primarily studied from a
sion. It is an approach to reasoning with changing beliefs
theoretical perspective, e.g. in classical logic [4], prob-
under the assumption that the world did not undergo a
ability and possibility theory [5], ontologies [6] and
fundamental change. It is characterised by a belief set 𝒦,
abstract argumentation [7]. Our first hypothesis is that
a revision operation * and reasoning rules referred to as
NMR’22: 20th International Workshop on Non-Monotonic Reasoning, postulates. A belief set is a set of propositional formulas
August 07-09, 2022, Haifa, Israel closed under logical consequence. A revision operation
*
Corresponding author. allows a reasoner to add new information to his beliefs
$ bkrcla003@myuct.ac.za (C. K. Baker); tmeyer@cs.uct.ac.za
if the new information is consistent with his beliefs. A
(T. Meyer)
https://tinyurl.com/5n6vuyp7 (C. K. Baker); revision operation also allows a reasoner to add an ex-
https://tinyurl.com/5ejbf2vr (T. Meyer) ception to his beliefs to account for the situation where
� 0000-0002-3157-9989 (C. K. Baker); 0000-0003-2204-6969 this exception or new information is inconsistent with
(T. Meyer) his beliefs. Moreover, the result of a revision operation
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License
Attribution 4.0 International (CC BY 4.0). must always be that a reasoner’s beliefs do not contradict
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
139
�one another. There are eight postulates in the AGM [8] then it is a partial preorder. A preorder is total if 𝑎 ≤ 𝑏
belief revision framework. (R1)–(R6) correspond to the or 𝑏 ≤ 𝑎 for all 𝑎, 𝑏 ∈ 𝑃 . A revision operator satisfies
core rationality postulates and the (R7)–(R8) correspond postulates (R1)–(R6) using the notion of a total preorder
to supplementary postulates. on interpretations while an update operator satisfies pos-
tulates (U1)–(U6) using the notion of a partial preorder
(R1) 𝒦 * 𝜇 implies 𝜇
on interpretations. By replacing postulates (U6) and (U7)
(R2) If 𝒦 ∧ 𝜇 is satisfiable, then 𝒦 * 𝜇 ≡ 𝒦 ∧ 𝜇 with a new postulate (U9), the class of update operators
(R3) If 𝜇 is satisfiable, then 𝒦 * 𝜇 is also satisfiable can be designed using total preorders. The second and
(R4) If 𝒦1 ≡ 𝒦2 and 𝜇1 ≡ 𝜇2 , then 𝒦1 *𝜇1 ≡ 𝒦2 *𝜇2 more important difference between revision and update
(R5) (𝒦 * 𝜇) ∧ 𝜑 implies 𝒦 * (𝜇 ∧ 𝜑) is that, in the case of update, a different ordering is in-
(R6) If (𝒦 * 𝜇) ∧ 𝜑 is satisfiable, then 𝒦 * (𝜇 ∧ 𝜑) duced by each model of 𝒦, while for revision, only one
implies (𝒦 * 𝜇) ∧ 𝜑 ordering is induced by the whole of 𝒦.
(R7) If 𝒦 * 𝜇1 implies 𝜇2 and 𝒦 * 𝜇2 implies 𝜇1 , then
𝒦 * 𝜇1 is equivalent to 𝒦 * 𝜇2
(R8) (𝒦 * 𝜇1 ) ∧ (𝒦 * 𝜇2 ) implies 𝒦 * (𝜇1 ∨ 𝜇2 )
3. Methodology
Our empirical investigation took place through four ex-
2.2. Belief Update periments. In the first experiment, we prepared a survey
of 30 general statements about the world for participants
The next form of belief change we investigated was up- to evaluate for clarity and bias. 7 participants had to com-
date. It is an approach to reasoning with changing beliefs plete a table in which they identified statements with am-
after some fundamental shift in the world occurred. It biguous language and biased examples. In the second ex-
is characterised by a belief set 𝒦, an update operation ◇ periment, we prepared a survey of 30 general statements
and postulates for reasoning. As with revision, 𝒦 refers about the world taken from refining the material in the
to a logically closed set of propositional formulas. When first experiment. 30 participants evaluated the degree to
we update 𝒦 with new information 𝜇, we are saying that which they believed each of the statements in the survey
we used to believe 𝒦, we know now that 𝜇 holds, and and explained their answers. In the third experiment, we
we need to modify 𝒦 by adding 𝜇, acknowledging that prepared a survey of English statements corresponding
we may have been wrong if 𝜇 contradicts 𝒦. There are to translations of the AGM postulates for belief revision.
nine postulates in the KM [9] belief update framework. 35 participants on Mechanical Turk (MTurk) evaluated
(U1) 𝒦 ◇ 𝜇 implies 𝜇 the degree to which they believed each statement in the
(U2) If 𝒦 implies 𝜇 then 𝒦 ◇ 𝜇 is equivalent to 𝒦 survey. We tested our hypothesis statistically and deter-
(U3) If both 𝒦 and 𝜇 are satisfiable then 𝒦 ◇ 𝜇 is also mined whether the association between the premises and
satisfiable the conclusion for each postulate holds for the general
(U4) If 𝒦1 ↔ 𝒦2 and 𝜇1 ↔ 𝜇2 then 𝒦1 ◇ 𝜇1 ↔ English-speaking reasoner. In the last experiment, we
𝒦2 ◇ 𝜇2 used the same material from the belief revision experi-
ment to instantiate the KM belief update postulates. The
(U5) (𝒦 ◇ 𝜇) ∧ 𝜑 implies 𝒦 ◇ (𝜇 ∧ 𝜑)
experimental setup followed a similar approach to the
(U6) If 𝒦 ◇ 𝜇1 implies 𝜇2 and 𝒦 ◇ 𝜇2 implies 𝜇1 then belief revision experiment. We obtained ethical clearance
𝒦 ◇ 𝜇1 ↔ 𝒦 ◇ 𝜇2 from the Faculty of Science Ethics Research Committee
(U7) If 𝒦 is complete then (𝒦 ◇ 𝜇1 ) ∧ (𝒦 ◇ 𝜇2 ) implies at the University of Cape Town. We include the consent
𝒦 ◇ (𝜇1 ∨ 𝜇2 ) forms and a link to our data management plan in our
(U8) (𝒦1 ∨ 𝒦2 ) ◇ 𝜇 ↔ (𝒦1 ◇ 𝜇) ∨ (𝒦2 ◇ 𝜇) Github project repository, linked in Appendix A. For
(U9) If 𝒦 is complete and (𝒦 ◇ 𝜇) ∧ 𝜑 is satisfiable then the bulk of our reasoning experiments, we used Google
𝒦 ◇ (𝜇 ∧ 𝜑) implies (𝒦 ◇ 𝜇) ∧ 𝜑 Forms to design our surveys, and we used Mechanical
Revision and update differ from non-monotonic logic Turk to crowdsource our data collection.
using the concept of orders on interpretations. A ho-
mogeneous relation ≤ on some given set 𝑃 , so that by 4. Results
definition ≤ is some subset of 𝑃 × 𝑃 and the notation
𝑎 ≤ 𝑏 is used in place of (𝑎, 𝑏) ∈ 𝑃 , is called a preorder In this work, we investigated the endorsements of each
if the relation is also transitive and reflexive. A reflexive component of the AGM postulates when formulated as
relation has the property that 𝑎 ≤ 𝑎 for all 𝑎 ∈ 𝑃 . A material implication statements. We found evidence for
transitive relation has the property that if 𝑎 ≤ 𝑏 and whether or not the participants found our concrete instan-
𝑏 ≤ 𝑐 then 𝑎 ≤ 𝑐 for all 𝑎, 𝑏, 𝑐 ∈ 𝑃 . If a preorder is also tiations of the AGM postulates plausible. We determined
anti-symmetric, that is, 𝑎 ≤ 𝑏 and 𝑏 ≤ 𝑎 implies 𝑎 = 𝑏, whether the postulates hold in general. The results show
140
�that the participants’ reasoning tends to be consistent tionships. The empirical part will focus on identifying
with the 9 AGM postulates, with the significance of the as- representations of the postulates that support both the
sociation between the endorsement of the premises and theory and the beliefs of human reasoners. It will in-
the endorsement of the conclusion ranging from non- volve the development of an online reasoning tool that
significant to highly significant. The number of logical automates the production and presentation of structured
violations per postulate is generally low with a range of reasoning examples in a survey setting. The structured
0 to 4 violations (< 12% of participants). The exception reasoning examples will depict the static and dynamic
is postulate (R5) with 54,29% (19 participants) endorsing nature of changing beliefs in terms of a revision and an
the premises, but not the conclusion. update, respectively. In turn, the tailored surveys created
We also investigated the endorsements of each compo- using the reasoning tool can be used to elicit responses
nent of the KM postulates when formulated as material from human reasoners, the design of which improves
implication statements. We found evidence for whether upon the limitations of question types from conventional
or not the participants found our concrete instantiations survey platforms like Google Forms and Microsoft Forms.
of the KM postulates plausible. We determined whether Furthermore, non-parameterised statistical methods, that
the postulates hold in general. The results show that the is, methods that do not assume how the sample data is
participants’ reasoning tends to be consistent with the 9 distributed, e.g. the Wilcoxon signed-rank test [12, 13],
postulates of KM belief update, with the significance of will be used to interpret the significance of the postulates
the association between the endorsement of the premises of belief change, as found by human reasoners.
and the endorsement of the conclusion ranging from non-
significant to highly significant. The number of logical
violations per postulate is generally low as well with a Acknowledgments
range of 0 to 8 violations (< 23% of participants). The
We wish to express our sincere gratitude and appreciation
exception is postulate (U8) with 48,57% (17 participants)
to the DSI – CSIR Interbursary Support (IBS) Programme
endorsing the premises, but not the conclusion.
and the Centre for Artificial Intelligence Research (CAIR)
for financial support.
5. Conclusions and Future Work
Our work builds on previous empirical studies involv- References
ing human subjects who are tasked with reasoning non-
[1] R. Neves, J. Bonnefon, E. Raufaste, An empirical test
monotonically. We created a reproducible approach for
of patterns for nonmonotonic inference, Annals
empirically investigating the plausibility of postulates
of Mathematics and Artificial Intelligence 34 (2002)
of belief change. This approach accounts for the effect
107–130.
of the premises and conclusion of each postulate and
[2] M. Ragni, C. Eichhorn, T. Bock, G. Kern-Isberner,
determines whether the overall postulate is found plau-
A. Tse, Formal nonmonotonic theories and prop-
sible with statistically significant evidence. We applied
erties of human defeasible reasoning, Minds
this approach to the formal theory of belief revision and
and Machines 27 (2017) 79–117. doi:10.1007/
update. We hypothesised that human belief change is
s11023-016-9414-1.
consistent with the AGM postulates of belief revision
[3] R. M. Byrne, Suppressing valid inferences with
and the KM postulates of belief update. The results show
conditionals, Cognition 31 (1989) 61–83.
that the participants’ reasoning tends to be consistent
[4] F. Lévy, A survey of belief revision and updating in
with the 8 AGM postulates, (R1)–(R8), with the signifi-
classical logic, International Journal of Intelligent
cance of the association between the endorsement of the
Systems 9 (1994) 29–59.
premises and the endorsement of the conclusion ranging
[5] D. Dubois, H. Prade, A survey of belief revision
from non-significant to highly significant. The results
and updating rules in various uncertainty models,
also show that the participants’ reasoning tends to be
International Journal of Intelligent Systems 9 (1994)
consistent with the 9 KM postulates, (U1)–(U9), with the
61–100.
significance of the association between the endorsement
[6] C. Boutilier, N. Friedman, J. Y. Halpern, Belief revi-
of the premises and the endorsement of the conclusion
sion with unreliable observations, in: AAAI/IAAI,
ranging from non-significant to highly significant.
1998, pp. 127–134.
In future work, we will refine our approach in the
[7] S. Doutre, J.-G. Mailly, Constraints and changes: A
following way. We will conduct a theoretical and em-
survey of abstract argumentation dynamics, Argu-
pirical investigation of the postulates of belief revision
ment & Computation 9 (2018) 223–248.
and update. The theoretical part will build on this work
[8] C. E. Alchourrón, P. Gärdenfors, D. Makinson, On
by exploring inter-postulate and inter-framework rela-
the logic of theory change: Partial meet contraction
141
� and revision functions, Journal of Symbolic Logic
50 (1985) 510–530. doi:10.2307/2274239.
[9] H. Katsuno, A. O. Mendelzon, On the difference
between updating a knowledge base and revising
it, Belief revision (1991) 183.
[10] C. Baker, C. Denny, P. Freund, T. Meyer, Cognitive
defeasible reasoning: the extent to which forms
of defeasible reasoning correspond with human
reasoning, in: Proceedings of the First Southern
African Conference for Artificial Intelligence Re-
search (SACAIR 2020), CCIS, Springer, 2020, pp.
119–219.
[11] C. Baker, T. Meyer, Belief change in human rea-
soning: An empirical investigation on mturk, in:
Proceedings of the Second Southern African Con-
ference for Artificial Intelligence Research (SACAIR
2021), 2021, pp. 520–536.
[12] X. Li, Y. Wu, M. Wei, Y. Guo, Z. Yu, H. Wang, Z. Li,
H. Fan, A novel index of functional connectivity:
phase lag based on wilcoxon signed rank test, Cog-
nitive Neurodynamics 15 (2021) 621–636.
[13] L. Zhou, Performance of corporate bankruptcy
prediction models on imbalanced dataset: The effect
of sampling methods, Knowledge-Based Systems
41 (2013) 16–25.
A. Additional resources
The Github repository for this work, containing supple-
mentary material and code scripts, can be accessed via
this URL, https://tinyurl.com/2p98m76n.
142
�