Moacyr Machado Cardoso Junior
Instituto Tecnológico de Aeronáutica, Brazil
E-mail: moacyr@ita.br
Submission: 04/09/2018
Revision: 25/09/2018
Accept: 01/10/2018
ABSTRACT
“Black swan” events
represent a critical issue in risk analysis. Events with extremely low
probability of occurrence are in general discarded from the risk analysis
process. This paper aims to identify and characterize four accidents that
occurred in Brazil into the following classes: “not a black swan”, “black swan:
unknown-unknown”, “black swan: unknown-known” and “black Swan: not believed to
occur”, by obtaining from experts the distribution of belief for the real
probability of each class. Results showed that, throughout all cases analyzed,
the class “black swan: unknown-unknown” was never reported, which means that
none of the cases studied were a complete surprise to anyone. The method used
was able to assign all accident events to the remaining classes. Probability
distribution elicited from experts showed large disagreement among them, and
the expected value was considered low. Nevertheless, the elicited distributions
can be utilized in future risk analysis as a priori distribution in a Bayesian approach.
Keywords: Black Swan; Expert elicitation;
Technological accidents; Risk analysis
1. INTRODUCTION
In
risk assessment context, specialists are demanded for the risk assessment
process, which involves identification, analysis, evaluation, and decision-making
about possible controls and/or mitigating measures from inherent risks of a
process, system, equipment, and others (ABNT-ISO, 2009).
In
the identification phase, all threats and/or hazards inherent to the system are
raised. This phase is crucial because unidentified potential events will not be
analyzed and will not be part of the decision-making process. In the risk
analysis phase, causes and their sources are evaluated, and the consequences of
an unwanted event are estimated. The threefold cause-event-consequence defines
one risk, and for each risk, its event likelihood is determined associated to a
cause and the consequences of its materialization. In the last phase, risk
evaluation, tolerance criteria, or attitude towards risk are compared with
values obtained in the risk analysis phase. This comparison helps the
decision-making process about risk treatment, as well as its prioritization.
Events
described by Aven, (2013) and Flage and Aven, (2015) as Black Swans are
characterized by the following attributes: They are considered outliers, that
is, they represent extremely improbable events with extreme consequences, and
as a rule, after their materialization they are perfectly explicable and
predictable by the experts, which represents a paradox.
These
events represent a real threat to the risk assessment process, as they will not
be part of the analysis, or if they are identified, they will be discarded
after that because of their extremely low likelihood. Therefore, new strategies
have to be developed to incorporate and maintain these events in risk analysis
process as described by (AVEN, 2015).
A
fundamental step towards this direction is to understand how these events are
perceived by experts. Thus this paper aims to identify and classify different
black swan events that occurred in technological accidents in Brazil based on
elicitation from experts, and consequently, point to general strategies that
could help risk analysts to cope with “Black Swans”.
This
paper is structured as follows: Beyond this short introduction and
contextualization, are sections: Section 2 with the theoretical foundation
about black swan events and elicitation tools. In section 3, the research
method is presented. Section 4 contains the results and discussion. Finally in
section 5, the final considerations and proposals for new research on this
theme are presented.
2. THEORETICAL FOUNDATIONS
2.1.
“Black
Swans”
Black
swan event theory is derived from a metaphor that describes an event that comes
as a surprise and has a huge negative impact. The term "Black Swan"
comes from the idea that these animals would not exist in nature, which was
later contradicted by the discovery of these animals in Australia in 1967 (TALEB,
2010).
According
to Aven, (2013), the term was popularized by Taleb, (2007) in his book
"The Black Swan", and highlighted three attributes associated with
the term: (i) It is an outlier, in the sense that one cannot expect the
occurrence of that event based on previous events; (ii) It involves an extreme
impact; and (iii) After its occurrence people and experts invent logical
explanations, making the event explicable and predictable.
The
first attribute is questioned by some researchers because if the event is an
outlier in the interpretation of probability theory, even small probabilities
of occurrence can be expected to occur given the time-space considered.
The
concept of "Black Swan" can be seen in two ways according to Aven, (2013):
(i) a rare event with extreme consequences; and (ii) an extreme, astonishing
event concerning current knowledge and/or belief, and that the latter may be
more appropriate.
"Black
Swan" events can be classified into three basic types: (i) events that are
completely unknown in the scientific milieu, termed unknown-unknown, as a reference
to the fact that they are unknown to risk analysts and science; (ii) Events
that are not on the list of risk analysts, but are on the list of experts
and/or science (unknown-known); (iii) Events known to risk analysts, but judged
to have negligible probability of occurrence and therefore are not amenable to
analysis because they are judged as "not believed to occur" (FLAGE; AVEN,
2015).
The
authors also cite a few examples of the three types: The use of thalidomide in
1957, a drug that caused congenital malformation of the upper limbs, which was
totally unknown to physicians and scientists. The attack on the Twin Towers on
September 11 can be considered as type (ii), and finally the tsunami that
destroyed the Fukushima nuclear reactor is an example of type (iii).
In
Figure 1, the conceptual map for the focal question is presented: what are
"Black Swans"? Which summarizes what has been discussed up to this
point.
Figure 1: Conceptual
map of “Black Swan”.
Another
concept is "Perfect Storm" to designate a rare event involving
uncertainty, and this is represented by the randomness of joint but known
events. It differs from the term "Black Swan" because the latter
involves epistemic uncertainty or lack of knowledge, i.e. not only the lack of
knowledge of the distribution of probabilities but the ignorance of the
phenomenon itself (PATÉ-CORNELLl, 2012). The author concludes that proactive
management with early warnings, quick detection, and mainly agile responses
enable analysts to comply with Augustine's
Law XLV, i.e. "One should expect that the expected can be avoided,
but the unexpected should have been expected".
2.2.
Probability
elicitation
Elicitation
is a process of constructing probability distributions from the extraction of
beliefs and expert knowledge about one or more uncertainties.
Much
of the literature on elicitation is concerned with constructing a probability
distribution to model uncertainties when there is insufficient data to
construct a model (GARTHWAITE; KADANE; HAGAN, 2005). As an example, the authors
cite the case of decision making where the uncertainty regarding the theme must
be represented by a distribution of probabilities in order to maximize the
expected utility.
People
are affected by heuristics and biases in how they respond to uncertainty issues
(GARTHWAITE et al., 2005). Some
heuristics that influence the elicitation process are described by Burgman, et al., (2006). One of them is the
representativeness, that is, as the number of details in a given scenario
increases, its probability may only decrease, although due to its
representativeness apparent probability grows. Another heuristic cited by the
authors is availability, that is, common events, or more likely, or more recent
or even those that have been very explored by the media. Finally, the anchorage
heuristic states that when a person is asked to estimate a number, percentage
or range of values, people anchor in values that have been previously suggested
or have arisen from other judgments.
The
great advantage of the elicitation process is to use the results for the
decision-making process, and in this context, the capture of expert opinion is
fundamental as well as the possibility of constructing the prior distribution
for inference in a Bayesian process (GARTHWAITE et al., 2005).
Several
authors used probability elicitation in different applications like
probabilities of explosion in different scenarios (MACDONALD; SMALL; MORGAN,
2008). They asked experts for estimates of the upper (U) and lower (L) bounds
of probability, and next asked them to give the mode or most likely value, and
then divided the interval [L,U] into six subintervals. The authors found lack
of consensus among experts. Predictive elicitation of subjective probability
distributions was used to evaluate the effectiveness of Risk Control Option
(RCO) for reducing the risk of ship collisions in Australia's Territorial Sea
and Exclusive Economic Zone (HOSACK; HAYES; BARRY, 2017).
Ioannou
et al., (2017) and elicited 13
international experts on the responses of a generic mid-rise cast-in-place
reinforced concrete frame when exposed to different fire intensities, and then
asked them to judge the level of response that would be required to cause a
given level of damage. Elicitation offers a feasible method to generate
evidence for the missing information, but a number of key issues must be analyzed in a real elicitation
process, like weighting, aggregation, and others (BOIKE et al., 2010).
In an
expert elicitation of climate, energy, and economic uncertainties, (USHER;
STRACHAN, 2013) found that while experts agreed on the structure of the
uncertain parameters, the shape of the distributions representing their beliefs
varied widely, which reflects the different perspectives of the interviewees.
Decomposing the structure of the parameters and exploring the influence of
dependence on expert responses may help explain some of these differences.
However, the pooled beliefs are insensitive to the weighting assumptions that
compensate for bias and correlations within and among experts.
One
of the usual methods for probability elicitation is quartiles, in which experts
are asked to assign the median value for the distribution and then evaluate
other points of the distribution. Quartile methods are the best for damping
heuristics and biases from experts, according to (GARTHWAITE; KADANE; HAGAN,
2005; MORRIS; OAKLEY; CROWE, 2014)
Aggregation
of expert distribution typically rely on very simple combination schemes, such
as ascribing equal weight to all the participating experts. (ASPINALL, 2005).
Although other forms like Cooke´s classical method, which is a weighted
arithmetic average of the experts’ probability distributions, many simulated
studies were no better than the simplest simple arithmetic average (CLEMEN, 2008).
3. METHOD
To
identify and characterize Black Swan events, an expert elicitation research
questionnaire was carried out based on technological accidents segregated by
type: Aircraft accident – GOL-Legacy in 2006; Construction Industry - Tim Maia
bicycle pathway in 2016 and Metro-SP in 2007; and Fire - Terminal Alemoa in
2015, which are explained below.
3.1.
Event:
Aircraft Accident - Gol 1907 – Legacy – year: 2006
Gol
Flight 1907 was a domestic commercial route, operated by Gol Airlines, using a
Boeing 737-8EH. On September 29, 2006, the aircraft departed from the Eduardo
Gomes International Airport in Manaus to the Galeão International Airport in
Rio de Janeiro and scheduled a stop at the Juscelino Kubitschek International
Airport in Brasilia. As it flew over Mato Grosso state, it collided mid-air
with an Embraer Legacy 600. All 154 passengers and crew aboard the Boeing 737
died after the aircraft collided in the air and crashed into a closed forest
area. However, the Legacy, despite having suffered severe damage to its wing
and horizontal left stabilizer, landed safely with its seven uninjured
occupants at the Air Base of Cachimbo. CENIPA concluded that the accident was
caused by both air traffic controllers and Legacy pilots errors, while the
National Transportation Safety Board (NTSB) determined that all pilots acted
correctly and were placed on a collision route by a variety of air traffic
controller errors (WIKIPEDIA, 2017).
3.2.
Event:
Construction Industry: Accident on the Tim Maia bicycle pathway in Rio de Janeiro-
year: 2016.
An
section of the Tim Maia bicycle pathway, which was opened in January 2016 in
the São Conrado neighborhood of Rio de Janeiro, collapsed on the morning of
January 21. Two men died. According to one witness, a series of strong waves could
have hit the bike lane before the crash. "The wave swept the track, which
fell apart like paper," the man explained. Another witness said he saw
three people fall into the sea after the collapse (UOL, 2016).
According
to Oliveira, (2016), one of the main conclusions of the civil engineers
involved in the study is that the design of the project failed, because
preliminary oceanographic studies of the effects of waves on the structure of
the bicycle lane were lacking.
3.3.
Event
- Construction Industry: Accident in the
work of the Metro – SP – year: 2007
On
January 12, 2007, the work of the future Pinheiros station of line 4 (yellow)
of the São Paulo subway collapsed. The accident, according to the builders
responsible for the work, occurred due to the instability of the region's soil,
aggravated by the heavy rains that hit the city days before. The event
culminated in the deaths of 7 people and a total of 271 people were affected due
to the need for resettlement (FOLHA DE SÃO PAULO, 2007).
According
to the G1 news portal, the defendants were cleared of any criminal offense in a
court decision in May 2016. The prosecutor's office defended in the complaint
that the employees were negligent. The complaint states that problems were
detected in the tunnel in the month prior to the tragedy and, the day before,
the decision of those responsible for the work was to install reinforcing
structures. However, the work continued without the installation of those
structures (G1, 2016a).
A
report in the same news portal stated that the judge said the evidence proved
that the accident would probably have happen even with the installation of the
reinforcing structures. He also said that there was no indication of the
accident and that those responsible took the necessary care.
"Now
the accused had no way of predicting the accident, because of all the
circumstances. The execution of the work project was within normalcy, all the
teams carefully monitored each step of the execution and did not point out any
situation that indicated the possibility of an accident."
The
prosecution said in 2011 that there was recklessness, malpractice, human and
technical error. The process questioned the quality of the material used and
the neglect of preventive measures and failures in soil analysis. In other
words, for the prosecution, the tragedy could have been avoided if those
responsible had alerted the authorities and interrupted the work in time.
3.4.
Event: Fire in Alemoa Terminal –
year: 2015
The
fire at the company Ultracargo began around 10 am on April 2 and was
extinguished on April 9, 2015. Six fuel tanks were hit, but nobody was injured.
At the start of the fire, the temperature reached 800 °C. Federal Government
assistance was required, and fire-fighting products had to be imported to stop
the flames. The Environmental Company of the State of São Paulo (Cetesb) fined
the company R$ 22.5 million for the fire and Santos City Hall imposed a fine of
R$ 2.8 million (G1, 2016b).
According
to the same portal, the cause of the accident was a pump connected to the
valves that were closed, and due to the pressure caused, the tanks exploded.
"The
pump should not have been put into operation because it had the inlet and
outlet valves closed, and an operator inadvertently had it run. This caused the
pump to run without the circulation of fuel until it generated the explosion by
the pressure buildup".
For
each of the events listed, experts were chosen based on their professional
background, so that the analyzed events were related to their area of expertise.
Snowball sampling was used as each expert was invited to nominate another
expert who could participate in the survey. Google forms were used to send the
questionnaires. A brief contextualization of the research, highlighting what
Black Swan events are and the accidental scenario according to the expert's
area of knowledge, was presented so that they estimate the probability that the
event described is framed in one of the classes: “Not a Black Swan”, “Black
Swan: unknown-unknown”, “Black Swan: unknown-known” and “Black Swan: not
believed to occur". An example of this form is in Appendix A.
The
quartile method was used in the elicitation process, to minimize the expert´s
biases and heuristics. The fitting of the individual probability distributions
and aggregation was performed with the SHELF: Tools to Support the Sheffield
Elicitation Framework (OAKLEY, 2017) software package R (R DEVELOPMENT CORE TEAM,
2017). Equal weights were assigned to the different experts so that the final
probability distribution was obtained by the linear aggregation algorithm.
4. RESULTS
The
results obtained within the previously described method for each type of
technological accident studied will be presented in this section. The quotes
from experts are translated from Portuguese.
4.1.
Event:
Aircraft Accident - Gol 1907 – Legacy – year: 2006
This
accident was analyzed by 23 experts, including air traffic controllers,
aircraft pilots, and aviation safety experts. Of these, only 1 was discarded
from the analysis due to incoherence in the probability elicitation process.
As a
general result, the experts classified the accident as shown in Table 1. The vast majority (70.8%) considered this
type of event to be perfectly predictable, but with a very low probability, and
thus it represents an unbelievable event, classifying it as “Black Swan: not
believed to occur". This view can be confirmed in the justifications of
the experts, among which we highlight the following:
·
Expert A: “I consider it of known causes but of very
small probability.”
·
Expert B: “I believe that the lack of training and
investment has made the possibility of an accident to be underestimated,
despite all the statements by the responsible bodies that this care is one of
the pillars of the aviation and control system in the country.”
·
Expert C: “because it had never happened before this
way.”
·
Expert D: “With the current technologies of (Traffic
Collision Avoidance System – (TCAS), among others, those responsible assume
that the collision between two aircraft in flight is something almost
impossible to occur and end up denying the risks of this situation.”
·
Expert E: “Everyone knew it could happen, but no one
believed it would happen.”
·
Expert F: “Due to the existence of transponder-type
equipment and TCAS for example, this type of accident is known to science and
risk analysts, but its likelihood of occurrence is very unlikely.”
Table 1: Classification of the event that occurred
with the Aircraft Accident - Gol 1907 – Legacy – year: 2006.
“Not
a Black Swan” |
“Black
Swan: Unknown-Unknown” |
"Black
Swan: Unknown-Known” |
"Black
Swan: not believed to occur” |
20.8% |
0% |
8.3% |
70.8% |
The
second class, with 20.8% was classified as not being a Black Swan by the
experts, and the justifications were:
·
Expert A: “The technological evolution of the air
navigation provided an almost perfect route in the middle of the airways with
almost no lateral deviation. Then the aircraft, due to the advanced navigation
equipment on board, maintaining the same level of flight, passed at the same point.”
·
Expert B: “Several reports of hazard (RELPREV), now
known as Prevention Reports, completed prior to the accident, already indicated
that something like this could occur. I have filled in many.”
·
Expert C: “The factors that caused the accident are
known, but the conjunctures of the system (human and equipment failures) were
determinant for it to occur.”
And
finally, the lowest of the groups (8.3%) was classified as being a Black Swan
of unknown nature by the analysts and known to the experts.
Some
experts reported:
·
Expert A: “The indication of TCAS OFF with low
visibility could be associated with human error. The coincidence of the routes
was predictable by the precision of the technology. It is the case of joining
the facts together, which is not trivial.”
·
Expert B: “Traffic operators generally knew the risks
and avoided them.”
It is
noteworthy that no expert considered the event to be of the Unknown-Unknown
Black Swan type.
We
also obtained the results of the elicitation process of the distribution of
subjective probability of the experts in each classification, according to
Table 2.
Table 2: Probability distribution elicitation for “p”
in the event Aircraft Accident - Gol 1907 – Legacy – year: 2006.
Black Swan type |
Expert |
Best fitting (least square) |
Model parameters |
“Not a Black Swan” |
A |
Beta |
𝜶 = 0.9999813 𝜷 = 1.000008 |
B |
Beta |
𝜶 = 1.6925468 𝜷 = 1.196452 |
|
C |
Beta |
𝜶 = 2.4078168 𝜷 = 1.310733 |
|
D |
Normal |
µ = 0.1538458 σ = 0.1140464
|
|
E |
Beta |
𝜶 = 4.3184044 𝜷 = 2.683653 |
|
“Black-Swan
Unknown-Known” |
A |
Normal |
µ = 0.2499995 σ = 0.1853250 |
B |
Beta |
𝜶 = 0.4265952 𝜷 = 0.5049448 |
|
“Black Swan not
believed to occur” |
A |
Beta |
𝜶 = 0.8971646 𝜷 = 0.3533580 |
B |
Beta |
𝜶 = 7.6919001 𝜷 =1.5864494 |
|
C |
Beta |
𝜶 = 1.6352622 𝜷 = 0.6040447 |
|
D |
Normal |
µ = 0.4285489 σ = 0.2115726 |
|
E |
Beta |
𝜶 = 0.7982027 𝜷 = 1.4190533 |
|
F |
Beta |
𝜶 = 1.1738755 𝜷 = 0.6334071 |
|
G |
Beta |
𝜶 = 0.9999813 𝜷 = 1.0000085 |
|
H |
Beta |
𝜶 = 0.9999813 𝜷 = 1.0000085 |
|
I |
Beta |
𝜶 = 0.5234347 𝜷 = 1.2619953 |
|
J |
Beta |
𝜶 = 0.9999813 𝜷 = 1.0000085 |
|
K |
Beta |
𝜶 = 0.7329743 𝜷 = 0.3972961 |
|
L |
Beta |
𝜶 = 0.9999813 𝜷 = 1.0000085 |
|
M |
Beta |
𝜶 = 0.6203376 𝜷 = 1.0806761 |
|
N |
Beta |
𝜶 = 0.3533580 𝜷 = 0.8971646 |
|
O |
Beta |
𝜶 = 0.9999813 𝜷 = 1.0000085 |
|
P |
Beta |
𝜶 = 0.9999813 𝜷 = 1.0000085 |
Table
2 presents, for each Black Swan class, the distribution that provided the best
fit according to the least squares criterion between the adjusted model and the
data of each expert, as well as, the values obtained for the parameters. Three
models of probability were used in this adjustment: Beta, Gamma, and Normal
distributions that have their functional forms defined by:
Beta: p ~ Beta( α, β), (Eq.1).
where
Probability-Density Function - pdf is:
|
Eq. 1 |
α, β > 0 ; p ∈ [0;1]
α, β are Beta shape parameters.
Gamma: p ~ Gamma (α, β), (Eq.2).
|
Eq. 2 |
α, β > 0; p > 0
α: Gamma shape parameter.
β: Gamma rate parameter.
And Γ: is the gamma function (Eq.3), defined as:
|
Eq. 3 |
Normal: p ~ Normal(µ, σ^2)
where:
|
Eq. 4 |
µ, σ: represents distribution mean and standard
deviation, respectively.
Figure
2 presents, as an example, the result for the elicitation process of the class
"not a Black Swan".
In
class “not a Black Swan”, four of five best fitting were Beta and only one was
Normal distributed, and so we decided to consider Beta distribution for all
cases, including “Black Swan: unknown-known” and “Black Swan: not believed to
occur”. This decision is possible because the values for “p” are ranged in
[0,1] and strictly positive, so Beta is technically better than Normal.
Figure 2: The
Beta probability distribution for "not a Black Swan", in Aircraft
Accident - Gol 1907 – Legacy – year: 2006, for all experts considered.
In
Table 3, the quantiles values for the experts are provided considering Beta
distribution. There is not much
agreement among experts for “p”, and the class of Black swan considered was not
strongly evidenced.
Table 3: Probability “p” that the Aircraft Accident -
Gol 1907 – Legacy – year: 2006 was “not a Black Swan”, fitted by Beta
distribution, for 0.25; 0.50; 0.75; and 0.95 quantiles.
Quantil |
Expert |
||||
A |
B |
C |
D |
E |
|
0.25 |
0.250 |
0.394 |
0.492 |
0.0801 |
0.497 |
0.50 |
0.500 |
0.608 |
0.676 |
0.1470 |
0.628 |
0.75 |
0.750 |
0.796 |
0.828 |
0.2380 |
0.748 |
0.95 |
0.950 |
0.949 |
0.954 |
0.3990 |
0.879 |
The
average of the expected values of experts were calculated, and the following
values were obtained: E[p] = 0.5193, for the Black Swan “not believed to occur”
type; E[p] = 0.5042 for “not a Black swan”; and E[p] = 0.3644 for “Black Swan:
Unknown-Known” type. The aggregation of the expected value of different experts
was performed by linear aggregation.
The
elicitation process was able to produce the aggregate distribution of the “p”
values. These values can be used in the future as a priori distribution for
future studies in risk analysis of events of this nature.
The
conclusion is that although the experts classified the event into different
types of Black Swan, they were not able to produce a “p” value consistent with
their classification, denoting the great discrepancy among the experts.
4.2.
Event:
Construction Industry: Accident on the Tim Maia bicycle pathway in Rio de
Janeiro- year: 2016.
This
accident was analyzed by 7 experts, including safety engineers, oceanographers,
and risk analysts.
As a
general result, 43% of the experts judged that this type of event was “not a
Black Swan” and 57% classified it as “Black Swan: not believed to occur”. This
classification was justified by each group of experts as follows:
For
the group that considered it to be “not a Black Swan”:
· Expert
A: “The event was not "unknown", I believe it was overlooked by human
failure.”
· Expert
B: “The event is perfectly predictable by civil engineering.”
· Expert
C: “A work on the shoreline should consider the possibility of bad weather and
the action of waves, in other words, have there never been waves of this nature
that were known to the authorities?”
And
for “Black Swan: not believed to occur”:
· Expert
A: “There were elements to at least take a sufficient storm surge to project
waves to the walkway; The locking system of the chosen structure, while
suitable for the intended load, disregarded this possibility.”
· Expert
B: “This is a predictable event since there is a known hazard (waves crashing
on the rocks and history of the height and intensity that they can reach). With
known design data, it would be possible to evaluate, even qualitatively, the
occurrence of the event and the intensity sufficient to move the walkway. This
evaluation could be done in the design stage, as previously determined, for
example, the best type of mooring of the boards to the pillars.”
· Expert
C: “I imagine that the work could have been designed and constructed to withstand
the scenario, but it was not for its probability was considered low..
· Expert
D: “The variation in sea outflow is somewhat predictable.”
For
this event, none of the experts considered the Black Swan classification of the
Unknown-Unknown or Unknown-Known type.
We
also obtained the results of the elicitation process of the subjective
probability distribution of the experts in each classification, according to
Table 4.
Table 4: Probability distribution elicitation for “p”
in the Accident on the Tim Maia bicycle pathway in Rio de Janeiro- year: 2016.
Black Swan Type |
Expert |
Best
fitting (Least square) |
Model parameters |
“Not a Black Swan” |
A |
Gamma |
Shape = 1.231008 Rate = 4.416544 |
B |
Beta |
𝜶 = 0.9079558 𝜷 = 1.068221 |
|
C |
Beta |
𝜶 = 1.6925468 𝜷 = 1.196452 |
|
“Black Swan: not
believed to occur” |
A |
Beta |
𝜶 = 0.5095363 𝜷 = 1.0934487 |
B |
Beta |
𝜶 = 0.9999813 𝜷 = 1.0000085 |
|
C |
Beta |
𝜶 = 0.8391531 𝜷 = 0.9077006 |
|
D |
Beta |
𝜶 = 0.5716223 𝜷 = 0.9911040 |
For
the expert fitting group “not a Black Swan”, two had their results of the
elicitation process best represented by the Beta distribution and 1 by the
Gamma distribution, by the least squares criterion. Again, to obtain the
aggregate distribution, the Beta distribution was used for all the members of
the groups, resulting in Table 5 and Table 6 with the summarized results of the
elicitation process of the groups “not a Black Swan” and “Black Swan: not
believed to occur”.
Table 5: Probability – “p” that the Accident on the
Tim Maia bicycle pathway in Rio de Janeiro was “not a Black Swan”, fitted by
Beta distribution, for the 0.25; 0.50; 0.75; and 0.95 quantiles.
|
Expert |
||
Quantil |
A |
B |
C |
0.25 |
0.0924 |
0.2050 |
0.394 |
0.50 |
0.2140 |
0.4440 |
0.608 |
0.75 |
0.3880 |
0.7040 |
0.796 |
0.95 |
0.6600 |
0.9340 |
0.949 |
Table 6: Probability-“p” that the Accident on the Tim
Maia bicycle pathway in Rio de Janeiro was a “Black Swan: not believed to
occur", fitted by Beta distribution, for 0.25; 0.50; 0.75; and 0.95
quantiles.
Quantil |
Expert |
|||
A |
B |
C |
D |
|
0.25 |
0.05930 |
0.25 |
0.2100 |
0.08940 |
0.50 |
0.23400 |
0.50 |
0.4720 |
0.30000 |
0.75 |
0.53100 |
0.75 |
0.7460 |
0.60800 |
0.95 |
0.88000 |
0.95 |
0.9560 |
0.91600 |
Graphical
results are also shown in Figures 3 and 4. The linear aggregate results are
plotted.
|
|
Figure 3:
The result of the Beta probability distribution elicitation for the class
“was not Black Swan”, for the 3 experts considered. |
Figure 4:
The result of the Beta probability distribution elicitation for the class
“Black Swan: not believed to occur”, for the 4 experts considered |
Considering
the expected value of the aggregate distribution of the experts, for “not a
Black Swan” class, the expected value was E[p] = 0.43; and for the “Black Swan:
not believed to occur”, the expected value was E[p] = 0.416 for the value of
“p”. The fact once again demonstrates that despite the classification given by
the experts for each group, they were not able in the elicitation process to
make this evidence clear, as the probability “p”, that the event is of the
class considered for the two cases is low. Although general fitting is much better
for “Black Swan: not believed to occur”, in which the differences among
distributions of experts are lower (Figure 4).
4.3.
Event
- Construction Industry: Accident in the
work of the Metro – SP – year: 2007
This
accident was analyzed by 9 experts, including civil engineers, geologists, and
experts in risk analysis.
As
general results, 44% of experts judged that this accident was “not a Black
Swan”, 33% classified as “unknown-known type of Black Swan”, and 22% as “Black
Swan not believed to occur”. The justifications of each expert for
classification are transcribed as follows:
“Not a Black Swan”:
· Expert
A: “The risk of collapse exists in the engineering environment and an action
plan must be in place in case of heavy rains.”
· Expert
B: “All the conditions of the terrain could have been preliminarily evaluated
by the technicians considering the normal conditions and with soil wet and
other variables.”
· Expert
C: “Although it is an extreme event, it is not unknown to risk analysts and
science.”
· Expert
D: “There was no (presented) calculation worksheet, execution without control,
in the placements of hangers.”
And
for the group that considered it a “Black Swan of the type not believed to
occur”:
· Expert
A: “It is possible that the analysts did not believe in the possibility of the
accident, and although the workers noticed the risks, they did not carry on
because they also believed that it would not happen.”
· Expert
B: “The soil characteristics were known, and were not monitored in an
apparently adequate way.”
Finally,
the “unknown-known Black Swan” classification was justified as follows:
· Expert
A: “I think the engineer in charge did not see the need to call a geologist,
but this may have seen signs of overload.”
· Expert
B: “It is possible that it is not a Black Swan, because in this type of work
collapse is one of the main risks and risk analysts are expected to have
identified it. However, it is also possible that they were unaware of
particular soil characteristics of the region, which favored the occurrence of
the catastrophe, hence an Unknown-Known Black Swan.”
· Expert
C: “Many times work is carried out with unqualified technical staff, and/or the
risks of accidents are diminished by its directors determined to increase of
the profit from the work. That is, very low risks are greatly amplified in this
scenario.”
We
also obtained the results of the elicitation process of the subjective
probability distribution of the experts in each classification, according to
Table 7.
Table 7: Probability distribution elicitation for “p”
in the Metro–SP Accident – year: 2007.
Black Swan Type |
Expert |
Best
fitting (Least
square) |
Model parameters |
“Not a Black Swan” |
A |
Beta |
𝜶 = 0.6809183 𝜷 = 1.1100828 |
B |
Beta |
𝜶 = 0.3878157 𝜷 = 0.6960759 |
|
C |
Beta |
𝜶 = 0.3978971 𝜷 = 0.6276588 |
|
D |
Beta |
𝜶 = 0.9999813 𝜷 = 1.0000085 |
|
“Black Swan - not
believed to occur” |
A |
Beta |
𝜶 = 0.8238680 𝜷 = 1.0816977 |
B |
Beta |
𝜶 = 0.9498462 𝜷 = 0.4723515 |
|
“Black Swan -
Unknown-Known” |
A |
Beta |
𝜶 =0.8105315 𝜷 =1.1023002 |
B |
Beta |
𝜶 =0.3765031 𝜷 =0.6307464 |
|
C |
Beta |
𝜶 =0.9999813 𝜷 =1.0000085 |
For
all three classes of Black Swan, the Beta distribution was the one that
presented the best fit according to least square criterion. Again, to obtain
the aggregate distribution, the Beta distribution was used for all members of
the groups, resulting in Tables 8, 9, and 10 with the summarized results of the
group elicitation process, “Not Black Swan”, “Black Swan: not believed to
occur”, and “Black Swan: unknown-known” type, respectively.
Table 8: Probability-”p” that the Metro – SP Accident
was “not a Black Swan”, fitted by Beta distribution, for 0.25; 0.50; 0.75; and
0.95 quantiles.
|
Expert |
|
||
Quantil |
A |
B |
C |
D |
0.25 |
0.117 |
0.0453 |
0.0569 |
0.25 |
0.50 |
0.329 |
0.2570 |
0.3010 |
0.50 |
0.75 |
0.613 |
0.6430 |
0.7010 |
0.75 |
0.95 |
0.904 |
0.9590 |
0.9740 |
0.95 |
Table 9: Probability-”p” that the Metro – SP Accident
was “Black Swan: not believed to occur”, fitted by Beta distribution, for 0.25;
0.50; 0.75; and 0.95 quantiles.
Quantil |
Expert |
|
A |
B |
|
0.25 |
0.172 |
0.434 |
0.50 |
0.405 |
0.756 |
0.75 |
0.675 |
0.943 |
0.95 |
0.925 |
0.998 |
Table 10: Probability-”p” that the Metro – SP Accident
was “Black Swan: unknown-known”, fitted by Beta distribution, for 0.25; 0.50;
0.75; and 0.95 quantiles.
Quantil |
Expert |
||
A |
B |
C |
|
0.25 |
0.165 |
0.0469 |
0.25 |
0.50 |
0.393 |
0.2760 |
0.50 |
0.75 |
0.664 |
0.6810 |
0.75 |
0.95 |
0.920 |
0.9720 |
0.95 |
Figures
4, 5, and 6 present the individual expert and aggregate fitting.
|
|
Figure 4:
The result of the Beta probability distribution for the class "was not
Black Swan", for the 4 experts considered. |
Figure 5:
The result of the Beta probability distribution for the class "Black
Swan – not believed to occur", for the 2 experts considered. |
Figure 6: The result of the Beta probability
distribution for the class "Black Swan Unknown-Known", for the 3
experts considered.
Figure
6. The result of the Beta probability distribution for the class "Black
Swan Unknown-Known", for the 3 experts considered.
Considering
the expected value of the aggregate distribution of the experts, for the class
“not a Black Swan”, the expected value is E[p] = 0.40, and for the “Black
Swan: not believed to occur”, the
expected value is E[p] = 0.55. Finally, the “Black Swan: unknown-known”
resulted in E[p] = 0.43 for “p”. Once again, despite the classification given
by the experts for each group, they were not able in the elicitation process to
make this evidence clear, because the probability “p”, that the event is of the
class considered for the three cases is low. However, in all three cases, the
distributions show a similar behavior.
4.4.
Event
- Fire in Alemoa Terminal – year: 2015
This
accident was analyzed by 10 experts, including civil engineers, safety
engineers, and experts in risk analysis.
As
general results, 20% of experts judged that this accident was “not a Black
Swan”, 30% classified as “unknown-known type of Black Swan”, and 50% as “Black
Swan not believed to occur”. The justifications of each expert for
classification are transcribed as follows:
“Not a Black Swan”:
· Expert
A: “This type of installation is already considered a fire risk, lack of
preparation, equipment, and products to extinguish fire in the beginning.”
· Expert
B: “There are several similar/equivalent events that can be identified in the
world.”
And
for the group that considered it a “Black Swan of the type not believed to
occur”:
· Expert
A: “Totally predictable to happen.”
· Expert
B: “Tank fire is possible, though unlikely.”
· Expert
C: “The risk is known, but very low, a sequence of adverse events is required
to occur.”
· Expert
D: “It was an extreme event in which the company did not have a fire prevention
and improvement plan due to its low probability of occurrence.”
· Expert
E: “Every storage location for flammable material must always contain
contingency plans, such as containment barriers, distance between them.”
Finally,
the “unknown-known Black Swan” classification was justified as follows:
· Expert
A: “The probable reason was a human fault.”
· Expert
B: “probability of occurrence is difficult.”
· Expert
C: “Known risks; but not controlled.”
We
also obtained the results of the elicitation process of the subjective
probability distribution of the experts in each classification, according to
Table 11.
Table 11: Probability distribution elicitation for “p”
in the Alemoa terminal fire – year:
2015.
Black Swan Type |
Expert |
Best
fitting (least
square) |
Model parameters |
“Not a Black Swan” |
A |
Beta |
𝜶 = 0.6203376 𝜷 = 1.080676 |
B |
Beta |
𝜶 = 7.6919001 𝜷 = 1.586449 |
|
“Black Swan - not
believed to occur” |
A |
Beta |
𝜶 = 1.0832013 𝜷 = 0.8756585 |
B |
Beta |
𝜶 = 0.4543275 𝜷 = 0.8756081 |
|
C |
Beta |
𝜶 = 0.9429176 𝜷 = 0.4615730 |
|
D |
Beta |
𝜶 = 1.1100828 𝜷 = 0.6809183 |
|
E |
Beta |
𝜶 = 2.3025908 𝜷 = 0.9229045 |
|
“Black Swan -
Unknown-Known” |
A |
Beta |
𝜶 =0.9999813 𝜷 =1.0000085 |
B |
Beta |
𝜶 =0.6569452 𝜷 =0.8231489 |
|
C |
Beta |
𝜶 =0.9877243 𝜷 =1.1428279 |
For
all three classes of Black Swan, the Beta distribution was the one that
presented the best fit according to least square criterion. Tables 12, 13, and
14 present the summarized results of the group elicitation process, “Not Black
Swan”, “Black Swan: not believed to occur”, and “Black Swan: unknown-known”
type, respectively.
Table 12: Probability-”p” that the Alemoa terminal
fire was “not a Black Swan”, fitted by
Beta distribution, for 0.25; 0.50; 0.75; and 0.95 quantiles.
Quantil |
Expert |
|
A |
B |
|
0.25 |
0.0983 |
0.762 |
0.50 |
0.3040 |
0.853 |
0.75 |
0.5970 |
0.920 |
0.95 |
0.9030 |
0.975 |
Table 13: Probability-”p” that the Alemoa terminal
fire was “Black Swan: not believed to
occur”, fitted by Beta distribution, for 0.25; 0.50; 0.75; and 0.95 quantiles.
Quantil |
Expert |
||||
A |
B |
C |
D |
E |
|
0.25 |
0.310 |
0.0557 |
0.438 |
0.387 |
0.569 |
0.50 |
0.574 |
0.2510 |
0.763 |
0.671 |
0.761 |
0.75 |
0.809 |
0.5890 |
0.947 |
0.883 |
0.898 |
0.95 |
0.970 |
0.9260 |
0.998 |
0.989 |
0.983 |
Table 14: Probability-”p” that the Alemoa terminal
fire was “Black Swan: unknown-known”,
fitted by Beta distribution, for 0.25; 0.50; 0.75; and 0.95 quantiles.
Quantil |
Expert |
||
A |
B |
C |
|
0.25 |
0.25 |
0.149 |
0.219 |
0.50 |
0.50 |
0.414 |
0.451 |
0.75 |
0.75 |
0.724 |
0.700 |
0.95 |
0.95 |
0.959 |
0.926 |
Figures
7, 8, and 9 present the individual expert and aggregate fitting.
|
|
Figure 7:
The result of the Beta probability distribution for the class "was not
Black Swan", for the 2 experts considered. |
Figure 8:
The result of the Beta probability distribution for the class "Black
Swan – not believed to occur", for the 5 experts considered. |
|
Figure 9:
The result of the Beta probability distribution for the class "Black
Swan Unknown-Known", for the 3 experts considered. |
Considering
the expected value of the aggregate distribution of the experts, for the class
“not a Black Swan”, the expected value is E[p] = 0.59, and for the “Black Swan:
not believed to occur”, the expected value is E[p] = 0.58. Finally, the “Black
Swan: unknown-known” resulted in an E[p] = 0.47 for “p”. Once again it was
demonstrated that despite the classification given by the experts to each
group, they were not able in the elicitation process to make this evidence
clear, because the probability “p”, that the event was of a considered class
for the three cases is low. In the last two cases, the distributions show a
similar behavior among experts.
5. FINAL REMARKS
This
article reached its objectives of identifying and classifying different types
of “Black Swans”. For all four events analyzed, it was possible to extract from
the different experts the classification of “not a Black Swan”, “unknown-known
Black Sawn”, and “Black Sawn not believed to occur”. For all cases,
“unknown-unknown Black Swan” was never reported by any of the experts. It is
possible to conclude that events never represent completely unknown events,
both by risk analysts and science, that is, none of the events studied
represents a surprise to anyone. This fact contradicts type (ii) – an extreme,
astonishing event concerning current knowledge and/or belief, as stated in
Aven, (2013).
One
of the attributes of Black swan events is that after their occurrence, people
and experts provide logical explanations, making an event hitherto unknown into
a perfectly predictable event, which is in agreement with (AVEN, 2013).
Most
of the times, experts classified events as “Black Swan not believed to occur”,
which means that the events are in general well known, but since their
probability is judged to be extremely low, the events are discarded in risk
analysis. This leads to the conclusion that the analyzed events are sometimes
classified as real outliers and this represents an issue to risk analysis.
The
elicitation process showed large disagreement among experts. The expected
value, meaning the most probable, was consistently low throughout all cases
analyzed. This issue could represent a failure in elicitation, which will need
further research to confirm or discard. However, we still believe the process
was useful and could serve as a basis for future risk analysis studies, such as
a priori belief distribution in a Bayesian analysis.
Future
works should analyze other events in different contexts, and maybe analyze
Black Swan occurrence using Bayesian belief networks, that is, studying factors
that affect expert beliefs in the elicitation process.
6. GENERAL ACKNOWLEDGMENTS
We
are very grateful to all anonymous experts who spent time in responding to the
specific forms, providing their valuable contribution to the accomplishment of
this work.
REFERENCES
ABNT-ISO.
(2009) NBR-ISO 31000: Gestão de
Riscos – Princípios e Diretrizes.
Aspinall, W. P. (2005) Chapter 2 Structured Elicitation of Expert
Judgment for Probabilistic Hazard and Risk Assessment in Volcanic Eruptions.
In Mader, H.M.; et al. (Eds.), Statistics in volcanology. Special Publication
of IAVCEI (p. 1–31) Geological Society of London.
Aven, T. (2013) On the
meaning of a black swan in a risk context. Safety
Science, v. 57, p. 44–51. Available: https://doi.org/10.1016/j.ssci.2013.01.016
Aven, T. (2015)
Implications of black swans to the foundations and practice of risk assessment
and management. Reliability Engineering
and System Safety, v. 134. Available: https://doi.org/10.1016/j.ress.2014.10.004
Bojke, L.; Claxton, K.; Bravo-Vergel, Y.;
Sculpher, M.; Palmer, S.; Abrams, K. (2010) Eliciting distributions to
populate decision analytic models. Value
in Health, v. 13, n. 5, p. 557–564. Available:
https://doi.org/10.1111/j.1524-4733.2010.00709.x
Burgman, M.; Fidler, F.; Mcbride, M.;
Walshe, T.; Wintle, B. (2006) ACERA Project 0611 Eliciting Expert Judgments.
Risk Analysis, v. 1 p. 71.
Clemen, R. T. (2008)
Comments: Comment on Cooke’s classical method. Reliability Engineering and System Safety, v. 93, n. 5, p. 760–765.
Available: https://doi.org/10.1016/j.ress.2008.02.003
Flage, R.; Aven, T. (2015) Emerging
risk – Conceptual definition and a relation to black swan type of events. Reliability Engineering & System Safety,
v. 144 (August) Available: https://doi.org/10.1016/j.ress.2015.07.008
FOLHA
DE SÃO PAULO. (2007) Entenda como
aconteceu o desabamento nas obras do metrô em SP. São Paulo. Retrieved from
http://www1.folha.uol.com.br/folha/cotidiano/ult95u130487.shtml
G1.
(2016a) Justiça inocenta 14 réus por cratera
que matou 7 no metrô de São Paulo. São Paulo. Retrieved from
http://g1.globo.com/sao-paulo/noticia/2016/10/justica-inocenta-14-reus-por-cratera-que-matou-7-no-metro-de-sao-paulo.html
G1.
(2016b) MP divulga laudo e pede multa de
R$ 3,6 bilhões por incêndio na Ultracargo. G1. Santos.
Retrieved from
http://g1.globo.com/sp/santos-regiao/noticia/2016/11/mp-divulga-laudo-e-pede-multa-de-r-36-bilhoes-por-incendio-na-ultracargo.html
Garthwaite, P. H.; Kadane, J. B.; Hagan, A.
O. (2005) Statistical Methods for Eliciting Probability Distributions. Journal of the American Statistical
Association, v. 100, n. 470, p. 680–701.
HOSACK, G. R.; HAYES, K. R.; BARRY, S. C. (2017) Prior elicitation for
Bayesian generalised linear models with application to risk control option
assessment. Reliability Engineering and
System Safety, n. 167, p. 351–361.
https://doi.org/10.1016/j.ress.2017.06.011
IOANNOU, I.; ASPINALL, W.; RUSH, D.; BISBY, L.; ROSSETTO, T. (2017)
Expert judgment-based fragility assessment of reinforced concrete buildings
exposed to fire. Reliability Engineering
and System Safety, n. 167, p. 105–127.
https://doi.org/10.1016/j.ress.2017.05.011
MACDONALD, J. A.; SMALL, M. J.; MORGAN, M. G. (2008) Explosion
probability of unexploded ordnance: Expert beliefs. Risk Analysis, v. 28, n. 4, p. 825–841.
https://doi.org/10.1111/j.1539-6924.2008.01068.x
MORRIS, D. E.; OAKLEY, J. E.; CROWE, J. A. (2014) A web-based tool for
eliciting probability distributions from experts. Environmental Modelling and Software, v. 52, p. 1–4.
https://doi.org/10.1016/j.envsoft.2013.10.010
OAKLEY, J. (2017) SHELF:
Tools to Support the Sheffield Elicitation Framework. Retrieved from
https://cran.r-project.org/package=SHELF
OLIVEIRA, N.; (2016) Crea-RJ: falha de
projeto é uma das causas do desabamento da Ciclovia Tim Maia. EBC Agência
Brasil. Rio de janeiro. Retrieved from
http://agenciabrasil.ebc.com.br/geral/noticia/2016-05/crea-rj-falha-de-projeto-foi-uma-causas-do-desabamento-de-ciclovia-no-rio
Paté-Cornell, E. (2012) On “Black Swans” and “Perfect Storms”: Risk
Analysis and Management When Statistics Are Not Enough. Risk Analysis, v. 32, n. 11, p. 1823–1833.
https://doi.org/10.1111/j.1539-6924.2011.01787.x
R DEVELOPMENT CORE TEAM. (2017) R:
A Language and Environment for Statistical Computing. Vienna, Austria: R
Foundation for Statistical Computing. Retrieved from https://www.r-project.org/
Taleb, N. N. (2010) The Black
Swan: The Impact of the Highly Improbable (2nd ed.) Random House.
UOL.
(2016) Trecho de ciclovia inaugurada em
janeiro desaba no Rio. São Paulo. Retrieved from
https://noticias.uol.com.br/cotidiano/ultimas-noticias/2016/04/21/ciclovia-inaugurada-em-janeiro-em-sao-conrado-desaba-no-rio.htm
Usher, W.; Strachan, N.
(2013) An expert elicitation of climate, energy and economic uncertainties. Energy Policy, v. 61, p. 811–821. Available:
https://doi.org/10.1016/j.enpol.2013.06.110.
Ranges for “p” |
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
10 |
(0.00-0.25] |
|
|
|
|
|
|
|
|
|
|
(0.25-0.50] |
|
|
|
|
|
|
|
|
|
|
(0.50-0.75] |
|
|
|
|
|
|
|
|
|
|
(0.75-1.00] |
|
|
|
|
|
|
|
|
|
|