Research and Technology Risks: Part III – Risk Classification

This article describes how research and technology risks could be classified. This risk classification is the basis for the attribution of appropriate and proportionate legal obligations in the prototype regulation presented in the following blogpost.

I. The purpose of risk classification

1. Not all research projects or technology are dangerous. The vast majority are not. But some are, and they are so, to differing degrees. This should be reflected by regulation. It would be disproportionate to burden low-risk research projects or technologies with heavy requirements (e.g. on risk control), whilst it might be fully justified to burden high-risk research projects or technologies.

2. In theory it would be conceivable to have an endlessly fine-tuned risk scale and to develop appropriate requirements in regulation. In practice it is not. To reflect different risks, the best possible approach is to create risk classes and to establish requirements which are proportionate to the risk class.

3. Based on the previous blogposts on research risks and on technology risks, the next blogpost in this series will present a prototype regulation. To prepare this final and fourth blogpost, this blogpost presents the missing preparatory element, the risk classification. The risk classes shall be cast in such a way that, together with the scaled obligations under each risk class, research and technologies shall not be disproportionately hindered or even just slowed down, but still efficiently controlled so as to reduce the high risks linked to them.

II. Basic questions of risk classification

4. When regulators evaluate research undertakings and technologies, they might apply two different goals:

– Reducing harms or risks;

– Risk-benefit optimisation.

5. The problem with risk-benefit optimisation is that the benefits occur to other persons than the risks, as do financial benefits and costs. Furthermore, it is even more difficult to evaluate benefit than to evaluate risks. Hence it is much easier not to take account of benefit at the level of procedures to be applied to research and technology risks. The benefit will thus not be reflected in our risk classification. However, it is fully rational to take account of potential benefit at the level of decision making, and be it in view of providing a derogation to undertake research or apply technologies despite risks which, otherwise, would be deemed to be too high to be accepted. We will therefore integrate a special derogation clause in the prototype regulation developed in the next and final article of this series.

3. When pursuing the goal of reducing harms or risks, the question arises how the harm or the risk shall be evaluated or which definition of harm or risk shall be applied. In the international standards on risk reduction and other literature, we have identified the following most frequently used parameters for evaluating harm or risk or for defining harm or risk:

Severity: Seriousness of harm

Scope: Number of persons affected by harm

Likelihood: probability of the harm occurring

Lasting of harm: unless taken into consideration by Severity or Scope

We will base our further modeling on these four parameters.

The expressions harm and risk are sometimes used synonymously. Mostly, however, risk is defined or implicitly understood as harm multiplied by probability. We use here the second definition. We use the term “risk” also in the following way:

Death risk = a likelihood of death >0

Risk = likelihood of harm >0

4. Often it is not sure how the future will evolve. Where different scenarios are conceivable, one can evaluate them separately and attribute likelihoods to them. This is one way to cope with scientific uncertainty. Other ways consist of factoring in safety margins or safety factors. Whilst this is all reasonable, we will not enter this debate here, but focus on the modeling as such.

5. Before coming to the models as such, please read carefully a few preliminary remarks which apply to all of them:

– Indirect risks need to be taken into account. E.g., a computer virus might indirectly cause the dis-functioning of an electricity grid, thus affect a hospital and cause deaths by disrupting life-preserving machines.

– In case of multiple harms for the same victims, such as suffering followed by death, it is appropriate to evaluate both and to classify risks separately.

– Regulators who give a value to sentient beings other than humans or to nature, can introduce additional rows to cover harms affecting them.

– A particular risk assessment is needed when research or technologies might lead to the extinction of mankind. The extinction of mankind would stop the potential not only of billions, but trillions or even quadrillions of humans who could live over the next millions of years. This is not reflected by any of the models below. For further reflection on this aspect, we recommend the writings of the existential risk pioneer Nick Bostrom and in particular his article Existential Risk Prevention as Global Priority.

6. Furthermore, before coming to the models as such, we provide here a limited preview on the obligations that we will recommend to link to each of the five risk classes:

Class I: Self-certification without quality management system followed by registration in a public database

Class II: Self-certification with quality management system followed by registration in a public database

Class III: Certification of the quality management system by an entrusted body, self-certification for other aspects, both followed by registration in a public database

Class IV: Certification of the quality management system by an entrusted body + State authorisation procedure with consultation of national experts.

Class V: Certification of the quality management system by an entrusted body + State authorisation procedure with consultation of international experts.

III. A few key models of risk classification

7. Evidently, it is very cumbersome for regulators to develop a multi-parametric risk classification system based on the factors identified. In order to facilitate the task, we present here a few models which regulators can take as a basis.

8. To do so, we come first back to the four key parameters presented above:

Severity: Seriousness of harm (which can be roughly represented by the types of harm: death / health loss / financial loss)

Scope: Number of persons affected by harm

Lasting: duration of harm

Likelihood: probability of harm

Risk = likelihood of harm >0

9. Outline of MODEL A:

Class I: No death risk (0 likelihood of death)

Class II: Death risk for less than 10 persons (>0 likelihood of death for less than 10 persons)

Class III: Death risk for 10 to 1000 persons

Class IV: Death risk for 1000 to 1.000.000 persons

Class V: Death risk for more than 1.000.000 persons

10. The most simple model for risk classification focuses on the risk of death and takes just the parameter “scope” into further consideration. The death risk indirectly also mirrors to some extent the risk of being hurt or losing health. The most simple model avoids to a large extent the complicated and debatable assessment of the likelihood that will play a role in the following models.

11. Evidently, the thresholds (10, 1.000, 1.000.000) are somehow arbitrary and thus can be modified.

12. Outline of MODEL B

Class I

Class II

Class III

Class IV

Class V

Death risk for

None

Less than 10 persons

10 to 1000 persons

1000 to 1.000.000 persons

More than 1.000.000 persons

Health risk / Risk of injury for

Less than 10 persons

10 to 1000 persons

1000 to 1.000.000 persons

1.000.000 to 1.000.000.000 persons

More than 1.000.000.000 persons

Risk of income loss / wealth loss for

Less than 1000 persons

1000 to 1.000.000 persons

1.000.000 to 1.000.000.000 persons

More than 1.000.000.000 persons

Mankind

13. Model B expands the simple Model A by additional types of harm. Thereby, we shift from a one-dimensional to a two-dimensional structure. But the severity of the harms are not yet taken into account. To do so, we turn to Model C.

14. Outline of MODEL C

Class I

Class II

Class III

Class IV

Class V

Death risk for

None

Less than 10 persons

10 to 1000 persons

1000 to 1.000.000 persons

More than 1.000.000 persons

Risk of severe health damage or injury for

Less than 10 persons

10 to 1000 persons

1000 to 1.000.000 persons

1.000.000 to 1.000.000.000 persons

More than 1.000.000.000 persons

Risk of medium severe health damage or injury; risk of important loss of income or wealth for

10 to 1000 persons

1000 to 1.000.000 persons

1.000.000 to 1.000.000.000 persons

More than 1.000.000.000 persons

Mankind

Risk of light health damage or injury; risk of medium important loss of income or wealth for

1000 to 1.000.000 persons

1.000.000 to 1.000.000.000 persons

More than 1.000.000.000 persons

Mankind

Risk of not so important loss of income or wealth for

1.000.000 to 1.000.000.000 persons

More than 1.000.000.000 persons

Mankind

15. Here we see in the left column a modification when compared with Model B. Instead of referring just to types of losses as indicator for severity, we add qualitative descriptions: “important”, “medium important”, “not so important”. Evidently, the qualification is prone to interpretation disputes, as always when we refer to verbal description. This is a downside of this model. Furthermore, the increased complexity might be regarded as disadvantage as such. However, we are still far away from the upper end of conceivable complexity. We will go only one steps ahead with Model D. Model D integrates the parameter of likelihood.

16. Outline of MODEL D

Class I

Class II

Class III

Class IV

Class V

High or medium death risk for

None

Less than 10 persons

10 to 1000 persons

1000 to 1.000.000 persons

More than 1.000.000 persons

Low death risk; high risk of health damage or injury for

Less than 10 persons

10 to 1000 persons

1000 to 1.000.000 persons

1.000.000 to 1.000.000.000 persons

More than 1.000.000.000 persons

Medium risk of health damage or injury; high risk of loss of income or wealth for

10 to 1000 persons

1000 to 1.000.000 persons

1.000.000 to 1.000.000.000 persons

More than 1.000.000.000 persons

Mankind

Low risk of health damage or injury; medium risk of loss of income or wealth for

1000 to 1.000.000 persons

1.000.000 to 1.000.000.000 persons

More than 1.000.000.000 persons

Mankind

Low risk of loss of income or wealth for

1.000.000 to 1.000.000.000 persons

More than 1.000.000.000 persons

Mankind

17. Model D integrates the parameter of likelihood without losing the parameter of severity if risk is understood in the following way:

Risk = Severity x Likelihood

We recommend, for the context of this Model D, to understand risk in this way so as to make Model D more comprehensive.

18. Furthermore, if we so wish, we can interpret severity in such a way that “duration” is taken into account. Duration is an important factor above all when assessing health damage or injury.

19. So far, we have integrated one parameter after the other. But what can regulators do, when they wish to discard the basic parameter of Model A to D, the scope? They can simplify the Models B, C or D by eliminating the parameter scope and placing the entries of the left column directly under the heading Class V, Class IV, Class III, Class II and Class I. The chart for the simplified Model D (= Model E) would look as follows:

20. Outline of MODEL E (= Model D simplified)

Class I

Class II

Class III

Class IV

Class V

Low risk of income or wealth; no risk of health damage or injury or death

Low risk of health damage or injury; medium risk of loss of income or wealth

Medium risk of health damage or injury; high risk of loss of income or wealth

Low death risk; high risk of health damage or injury

High or medium death risk

IV. Results of Models A to E

21. The Models A to E are abstract. They must be applied to the various research and technology risks. We do not claim that we dispose of a comprehensive list of research and technology risks. By the following, we just wish to illustrate how various research and technology risks could be classified under the Models A to E. The risks are mostly identical, but in some cases the readers will observe deviations.

Research undertaking / technology application

Model A

Model B

Model C

Model D

Model E

Setting free or other uncontrolled dissemination of organisms falling under the definition of biological arms

Class V

Class V

Class V

Class V1

Class V2

Setting free or other uncontrolled dissemination of genetically modified or artificially created viruses, bacteria or fungi

Class V

Class V

Class V

Class V3

Class V

Genetical modification or artificial creation of viruses, bacteria or fungi in a controlled environment4

Class V

Class V

Class V

Class V5

Class V

Setting free or other uncontrolled dissemination of genetically modified or artificially created plants or animals

Class V6

Class V7

Class V8

Class IV9

Class IV

Genetical modification or artificial creation of plants or animals in a controlled environment10

Class V

Class V

Class V

Class IV

Class IV

Research on or creation of IT viruses11

Class II

Class III12

Class III

Class II13

Class II

Release of artificially intelligent software or hardware which is self-replicant or which has the potential to control an undetermined number of other software or hardware

Class IV

Class V14

Class IV15

Class IV

Class V

Research on or creation of artificially intelligent software or hardware which is self-replicant or which has the potential to control an undetermined number of other software or hardware in a controlled environment

Class IV

Class V16

Class IV17

Class IV18

Class IV

Research on or creation of artificially intelligent weapons

Class II

Class II

Class II

Class II

Class II

Changing the ground below the surface of the earth for a space larger than 1.000.000 cubic meters, but not more than 1.000.000.000 cubic meters

Class II

Class II

Class II

Class II

Class II

Changing the ground below the surface of the earth for a space larger than 1.000.000.000 cubic meters (= 1 cube km)

Class III

Class III

Class III

Class III

Class IV

Changing the ground below the surface of the earth for a space larger than 1.000.000.000.000 cubic meters (= 1000 cubic km)

Class IV

Class IV

Class IV

Class IV

Class V

Establishment of a nuclear installation which can explode

Class IV

Class IV

Class IV

Class IV

Class IV

Establishment of a nuclear installation which can otherwise release nuclear material

Class III

Class III

Class III

Class III

Class III

Operations which might change the weather or the climate in a zone larger than 100 square kilometers (e.g. 10 x 10 km)

Class II

Class II

Class II

Class II

Class III

Operations which might change the weather or the climate in a zone larger than 10.000 square kilometers (e.g. 100 x 100 km)19

Class III

Class III

Class III

Class III

Class IV

Operations which might change the weather or the climate in a zone larger than 1.000.000 square kilometers (e.g. 1000 x 1000 km)20

Class IV

Class V21

Class IV22

Class IV

Class V

Operations which might change the weather or the climate in a zone larger than 100.000.000 square kilometers (e.g. 10.000 x 10.000 km)

Class V

Class V

Class V

Class V

Class V

Operations which might change the weather or the climate globally23

Class V

Class V

Class V

Class V

Class V

22. In case of doubt, research undertakings and technologies were attributed to the highest debatable risk class, applying the precautionary principle. For some research undertakings and technologies, it should be possible to determine sub-groups for which a lower risk-class could be assigned, e.g. by lower level regulatory acts to be adopted by the administration. Therefore, we recommend foreseeing empowerments for these lower level regulatory acts to be adopted.

23. Applied to the research undertakings and technologies listed above, the risk for health and of injury did not change the outcome in the previous chart. Thus it seems that the risk of death is a very good indicator for the risk for health and of injury. Still we are not sure that other assessors will necessarily come to the same result. Furthermore, other research undertakings or technologies might well trigger a risk for health or of injury without triggering any risk of death. Therefore, we kept health and injury in Models B to E.

V. Comparing the Models in light of the outcomes

24. The fact that the risk for health and of injury did not change the outcome lets us suppose that the risk of death is a strong indicator for the risk for health and of injury. More surprisingly, the risk of death is also strongly correlating with the risk of loss of income or of wealth. In but a few cases the risk of loss of income or of wealth led to the assignment of a higher class than the risk of death. These two facts might console those regulators who would like to establish a risk classification model based on various risks, but who cannot do so because of technical or political obstacles. Already Model A which is exclusively based on risk of death as risk class assignment criterion does quite well even when measured against “the ideal” complex Model D.

25. Inversely the findings described in the previous paragraph permit regulators who are primarily interested in simplicity to discard other criteria than the risk of death because the sidelining of other criteria leads only to a few unjustified “misses”. The “misses” occur with regard to loss of income or of wealth only.

26. Model E is another quite simple model. The results of Model E differ however more from Models B to D than the results of Model A. Compared with Model A, Model E has the advantage of taking account of more parameters. However, it is weaker for not taking into account the number of victims. As the death risk is such a good indicator for other risks, we prefer Model A to Model E of the two simple models.

27. Is it at all ethically justified to apply the criterion “loss of income or of wealth”? We think it is because income are to some extent de facto used to increase the life expectancy or to prolong life. This is evident in poor countries where there is a lack of availability of medical treatments due to a lack of income. But even in some relatively rich countries the access to medical treatment depends on wealth or income. Finally, at the very high end, in the societies of the rich people in the rich countries, income or wealth are also invested in life-prolonging preventive screenings or treatments and better quality curative treatments. For these reasons, we prefer the ever more complex Models B to D to Model A. Evidently, Model D is the most comprehensive and complex, but also the most difficult to handle. The range from Model A to Model D is thus a range which offers ever more preciseness on one hand but requires ever more complex handling on the other. The right Model can only be chosen in the light of the capacity of the respective jurisdiction.

VI. Final remarks on the broader context

28. There are a variety of risks which have not been covered by this chart, e.g. risks linked to cognitive enhancement of humans or risks linked to the use of nano-technologies, an expression which covers a range of technical approaches and which suffers from intense debates on the right definition. The purpose of this table is not to show a complete list of risks, but to demonstrate how the various risks might be classified under the different Models. Regulators are invited to complete the list and, evidently, make their own assessment regarding the appropriate risk classes. We aim to demonstrate a methodology, not to prescribe or even recommend any results.

29. The risk classification is only one step in a multi-step process. We recommend proceeding in the order set-out below, but double-checking at the end of the process whether the right classification model has really been chosen. This has to be assessed in accordance with the results of the risk attribution and the corresponding sets of legal requirements. Some fine-tuning of the sets of legal requirements can conclude the process. We recommend thus the following order:

1. The selection and fine-tuning of classification models;

2. The attribution of the different risks to risk classes;

3. The determination of sets of legal requirements to be attributed in a proportionate way to the various risk classes.

4. The identification of exceptions to be made, e.g. exceptions at the level of the risk classification or exception with regard to the legal requirements. Evidently, if a need for very many exceptions is felt, the regulators should check whether something went wrong at the level of risk classification or establishment of legal requirements.

30. The method described in this blogpost and resumed in the previous paragraph can evidently be applied to other spheres as well. By no means is it limited to research and technology risks in general. The method can be transposed to other spheres as it can be downscaled to specific research undertakings or specific technologies.

1Although the death risk is low, it may affect more than 1.000.000.000 persons. Hence we keep Class V.

2In Model E, the number of potential victims does not count. A medium likelihood for one victim is thus sufficient for classification in Class V.

3Although the death risk is low, it may affect more than 1.000.000.000 persons. Hence we keep Class V.

4However, a lower class might be justified if the control of the environment is of extremely high quality so that the likelihood of dissemination is extremely low.

5Although the risk is low, it may affect more than 1.000.000.000 persons. Hence we keep Class V.

6Provided they cause a risk of death. Otherwise: Class I.

7Even if they do not cause a risk of death, because they may affect the survival of crucial plants or animals (e.g. bees) which then affects the world market for food.

8See previous footnote.

9Taking account of likelihood justifies lower Class under Model D than under Models A to C.

10See footnotes of previous line. However, a lower classification might be justified if the control of the environment is of extremely high quality so that the likelihood of dissemination is extremely low.

11Including indirect effects as dysfunctionning of devices used in hospitals.

12The wealth effect justifies a higher class than under Model A.

13Taking account of likelihood justifies lower Class under Model D than under Models A to C.

14Might cause an increase in the price of food or commodities on the worldmarket, affecting everybody. This justifies a higher Class as under Model A.

15However, this increase in the price of food or commodities will not touch upon the medium and upper incope persons in a significant way. Accordingly, it is not the entire mankind which is affected in a significant way. Hence the risk falls back to Class IV under Model C.

16See footnote 14.

17See footnote 15.

18However, a lower class might be justified if the control of the environment is of extremely high quality so that the likelihood of dissemination is extremely low.

19Including indirect effects as spreading of deadly viruses.

20See last footnore.

21See footnote 14.

22See footnote 15.

23Though the results for this line are identical to the results for the previous line, we keep the two lines separate as some readers might disagree with the assessments made. Hence for these readers it might make sense to keep the two lines separate.

Leave a Reply