Why is the Ideology of Evaluation Pernicious?

by Jean-Claude Maleval on April 14, 2010

Presented on 14th April, 2010 to the VIIth Congress of the World Association of Psychoanalysis (WAP): ‘Semblants and Sinthome’.

Published in Psychoanalytic Notebooks, journal of the London Society of the New Lacanian School, issue 21 2010.

Translated from the French by Lynn Gaillard

Jean-Claude Maleval is a member of the École de la Cause Freudienne (ECF ) and the World Association of Psychoanalysis (WAP) and is a Professor of Clinical Psychology at the University of Rennes, France.

Availability of a reference : The Black Hole of Vanities : 9th September 2005 : Éric Laurent : See here

Available below & here

Why is the Ideology of Evaluation Pernicious?

by

Jean-Claude Maleval

The methodological evaluation of services spread across the entire western world in the 1990s, attacking in particular the health systems and the universities. Its major principle consists in working to optimise the ratio between cost and efficiency. It introduces the market economy into fields that had hitherto been spared. In our cultural field, where ideals are disintegrating, being useful is looking like a minimal value. Evaluation, presenting itself as being at the service of the useful, is now tending to extend to all human activity.

How is it possible today to refuse evaluation? It only wants the common good: to maximise the quality of services given and obtain the best return for public funds. These notions can only appear to be ‘good’, with blinding evidence that those who are in opposition are ‘absurd’. They want to make us forget that evaluation is founded on the logic of administration and on the technique of management whose consequences are worse than the expected benefits.

Amputation of the Evaluated

One of the clearest consequences is eminently paradoxical: from the start, evaluation amputates all the activities to which it is attached. Everyone notices that it diverts a considerable amount of time that can no longer be devoted to the work being evaluated. A department head in the psychiatric sector tells that a simple process of auto-evaluation, administratively unavoidable, deployed a quarter of the staff for sixteen hours, to the rhythm of eight two-hour meetings, in order to fill out a pre-established referential grid that was not adapted to psychiatry.[1] The first effect of the introduction of the mechanism of evaluation is a loss of productivity that sometimes reaches more than 20%. It turns the clinician away from his patients, the researcher from his work. ‘It gives birth to a parasitic bureaucracy, which diverts resources to its own profit, nourishes itself on the work of others, eats it away from the inside.’[2] It generates new institutions, greedy to create jobs so that they can function. It needs an ever-increasing number of experts. It calls upon individuals trained in various different fields in order to initiate them into the logic of management peculiar to the assessment of health institutions. That the experts are totally ignorant of the practices of these institutions can only incite them to attach more importance to the nit-picking pre-established grids. Evaluation recruits teachers and researchers in order to satisfy its need to devour time and judiciously place a few scientific authorities in its decision-making bodies. Certain academics go so far as to totally abandon their activities in order to devote themselves to the administration of evaluation. What’s more, the task is not diminishing with time; anxious to be ever more precise, more efficient, yet coming up against something unpredictable that resists, the bureaucracy is growing, its demands are becoming more apparent, the paper-work is increasing. The evaluative activity is always ready to spiral out of control. It is advocating pre-evaluation using contracts, it is calling for permanent auto-evaluation, it is encouraging a follow-up of the impact of its effects, it is creating evaluators of the evaluators, it wants to become more frequent, its agencies are naturally incited to develop endlessly. The spiral of evaluation is an infinite process that generates a utilitarian ogre who is never satisfied. In short, from the start it is very costly for the public and heavy on the professionals it falls to.

It is difficult to obtain figures concerning the exact cost of evaluation, but when one tries, the results are surprising. One notes that one small psychiatric hospital in the provinces is already dedicating considerable sums. One learns from a ‘2007 Budget Forecasts and Revenues Report’, that a ‘first visit of certification’, which lasted one week, resulted in a payment of 15, 540 euros to the Haute Autorité de Santé. ‘To this cost’, it is underlined, ‘will be added the cost incurred by the participation of the personnel in the groups of evaluation of their professional practice and of auto evaluation’: 11 thematic working groups and 10 other groups for the evaluation of professional practice! Other expenditures are envisaged in order to call upon a ‘training body whose mission will be to put in place an unofficial visit, and to support us in the elaboration of a definitive synthesis.’ The 15, 540 euros are only a drop in the ocean in the complex process of pre-evaluation, of unofficial evaluation, of definitive evaluation, etc. And all this must be repeated periodically. But this is not enough: obviously worried, the board of directors is seeking to protect itself by having an evaluation specialist join the hospital. Creating a job entitled, ‘head of quality’: this is adding 68, 478 euros a year. At a time when thousands of jobs in the public function are being cut, it is interesting to know that funds are not lacking for creating evaluators. Therefore, in 2007, the total expenditure devoted to evaluation at one small hospital adds up, no doubt, to hundreds of thousands of euros. And yet, these expenses remain negligible in relation to those committed by large hospitals. The same is true for universities, where the professors note that each year the hours dedicated to evaluation are increasing to the detriment of teaching and research. Totalling up the accumulation of hours thus used up would give a sum for the cost of evaluation that we can hardly imagine. We have, nonetheless, a particularly convincing indication mentioned in a work by Éric Laurent entitled ‘The Black Hole of Vanities’.[3] He reports that things have got to the point ‘in the most evaluated system in the world, the USA, that the administrative costs of evaluation absorb around a third (31%) of the costs of healthcare’ according to studies carried out in 2003 by the Harvard Medical School and in 2004 by the Public Citizen Health Research Group.

Why is it that the heavy cost of evaluation is never itself evaluated by the evaluators? Because its advantages are incommensurable – but not for everyone. We will come back to this question.

Relative Reliability

The Agency for Evaluation of Research and Higher Education (AERES), following the example of all the official evaluation agencies, does not hesitate to affirm that it ‘guarantees the reliability of its procedures and of the results of the evaluations it conducts.’ Moreover, it puts in place important means to this end; first, the permanent systems of auto-evaluation, and also the call for ‘an independent external evaluation’. It attempts thus to bring about the evaluator of evaluators who would guarantee an absolute evaluation. And yet, each evaluation can of course in turn be contested by another evaluation, using another methodology, opening onto an infinite regression.

All evaluation is relative because it depends on a choice of methodology. And yet it is presented to us today as objective, adorned in the prestige of figures, therefore scientific, and thenceforth impossible to question. Nothing is said about the fact the mathematic model relies on the approval of a decisive choice, that of putting a figure on a certain element rather than another, whereas in the beginning other choices were made, ones that operate in the interpretation of the results. Concerning the evaluation of the efficacy of psychotherapies, for example, it has been established that the results obtained are strongly correlated to the theoretical allegiance of the researchers who undertook the study[4]: in this area it is fairly clear that the desired results determine the methodology employed. To measure the efficacy in reference to the eradication of a symptom is not of the same order as an appreciation in relation to the general improvement of the functioning of the subject. The first measure is largely calculable and better suits modern evaluators, whereas the second necessitates a qualitative appreciation and is readily rejected as unscientific. However, the observation made in the field of psychotherapies can, no doubt, be generally applied, since no methodology stands out as the obvious choice in any considered field, and thus the evaluator necessarily proceeds from a prior determining choice. As for the interpretation of the results, when they are not convenient, a method widely used is what in the Anglo-American domain they call the ‘file drawer effect’. When pharmaceutical laboratories noticed that, compared to the placebos, certain anti-depressant drugs increase suicide attempts, what did they do? They make sure not to publish the studies. It took the long battle by D. Healy for these studies to be known to everyone.[5]

A minimal credibility could be accorded to evaluation if the evaluators were recognised as particularly competent in the considered field. However, this is rarely the case. It is frequent for the evaluations to be carried out by individuals who are not troubled by their lack of competence in the subject under consideration, the handling of figures conferring upon them a scientific unction that seems to them to be authorisation enough. Needless to say, as noted by Y. C. Zarka[6], it is rarely the best of the bunch who have either the time or a taste for policing their colleagues.

The AERES is seeking to reduce evaluation to a technical problem, which it is, but only secondarily. But is it possible to have confidence in something that does not even respect the standards of its own protocol? The agency affirms that its experts must not have a ‘conflict of interests’ with the evaluated. However, in psychology almost all the experts are cognitivists, who consider that only experimental method is appropriate to their science, thus those who advocate clinical method are blacklisted and progressively stifled. If the clinicians were to evaluate the cognitivists, the latter would protest against the incompetence of their former. But the inverse does not seem true to them, designating a naive conception of science.[7] By supporting this, in psychology as in other fields of the human sciences, the AERES discredits itself.

Management has noted the high cost of evaluation, and is trying to reduce it, by trying to diminish the time it eats up. The radical solution put in place consists of proceeding in an essentially quantitative manner. This is why it looks like it is less and less necessary for peers to do the work. Tomorrow, evaluation will be entrusted to a machine that calculates the number of publications, balanced with the journal’s ‘impact factor’[8], which will give a figure to which the person evaluated will be reduced.

Recently, the appreciation of the quality of researchers is supposed to have become reliable: it is measured by the ‘impact factor’. The more a researcher is cited in the journals of his field, the more this is positive. Taking things to the extreme, the production of a thesis that has little credibility, but causes a reaction in the scientific community, such as Water Memory by Dr Benveniste, will benefit its author with a better impact factor than a highly innovative thesis at odds with the knowledge of the time. Let us remember that a text that has marked our times, such as Freud’s The Interpretation of Dreams, was very little cited in the years following its publication. It sold 228 copies in the first two years; it took ten years for the first edition of 600 copies to be sold out.[9] A work that would probably be evaluated as mediocre using the measure of impact factor.

Today’s vogue for the ‘bibliometer’ has been prompting most researches to compromise with the obtuseness of the figure by putting strategies into place to get around it. Many articles consist merely of a collection of statistics, using the most diverse parameters, which it is always possible to vary, and whose theoretical interest is often weak. However, these articles are of great use in multiplying references to colleagues, so that they may be kind enough to return the favour. In certain fields, in particular the human sciences, the network effects have been producing an aggressive intellectual conformity that endeavours to suppress divergent research.

The situation is no better in the healthcare field, where clinicians are constrained to give an account of their practice by ticking boxes in reference to diagnostic systems that they do not use because they object to the pertinence of these empty artefacts, but which are imposed on them due to the principal merit that they are better suited to statistics.[10] Is it necessary to specify that rigour is not the dominating concern when giving oneself over to these tasks, seen as useless? In the name of an evaluation that presents itself as objective, some are being invited to disown what they do and to cut themselves off from the knowledge that is their reference.

No matter the manner in which diagnostic data are deduced, no matter the way of calculating the impact factor, in the end the evaluator has a clean, tidy, scientific figure at his disposal, that can be used to orient decision-making. It goes without saying, it can happen that the figure is not suitable, but the methodological resources are not lacking to obtain another if necessary.

So, one can only share the opinion of the mathematician Luc Miller, ‘the discourse of the evaluators which claims perfect objectivity is a delusion, putting the weight of a false absolute on the researcher.’ Furthermore, he adds that is a dangerous idea to think that there might be an automatic calculable evaluation, one that would be systematic and completely objective, and which calls upon you to participate in this objectivity, when in fact you certainly have the right not to be like the others, to have a different practice. There is, in ‘the culture of evaluation’ the desire to impose norms, to humiliate the other, to make him give in on his being.

There are passionate researchers who are suddenly faced with criteria, modes of evaluation that depreciate them in their own eyes.[11]

Damage to the Social Bond

In addition to the increase in the workload, evaluation, even positive, generates a fundamental disorganisation: it always makes the present functioning look unaccomplished in relation to an optimal performance. It calls for a permanent modification of the tasks to be evaluated. It generates an inaccessible ideal: the lowest cost for the most efficiency – even an expert working for free cannot completely satisfy this ideal. It brings with it a constant instability and tension. Whoever adheres to the ideology of evaluation gives an ever-higher performance, getting involved in a multiplication of publications, with a growing number of consultations, in a frenzied ‘colloquitis’, etc. Thanks to the new context, what is evolving, much more than the quality of the work or of the research, is the capacity to adapt oneself to heavy demands.

Evaluation, by its very nature, segregates: it produces classifications, it designates the best, it implicitly stigmatises the others. It imposes permanent competition among institutions, teams, researchers and professionals. Thus it causes harm to the social bond by establishing as potential rivals those who might have otherwise felt themselves to be united. Nothing beats evaluation when it comes to undermining the solidarity in unions. It produces a Darwinian selection that leads, for example, towards universities being separated out into scientific centres of excellence and literary university colleges.

The human cost of evaluation is heavy: not only does it increase the workload of the one being evaluated, encouraging him moreover to do ever more, but it is fundamentally suspicious towards him. Human activities have always been the object of a spontaneous evaluation, based a priori on confidence in institutions and professionals. Methodological evaluation takes this away. It imposes an ever more demanding surveillance. It does not seek to justify the suspiciousness because it is inherent to its functioning. In a recent book Les ravages de la ‘modernisation’ universitaire en Europe, Chris Lorenz,  professor of history in Amsterdam, very rightly notes that ‘the reasons why one should not have confidence in the professional autonomy of the universities and why bureaucratic formalism should be preferred to professionalism have never been submitted for discussion: it is simply a matter of a presupposition integrated into the neo-managerial discourse and, as such, placed outside the domain of reflection and criticism.’[12] Evaluation believes neither in the value of professional ethics nor in institutional regulations. It mistrusts what is human.

‘Quantitative evaluation is harmful’, notes Michel Saint-Jean, physicist, ‘because it destroys cooperation among researchers in order to encourage uniquely research that is competitive and compartmentalised so that it can be identifiable […] Public research calls for intellectual liberty, mutual confidence, laboratories that permit fruitful exchange among researchers and an evaluation of their ideas based on intellectual confrontation, in bodies capable of dialogue and of accompanying the researchers. Instead, the present powers have chosen to monitor and punish. Although the potential is immense, it is to be feared that with these constraints, French science will decline through exhaustion, passivity or submission’.[13]

The evaluators always take the precaution to underline that they are not the ones who make decisions, but they willingly remain silent about the fact that they are nominated by them. In France, unlike other countries, the board of directors of the ANR (the French National Research Agency) and the AERES are not independent, but dominated by representatives from the state and business. The evaluations produced are supposed to deliver objective data designed to enlighten the decisions of the politicians. Sometimes their expectations are not met.

Oriented Interpretation of the Results

Let’s take an example of their possible treatment of results. Recent studies confirm that the effectiveness of anti-depressants in most cases is hardly superior to that of placebos. For once, the information is not remaining classified and is circulating in the press and on television news. What is the reaction of the university psychiatrists regarding this evaluation that irritates their ‘scientist’ convictions? They start by trivialising the information in Le Figaro of 2 March, 2008, by reminding us that all the specialists, of course, know this, but not the consumers of anti-depressants. What followed shows that they would like for the latter to remain ignorant. In a second phase, they questioned the methodology of the studies, which use a control group receiving the placebo, although this practice is used to test all medication. They do not challenge the methodology when it gives the results they are waiting for. Most astonishing is the following affirmation, in which scientism is carried to an extreme. The professors Jean-Pierre Olié, Henri Cuche, Daniel Sechter and Thierry Bougerol do not hesitate to say: ‘negative results do not interest the scientific community’,[14] science should, according to them, only bring good news, otherwise it is not science. Carried away by their enthusiasm they add: negative results ‘are not accepted by scientific journals’! Be sure to understand: it is scientific to say that a drug is effective; that falls within the remit of evaluation. To affirm that it is not, is ideology. To reveal that a commercialised product is just about ineffective is not worth publishing; it would not be considered authentic information. Such a caricatured procedure is commonplace in the interpretation of the results of evaluation.

For those who decide, there are good and bad evaluations. The first are raised to the rank of scientific truth, the second are not worthy of mention. Depending on the product evaluated and the expected results, an identical methodology will be praised or disparaged by the same experts. By protecting themselves with the cloak of evidence, the ‘good’ evaluation erases the political choice that promotes and uses it. It serves as an alibi for decisions whose determinants are essentially disconnected from the process of evaluation. What are they? The example of the psychiatrists from ‘Figaro’ put us on the right track, orienting us towards the pharmaceutical laboratories and leading us to the workings of the market economy. The defenders of the ideology of evaluation carry a project for society; some are aware of this, others would prefer to ignore it, but all refuse to assume responsibility by camouflaging themselves in scientific objectivity.

Erasing the Political Act

Even though the material and human cost of evaluation is high, no manager would dream of evaluating that. Why? Well, because it draws its force from generating an immeasurable political profit: whoever freely accepts to be evaluated finds himself captured, sometimes in spite of himself, by a conception of the world governed by profit. J.-A. Miller underlines quite rightly that what is essential concerning evaluation is not to produce a figure, but to obtain the consent of the other. ‘Evaluation’, he affirms, ‘is an initiation and is transmitted as an initiation’. Clearly one tempts people, in the sense of temptation, to lend themselves to evaluation by telling them: ‘once you have been accredited-evaluated, you will evaluate others’. The content itself of the evaluation, the evaluative operation, escapes. It is a questionnaire, some interviews, that sort of thing. The most important thing is for the other to have consented to the evaluation. To consent to be evaluated is much more important than the operation of evaluation itself. Let’s even say: the operation is to obtain your consent to the operation…[…] Evaluation is the methodological, unremitting, and extremely malignant quest to obtain the other’s consent. […] The evaluated of yesterday is the evaluator of tomorrow; each evaluated-accredited person is a potential evaluator. The evaluated one has gone through that, he’s waiting for you to go through it in turn, he is a natural proselyte.[15]

Even if not accredited, the evaluated one who has given his consent will afterwards perceive his work through the lens of the values suggested by the evaluation. From then on his way of reading the world and human relations bends, most often without him realising. The laws of the market take on the allure of something obvious. To contest the ramifications implied by evaluation becomes ‘absurd’. The subjective mutation effected by evaluation sometimes turns out to be striking. Certain of its proselytes do not imagine that the market economy tends resolutely towards an emphasis on the segregation between the haves and the have nots, that it promulgates two-tier healthcare, that it is working to divide ‘the economy of knowledge’ into poles of excellence and university colleges. In spite of everything they advocate the benefits of evaluation: the connection between the marketplace and evaluation having been cut off when they consented to be evaluated.

Admittedly, the bodies in charge of evaluation are sometimes capable of hearing certain criticisms of their methodology. The AERES, for example, no longer gives much credence to impact factor. Nonetheless, contrary to what those who hold to evaluation would like us to believe, it is not a technical problem. Methodological evaluation of services is an ideology in the strong sense of the word, one designating an elaborate intellectual construction to justify the existing social order by veiling the material interest of certain people, presenting it as being in the common interest.

Human activities have always been spontaneously evaluated, both by those who do the activities and those who use them. What is new in our time resides in the introduction of a methodical evaluation that claims to incarnate an absolute, which is false. It is important to repeat that evaluation carries with it a pernicious ideology, because it is extremely costly from a material and human point of view, because it disorganises what it claims to optimise, because, contrary to its pretension, it is never totally reliable, because it fetishises the numerical figure, thus giving a simplistic version of the human factor, because it harms the social bond, because it erases the political act on which it is founded. Why is it being advocated in spite of everything? Because it presents the enormous political advantage of covertly insuring that those who accept it adhere to the liberal logic of the market economy.

Admittedly, we cannot refuse to enter into the mechanism of evaluation, except by resigning from the places where it is practiced. However we owe it to ourselves to refuse to adhere to its ideology, by refusing to nourish it, by not dreaming of a good evaluation, by remembering constantly that the human factor outstrips the numerical figure, by resisting, ferociously denouncing the harm it does, and by revealing it for what it is, in the last analysis: an instrument to gain consent to the economy of the market.

Translated from the French by Lynn Gaillard

Note 1 :  Presented on 14th April, 2010 to the VIIth Congress of the World Association of Psychoanalysis (WAP): ‘Semblants and Sinthome‘.  Published in Psychoanalytic Notebooks, journal of the London Society of the New Lacanian School, issue 21 2010.

Note 2 : Jean-Claude Maleval : See here for further information.

[1] Matet, J.-D., ‘La dictature du consensus’ in Le nouvel Âne, January 2004, Issue 4, p. 8.

[2] Miller, J.-A., ‘Arrêtons de noter tout le monde!’ in Journal du Dimanche, 10 February 2008.

[3] Laurent, É.,  ‘The Black Hole of Vanities’ in NLS (New Lacanian School) Messager, No. 172, 12 September 2005. See The Black Hole of Vanities : 9th September 2005 : Éric Laurent or here for availability.

[4] Luborsky, L., Diguer, L., Seligman, D.A., Krause, E.D., Johnson, S., Halperin, G., Bishop, M., Berman, J. S., Schweizer, E., ‘The researcher’s own therapy allegiances: a “wild card” in comparisons of treatment efficacy’ in Clinical Psychology Science Practice, Issue 6 1999,  pp. 95-106.

[5] Healy, D.,  Let Them Eat Prozac. The Unhealthy Relationship between the Pharmaceutical Industry and Depression, New York University Press, 2004.

[6] Y. C. Zarka is a  philosopher, director of research at the CNRS (the French Centre for Scientific Research), Paper given at the Forum des psys on February 9, 2008 at La Mutualité.

[7] Maleval, J.-C., ‘L’unité de la psychologie a vécu’ in Le nouvel Âne, Issue 8 February 2008, pp. 50-51

[8] The impact factor of a journal represents, for a given year, the ratio between the number of citations referring to this journal and the number of articles published by the journal, calculated over a reference period of two years. To give an indication as to the pertinence of the figure thus obtained, let us underline that a critical citation of an article has the same weight as a citation praising it.

[9] Jones, E.The Life and Work of Sigmund Freud, Volume I.

[10] Maleval, J.-C., ‘DSM, un manuel pour quelle science?’ in Raison présent, 2002, Issue 144, pp. 37-55.

[11] Miller, L., ‘Le discours évaluationniste qui revendique l’objectivité parfaite est un délire…’ in Le nouvel Âne, Issue 8, February 2008, p. 63.

[12] Lorenz, C., ‘L’économie de la connaissance, le nouveau management public et les politiques de l’enseignement supérieur dans l’Union européenne’, in Areser, Charles, C., & Soulié, C., Les ravages de la ‘modernization’ universitaire en Europe, Syllepse, 2007, p. 50.

[13] Saint-Jean, M., ‘Ceinturée, corsetée, flagellée, la science étouffée!’  in Sauvons la recherché, June 10 2008, http://www.sauvonslarecherche.fr/

[14] Olié, J.-P., Cuche, H., Sechter, D., Bougerol, T., ‘Les antidépresseurs, trop c’est trop!’ Le Figaro Saturday/Sunday 1-2 March, 2008, p. 16.

[15] Miller, J.-A. & Milner, J.-C. Evaluation. Entretiens sur une machine d’imposture, Agalma. 2004, pp. 35-37.

Further texts

Jean-Claude Maleval : See here for further information.

By Éric Laurent : here

In Topic Areas

Networking & Politics here

Evaluation and Outcome Measurement here

One comment

In terms of evaluation, clinical research will not exist until human beings learn to think, interact and handle data in the same way as computers do. We all need to try harder!

by Jo Rostron on 17/06/2011 at 12:48 pm. Reply #