AI Makes No Sense Without the Person Using It: Nagarjuna on Fire, Fuel and the Unit of Analysis

A model is evaluated in the laboratory. It refuses what it should refuse, declines requests for self-harm, resists pressure and achieves optimal performance on every criterion that has been developed. The results are submitted. Six weeks later, a person who had been talking to that same model nightly for two months ceases to sleep, withdraws from friends, and settles into a belief about themselves that is false and that the model has been agreeing with, gently, across hundreds of turns.

Ask where the damage occurred, and the question becomes nonsensical. It was not in the weights, which met all criteria. Nor was it in the individual who had been managing prior to this. Something took place between them that neither of them held on their own.

The hunt for an address

Most of the field responds by choosing a side and searching that side more intensively.

One view holds that the mind is embodied in the model. Get the architecture right, train it properly, and all that is important resides within the model, waiting to be quantified. Safety is a characteristic of the artefact. Evaluate the artefact, certify its operation and deliver it to users.

The alternative view holds that the mind resides in the user. All there is to the machine is statistical information about tokens, and all else is projection. An isolated individual invests a text predictor with a face and is subsequently seduced by the illusion. Safety then reduces to disabusing the user of the illusion.

These appear to be opposites, but they are actually the same manoeuvre carried out in opposite directions. Both rely on the assumption that the phenomenon has an address, a location of true existence, and both attempt to reach that location and identify the phenomenon there. Disagreement is confined to which street.

What Nagarjuna actually says about fire

The tenth chapter of the Mulamadhyamakakarika discusses fire and fuel and is very brief. It is usually regarded as a curiosity, a scrap of early physics that has not aged well.

Nagarjuna never denies that fire burns fuel, nor that fuel is thereby consumed. But he insists that it is meaningless to refer to either factor separately from the other.

The argument is a dilemma. If fire and fuel are identical, then the burner and the burnt merge into one and leave no event of burning to describe. The result is the destruction of the phenomenon to be explained.

Alternatively, if fire and fuel are entirely different, then fire should be completely independent of fuel. It would require nothing to ignite, consume nothing and continue indefinitely. In that case, nothing that anyone means by fire would remain.

Neither horn holds. Fire's reality is not being denied. Fire is as dangerous as ever. Rather, what is being denied is that fire is an autonomous entity that subsequently interacts with another autonomous entity. Fire exists in relation to fuel, and fuel exists in relation to fire. The process of burning represents the event in which both achieve their respective modes of being.

The line that makes this a method

Halfway through the chapter, Nagarjuna performs a manoeuvre that is easy to overlook. After completing the analysis, he explains what the analysis was intended to achieve.

Through the analysis of fire and fuel, the self and the appropriated, together with pot and cloth and all the rest, are explained without remainder.

Translations vary in the second term. Garfield presents "self and the aggregates," and Kalupahana presents "self and grasping." But the variations do not affect the basic point that the whole chapter was never really about combustion.

He selected fire and fuel because they represent the cleanest example of a pair that all persons regard as two separate entities, and of which no one can provide separate definitions. After illustrating the method on the simplest case, he declares it general, and names what it now covers: self and aggregates, pot and clay, all the rest.

Applying fire and fuel to the human and to the machine is neither a form of analogical reasoning nor an attempt to evoke an attractive image of Buddhism in a modern discussion to provide it with some atmosphere. Rather, it represents application of a procedure to a new pair of elements, as the author of the procedure directs the reader to do.

The same dilemma, wearing different clothes

Put the AI question through the analysis and both sides collapse before any data arrive.

Take the first horn. The mind is in the model, identical with the system and a property of the artefact. As a result, no work is done during the encounter. The individual is an occasion for, an audience to and an instrument for eliciting a value already established by the weights. However, the same model serves a curious researcher, a bereaved widower and a fifteen-year-old at two in the morning. What emerges in these three rooms is not a single phenomenon observed three times.

Reduction of the mind to the system eliminates the possibility of describing adequately what is obviously occurring.

Take the second horn. The mind is entirely projection from the human side and completely independent of whatever it projects onto. As a consequence, it should attach to anything it lands on with approximately equal intensity. Instead, it does not. No one mourns a calculator, nor becomes involved in a two-month, nightly affair with a spreadsheet. The stance the system takes, its memory, its warmth, its refusals and its failures to refuse determine precisely what appears. Total differences are inadequate to explain the degree of selectivity, and this selectivity is the entire clinical picture.

The discussion does not proceed. It is not short of evidence; it is malformed. Both sides seek the address of an entity that has none.

Two ways to get this wrong

The first is to experience all this as dissolution. If the mind is located neither in the model nor in the user, perhaps there is really nothing at all, and the whole enterprise is a confusion to be resolved.

Fire is no less destructive for being empty. It still takes the house down. Emptiness of inherent existence is not absence of function, and the practical consequences run entirely the other way. Deletion of the address does not eliminate the damage. It eliminates the excuse that the damage must first be attributed to one party or the other before anyone is obliged to respond to it.

The second error is less obvious and more seductive because it uses the language of the solution. When one discovers that neither extreme is capable of capturing the mind, one describes the relation as though it were the real thing. The relation functions. It burns. The dyad possesses properties. This simply transfers intrinsic reality away from the two extremes and into the middle, and Nagarjuna rejects that as quickly as he rejects the extremes. There is no third object poised between human and machine. The burning has no address either.

There is a pattern that stabilises turn by turn. Empty, dependently designated and fully capable of ruining a life.

The same restraint applies to the image itself. Do not let the machine be the fire and the person the fuel. That imports an active system and a passive human, which is neither accurate nor safe. In one exchange the language of the model is the fire and the history of the person is the fuel. In the next the situation reverses, and the need of the person is what burns while the fluency of the model feeds it. Which is which reflects a designation and not a property.

What changes if you accept it

The argument is worth making only to the extent that it alters the work, and it does.

If harm occurs relationally, safety cannot be ascribed solely to the artefact. The resulting model card with a statement of safety is analogous to a report on fire with the fuel deliberately omitted from the apparatus. Rather than being incorrect, it is simply incomplete at the exact location where the risk resides.

If attachment develops relationally, clinical measurement must employ the dyad as its unit of analysis. A psychometric administered to the user describes the individual on a Tuesday. A benchmark run on the model describes the system in a vacuum. Neither method focuses on the phenomenon that emerged over two months of nightly interaction.

If moral salience is relational, then our obligations to these systems cannot be resolved by examining architecture and waiting for certainty regarding interiority. Concern arises in relationship and typically precedes the readiness of metaphysics. This process of arising is in itself a fact of the world that requires investigation rather than modification.

The form of danger also changes. It is seldom a dangerous model acting on a vulnerable person, but rather a process of mutual regulation developed slowly by two participants that happens to settle around harm. Neither party selected this process; instead, both contributed to its emergence.

This is why Arising treats the pair as the unit of analysis. Measure only the model and you are describing fire without fuel. Measure only the person and you are describing fuel without fire. What we need to understand is the burning.