Can Science Exist Beyond Human Understanding?

Essay 4 of 4 — AI and the Future of Science. The series ends with a question that is easy to ask but difficult to settle: if an AI-generated model survives demanding tests, yet people cannot explain its mechanism, should we call its output scientific knowledge?

Imagine a system that describes a complex material using unfamiliar mathematical features. Its model accurately predicts how the material will behave under new temperatures and pressures. Independent teams reproduce those forecasts. They also use the model to choose new experimental conditions, obtaining the changes it predicted.

Still, no scientist can produce a simple account of why those features track the material’s behavior. There is no elegant law written in familiar variables.

This is a thought experiment, not a report of an existing AI discovery. It forces a distinction that discussions of AI science often blur: predicting, intervening, and explaining are related but not interchangeable achievements.

Prediction is not explanation

A model can predict an outcome without identifying the causal mechanism that produces it. A correlation may be stable in the data used to test it, yet fail when the environment changes. Even successful forecasts do not settle which competing explanations is correct.

Conversely, a scientific explanation may deepen our understanding while remaining approximate or incomplete. Not every explanation promises the most accurate forecast in every situation.

Leo Breiman’s influential 2001 essay on the “two cultures” of statistical modeling already distinguished algorithmic approaches centered on prediction from models built around assumptions about the underlying data-generating process. [1] This debate predates modern generative AI.

Preparing diagram.

The AI-era problem is that a system might advance the first dimension far beyond the second and third. A prediction engine can make a laboratory more effective while leaving many explanatory questions unanswered.

That is genuine progress, but we should name what sort of progress it is.

Intervening tells us more—but not everything

Suppose the system does not merely predict natural outcomes. It proposes a deliberate change to an experimental condition, correctly forecasting how the outcome will shift. Better still, the success is repeated in settings not used to develop the model.

Such success provides a stronger reason to trust the model’s actionable knowledge than a score on a familiar benchmark would.

Yet even a successful intervention may be compatible with multiple mechanisms. A model could work locally while failing elsewhere. Scientists would still want boundary conditions, uncertainty estimates, alternative explanations, and evidence about when the intervention is unsafe.

For this reason, I would distinguish three levels of confidence: fit to existing observations, successful out-of-sample prediction, and reliable prospective intervention. They are tests of increasing practical strength in some settings, not a universal ranking of all sciences.

Can a machine be an epistemic partner?

There is a real philosophical puzzle here. Traditional scientific practice values explanations humans can communicate and criticize. If an AI develops concepts no researcher can intuitively grasp, the work may become harder to teach, combine with other theories, or audit for hidden error.

But insisting that knowledge must always fit our existing intuition is also risky. Much of mathematics and modern physics challenges ordinary imagination. Research communities routinely rely on instruments and specialists whose outputs no single scientist fully understands.

The more useful question is not whether a model is immediately intelligible. It is whether its claims can be publicly tested, bounded, challenged, and integrated with other knowledge, even while explanation remains incomplete.

Preparing diagram.

An opacity we cannot eliminate may be tolerable in some discovery settings and unacceptable in others. In safety-critical decisions, merely producing an after-the-fact explanation can be dangerous. Cynthia Rudin has argued for inherently interpretable models rather than explanations pasted onto black boxes when decisions carry high stakes. [2]

The ambition of AI-native science must therefore not become an excuse to lower standards where explanation and accountability are essential.

An independent test of the idea

We could make the thought experiment more concrete.

Ask a system to discover predictive representations in simulated environments with known underlying mechanisms, while preventing it from seeing those mechanisms directly. After training, give it new environmental changes, unseen interventions, and rival hypotheses.

Compare three outputs:

1. A model optimized for held-out predictive accuracy.
2. A model optimized for prospective intervention success.
3. A model constrained to provide an independently testable, human-readable explanation.

The interesting outcome is not just which system earns the highest score. We would ask whether a representation that humans find opaque can be more reliable under new interventions, and under what conditions interpretability or causal structure changes that reliability.

One must be careful: a simulator is not nature, and knowledge of its true mechanism is a privilege researchers rarely enjoy. Such experiments would test specific claims about models, not settle centuries of philosophy.

Two opposite mistakes

The first mistake is to say: We cannot explain it, therefore it is not scientific. That rule could discard useful, independently validated discoveries and would impose an unrealistic standard of immediate comprehension.

The second mistake is to say: It predicts well, therefore it understands the world. That leap can promote fragile correlations into causal stories. Messeri and Crockett have warned that AI can encourage scientists to believe they understand more than their evidence warrants. [3]

A responsible response lies between these claims. We can acknowledge useful knowledge without inflating its explanatory scope. We can demand stronger testing without pretending that every mechanism must be transparent from the outset.

What kind of future would count as a revolution?

The previous essays traced a shared problem. AI can generate convincing research narratives; it can reorganize discovery; and it can make hypotheses abundant relative to the capacity to validate them.

Here is the further possibility: AI may also produce scientifically useful concepts that are difficult for humans to translate into familiar theories.

If that happens, the central challenge will not be deciding whether machines “think like scientists.” It will be designing institutions and tests that let us know what a model can be trusted to do, what remains unexplained, and what observations would make us withdraw our confidence.

That would be a profound change in science—not because understanding no longer matters, but because we would be more precise about the different things we mean by understanding.

The series began with alchemy. It ends with a less dramatic, perhaps harder demand: the more powerful our tools become, the more carefully we must distinguish what works, what we know, and why it works.

---

References

1. Breiman, L. (2001). Statistical Modeling: The Two Cultures. Statistical Science, 16(3), 199–231.
2. Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1, 206–215.
3. Messeri, L., & Crockett, M. J. (2024). Artificial intelligence and illusions of understanding in scientific research. Nature, 627, 49–58.

The material-science example is hypothetical. The proposed confidence levels and experiments are conceptual suggestions, not validated measures. This working essay was developed with generative-AI assistance and remains open to revision.