How well does an Artificial Intelligence (AI) perform when an historian of astronomy queries it about the history of astronomy? I have been working for a while now to find out the answer to that question by querying different AIs, especially in regard to the specific areas of the history of astronomy that I know the most about. I have been sharing my results with Sacred Space Astronomy readers in various posts over time. You can click here for those AI-related posts.
Of all the AIs, ChatGPT seemed to do the best. In that sense, it somewhat lived up to the ChatGPT hype. But that is not to say that ChatGPT did well. It got a lot of things wrong. At times it went bonkers, generating stuff that made absolutely no sense at all — in other words, generating misinformation. It started talking, for example, about Tycho Brahe’s ideas regarding stars and their gravity; but Tycho Brahe saw the universe in more or less Aristotelian terms. To him stars were objects made of a celestial ether unlike any substance found on Earth. Rocks had gravity; stars did not have gravity.
ChatGPT had other issues as well, but its biggest issue was a general one. Like all the AIs I have tested, it tended to merely re-tell what “everyone already knows”. I queried AIs about the Copernican revolution, Galileo, and so forth, and the AIs came back with the usual stuff you hear in, say, books for children or the “Cosmos” TV shows: “Galileo’s observations provided compelling evidence for the heliocentric model and laid the groundwork for modern astronomy.”; “The Church declared heliocentrism heretical, and Galileo was warned not to advocate for it.” The most surprising example of this was the Bing AI talking about Galileo being told by the Church that he could not be a scientist and a Catholic (click here). Bing had to turn to material written by some turn-of-the-century schoolgirl for that statement. The remarkable thing is that somehow the Bing AI was driven to include that idea, even though it had to reference the work of a child in the process.
I had been hoping that these AIs might be decent research tools — that their having been trained on the vast pile of information that is The Internet might enable them to somehow dig up something interesting that I did not know, or that would cause me to think about something in a new way. That has certainly not been the case.
For example, even when ChatGPT was at its best, I had to drag better information out of it (“better” meaning information that is both correct and goes beyond just the usual stuff). In discussing heliocentrism and Tychonic geocentrism, ChatGPT talked readily about Galileo and “compelling evidence” for heliocentrism (not so compelling, since plenty of astronomers replicated Galileo’s groundbreaking observations and saw what he saw, but were not compelled toward heliocentrism), but it took me multiple tries to get the stupid AI to talk about the problem of parallax and the stars (parallax being the effect of a moving Earth on our view of the stars). The fact that no parallax was observed, whereas a moving Earth required that parallax must exist, was a huge deal. It led to stars in a heliocentric universe having to be at vast distances and (given the knowledge of the time) to have vast sizes. Parallax is one of the reasons Galileo’s observations were not so compelling. But to get ChatGPT to talk about parallax, I had to already know about parallax. Someone not familiar with the things that made Galileo’s observations less than compelling would have never heard anything about parallax from ChatGPT.
I have read that AIs like ChatGPT suck up data from the internet and produce a statistically likely stream of words in response to words you give it — a far more complex version of “complete this sentence: my favorite cartoon character is a carrot-chomping rabbit who says, ‘eh, what’s _______?’” What I see with AIs certainly supports what I have read. They spit out what is expected: “up Doc”. Insofar as AIs can polish up the usual stuff and make a “new” article about it, I fear they are simply going to replicate and reinforce what “everyone already knows”.
I will try out more things with AIs and the history of astronomy as I think of things to try out. And who knows? — maybe the AIs will get better with time. That will be interesting to monitor. Stay tuned.
