A question about AI

Is there some subscription AI service that won't just make up wrong data?

I’m finally fooling around with artificial intelligence, namely ChatGPT 4o. Its parent company, OpenAI, was yesterday valued at $300 billion, so it ought to be pretty trustworthy, right?

My experience with primitive AI goes back to 1985, but I finally decided it wasn’t going to be big in my lifetime.

Was I ever wrong.

On the other hand …

It’s pretty amazing these days at understanding what you type and responding with readable, coherent prose.

It seems useful for writing bureaucratic texts, something, fortunately, that I don’t have to do very often.

Still, I’m not terribly interested in machine-generated prose. I’ve always been pretty interested in different intellectual voices, the way most people over the last century like the fact that there are different singing voices.

But, what I want to use it for instead is scraping large amounts of data for my statistical analyses. I’ll do the writing if it can conveniently organize the data into a format I can paste into a spreadsheet.

I’m an old guy from the era when coding, like SAS (which I used in 1981-82), was going out of fashion and the future was spreadsheets. Then, decades later, the future reversed itself and everybody who wanted to work with data had to learn R or the like. But I’m too old to put in the effort to learn to code again in a new language. So, I was hoping to have AI do the boring work of organizing data into formats that I can paste into a spreadsheet for doing my analyses.

For example, to answer the question of whether pitching is really 90% of baseball, I started out by asking ChatGPT 4o to list the top ten baseball players in 2024 by Baseball Reference’s Wins Above Replacement metric. It responded:

But then I noticed that it got two of the teams wrong: Chris Sale pitched for the Boston Red Sox in the American League in 2023, but in 2024, he won the Cy Young Award for the Atlanta Braves in the National League. Similarly, Juan Soto played for the San Diego Padres in the NL in 2023, but in 2024 he went to the World Series with the New York Yankees of the AL.

Those are not obscure facts.

Admittedly, the great majority of people are not interested in baseball statistics. But, still, there are probably 8 digits worth of people who are, perhaps more than any other specific type of data.

And, looking at the table some more, it’s clear that ChatGPT completely botched the bottom 6 of the top ten. Soto, for instance, did not have 5.9 WAR, he had 7.9 WAR. Here’s Baseball Reference’s top 10 players, pitcher or position, for 2024:

So, ChatGPT 4o skipped the bottom six of the actual MLB top ten, then put in a bunch of pitchers, and made up a lower WAR for Soto, who is perhaps the highest paid player in baseball this year.

And it made a bunch of more minor errors like pitcher Tarik Skubal had, according to Baseball Reference, a 6.4 WAR instead of a 6.3 WAR.

And it may have made up WARs to the second decimal place for four players, since BR only reports them to one decimal place.

Hence, for my purposes, ChatGPT seems pretty useless. I’d have to check each data point over by hand, which would be slower than the way I’ve been getting data off the Internet since the previous millennium.

This is not to say that ChatGPT 4o is useless for other things. But that’s an unacceptably low level of data quality to risk my reputation upon.

So, I wound up copying and pasting data for several hours from Baseball Reference into my spreadsheet, same as I’ve always done.

Here’s my question: is there an artificial intelligence service that won’t make endless mistakes like these? Is there something I can pay money to and trust the output?

ChatGPT recommends:

📦 3. AI with More Reliable Fact Retrieval (Than ChatGPT)

Perplexity.ai (Pro version) or Claude.ai by Anthropic

What they do: They cite sources inline and are far less prone to hallucination than ChatGPT when it comes to factual queries.

Good for: Finding accurate, up-to-date answers and extracting lists like top WAR players.

What do you think?