ChatGPT Gets an 'F' in Factual Accuracy and a 'See Me' in Grammar

Anyone with an internet connection has read something about ChatGPT since OpenAI released it to the public last fall as a free "research preview." The AI chatbot produces detailed, conversational, human-sounding answers to almost any prompt, and does so with remarkable speed and versatility.

Did you ever imagine needing the recipe for banana bread written out as a poem? Well it's out there — only a login page and roughly eight seconds stand between you and...

Flour, baking soda, and salt as well

All-purpose or whole wheat, can't you tell?

ChatGPT's massive wow factor has experts in dozens of industries and professions forecasting its role as a potential mega-disruptor that will change the way people work, students learn and businesses evolve. Is homework obsolete? Are writers being automated?

I put ChatGPT to the test — a very basic test — to see how it performs when given straightforward prompts on familiar topics. If the chatbot really is coming for my writing job like some predict, then I'd like to size up my competition. Based on the results, I'm not concerned. As an editor, I gladly welcome the flood of flawed content.

Here's why: ChatGPT fails to get simple facts straight. And it's surprisingly careless with grammar.

As a longtime golf writer, I prompted ChatGPT to "write an article about the history of Eisenhower Park in New York." The first red flag went up before the opening paragraph was complete.

The park was named after Dwight D. Eisenhower, the 34th President of the United States, who grew up in nearby Garden City.

What's that now? Dwight Eisenhower, a Nassau County boy? No, not true, or even close. President Eisenhower did not grow up in Garden City. He did not grow up anywhere near New York or the East Coast. A brief Google search turned up no conceivable link between young Ike and Long Island. So why would ChatGPT connect them?

Moving on, some of the subsequent material seemed a little off, as if ChatGPT was pulling historical nuggets from nearby parks and golf clubs and folding them into Eisenhower's past. Now I had to Google again, and with that, ChatGPT's ease of content creation became a burden for me to sort out on my own.

Then I had a little fun and asked it to write a poem about the history of Montauk Downs.

Built in the thirties, with care and skill... (Wrong.)

And the more concerning mistake:

It's fairways green, and sand traps deep...

An error in basic grammar? For a powerful, world-changing language tool? See me.

What's cool about ChatGPT is that it can react to follow-up questions and criticism, so a "see me"-type response can actually be productive. Take a look:

ChatGPT is a fantastic tool for launching and inspiring ideas and generating basic content. It should make staring at a blank page while formulating thoughts a thing of the past. It might not write a great opening paragraph, but it could spark something that grows into a wonderful, human-crafted one.

In the end it cannot be relied upon for either factual or grammatical accuracy. If it can make one fundamental error, it can make many. Human editors and writers are still needed today and will be needed long into tomorrow. ChatGPT is an incredible example of technological achievement, but its wow factor should not obscure its flaws and limitations.