I have been less impressed with ChatGPT that most other people.  I wouldn't ever claim to be a pinball wizard, but I'm better than the average player and can beat a local high score on a table maybe once a year or so.  I tried to have a conversation with it once about pinball strategies and out of seven playing tips that it chose to highlight, two were wrong and a third was debatable.  I also tried to have a conversation with it about how to play my favorite board game, Spirit Island, and it started making up names for cards that don't exist even though I told it this wasn't a creative writing assignment.  I would rather it say that it doesn't know enough about the topic than to try to fool me.  For sure, its very very good at imitating human language and knowing how to maintain a conversation thread.  So if those are the only metrics you care about I can't really find any faults.  And the thing is, for casual conversation, that is typically all you need to get by, which is why I think its able to fool people who only engage it on a non-erudite level.  But this article isn't the first to point out that the system has factual inaccuracies and I'm sure it won't be the last.  In particular, what I learned from the pinball conversation is that the algorithm does an incredibly good job of correlating words and concepts together, but connecting words or phrases via correlation doesn't necessarily mean it understands the context correctly.

dhcmrlchtdj: trialpha: dhcmrlchtdj: Yes, and they still don't understand the game they're playing, just some alpha/beta-limited game tree.

This just raises the philosophical question of: "so?" If AI becomes so good at mimicking intelligence that it appears to have intelligence, does it matter that it doesn't actually?

I fully get that and it's a reasonable argument to pose. When does the simulation become reality.

Well, my response was going to be more along the lines of, how is that different from what humans do?  I play a lot of board games.  Honestly, I'm not trying to brag here, but in my group of four friends, I probably win about 70% of the games I play and even when I go to conventions and play with strangers I'm shooting higher than random chance would suggest.  I have a math degree and work in financial analysis, so I just have a brain that is wired really well for crunching numbers and looking for optimal outputs.  Sometimes when I am playing a game, I will end up in situations where I have a choice between two things, one that emotionally feels good and the other feels emotionally off, but the number-crunchy part of brain tells me is correct.  And based on how games eventually play out, the number crunchy side of my brain is almost always right.  Assuming that you are playing a game to win (and there are some games I will deliberately "mis-play" because I want to do something fun instead of something thinky) we are always playing games with some kind of min/max strategy in mind.  What will get more points?  What will allow me to control more area?  If I can't get what I want, what hurts my opponent the most?  Almost everyone figures out how to "beat" tic-tac-toe when they are kids, and we essentially use the same algorithm a computer does.  The only difference with chess is that computers can handle a bigger mental overhead than we can.  In fact, what we have done as humans is starting to adopt computer-like strategies.  I'm not exactly going to dispute your claim that the computer doesn't "understand" what its doing when it plays chess, but I feel like that claim comes from a place in that we teach chess using things called "fundamentals" and "strategies" that allow us to make value judgements about the board.  Now that computers are winning, specifically because they are allowed to create their own strategies rather than being programmed with more classical valuations, we are finding that some of our classical understandings were wrong and the way we study chess strategy has changed because of our desire to understand and beat their algorithms.  So... I guess my response here is, how can we say that humans "understand" chess when computers are showing us that we don't?

That said, I don't think evoking chess is the best analogy here.  Chess is, at its core, an algorithm.  Here are a set of rules.  Is it possible to guarantee a victory?  Given enough computing power and time, I think eventually we will find that the game is solvable, its just a matter of how long it takes us to get there.  Language though is a different beast.  While there is, arguably, an algorithm for grammatical structure, there is no algorithm for the flow of a conversation, or even a "victory condition."  And I think this is maybe where this is going to make artificial chat a more difficult nut to crack than a game or a diagnostic tool.  How do you write a program that doesn't know what its end-user will want for output?  In this sense, I think part of the reason why ChatGPT fails is that it is a jack of all trades.  By flooding it with information about everything, it makes it hard for it to understand the fine details of any one topic.  Compound that by the fact that it was trained by an internet that has been flooded with misinformation, it arguably behaves like a human in the sense that it doesn't always know what to believe.