I get that it’s usually just a dunk on AI, but it is also still a valid demonstration that AI has pretty severe and unpredictable gaps in functionality, in addition to failing to properly indicate confidence (or lack thereof).
People who understand that it’s a glorified autocomplete will know how to disregard or prompt around some of these gaps, but this remains a litmus test because it succinctly shows you cannot trust an LLM response even in many “easy” cases.
valid demonstration that AI has pretty severe and unpredictable gaps in functionality
These comments are driving me nuts - what gaps in functionality do you see demonstrated in this example? It answered both the literal question and the intended question, recognizing the misspelling. There are certainly lots of examples of AI screwing up, but to me this one seems like a demo of excellent functionality.
I get that it’s usually just a dunk on AI, but it is also still a valid demonstration that AI has pretty severe and unpredictable gaps in functionality, in addition to failing to properly indicate confidence (or lack thereof).
People who understand that it’s a glorified autocomplete will know how to disregard or prompt around some of these gaps, but this remains a litmus test because it succinctly shows you cannot trust an LLM response even in many “easy” cases.
These comments are driving me nuts - what gaps in functionality do you see demonstrated in this example? It answered both the literal question and the intended question, recognizing the misspelling. There are certainly lots of examples of AI screwing up, but to me this one seems like a demo of excellent functionality.