I try, every now and then, and I'm fairly consistently disappointed. Every time I end up finding out that the output is wrong. Not in the sense of aesthetics or best practices, but truly incorrect in the sense that it either doesn't work or that it's a falsehood. And there's two explanations for this that I can think of.

The first is that I'm using them wrong. And this seems likely, because some people I respect swear by them, and I'm an idiot. Instead of asking "how does mongoDB store time series data" or "write a small health check endpoint using Starlette" maybe there's some magic invocationsor wording that I should be using which will result in correct answers. Or maybe I'm expecting the wrong things from LLMs, and these are not suitable usecases.

The other possibility is that my prompts are right, and I'm expected to correct the LLM when it's wrong. But this of course assumes that I already know the answer, or that I'm at least well-versed enough to spot issues. But then all LLMs automate away is typing, and that's not my bottleneck (if it were, what a boring job I would have).

I think a key thing I'm doing wrong is occasionally forgetting that this is ultimately fancy autocomplete and not a source of actual knowledge and information. There's a big difference between answers and sequences of words that look like answers, but my monkey brain has a hard time distinguishing between the two. There's an enormous, truly gigantic, insurmountable, difference between

"Ah yeah we've used terraform in production for 5 years, best way to go is really not putting your state file under version control for ..."

and

"Sure! When using terraform it is generally considered a bad practice to put your state in version control for these reasons

<bunch of bullet points and bold words>

But I'm only human, and it's really easy to trick me into forgetting this.