- cross-posted to:
- programming@programming.dev
- cross-posted to:
- programming@programming.dev
Four months ago, we asked Are LLMs making Stack Overflow irrelevant? Data at the time suggested that the answer is likely “yes:”
You must log in or register to comment.
Ai haters out there, doesn’t this give a valid use case for LLM?
Four months ago, we asked Are LLMs making Stack Overflow irrelevant?
“That’s a stupid question, marked as solved.”
marked as duplicate, see <other question from 2005, before LLMs were invented>
SO is a collaborative encyclopedia of technical discussion that tries to be relevant, be practical, and to not constantly repeat topics.
LLMs can’t provide that structure, they just shit out answers.
Most people think SO is a help desk and don’t appreciate the structure and just want it to shit out an answer.
Maybe SO isn’t dying so much as a cancerous growth is being treated.
It’s not dead until I stop getting 10 year old outdated answers in my searches!
I had a decently awarded account on SO because I joined it in 2012. I asked and answered questions. For the first few years it was fucking awesome as a professional developer. Then it’s popularity on google search results ended up making it too well known and the comment quality dropped substantially. Then the fucking powerusers popped up and started flagging almost everye one my questions as duplicates while pointing to unrelated questions. The last I really used SO was around 2017. I got too fed up to participate in the platform because when I spent the time to make a well formed question, it would just got shut down and my time wasted.
Had the same experience, almost exactly.
I wonder how well LLMs would do without SO’s data
Exactly this. LLMs are already bad at answering questions about very old technology, and I assume they are equally bad at answering questions about very new technology. Also bad at answering questions that have never been asked. We’re doomed.
Or Reddit, or Linux forums.
Stack Overflow hasn’t been useful for at least 10 years, if not longer.
The flagged “correct” answer is almost always wrong due to idiotic power-users and the vast horde of idiots who upvote obviously wrong answers because they’re bootlickers. The real answer is usually buried in between the posts by gatekeepers, pedants, idiots with something to prove, wannabe admins, egotistical idiots, the highly opinionated technologically insecure, etc ad nauseam. Reddit is just as bad for tech questions, if not worse.
Since I started using LLMs (running on my own inference server) I haven’t used anything else for tech questions that wasn’t opinion-based. Much, much more useful, and it requires you to think seriously about the problem to come up with a good prompt – which often gives you the answer before you even finish the prompt.
This is interesting because a huge amount of AI “knowledge” comes from stack exchange.
Now I’ll go read the other comments and article to see if that’s already been mentioned :)
Sucks because I prefer stack overflow in searches because I get more of a human explanation and wisdom. With llm i have to figure out what it’s_trying to do_ , debug it, and god forbid you want various ways of doing the same thing. I hate LLMs for coding. I hate clients for trying to force me to use it when most of the time now they admit they’re hiring me because AI failed in the first place
It will endure as long as the LLM’s on there know how to misinterpret the question and fire back snarky unhelpful answers about how clueless you are for asking in the first place.
Like it or hate it (personally I prefer the latter, posting there I felt like a middle schooler with a PUNCH ME sticker on my face) it was a great source of indexable data on programming.
I wonder how will this affect future search and llms, now that all similar questions are being asked in private llm threads.
I never once actually asked a question there. Partly because most of the time, the question I was asking had always been asked.
However, I have found the correct answer to 100s of questions there. Usually through google/ddg/kagi searches.
Anyone remember experts-exchange?
I remember when it didn’t have a dash. Until people started making fun of the old URL…
So easily avoided too
Ah yes, the place that never answered anything.
The sloppiest of slops before we got AI slop.
It was the pinterest of answering stuff
Or if they had an answer, they paywalled it, until Google got pissed at them for including the answer in their SEO but blocking it once the user clicked through. Then they maliciously complied with Google’s demand to not censor by burying the answer under layers upon layers of ads and other “related” questions.
I was so glad to see SO eat their lunch.
I used it in earnest! (to write shitty VB scripts and PHP websites)
My experience with SO is that I’ll look up a question about how to do something using X method and all the answers are like “why are you using X?” or “here’s how to do it using Y.”. You rarely find people answering the questions and instead find people trying to spread gospel about a certain tech that you aren’t using.
This was the majority of my experience as well. As a newer programmer, I’m more than happy to always know a better option. But if the way I’m looking to solve my problem is wrong, don’t just give me Y, explain to me why it may not work how I think it will. Tell me about X and some pitfalls or reasoning for it not going to work, then recommend Y. Because if others only see the Y answer to my question about X, they’ll probably just keep searching for a solution to X not knowing it may not work like I didn’t know.
That’s strange. It’s almost never my experience on stack overflow.
What you’re describing happens mostly on reddit and lemmy.
Thats been my experience as well.
On SO it seems much more likely that the answers answering a different question have a negative score.
In my experience has been like “that’s a bug and was solved on version 2.1, update” and I’m having the exact problem in version 2.2 so what now?
Or I don’t actually get to update the version my company is using, is there a workaround?
My experience with SO is somewhat the same, but sometimes (actually maybe most times) you’re trying to use a hammer to screw in a screw… If you read the suggestions and take them into account you can often find the actual question, and then the actual answer.
I’ve decided the best way to deal with someone asking an XY question is the following.
- Answer it. I don’t know what this person is doing, maybe they do really need to do some super weird thing and they are 4 weeks deep into “getting this project to work” and they don’t need me giving them the idea they also immediately thought of and can’t do for a bunch of reasons they are too exhausted to go into.
- See if this is an XY problem.
I have found this to be infinitely more well received. I think because by answering the question upfront without any annoying back and forth about why exactly they need to OCR a pdf in JavaScript, they are much more likely to be willing to have a dialog if their immediate question has been met.
The only danger is that some noob might stop reading after the answer and not engage with the deeper design issue, but by gatekeeping the answer behind a “you must convince the council of elders that you are doing something reasonable first” all we’ve done is push those people into ChatGPTs cheery answer first even if you have to make it up hands.
I very rarely ask questions on stack overflow but I appreciate much more as a sanity check on what I’m attempting to do.
In my experience, the majority of people have a flawed initial approach to what they’re trying to do, and if they all follow it they’ll produce a lot of really shitty software and learn very little in the process.
But they’re likely gonna anyway and didn’t even appreciate the sanity checks, so I fully expect software quality will continue to go down.
Yea I just think too many people end up forcing a sanity check before they will answer the question and it tends to make the question askers grumpy.
I’ve just noticed that if I answer their question first and then ask them a sanity check, they will more often engage with my sanity check.
Humans are tribal animals to a great degree, and the older I get the more I just accept that. And so if someone comes and asks me a question and I know they are more likely to accept pointed questions from someone they consider part of their tribe, answering the question first is an easy way to get them to put down their guard and engage.
I think what’s interesting about the ascent of LLMs is that they show that people are hungry for something to just answer their question. So much so that they are willing to deal with getting a completely wrong answer and having to come back and go “that function you suggested doesnt exist” a half dozen times.
I also moderate a couple technical discords and there are always members of the community that want to catalog and organize questions so they never have to answer the same question twice. And I get that impulse, but the thing I realized is that question askers want help.
I made it a point to make a culture around just answering questions and those communities are thriving. We don’t tell people to go search, we don’t tell people to explain themselves. Step one is always, answer their question. Then you are free to ask them why and see if there’s a better approach, but if someone wants to reverse flat map a list, show them how, and then they will be much more receptive to you asking why.
Yeah I mean that all sounds reasonable. I’m rooting for stack overflow to continue because it’s frequently helpful. I’ve basically never found chatgpt to be helpful.
My experience has been more like this:
OP: I’m trying to make lasagna from scratch but my noodles aren’t turning out right. Here’s my noodle recipe and settings for my pasta machine.
Mod: duplicate post of “How to make canned spaghetti bolognese” thread locked.
Yep, they aggressively XY problem your question until you give up. Also why many questions do not give the answer to the problem what most people asking that question would ask.
Then the author marks the question as answered because doing Y solves their problem…
Good for you, but I actually need to do X and Y wouldn’t work for me. At least change the title so it doesn’t come up as the top result in search engines.
I think all that needs to be said is if you search how to install a new CA in a given runtimes cert store, odds are the first and accepted answer will almost without fail describe how to disable ssl.
A lot of times the accepted answer on a locked question will be extremely outdated and/or not even functional anymore.
Modern tech charges at a break neck pace and stack overflow can’t keep up because the people who run the community created rules that artificially led to it not keeping up
This is honestly the reason why it’s going downhill, forcing people to do Y or use Z because of some problem irrelevant to the question being asked.
It limits creativity and depth of discussion on a forum designed to discuss all principles of programming
I’ve been in your position and in the other person’s position many times. It can be frustrating but we need to think about the big picture. It’s possible you hadn’t considered a certain approach, and it’s probable that many other future readers will not have considered a certain approach. So even though you might have said that you want to do something specific, it’s often helpful to some people to provide general information of another way to tackle the same issue.
And of course you know your own situation, so now there are these comments that appear off topic, and they kind of are, for you, and that’s just how it is on forums.
The other situation that comes up a lot is that people are doing it wrong. They are misusing some piece of technology and while their kluge might kind of work right now, it’s setting themselves up for bigger issues in the future. Of course no one appreciates it when you tell them they’re doing it wrong.
People don’t like when you don’t answer their question because it doesn’t give them an answer to their question. Just answer the question first and then hop on your high horse to tell them why it’s not going to work.
Make no mistake. LLMs aren’t killing stackoverflow. LLMs just arrived to finish it off. The stuff that was killing it are the regular posters there, and their passive aggressive bullshit
Nothing passive about them it was just regular aggressive. Made my programming coursework so much worse. Indian guys on YouTube however, now those guys were helpful!
Yup. I once decided to spend an afternoon answering questions on a framework I was expert in, as a kind of profile-building exercise to help with job hunting, and after around the third smug self-satisfied comment picking me up on some piece of irrelevant bullshit I deleted my account.
I hate how cathartic it is to watch that mountain of bullies burn to the ground 😌
I never asked a question, despite using it daily. Too afraid of being berated 😅
Question closed as off-topic.
Question closed as off-topic.
Removed as duplicate of #264826376: “Question closed as duplicate.”
Sometimes my jokes need explaining...
I’m pointing out that questions on SO too often get closed as duplicates of adjacent (but distinctly different) questions, and I did so in the most confusing, recursive way possible.
Ever ask a question on SO? I tell my students to search there but never, ever ask a question. The unmitigated hostility is not what new developers need or deserve. ChatGPT won’t humiliate you for asking a question that someone else has already asked.
That’s why I only post questions for bleeding-edge languages and code libraries. I have to answer them myself.
ChatGPT won’t humiliate you for asking a question that someone else has already asked.
I don’t know, being told what a good question that was and what a good boy I am everytime I ask a stupid question feels pretty humiliating.
(Still better than SO)
That’s a pretty recent development, isn’t it? I remember ChatGPT being a lot more matter of factly earlier on.
Yep, old ChatGPT was much more blunt and factual.
Don’t really like the recent trend of every LLM talking to me like I’m in kindergarten.
I forget where I heard the quote, but:
Stack Overflow is a great place to find answers. Stack Overflow is a terrible place to ask questions.
Their moderation approach is a big part of why it’s a great place to search for answers.
But if it results in edge issues that’re similar to another problem but not to the point of having the same solution being closed for being a duplicate, is it really helpful to the overall quality of the answers on Stack Overflow?
If LLMs just copied stack overflow they’d respond to every question with “Closed as duplicate. Question already answered.”
and link a slightly similar question, which’s answers can’t be used in your case, because of the small difference. also, it’s outdated since four years.
or 13 in case of python questions, and they are about python2
Problem being that someone else asked the question 10 years ago and the answer is now irrelevant due to version changes. People with high scores are just early adopters who answered all of the easy questions. Hostile users generally can’t understand the question. The issue with llms answering your question is that they are going to be stuck in the current time period. In the future their answers will also be irrelevant due to version changes.
Earlier today I googled how to toggle full screen in dosbox-x and the AI-generated answer said to use alt+enter. Tried it and it didn’t work, so I look in the documentation and it turns out that they changed it to F12+f a while ago (probably to avoid interfering with actual dos input).
This is definitely already a problem.
Every LLM is shit at dealing with version changes. They don’t understand it as a concept, despite all their training data.
I mean that is already a problem, if you ask a question you have to be ready for the answer to be a mismatch of version conflicts.
But that is ok. ChatGPT is a tool that can either help you or hurt you. I like to think of it like a power hammer. If you are doing a roofing job, it can help you get things done faster compared to a manual hammer, but you still need to know how to build a roof to get started.
ChatGPT is great at helping you organize your thoughts or finding an answer to some error message buried in some log file, but you still need to know what questions to ask and you need to be ready for it to give you a stupid answer and how to get around that.
I’ve never had an issue asking a question on stack overflow.
I’d wager a lot of ‘you people’ that have issues with it probably didn’t do enough research on your own.
There’s issues on both sides. A lot of people who ask questions are clearly just asking others to do their homework or otherwise haven’t made any effort, but there are also a lot of people who are unnecessarily hostile.
I definitely agree with this. I think the easier and kinder thing to do is to just not reply to posts like that.
I see this hot take often, and it isn’t entirely without merit, but it is mitigated by moderation; in some Stack communities better than others. I’ve been an active member for many years, and in my view it goes like this.
If you contribute a question without reading the rules and How to Ask a Good Question, you don’t provide minimal reproducible steps with code, post images of code, etc. you may get flamed out of town. And that may feel bad and it may be mean if the questioner didn’t know to read those. But they are there for you.
If, however, you ask a thoughtful question, give examples, show what you’ve tried, etc. you definitely can get quality, courteous help.
Doesn’t change that video killed the radio star here. The show is over.
Beginners are the least likely to ask thoughtful questions. We include slides in lectures about how to ask a question, but when there’s an assignment deadline and you’re inexperienced, it’s more likely you’re going to just blurt out “help me!” rather than provide a detailed explanation that doesn’t require repeated prompting. It takes time to learn how to work through an issue yourself before asking. Students are often facing time pressure and that can drive bad behavior. Correcting them is important, just don’t do it in a way that crushes their spirit.
100% understood and agreed. I don’t want to defend the bad behavior. It is out there among questioners and in the experienced community alike. Just saying it is possible to find quality help there.
For me, strict rules are what make this website useful. No threads named “help me” is why I like reading it.
For newcomers there is https://stackoverflow.com/staging-ground
Even for non newcomers, having threads marked as duplicates for problems introduced by version changes that aren’t considered in the original question/answers is a major issue.
Said the same thing
is giving marked-as-duplicate vibes
I’ve asked questions on S.O. I’ve answered some too.
What I’ve found works well on s-o is
- Researching a bit first
- Asking a question properly*
- Including that search attempt to prove you’ve done some due-diligence
I’ve found even a dick like me can get a lot of leeway by showing I’ve put in the effort and asked properly.
*Same as Usenet
