From my understanding its length of the conversion that causes the breakdown. As the conversation gets longer the original system prompt that contains the guardrails is less relevant. Like the weight it puts on the responses becomes less and less as the conversation goes on. Eventually the LLM just ignores it.
Been tempted to fuck with ChatGPT, see what I can push it to say, but I don’t want that on my “record”. If it was utterly private, I’d be pushing the envelope. Be interesting to experiment with!
I wonder if that’s part of why GPT5 feels “less personal” to some users now? Perhaps they’re reinjecting the system prompt during the conversation and that takes away that personalisation somewhat…
From my understanding its length of the conversion that causes the breakdown. As the conversation gets longer the original system prompt that contains the guardrails is less relevant. Like the weight it puts on the responses becomes less and less as the conversation goes on. Eventually the LLM just ignores it.
Been tempted to fuck with ChatGPT, see what I can push it to say, but I don’t want that on my “record”. If it was utterly private, I’d be pushing the envelope. Be interesting to experiment with!
I wonder if that’s part of why GPT5 feels “less personal” to some users now? Perhaps they’re reinjecting the system prompt during the conversation and that takes away that personalisation somewhat…