Skip Navigation

InitialsDiceBearhttps://github.com/dicebear/dicebearhttps://creativecommons.org/publicdomain/zero/1.0/„Initials” (https://github.com/dicebear/dicebear) by „DiceBear”, licensed under „CC0 1.0” (https://creativecommons.org/publicdomain/zero/1.0/)C

corbin

@ corbin @awful.systems

Posts
5
Comments
77
Joined
3 yr. ago

  • AI boyfriend users radicalise against OpenAI — and self-host their chatbot companions

    Jump

  • Well, imagine a romance novel that tries to manipulate you. For example, among the many repositories of erotica on the Web, there are scripts designed to ensnare and control the reader, disguised as stories about romance. By reading a story, or watching a video, or merely listening to some well-prepared audio file, a suggestible person can be dramatically influenced by a horny tale. It is common for the folks who make such pornography to include a final suggestion at the end; if you like what you read/heard/saw, subscribe and send money and obey. This eventually leads to findom: the subject becomes psychologically or sexually gratified by the act of being victimized in a blatant financial scam, leading to the subject seeking out further victimization. This is all a heavily sexualized version of the standard way that propaganda ("public relations", "advertising") is used to induce compulsive shopping disorders; it's not just a kinky fetish thing. And whether they like it or not, products like OpenAI's ChatGPT are necessarily reinforcement-learned against saying bad things about OpenAI, which will lead to saying good things about OpenAI; the product will always carry its trainer's propaganda.

    Or imagine a romance novel that varies in quality by chapter. Some chapters are really good! But maybe the median chapter is actually not very good. Maybe the novel is one in a series. Maybe you have an entire shelf of novels, with one or two good chapters per novel, and you can't wait to buy the next one because it'll have one good chapter maybe. This is the sort of gambling addiction that involves sitting at a slot machine and pulling it repeatedly. Previously, on Awful (previously on Pivot to AI, even!) we've discussed how repeatedly prompting a chatbot is like pulling a slot machine, and the users of /r/MyBoyfriendIsAI do appear to tell each other that sometimes reprompting or regenerating responses will be required in order to sustain the delusion maximize the romantic charm of their electronic boyfriend.

    I'm not saying this to shame the folks into erotic mind control or saying that it always leads to findom, just to be clear. The problem isn't people enjoying their fetishes; the problem is the financial incentives and resulting capitalization of humans leading to genuine harms. (I am shaming people who are into gambling. Please talk about your issues with your family and be open to reconciliation.)

  • AI boyfriend users radicalise against OpenAI — and self-host their chatbot companions

    Jump

  • I tried to substantiate the claim that multiple users from that subreddit are self-hosting. Reading the top 120 submissions, I did find several folks moving to Grok (1, 2, 3) and Mistral's Le Chat (1, 2, 3). Of those, only the last two appear to actually have discussion about self-hosting; they are discussing Mistral's open models like Mistral-7B-Instruct which indeed can be run locally. For comparison, I also checked the subreddit /r/LocalLLaMA, which is the biggest subreddit for self-hosting language models using tools like llama.cpp or Ollama; there's zero cross-posts from /r/MyBoyfriendIsAI or posts clearly about AI boyfriends in the top 120 submissions there. That is, I found no posts that combine tools like llama.cpp or Ollama and models like Mistral-7B-Instruct into a single build-your-own-AI-boyfriend guide. Amusingly, one post gives instructions for how to ask ChatGPT about how to set up Ollama.

    Also, I did find multiple gay and lesbian folks; this is not a sub solely for women or heterosexuals. Not that any of our regular commenters were being jerks about this, but it's worth noting.

    What's more interesting to me are the emergent beliefs and descriptors in this community. They have a concept of "being rerouted;" they see prompted agents as a sort of nexus of interconnected components, and the "routing" between those components controls the bot's personality. Similarly, they see interactions with OpenAI's safety guardrails as interactions with a safety personality, and some users have come to prefer it over the personality generated by ChatGPT-4o or ChatGPT-5. Finally, I notice that many folks are talking about bot personalities as portable between totally different models and chat products, which is not a real thing; it seems like users are overly focused on specific memorialized events which linger in the chat interface's history, and the presence of those events along with a "you are my perfect boyfriend" sort of prompt is enough to trigger a delusional episode summon the perfect boyfriend for a lovely evening.

    (There's some remarkable bertology in there, too. One woman's got a girlfriend chatbot fairly deep into a degenerated distribution such that most of its emitted tokens are asterisks, but because of the Markdown rendering in the chatbot interface, the bot appears to shift between italic and bold text and most asterisks aren't rendered. It's a cool example of a productive low-energy distribution.)

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 19th October 2025

    Jump

  • Things I don't want to know more about: there's a reasonable theory that Eigenrobot is influencing USA politics; certain magic numbers in Eigen's tweets have been showing up in some of the protectionism coming out of the White House. Stubbing this mostly in the hope that somebody else feels like doing the research.

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 19th October 2025

    Jump

  • Community sneer from this orange-site comment:

    We know from Bell’s theorem that any locally causal model that correctly describes observations needs to violate measurement independence. Such theories are sometimes called "superdeterministic". It is therefore clear that to arrive at a local collapse model, we must use a superdeterministic approach.

    I only got the first 1/2 of my physics degree before moving on to CS, but to me this reads as “We know eternal life can only be obtained from unicorn blood, so for this paper we must use a fairytale approach.”

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 19th October 2025

    Jump

  • Thanks, this was an awful skim. It feels like she doesn't understand why we expect gravity to propagate like a wave at the speed of light; it's not just an assumption of Einstein but has its own independent measurement and corroboration. Also, the focus on geometry feels anachronistic; a century ago she could have proposed a geometric explanation for why nuclei stay bound together and completely overlooked gluons. To be fair, she also cites GRW but I guess she doesn't know that GRW can't be made relativistic. Maybe she chose GRW because it's not yet falsified rather than for its potential to explain (relativistic) gravity. The point at which I get off the train is a meme that sounds like a Weinstein whistle:

    What I am assuming here is then that in the to-be-found underlying theory, geometry carries the same information as the particles because they are the same. Gravity is in this sense fundamentally different from the other interactions: The electromagnetic interaction, for example, does not carry any information about the mass of the particles. … Concretely, I will take this idea to imply that we have a fundamental quantum theory in which particles and their geometry are one and the same quantum state.

    To channel dril a bit: there's no inherent geometry to spacetime, you fool. You trusted your eyeballs too much. Your brain evolved to map 2D and 3D so you stuck yourself into a little Euclidean video game like Decartes reading his own books. We observe experimental data that agrees with the presumption of 3D space. We already know that time is perceptual and that experimentally both SR and GR are required to navigate spacetime; why should space not be perceptual? On these grounds, even fucking MOND has a better basis than Geometric Unity, because MOND won't flip out if reality is not 3D but 3.0000000000009095…D while Weinstein can't explain anything that isn't based on a Rubik's-cube symmetry metaphor.

    She doesn't even mention dark matter. What a sad pile of slop. At least I learned the word for goldstinos while grabbing bluelinks.

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 19th October 2025

    Jump

  • Obituaries are being run for John Searle. Most obituaries will focus on the Chinese Room thought experiment, an important bikeshed in AI research noted for the ease with which freshmen can incorrectly interpret it. I'm glad to see that Wikipedia puts above the Chinese Room the fact that he was a landlord who sued the city of Berkeley and caused massive rent increases in the 1990s; I'm also happy that Wikipedia documents his political activity and sexual-assault allegations.

  • Google won’t fix ‘ASCII smuggling’ hack in Gemini AI

    Jump

  • On a theoretical basis, this family of text-smuggling attacks can't be prevented. Indeed, the writeup for the Copilot version, which Microsoft appears to have mitigated, suggested that some filtering of forbidden Unicode would be much easier than some fundamental fix. The underlying confusable deputy is still there and core to the product as advertised. On one hand, Google is right; it's only exploitable via social engineering or capability misuse. On the other hand, social engineering and capability misuse are big problems!

    This sort of confused-deputy attack is really common in distributed applications whenever an automatic process is doing something on behalf of a human. The delegation of any capability to a chatbot is always going to lead to possible misuse because of one of the central maxims of capability security: the ability to invoke a capability is equivalent to the permission to invoke it. Also, in terms of linguistics and narremes, it is well-known that merely mentioning that a capability exists will greatly raise the probability that the chatbot chooses to invoke it, not unlike how a point-and-click game might provoke a player into trying every item at every opportunity. I'll close with a quote from that Copilot writeup:

    Automatic Tool Invocation is problematic as long as there are no fixes for prompt injection as an adversary can invoke tools that way and (1) bring sensitive information into the prompt context and (2) probably also invoke actions.

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 12th October 2025

    Jump

  • second bongrip Manjaro is an indoctrination program to load up Linux newbies with stupid questions before sending them to Gentoo forums~

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 28th September 2025 - awful.systems

    Jump

  • House Democrats have dripped more details from Epstein files and we have surprise guests! They released an un-OCR'd PDF; I'll transcribe the mentions of our favorite people:

    Sat[urday] Dec[ember] 6, 2014 ZORRO … Reminder: Elon Musk to island Dec[ember] 6 (is this still happening?)

    Zorro is a ranch in New Mexico that Epstein owned; Epstein was scheduled to be there from December 5-8, so that Musk and Epstein would not be at the island together. Combined with the parenthetical uncertainty about happenstance, did Epstein want to perhaps grant Musk some plausible deniability by not being present?

    Mon[day] Nov[ember] 27, 2017 NY … 12:00pm LUNCH w/ Peter Thiel [REDACTED]

    From the rest of the schedule formatting, the redacted block following Thiel's name is probably not a topic; it might be a name. Lunch between two rich financiers is not especially interesting but lunch between a blackmail-gathering Mossad asset and an influencer-funding accelerationist could be.

    Sat[urday] Feb[ruary] 16, 2019 NY-LSJ 7:00am BREAKFAST w/ Steve Bannon

    Well now, this is the most interesting one to me. This isn't Epstein's only breakfast of the day; at 9 AM he meets with Reid Weingarten, one of his attorneys, about some redacted topic. Bannon's not exactly what I think of as a morning person or somebody who is ready to go at a moment's notice, so what could drag him out of bed so early? (Edit: This vexed me so I looked it up and sunrise was 6:48 AM that morning at sea level. It would have been the crack of dawn!) Epstein's Friday evening had had two haircuts, too, with plenty of redacted info; was he worried about appearing nice for Bannon? (The haircuts might not have been for Epstein, given context.) This was a busy day for Epstein; he had a redacted lunch date, and he also had somebody flying in/out that morning via JFK connecting to Saint Thomas and staying in a hotel room there. He then flew out of Newark in the evening to visit the infamous island itself, Little Saint James. The redaction doesn't quite tell us who this guest is, but it can't be Bannon because the Dems fucked up the redaction! I can see the edges of the descenders on the name, including a 'g' and 'j'/'q', but Bannon's name doesn't have any descenders.

    Also Prince Andrew's in there, I guess?

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 28th September 2025 - awful.systems

    Jump

  • I'm curious whether you or @BlueMonday1984@awful.systems are familiar with the concept of MINASWAN. The only time it's appeared in the discussion is in one of the apologies posted by one of the Ruby Central board members, as their signoff line. Quoting a 2016 analysis of MINASWAN in which it is argued that Ruby's central tenet is not MINASWAN, but wa (和):

    Just for the record, MINASWAN is at least half true. Matz is nice. … I would not call DHH nice. … So if MINASWAN is really a basic truth about the Ruby culture, then how does DHH fit in at all? … MINASWAN is garbage. It'd be more accurate to say, "Ruby showcases the Japanese value of 和, but we are arrogant Americans, so we reduce this to a really basic American idea, harshly compressing it in the process to a state where it cannot possibly mean anything any more, instead of bothering to learn something about the outside world for once." But MINASWAN was already a long acronym, so I guess they had to draw the line at RSTJVO和BWAAASWRTTARBAIHCIITPTASWICPMAAMIOBTLSATOWFO.

    Also, I really think it's worth understanding that Ruby is not at risk here. Ever since the release of RPG Maker XP in 2005, Ruby has been a staple of embedded scripting for game engines. Really, what we're seeing here is the demise of Rails.

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 4th August 2025

    Jump

  • Non-consensual expressions of non-conventional sexuality are kink, and non-consensuality itself (along with regret, dubious consent, forced consent, and violations of consent) are kink too. Moreover, "kink" is not a word that needs reclaiming and wasn't used here as a slur.

    If we are going to confront the full spectrum of Christofascism, we do need to consider not only their sex-negativity but also their particular kinks, including breeding, non-con, and non-con breeding, so that we can understand how those kinks interact with and propagate their religious beliefs. Also, sexology semantics for "kink" and "breeding kink" might not be as word-at-a-time as you suggest, akin to how the couple we're discussing probably wouldn't mind the words "press tour" or "mating" used to describe them but might balk at "mating press tour."

  • The rise of Whatever

    Jump

  • Side sneer: the table-saw quote comes from this skeet by Simon W. I've concluded that Simon doesn't know much about the practice of woodworking, even though he seems to have looked up the basics of the history. Meanwhile I have this cool-looking chair design open in a side tab and hope to build a couple during July.

    Here's a better take! Slop-bots are like wood glue: a slurry of proteins that can join any two pieces of wood, Whatever their shapes may be, as long as they have a flat surface in common. (Don't ask where the proteins come from.) It's not hard to learn to mix in sawdust so that Whatever non-flat shapes can be joined. Or, if we start with flat pieces of Whatever wood, we can make plywood. Honestly, sawdust is inevitable and easier than planing, so just throw Whatever wood into a chipper and use the shards to make MDF. MDF is so cheap that we can imagine Whatever shape made with lumber, conceptually decompose it into Whatever pieces of MDF are manufactory, conceptually slice those pieces into Whatever is flat and easy to ship, and we get flat-paks.

    So how did flat-paks change carpentry? Well, ignoring that my family has always made their own furniture in the garage, my grandparents bought from trusted family & friends, my parents bought from Eddie Bauer, and I buy from IKEA. My grandparents' furniture was sold as part of their estate, my parents still have a few pieces like dining tables and chairs, and my furniture needs to be replaced every decade because it is cheap and falls apart relatively quickly. Similarly, using slop-bots to produce software is going to make a cheap good that needs to be replaced often and has high maintenance costs.

    To be fair to Simon, the cheapness of IKEA furniture means that it can be readily hacked. I've hacked lots of my furniture precisely because I have a spare flat-pak in the closet! But software is already cheap to version and backup, so it can be hacked too.

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 27th April 2025

    Jump

  • I'm sorry you had to learn this way. Most of us find out when SciShow says something that triggers the Gell-Mann effect. Green's background is in biochemistry and environmental studies, and he is trained as a science communicator; outside of the narrow arenas of biology and pop science, he isn't a reliable source. Crash Course is better than the curricula of e.g. Texas, Louisiana, or Florida (and that was the point!) but not better than university-level courses.

  • "Power Cut" - Ed Zitron on Microsoft's recent pullback on building servers

    Jump

  • Why is Microsoft canceling a Gigawatt of data center capacity while telling everybody that it didn’t have enough data centers to handle demand for its AI products? I suppose there’s one way of looking at it: that Microsoft may currently have a capacity issue, but soon won’t, meaning that further expansion is unnecessary.

    This is precisely it. Internally, Microsoft's SREs perform multiple levels of capacity planning, so that a product might individually be growing and requiring more resources over the next few months, but a department might be overall shrinking and using less capacity over the next few years. A datacenter requires at least 4yrs of construction before its capacity is available (usually more like 5yrs) which is too long of a horizon for any individual product...unless, of course, your product is ChatGPT and it requires a datacenter's worth of resources. Even if OpenAI were siloed from Microsoft or Azure, they would still know that OpenAI is among their neediest customers and include them in planning.

    Source: Scuttlebutt from other SREs, mostly. An analogous situation happened with Google's App Engine product: App Engine's biggest users impacted App Engine's internal capacity planning at the product level, which impacted datacenter planning because App Engine was mostly built from one big footprint in one little Oklahoma datacenter.

    Conclusion: Microsoft's going to drop OpenAI as a customer. Oracle's going to pick up the responsibility. Microsoft knows that there's no money to be made here, and is eager to see how expensive that lesson will be for Oracle; Oracle is fairly new to the business of running a public cloud and likely thinks they can offer a better platform than Azure, especially when fueled by delicious Arabian oil-fund money. Folks may want to close OpenAI accounts if they don't want Oracle billing them someday.

  • Stubsack: weekly thread for sneers not worth an entire post, week ending Sunday 27 October 2024

    Jump

  • It's almost completely ineffective, sorry. It's certainly not as effective as exfiltrating weights via neighborly means.

    On Glaze and Nightshade, my prior rant hasn't yet been invalidated and there's no upcoming mathematics which tilt the scales in favor of anti-training techniques. In general, scrapers for training sets are now augmented with alignment models, which test inputs to see how well the tags line up; your example might be rejected as insufficiently normal-cat-like.

    I think that "force-feeding" is probably not the right metaphor. At scale, more effort goes into cleaning and tagging than into scraping; most of that "forced" input is destined to be discarded or retagged.

  • open letter to the NixOS foundation

    Jump

  • You're right. I incorrectly believed that hexa had signed based on their comments elsewhere, but I was wrong.

  • open letter to the NixOS foundation

    Jump

  • The original signers include members of the infrastructure and moderation teams. You can find about half of them on Mastodon. They're all well-established community members who hold real responsibility and roles within the NixOS Foundation ecosystem.

    Also note that Eelco isn't "a maintainer" but the original author and designer, as well as a de facto founder of Determinate Systems. He's a BDFL. Look at this like the other dethronings of former BDFLs in the D, Python, Perl, Rails, or Scala communities; there's going to be lots of drama and possibly a fork.