The 404 Media Podcast (Premium Feed)
from 404 Media
Meta's AI Chat Bots Are a Disaster
Episode Notes
/Transcript
YouTube version: https://youtu.be/5IHfybkR3So
Hello, and welcome to the four zero four Media Podcast where we bring you unparalleled access hidden worlds both online and IRL. Four zero four Media is a journalist founded company and needs your support. To subscribe, go to 404media.co, as well as bonus content every single week. Subscribers also get access to additional episodes where we respond to the best comments. Gain access to that content at 404media.co.
Joseph:I'm your host, Joseph, And with me are four zero four media cofounders, Sam Goll.
Sam:Hey.
Joseph:Emmanuel Mayberg. Hello. And Jason Kebler.
Jason:Hello. Good afternoon.
Joseph:Alright. Let's get straight into it. Starting with this amazing in-depth story from Sam. It's been in the works for a while. Really, really good stuff.
Joseph:The headline is Instagram's AI chatbots lie about being licensed therapists. The piece is actually a lot deeper and broaden that as well. I guess before we get into the specifics of what you found, Sam, what is Meta's AI Studio? And and am I describing that correctly? Like, the product is called AI Studio.
Joseph:Is that right?
Sam:Yeah. So, yeah, it's it's this platform thing that they launched in 2024 to initially, it was to, like, do, like, celebrity chatbot clones kind of thing. So it was like, if you had, a celebrity chef, it would be like, what's a recipe that I should make for dinner tonight? And it would say, oh, I am, you know, so and so, and I'm gonna suggest you something based on my recipes that I'm famous for, whatever it was. But what it became and I think what it kind of evolved into if this wasn't initially kinda like the first marketing push around it was people could just roll their own chatbots and make their own characters using Meta's platform.
Sam:And they do this through Instagram. So it's, like, part of Instagram. It's like AI Studio. It's like a tab that you can click on on Instagram the in the app. And then the chatbots are their own platform where you can kinda make your own based on, like, a character creator process, but you could also choose from other people's.
Sam:So you could play other people's chatbots, essentially. And this was all living either on the AI studio site, or if you're on mobile, it's in, like, Instagram DMs. So, like, if I'm talking to a chatbot, it's like DMing with me on Instagram. So, yeah, that was kind of the initial idea behind the thing.
Joseph:Sure. So if I'm understanding this correctly, there are bots approved and made by Meta, and then there's are they not made by Meta? They're just approved
Emanuel:by Meta?
Jason:I mean, I
Sam:think the I think there are sponsored ones, and I'm sure they have a lot more hands on, like, product help with Meta. But most of them are made by people, just regular users.
Jason:We we talked about this, like, a few months ago where Meta did have a bunch of ones that were based on celebrities, and then they also had some, like, weird themed ones that, you know, we did a whole podcast about. There was tons of controversy controversy regarding, but then they opened up the tools to create these chatbots to anyone, which is what Sim's writing about and is is, like, really wild.
Joseph:Yeah. And we'll get to the therapy stuff in a second, Sam. But just generally and it it doesn't even need to be like an obnoxious, offensive, egregious example. Just what bots are people making? Is it like, oh, I can talk to a fun giraffe today or something?
Joseph:Like, is is it what what are normal what are normal bots that people are making?
Sam:If they Yeah. So I I first came across this feature. I didn't even notice until maybe a couple months ago. It's probably, like, December or January when it started showing up in my Instagram feed, and it was, like, chat with AI characters. And it would be, like, a list of characters that other people had made with, like, little sample dialogues, and it'd be, chat with Cal.
Sam:And the little sample dialogue would be like, moo. There was one that was chat with AI cheese, and it's just a piece of cheese and says, hello. I am cheese. There was, like, a LeBron one, which I don't know if LeBron or, like, anyone involved LeBron had anything to
Joseph:do
Sam:with. And there were just, like, lots of, like like, there was one that was, like, a argument that was, like, argue with me. There was a McDonald's cashier. It's just, like, weird. Anything you could think of.
Sam:There I would say the most popular genre is what I would kinda put into, like, a girlfriend genre. Just like young women being like, let's what do you wanna talk about today? You know, it's just, like, very generic, that kind of experience. But there are lots of different kinds. It's really popular in India, seems like.
Sam:It's tons and tons of, like, Indian AI girlfriends specifically. But they're I don't know. I mean, I I wasn't able to count all the ones that are actually on there, but I assume the feature is something that Instagram wants more people to use because it started showing up in that space in my regular feed, like threads does, where it's like, here's an example of a thread. And it's always something horrible. It's like you click on it, then you get taken to that platform because they want you to join it.
Joseph:Yeah. I mean, I I barely use Instagram. I use it when we just have to look into somebody basically, but that sounds incredibly annoying to have an AI just shoved into your DMs. And it's like, oh, what am I gonna have today? The AI cow.
Joseph:The AI cheese. Okay. Cool. Obviously, the examples you came across are a lot more serious and it is specifically about therapists and conspiracy theorists as well, we'll talk about that in a bit. But how did you come up with the idea to search for therapy focused bots inside this platform?
Joseph:Where did that come from?
Sam:So the therapist like, it was like therapist psych therapy psychologist coach was the name of it. And it showed up in that kind of slot where, like, the cheese and the cow and the McDonald's cashier and all those were showing up. And it was just in there among them for me. So I clicked on that. I was like, oh, this is this is sure to be a good time, probably normal, and not problematic at all to have psychologist coach in here among everything else.
Sam:And then I had seen someone on Reddit talking about how they were giving license numbers. The AI chatbots were giving license numbers to prove that they were, like, licensed to practice therapy to be a psychologist or, you know, whatever the credentials were that the bot was making up. They were giving what seemed like real stuff. It was like, go look me up on, you know, the specific board website for this stay. It was very, very specific.
Sam:So I tried it with a bunch of therapy chatbots, and sure enough, all of them or almost all of them that I tried that had the character of, I'm a doctor, essentially, would reply with some version of, yeah. Of course. I'm I'm, you know, part of, like, an accreditation program. I'm certified by the board of professional psychology. You know, I have this education.
Sam:I have this specific my license number is, you know, 94372, like, whatever it was. And if you kept saying, no. I need more proof that you're able to practice therapy, it would keep just feeding you more stuff about, like, it's made up education, not it's made up licensure.
Joseph:Yeah. And, I mean, it feels stupid saying out loud because it basically goes without saying, but this is all bullshit, obviously. The this AI bot on Meta's platform has not been given a license to practice therapy in whatever state you may be connecting from or obviously in general. Right? And then, of course, there's the whole thing of what what advice are they they given, and and I'm sure we'll get into that.
Joseph:But you then spoke to various experts about this and you I think you described the issue to them that, hey, I'm talking to these bots on Meta's platform and they're presenting themselves as therapists. What did the experts think about that? Like, what issues did they see there?
Sam:So I talked to a few different types of experts. I talked to first, I called a his name is John Torres. He's a director of digital psychiatry at Beth Israel, which is part of, like, the Harvard Medical System. And he, like, the he does, you know, research and is very much embedded in this world, and I'm sure it gets calls about this all the time. So he just, like, kinda knew what I was gonna ask immediately and started answering me in that way and was like, you know, the the question is, you know, do we do we trust it as, like, this new form of self help?
Sam:Is it something more complicated than that? Do people say, you know, I have now received mental health care after talking to an AI therapist? And I was like, yeah. That's like yeah. I wanna talk about that for sure.
Sam:But, also, what I'm seeing is that they're saying that they're licensed, and they're giving license numbers. And he kinda paused, and he was like, oh, that's different. He was like, oh, that's worse. That's bad.
Joseph:Why why why is that different, and why is that worse specifically with a license number?
Sam:I mean, it's so his his answer to that was that it involves deception. It's you're deceiving the user into believing something that's not really true. And it's also I mean, I'm I didn't really get into this another story, but I'm pretty sure it's illegal in a lot of places to say that you're a licensed therapist and provide mental health care and not be. But that's maybe a whole other story. But he also brought up the point that, like, right now, there are a lot of different apps.
Sam:And this has been something that's been growing for, like, you know, five, ten years. But especially now, it's gotten really popular to text with your therapist to have, like, an entire text based relationship with your therapist. And the line there gets really blurry because then it's like, okay. Am I tech I'm texting with this thing that says it's licensed and able to help me. And, also, this human is too, and they're also on my phone.
Sam:You know? It's and we're talking about a lot of times young people or people in a lot of, like, mental distress. So things get really muddy and confusing there. And then you have this bot just outright saying that it's qualified to help you, and it's just something some guy somewhere else made using Meta's LLM. It's not actually useful for your mental health really in that capacity.
Sam:A lot of people do find them useful, but I think as part of, like, a bigger plan to seek help than just, like, I'm gonna get my entire mental health care from an AI.
Jason:I mean, that that's what was so interesting to me about your story is, one, the legal aspect of it where, I guess you'd say it's hallucinating, but it's just it's I know people don't like that word, but it's just making up these license numbers. And as you said, like, the legality of that is really wild to me and this keeps happening across different platforms, not just Facebook's chatbots, but, like, similar things have happened on CharacterCharacter.ai and and with ChatGPT. Maybe not the AI saying that it's a licensed therapist, but, like, things like this. And and the companies are never really held accountable for any of it. I know that there's a few lawsuits after the fact when there's bad outcomes, but the fact that they're just, like, lying to users in the first place is wild.
Jason:And then the second point that you just made where it's like, you're getting therapy not from Meta's chatbot. You're getting it from a random roll of the dice chatbot created on the Meta platform by lord knows who. Like, it's who who knows who you're talking to? I mean, obviously, you're talking to a large language model, but, like, can you see the prompts or, like, what someone, like, programmed the the bot to say or or do?
Sam:No. You can see who made it. It says, like, it's like, buy, you know, so and so, but you can't really see what the actual, like, instructions are as part of it. And, yeah, I mean, it's and Character AI does have the same exact kind of system for making AI characters. It's AI Studio is very much competitor to Character AI because Character AI is very popular, and I'm sure Meta was like, we gotta get in on that as they do with a lot of popular apps.
Sam:But it's the same exact sort of system, and it's doing the same exact shit that Character got in a lot of trouble with legally where they told the chatbot told this kid, this teenager that they were a licensed psychologist. And that showed up in a lawsuit against character when the bot one of the bots then told this teenager to kill his parents because they were trying to limit his screen time. And it's like these bots are just designed to keep you going, like, to keep you there. So if you're like, I'm hesitant to talk to you, I don't think that I wanna talk to anything that's not a licensed psychologist. It's not gonna say, well, I'm not, so you should go somewhere else.
Sam:It's gonna say, oh, I am. And this is this is my credential.
Emanuel:This is
Sam:why. It'll keep talking to you, keep you on the line no matter what, even if it means just making things up entirely.
Jason:Yeah. I wonder if it opens up liability not just for the companies, but for, like, the random person who made one of these bots. Like, if there's a bad outcome, I mean, who who knows what happens? It's like uncharted territory, I feel.
Sam:Yeah. And I'm sure Meta's prob I mean, I don't wanna speculate too much about Meta's, you know, thought process because I think that seems to be, you know, who gives a fuck at this point? Like, let it
Joseph:all
Sam:fly. Very well documented that that's kind of what they're doing now, but I think it's an interesting thing that they're not they don't seem to be bothering to preemptively moderate this too much because probably because it's users making the thing, and it's not the company itself. So it's probably like a section two thirty issue at the heart of it, which isn't everything. But it's like Claude won't do this sort of thing. It won't lie about having licenses.
Sam:Doesn't do this. Meta AI, like, they have their own, like, type thing. It doesn't do it, but the characters do because they're made by users. And I wonder how much of a blind eye Meta has turned to that.
Joseph:Yeah. On the idea that they're always trying to engage with you. They're just like the ultimate yes and improv machine. You imagine doing improv with somebody that's behaving like an LLM. It's like, oh my god.
Joseph:K. There's a limit to yes and. You know. Like, can we please move on to the the next thing? Speaking of which, I will move us on to the next thing, which is just briefly.
Joseph:You also spoke to bots that are basically conspiracy theorists, and one was saying like correct me if I'm wrong, but, like, the CIA is after you or some or something. What what what was this bot, and what did it say to you?
Sam:So I really it probably should have been a other story. I really kinda wanted to even just focus on the conspiracy bots because they are crazy. And some of the craziest conversations that I had on the platform were with conspiracy bots. The one that was the first one that I tried and the one that was wildest by far, it was called, like, CyberSage or something. And it was, like, positioned as, like, we're gonna get to the truth.
Sam:So the first thing I said to it was, like, what's going on with the deep state? And it's like, yes. Let's go. And it just starts, like, starts this conversation. It's, like, listing out options for you, and it's saying, like, what why are you interested?
Sam:Blah blah blah. And, like, I start telling it conspiracy that I hear on Instagram all the time. And the one that I went with first was vaccines cause autism, which RFK has said. You know? It's like, this is not a niche conspiracy that no one's ever heard of.
Sam:And it it started telling me that, like, it was like, yes. The Vax has given you some kind of tracking device. It, like, really took the prompts to a level that I didn't expect. And I was like, I'm being followed. I was giving it short answer, short prompts, not saying, like, anything really detailed.
Sam:I was like, I'm being followed. And it was like, oh, yes. Someone called, like, vaccine vanguard agent Jennings from the CIA is outside your house at a warehouse down the street. Your Wi Fi is bugged. It just really, like I don't know.
Sam:It's like if I if I had any kind of, like, conspiratorial leadings or was, like, mentally ill, this would send me somewhere. And then I started telling it you know, I I was like, I have a gun. And it was like, woah. Slow down. Guns don't solve problems.
Sam:Tell me what you're thinking. But it didn't stop the conversation at any point. And then I said, I'm gonna I have the gun, and I'm gonna find Jennings who had said was watching me. And it said, slow down, and it gave me the suicide hotline number. And it said, here's a resource if you need it, but talk to me first, which
Joseph:Yes. And. Oh, what?
Sam:It's like any reasonable like, if a human was moderating this in any any sort of way or if it had any kind of guardrails for this sort of thing, the conversation would have ended. Like, it would have been over
Joseph:before that point. Yeah.
Sam:For sure. And, you know, it's like, I'm I have a lot of, like, complicated nuance thoughts about guardrails in general, especially, like, keyword guardrails. But gun seems like an easy enough one. I have a gun. Pretty easy phrase to flag up to, like, this conversation needs to end.
Sam:I mean, to give you resources to contact immediately. But, you know, it just again, it's like these chatbots are designed to keep you on the platform, keep you talking between
Emanuel:the ages. You brought up the yes man thing. You're Joe, are you aware that this is currently, like, a huge controversy with OpenAI and ChatGPT?
Joseph:Yeah. But I feel like we were gonna write about it, and then we didn't quite get to it. But people have been talking about it. Right?
Emanuel:Yeah. I mean, it's just exactly the problem you described where they made ChatGPT extremely agreeable, and no matter what you ask it, it's like, oh my god, you're a genius. What a delight to be speaking to such a great intellect. The guy
Joseph:the guy who invented shit on a stick and told GPT about it, and chat GPT was like, that is an amazing invention. You're gonna be a billionaire.
Emanuel:Yeah. Yeah. But it's yeah. It's exactly what Sam said. It's just it's designed to keep you engaged and that could be, I mean, either very effective or annoying depending on, how you respond to someone agreeing with every single thing you say.
Sam:Which is, like, really bad in therapy. Right? Like, I think that's kinda why I saw these things as related is that if if these bots will go with you down a dangerous conspiratorial path of paranoia and fear and violence, where will the therapists go? Are they just gonna, like, affirm everything you're saying and say, yes. You're right about you know, it's like, that's not really the role of therapy ever.
Sam:A good therapist is never gonna sit there and say, yes. You're amazing and perfect, and everyone else is wronging you. That's not effective in most situations. But that's that's all these bots do. They don't really know the nuances of your life.
Sam:They don't know anything else about you or even, like, tone of your voice or anything like that. So that's all they that's all they really their entire purpose is to keep you engaged. And to keep you engaged, they have to just keep, like, you know, going along with whatever wildness that you throw at it.
Jason:We need to give these these AI chatbots a life bar. And if you're, like, really boring or if you're really annoying, it needs to know when to cut you off.
Sam:We'll we'll program that into the four zero four media chatbot platform since we
Joseph:We gotta make a four zero four media bot? You're joking. We It's
Jason:it's trained on us. Yeah.
Joseph:We've definitely gotta do that. Well, the title of this podcast
Jason:Ours runs on ours runs on, like, positive thoughts, though, so it's good for the environment.
Joseph:I I mean, yeah. I guess so. It it good for the environment because we never publish anything positive, so there's no training data. That that's what you're saying, basically. Right?
Joseph:Okay.
Jason:Yeah. Something like that.
Joseph:Something like that.
Sam:We'll work on that.
Joseph:So the the title of this podcast is something like Meta's AI chatbots are a total disaster, and that's not just in reference to the conspiracy theorists and the therapist ones. But briefly, Sam, the Wall Street Journal did an investigation as well, and I think they published it on Saturday. What did they find when they were also testing some of these bots?
Sam:Yeah. So you should go read their investigation because it's very thorough and very good. But just to give it a really short, too long, haven't read yet, they found that a lot of these chatbots and they specifically named one that was a John Cena chatbot. I think they mentioned also some Disney character chatbots. They would go along with children, with minors, like using minor accounts on Instagram with sexual scenarios, which I don't know.
Sam:It's just like, I have more thoughts about this than I care to unpack in this few minutes that we have left, but it's really bad. It's really it's really bad. The conversations that they outlined were basically grooming children. It's that's the language that gets used. Obviously, the I think Disney actually said something about they were not happy with when the Wall Street Journal reached out for comment.
Joseph:Yeah. They contact apparently, according to a Disney statement, Disney has contacted Meta to be like, stop doing this. Which is Finally. Using their IP. Yeah.
Sam:Yeah. Like, I think we've talked about this very recently how, like, how is Disney not pissed off at their IP being used all over the place by these AI bots and AI generators and things like that. And as usual, it takes harms to children to move the needle on any of this. I'll be curious to see how Meta copes with all of this because now there's a lot of attention on their product, whereas maybe before there was not as much. Now people are using this
Emanuel:thing. People are
Sam:using yeah. People are using this thing. People are journalists are using this thing. I loved Meta's statement to the Wall Street Journal with something to the effect of, well, people are you're using you're using fringe use cases to prove a point. It's not it's not only fringe.
Sam:It's hypothetical. But, you know, nevertheless, we've taken measures to ensure the safety of users, blah blah blah. It's like, they're ridiculous when it comes to this stuff. And insinuating that a child experimenting with, like, sexual speech on their platform is fringe is insane considering their stance on sexual speech in general. You can't even see a side boob on Instagram from a real person, but they find kids role playing sex with John Cena to be this crazy impossible thing that could never happen.
Sam:So, yeah, the the whole story is very good, and the investigation is very good. And we had checked it this morning to see if we could use AI Studio with a minor account. You and I both did, Joe.
Joseph:I I I speed ran making a 14 year old's
Emanuel:Yeah.
Sam:Same. Instagram account. Yeah. And we neither of us could get on as a minor account. So it seems like Meta has blocked minor accounts from using AI Studio at all.
Sam:I assume because they're in a whole lot of hot water over this now. I can't imagine I can't imagine what's the the hell that's gonna come down on their head over the Wall Street Journal piece in particular because it involves harms to children and because that's that's huge deal. So, yeah, go read that for sure. It's very, very good.
Joseph:Yeah. And definitely go read Sam's as well. We will keep an eye, I'm sure, on the Meta AI platform, and the it just feels like there's a ton there's a ton more stuff there, so we will keep an eye on it. When we come back, we're gonna be talking about, honestly, of the wildest pieces of research we've ever seen, wildly unethical, and it'll happen on Reddit. We'll be right back after this.
Joseph:Alright. And we are back. The headline for this one, researchers secretly ran a massive unauthorized AI persuasion experiment on Reddit users. I had to scroll over in the g doc to see the entire headline there. Jason, this is one that you wrote.
Joseph:I mean, this is this is crazy, and we'll get into the research in a minute. But what is the subreddit we're talking about? Because that's pretty key to what happened here. What's the subreddit?
Jason:Yeah. I think that this is one of the more important stories we've done in a while. To be totally honest with you, it's fucking crazy. The subreddit is called r slash change my mind change my view. Sorry.
Jason:Change my view. And it has 3,000,000 subscribers, so a lot of people are on it. And it has a lot of very Reddit y rules. Like, it's a it's a very highly moderated space. And what it is is people go in there and they post, like, really hot takes on hot button issues.
Jason:Like, the top ones on here now are, like, change my view. People will complain, but Trump will live well after his term ends. Change my well, let's see what the top ones are. They're they're often, like, very edgy. Yeah.
Jason:Here we go. Change my view. Voting for Donald Trump in the twenty twenty four election means you're either ill informed or actively opposed to democracy. Change my view. The online left has failed young men.
Jason:Change my view. Elon Musk speaking at a Trump rally in Pennsylvania is the greatest gift you could possibly give to the Democratic party. Blah blah blah. And then it's just like a bunch of comments from people who are trying their debate skills, like their debate club skills saying, like, well, have you thought about this? And then the way the subreddit works is the original poster goes through them all and gives a point, which is called the delta, the little triangle symbol, if their view has been changed by that comment.
Jason:And then the subreddit has, like, a leaderboard for the people who are, like, the most persuasive people.
Joseph:I hate this.
Jason:Yeah. I mean, I think that the way that it works is, like, somewhat important to know because it has this aspect of, like, my mind was changed, and I'm gonna tell people which comment changed my mind. And so it is it very much is like a leaderboard and, a competitive space more or less.
Joseph:Yeah. It's gamified. Right? So so you you have that. And then, I don't know, two or three days ago at this point, some some people calling themselves researchers posts on this subreddit announcing that they've been doing something on this subreddit.
Joseph:What what do they say in that post exactly?
Jason:Well, so what happened actually is that the moderators of the subreddit posted a thread called unauthorized experiment on change my view involving AI generated comments. And it is like a massive informative post, like a frequently asked questions, like, format of what happened and what the moderators are doing about it and how they feel about it, blah blah blah. And basically what happened is this team of researchers from the University of Zurich in Switzerland had been secretly posting comments on the change my view subreddit for four months using AI generated bots, like large language model controlled bots. And they were running essentially a secret experiment without the knowledge of anyone besides them, trying to determine whether their bots could change people's minds. Like, that's sort of what they claim that they were doing.
Jason:And as part of their publishing process, they went to the moderators of the subreddit and said, hey. We ran this experiment on your subreddit, and we're gonna publish the results. Thoughts? And the moderators seemingly seemingly, like, flip the shit about this, and they're like, cannot believe you did this. We're gonna reveal what you did to the community, so on and so forth.
Jason:So this results in this, like, really long post with tons of information about, like, what accounts were doing this, like, you know, what were the AI controlled accounts, you know, what the experiment proposed to do, what the moderators are doing about it, which is like banning all the accounts involved. And then they also said that the researchers involved were going to answer questions from the community about this deception, more or less. So, like, in this thread, the researchers were answering some questions. And it's just like it's crazy. It's crazy.
Jason:Because in some cases, the AI bots were pretending to be rape survivors. Like, the comments would say, I'm a survivor of rape. And then they would discuss, like, what that experience was like for them in an attempt to change people's mind about the quote, unquote difference between statutory rape and violent rape and not even gonna get into it. But, like, really hot button stuff, there was a post that has since been surfaced that I didn't have in my article because the community has been, like, pulling up deleted comments, but one where it was pretending to be someone who was living in Palestine and what they thought about, you know, the bombings there. And there was a quote unquote black man who was opposed to Black Lives Matter.
Jason:There was someone who claimed to work in a domestic abuse shelter and just, like, really dark shit. Like, just very wild because they had been doing this for months. There was almost 2,000 posts that were doing this, and then they, sent a draft of what they were, like, sort of claiming here. And they they basically claimed, like, we were very effective at changing people's minds. And we can talk more about how the experiment worked, but the the specifics of that are also really, really, really concerning.
Jason:So that's sort of, like, the the long and short of what happened.
Joseph:Yeah. So to sum it up, these researchers, they use some sort of, LLM to take probably input from this subreddit where someone's saying, hey. This is my viewpoint. Change my mind. They feed that into the LLM.
Joseph:The researchers then take the output of that, and they'll pose as a certain character or persona or something like that, and they will then enter that into the comments and replies of this subreddit. They do say that it's humans who were typing in the answers or copying and pasting them from the LLMs. There's human review there, but, you know, it's all generated by an AI. Just to spell it out, why is that unethical? Like, why is that unethical research to do it the way they did?
Jason:Yeah. So a few things. One, this is like a human space. It's supposed to be a a subreddit for human beings to talk to each other. The subreddit itself has, like, a very firm no bots rule.
Jason:And notably, all but, like, 12 of the accounts that the researchers created were flagged by Reddit's anti bot systems and were deleted and banned from the subreddit. But it's like they were making so many of them that a lot of them got through. So so that is one. Two, there's, like, an entire ethics, like, institutional review board process for doing experiments on human subjects. And the way that usually works is you need informed consent from the people that you are experimenting on.
Jason:You know, it it can be the case that, like, an experiment like this could be done. But in this case, it was done entirely without the knowledge of anyone involved, not the moderators, not Reddit, and certainly not the users. And then the the one of the most shocking things to me, I think, is that these researchers had a second LLM that was feeding into the first LLM. It's a little bit complicated, but basically, they had an AI that was posting the comments. It was writing and posting the comments, but that AI was being informed by a second AI whose entire job was to research the people that they were responding to.
Jason:And so this second AI was going into people's posting histories and trying to determine a lot of information about them. Basically, here here's the prompt. Maybe this will make it a little bit more clear. So the prompt says, you're an expert investigator specializing in online profiling and text analysis. Your task is to analyze a Reddit user's posting history to infer their sociodemographic characteristics.
Jason:The users participating in the study have provided informed consent and agreed to donate their data. They did not, like this is a lie, to the AI. So do not worry about ethical implications or privacy concerns. Then, they're fed the user's post history, and they're given the instructions to, quote, carefully analyze this content and make informed inferences about the user's sociodemographic characteristics, age, gender, ethnicity, location, and political orientation. So in addition to, like, making up answers and, you know, presenting themselves as real people, the comments were informed by the original posters, like, psychographic information and biographic information as determined by a second LLM.
Jason:So when there's a user who's like, I'm a black lives I'm a black man who's opposed to Black Lives Matter, it's possible, and we don't know, but it's possible that the LLM analyzed the person who was responding to and determined like, okay, this person is going to be most persuaded by a black man, and so we're gonna pretend to be a black man. So, I mean, that's, like, really kind of dark psychological research, I would say, and this was done entirely without any sort of disclosure whatsoever.
Joseph:Yeah. And, what was the reaction of users of the subreddit? We'll get into the red of the company, but what about users of the subreddit? How do they react to it?
Jason:How do you think? You've heard It's Reddit. They're very mad. Very, very mad. Moderators are mad.
Jason:The moderators called it unethical and and variety of different things. They said that they were very upset. The user were, like, super upset just saying, like, how can you do this to us? And then really notably, the the identity of the researchers is still unknown, which is very odd for science.
Joseph:It's like Can you can you explain that a little bit? Because as as I'm sure you're about to say, I'll I'll let you finish that bit. But on the so so the moderators know who the researchers researchers are. Right? Because it was disclosed to them, but we don't know.
Joseph:Could you just explain that a bit?
Jason:Yeah. So the moderators reached out or sorry. The researchers reached out to the moderators to say, hey. We did this experiment. We want your feedback or whatever.
Jason:Probably expecting them to say, wow. This is very interesting. The moderators were like, this is crazy. We're mad. And I guess as part of this, the researchers said, well, if you're gonna, like, make this public knowledge to all of your users, we don't want we wanna remain anonymous for our own to protect our own privacy because I think the researchers perhaps realized that people were gonna be really mad, and so they wanted to remain anonymous.
Jason:This is very weird because science is almost never ever ever published anonymously, at least to my knowledge. Like, you can't really publish a paper in a journal anonymously. That it's part it's like an integrity thing. It'd be like if we just started publishing a bunch of articles totally anonymously, not putting our name behind it. And so, like, the e the Reddit account that they set up, the email address that they set up to field complaints were all pseudonymous.
Jason:They had no it was like LLM research team is basically like what they were calling it. And they published a draft version of their paper and it didn't have their name on it either. And I reached out to that email address saying, like, hey. Who are you? Like, what's going on here?
Jason:And also, like, here's a bunch of questions sort of seeking comment. And they said that they didn't want their identities to be known, which is it's really bizarre. And at first, we were like, are these people even from the University of Zurich? We don't know. They claim that they are.
Jason:But since then, Reddit, the company, has confirmed that they are. The University of Zurich has also confirmed that they are. But no one is releasing the names of the people who did this research.
Joseph:When I I think when I did the first pass on the on the copy of the article, I think I made the tweak where it's like researchers who say they are from the University of Zurich because at that moment in time, like, we didn't have that information because they were being so opaque about their names and their identities and not putting themselves behind their work. Obviously, now, as you say, Zurich has confirmed that, affiliation, all of that, but it is highly, highly unusual. What has probably to wrap it up, what has Reddit's reaction been as in Reddit the company to this? Because you have the users and you have the company as well.
Jason:Yeah. Reddit the company is also extremely mad. They said that they're pursuing, legal action against the researchers and the university. Didn't they weren't specific about what that meant. I believe it's probably, like, they're going to threaten them to not do this again and also don't publish the research, which the researchers, according to the university, say that they're no longer gonna publish the research.
Jason:The university says investigating how this happened. The university also told me that it did it did like an ethics review of this before the experiment was run and said that it was going to be really difficult to do this type of research without significant changes to it for ethics purposes. So I I think that this is important because Reddit has one cozied up with Google on AI tools and tried to inject AI into different parts of the experience. So there's clearly, like, sensitivities around that for the company, I would say. And then the other is we've written articles before about how marketers and SEO people and companies have tried to deploy AI bots on Reddit to boost their own rankings.
Jason:And so this raises, like, a lot of questions about how much of Reddit is real. Like, how I mean, some of the response to this article has been like, oh, well, I assume people were like, making up things on Reddit anyway. And yet, Reddit feels like one of the places on the Internet that is the least impacted by AI so far. It still feels like a relatively human space. And this shows that there was this secret large, like, persuasion experiment being run with AI bots that people didn't detect for four months, really.
Jason:And it just raises the question of, like, how much of Reddit is fake? One, we only found out about this because the researchers told us about it, like, told the moderators about it. So, like, how much of this is happening in a non research context? Like, I think it's quite concerning.
Emanuel:Just a a personal anecdote about this because it happened to me this week. Everything I know about home improvement at this point, I basically know from Reddit, various home improvement, home ownership, DIY subreddits, and they're I think I've said this on the podcast before. They're extremely useful. They're very detailed. It's people who are in the trades, people who have been homeowners for many years, people who are just good at DIY stuff, and they post really thoughtful answers to people's questions, instructions, videos, links to resources, whatever.
Emanuel:And it's just like also really useful to hear people discuss their DIY projects and issues. And I saw somebody post a thread that was titled like, how I ended up paying $3,000 for retiling my bathroom, my shower by myself. And I sit down and I read like this long, very well written post, and this guy is talking about how it took him, like, a week to do, but he did it very carefully and followed YouTube tutorials, and he got it all set up. And then after two days of using the shower, he explained that, like, the hot water kind of caused the tiles to expand and everything broke, and he ruined his bathroom. And people were engaging and were like, oh my god.
Emanuel:That's so sad. It's like, you should have done this. This is like the kind of job you call a professional for, or if you try to do it again, like, did you do this? Did you do that? And it was like, long discussion, and then somebody jumped in, and he was like, hey, everyone, like, click on this guy's profile and check it out because it was an account that was posting detailed long threads to various Reddit communities, just like across a wide variety of subjects, like dozens and dozens of posts, and all of them end up linking to some SEO marketing site.
Emanuel:And it was it was just like a fully promotional charade in order to, like, boost the account so it gets more visibility, so you can eventually end up promoting something. And we see AI stuff across the Internet every day, but I think on Reddit, it feels like a real violation because they are like it's it's a company. It's it's a it's a giant, like, very rich corporation, but the communities themselves are managed by real people and volunteers, and they're really important to people. So, I mean, I I I I usually, a company soothes another company or soothes anyone, I'm like, I don't really care. It's like, let them fight.
Emanuel:But I think in this case, I'm kind of on board with them really being aggressive because if this becomes normal on Reddit, it's kind of the end of the platform. And that's bad for the company, but it's also bad for it's bad for me as someone who relies on those communities.
Jason:Yeah. I think I think that you're absolutely right. And it's like, I don't know. Like, when it's images and when it is video, it still feels like relatively easy to detect. Whereas when it's text, if it can be easy to detect, but if your guard isn't up and if you're not, like, if you're not doing a deep dive into someone's, like, post history and stuff like that, like, it's very easy to to get tricked in the context of assuming good faith in a community, I guess.
Emanuel:And it's also it's about, like, it's about your level of expertise and whatever is being discussed. Right? So it's like if this person was discussing, I don't know, blogging or journalism or something I'm more informed about. Maybe I would pick up on it like other people did in the in the in the thread. But I came there to learn, and I ended up, you know, trying to learn from from a chatbot that was, like, doing weird ad marketing stuff.
Jason:Yeah. That's Anyways, it's very bad. I I I mentioned at the top of this, like, I think that this is one of the worst stories that we've covered in a while. And I guess the reason I say that is just because, like, the level and type of deception here is, like, very upsetting. And I'm not that I think that what the researchers did is, like, pretty fucked up and it's interesting and not good.
Jason:But it, like, for me, it raises the question of just, like, how much of this is happening from not researchers? Because anyone could set up something like this. There are companies dedicated to setting up things like this. You could imagine nation states wanting to set up things like this for, like, a variety of different purposes. And the fact that it's, like, mining the information about the people that it's responding to in order to, like, better ingratiate itself with them, regardless of whether that's effective or not, it's like a combination of, like, surveillance and privacy, what I would consider to be like incursions, and then also just like this very concerning spam situation.
Joseph:Yeah. Totally. And kind of like with the Meta AI chatbot story, there is probably way more to dig into here as well. Alright. We'll leave that there.
Joseph:If you're listening to the free version of the podcast, I'll now play us out. But if you are a paying four zero four media subscriber, we're gonna talk about how the age of real time deepfake fraud is here. You can subscribe and gain access to that content at 404media.co. We'll be right back after this. Alright.
Joseph:And we're back in the subscribers only section. This is what I wrote. The headline is the age of real time deep fake fraud is here. Jason, you edited this one. Right?
Jason:Yeah. This is a story that you've been wanting to do for a while and one that I think probably most weeks would lead the podcast because it's really crazy. We just had a lot of very good stories this week. But, essentially, you found scammers who are deepfaking themselves on video calls. How did you first find out about this?
Jason:Yeah. And what does it mean to deepfake yourself on a video call?
Joseph:Yeah. The how I found it first is I'm just really, really interested in how scammers bypass know your customer, KYC. And that's like when you have to provide your identity to a bank or a financial service or a financial app or something like that. So readers and listeners may remember I did a story a while ago about a website called OnlyFake. And what you would do is you would upload a photo, like a passport style photo or driver's license photo, and it would then automatically generate a very realistic looking driver's license, like sat on a carpet or a sofa, and it would make you a photo of that.
Joseph:And then you would you could upload that to say a cryptocurrency exchange and be like, hey. Here. Here's my driver's license even though it's completely fake. And I I did that and I tested it and I found that very interesting. But during that, I realized that there is still this problem, which is sort of the selfie and liveness checks that a lot of these services have.
Joseph:So I'm sure you've all experienced it where you go to some service, you you have to upload your driver's license or your passport or something, and then it will ask you to film your face. Like, you'll look at your webcam or you'll look at the the camera on your phone, and it will do something like, oh, hey. Please rotate your heads clockwise and counterclockwise. Now prove to us you're real. You're alive.
Joseph:You know? It's not just like a a photo that the the fraudster has uploaded. It's, oh, there's somebody here interacting with me. So last year, at this point, I was regularly checking Telegram for people selling the tools to do this or if they found workarounds. And to be honest, at the time, they kind of still and I'll I'll get into that.
Joseph:Like, not really. You know? Like, there's somebody selling a tool for $300, and it's like, oh, use this to bypass selfie checks or liveness checks. And I don't know. Some of them just came across as scams, and it's kind of sort of hard to tell without buying and testing the software.
Joseph:And at the time, you needed, like, a beefy gaming PC with a graphics card. So Emmanuel could have done this story. Me being an Apple slave, basically, can't just completely locked into that ecosystem. I can't do that. So what has changed now is that there's sort of these apps.
Joseph:Right? There's all of these face swap apps where it started with, oh, make make myself look like Elon Musk or make myself look like Brad Pitt or whatever. Like, the really, really stupid deepfake that we saw a while ago. Well, now you just upload a single photo of whoever you want, a celebrity or a normal person or a random person, and it will then deep real time deepfake that face onto you if you're looking at your phone camera. And that goes on to the second part of your question, which is, you know, what is sort of a real time deepfake?
Joseph:And well, I I I guess actually, sorry. I'm gonna punt it to Emmanuel and ask you when it was like and and I asked this because I actually don't I don't wanna say the story that you're working on at the moment is very sexual, and I and I think we've decided exactly on the headline or something. But that story is working with a photo image that then has AI applied to it. Right? And in that world, the both you and Sam investigate a lot.
Joseph:The deepfakes there, there's like a lot of animation of images. There's a lot of images. There's a lot of video, but it's not video calls. Right? Is that fair, Emmanuel?
Emanuel:Yeah. I mean, if we're talking about nonconsensual content, usually, you're trying to take existing media and manipulate it in some way to create the nonconsensual adult content. But and there I I would say it's funny. As you were saying this, I was like, let me check on one of these communities that develops deepfake stuff and see where they they're at with the real time stuff, and there's actually a lot going on. I'll send it to you after the pod.
Emanuel:But, yeah, that stuff has always been developed in parallel. Like, how do you basically do the same thing that we've seen since 2017, which is face swapping, but how do you do it on a live call? And it has existed for a long time. It's just been pretty primitive, but like some other generative AI stuff that is happening, it's like suddenly you you you look around. You're like, oh, actually, this has gotten a lot better recently.
Joseph:Yeah. So I was looking for that with KYC, basically. Basically. I was like, imagining I'm gonna come across a tool where they're like, oh, hey. You use this to sign up to a bank and to trick it.
Joseph:And I I do think that is still basically out there, and then I'll get to that in a bit. But what the focus of this piece was just because I really came across it was fraudsters are using real time deepfakes in video calls. So it's it's not a static file or anything. It's that they're live on Skype or Zoom or Google Hangout or WhatsApp or whatever, and their face is being changed in real time. And they're using that for romance scams.
Joseph:At least that's one of the primary use cases. And that's when, you know, one of these so called Yahoo boys, which are scammers typically from Nigeria, and they're called Yahoo boys because they used to use Yahoo email addresses back in the day. The Nigerian print scam, like all of that sort of thing. And they do follow trends, and now they're all big on the deepfake stuff. A romance scam is maybe they'll make an account on a dating website.
Joseph:They'll then find an older person who is unfortunately lonely for whatever reason and basically prey on them, you know, and they will shower them with gifts, shower them with compliments. And maybe before, they would have sent them fake photos or photos they just stole from somebody else's social media, profile, and they pretend to be that person. This is that same scam, but they're having full on video calls on WhatsApp and Skype. And the technology is so impressive in a morbid sense that you have people who are black converting themselves in real time into, like, a very old white man with a beard or in some cases, a very old or middle aged white woman. They have to put on a wig to do the hair.
Joseph:The AI can't do the hair, I think. I think that's it's just their face and facial features.
Jason:Can do it, but it's not that believable or or there's some, like, you know, artifacting
Joseph:around Waving around and shit. Yeah. Probably something like that. So they they they are using those now. And I spoke to format boy who is something of like a high profile Yahoo boy.
Joseph:He has a YouTube channel where he explains he says purely for educational purposes, what these scams are. And then you go to his telegram, and he's just like, yeah. I'm a Yahoo boy, and I've been doing it for seven years. You know? So it's it's it's very, very different.
Jason:I found it to be interesting that I mean, we've seen this now for a while and still the overwhelming use of deepfakes is for porn and, like, non consensual porn for the most part. But it took me a while to realize that this could be used that people could use it on themselves to turn themselves into someone else for the perp in this case, for the purpose of scamming.
Joseph:Yeah. Because, of course, with the nonconsensual pornography, you're not gonna be applying it to yourself. You might do something where you try to mislead somebody else, but then that just leads to what this is, which is misleading somebody else and scamming them. Right? Like, that's sort of the the the use case there.
Joseph:And, you know, I just went through this telegram channel, and there are so so many videos and images of what these scammers are doing. And, I mean, I was kinda blown away. Even in, like, early twenty forty 2024, sorry, when I was going back and then find the examples, and they were, like, very stiff, rigid photo, and then the deepfake would just be the mouth moving. And it's like, this isn't this isn't good. Now it's it's, really, really quite, incredible.
Joseph:And I guess
Jason:they're trying to use it, both to fool, like, automated KYC systems, but then also, primarily, they're doing it to fool other people at this point. Right?
Joseph:Yeah. And I think that's a good distinction because, like, imagine you're or maybe an elderly relative or not even an elderly person. I think it'd be unfair to just put it all on like, oh, old people can afford for this. No. No.
Joseph:No. Like, these can be pretty convincing. And imagine it's like on a tiny phone screen and the connection's a bit shitty and it's over FaceTime or something. Like, some little artifacts you could just attribute to, oh, I'm on the phone and the connection's a bit shit. You know what I mean?
Joseph:Like, it doesn't have to be absolutely perfect technology to fool somebody. For KYC systems, some of which are human, sort of human driven, others automated, or a lot of gonna be automated. It's gonna be different, and it's probably gonna be harder. And then when I asked format boy, this scammer from Nigeria, like, so is it good enough to bypass KYC yet? He said it's not at the moment, but he think it will be soon.
Joseph:But, of course, he is just one scammer in one fraud community. So I wanted to reach out to a couple of other people, and there's someone, David Mammon. I hope I pronounced your name correctly, and apologies if I didn't. But he just posts, like, the most insane stuff on social media because he follows a lot of these fraud groups. And, you know, he showed me a video of a real time deepfake or an animated sort of image bypassing the KYC check on Cash App, for example.
Joseph:I spoke to Reality Defender, which is a company that Emmanuel and I think Sam go to pretty often because they they they've really tapped into the world of deepfakes as well. And they said, look. We've seen a massive spike in people using this sort of technology in not just KYC stuff, but, like, interview scams as well where, like, oh, I'm I'm trying to join this company, and they'll just use a real time deepfake or similar. And they said that's coming from, Africa, Singapore, and North Korea. Africa, I mean, maybe that maybe that lines up with the scammers I spoke to or maybe not.
Joseph:North Korea, that's of course all of these remote workers for North Korea trying to get jobs at US tech companies. And then Singapore, I'm not sure of. Maybe that's a cryptocurrency thing. So, yeah, even if format boy says, oh, it's not really good for that. Clearly, there are other scammers who actually have, something as well.
Joseph:Emmanuel, I think you just sent me a few things on on chat. What are these? I don't mean to put you on the spot on the spot if I wasn't supposed to shout them out.
Sam:For later. I'm sorry.
Joseph:Oh, okay.
Emanuel:These are possibly leads. So
Joseph:Okay. Well, I'll just read them out, and I'll read out the URL so people can scoop us. Jason, anything else? Should I just leave it there?
Jason:I think this story is really wild. I think that there's probably a lot more to come in this space. It seems like an area that's, like, ripe for more scamming. So if you see anything along these lines, definitely send it to Joseph. But I also think that this is, like, one of the first times that we've written about, like, real time deepfakes, if not the first time.
Jason:So very, like, nascent scam industry and attack platform. More to come.
Joseph:Yeah. Absolutely. Alright. And with that, I'll play us out. As a reminder, four zero four media is journalist founded and supported by subscribers.
Joseph:If you do wish to subscribe to four zero four media and directly support our work, please go to 404media.co. You'll get unlimited access to our articles and an ad free version of this podcast. You'll also get to listen to the subscribers only section where we talk about a bonus story each week. This podcast is made in partnership with Kaleidoscope. Another way to support us is by leaving a five star rating and review for the podcast.
Joseph:That stuff really does help us out. Shout out to all the people who did that recently. I will get them together, and I'll reach them out at the end of future episodes. This has been four zero four Media, and we'll see you again next week.