The 404 Media Podcast (Premium Feed)

from 404 Media

A Massive Breach Reveals the Truth Behind 'Secret Desires AI'

Episode Notes

/

Transcript

We start this week with Sam's piece about a massive leak of an AI chatbot, and how it showed that people were taking ordinary women’s yearbook photos and using them to make AI porn. After the break, Jason explains how a recent change on X exposed a bunch of grifters all around the world. In the subscribers-only section, we talk about how our reporting contributed to the shut down of a warrantless surveillance program.

YouTube version: https://youtu.be/1B076Ci6at8

Timestamps:
1:23 - Intro - Please, please do our reader survey
3:57 - Story 1 - Massive Leak Shows Erotic Chatbot Users Turned Women’s Yearbook Pictures Into AI Porn
30:05 - Story 2 - America’s Polarization Has Become the World's Side Hustle
49:39 - Story 3 - Airlines Will Shut Down Program That Sold Your Flights Records to Government
Joseph:

Hello, and welcome to the four zero four Media Podcast where we bring you unparalleled access hidden worlds both online and IRL. Four zero four Media is a journalist family company and needs your support. To subscribe, go to 404media.co, as well as bonus content every single week. Subscribers also get access to additional episodes where we respond to their best comments. Gain access to that content at 404media.co.

Joseph:

I'm your host, Joseph, and with me are four zero four Media cofounders, Sam Cole

Sam:

Hello.

Joseph:

And Jason Kevlar.

Jason:

What's up?

Joseph:

As we're talking, I'm getting notifications that Jason's telling Emmanuel I don't know how to zoom in on my new camera. But maybe we'll save that for another time.

Jason:

Yeah. Okay. Joe's been setting up his camera for, like, three weeks now.

Joseph:

I do it bit by bit. As in I'm slowly improving the podcast setup.

Jason:

He's reading the instruction manual on how to turn camera on.

Joseph:

Well, we got that far.

Sam:

It looks really good.

Jason:

Oh, this is not it. This is not it.

Sam:

By Deb.

Jason:

This isn't the camera. No.

Sam:

What are you talking about?

Joseph:

The audio is new. This is just my standard webcam.

Sam:

You're just looking sharp today.

Jason:

Like He was showing me the inside of his nose moments ago.

Sam:

Oh, well, I missed that.

Joseph:

Well, that's the pivot. We're pivoting Discretion content

Sam:

is Yeah. Exactly. Instead of Joe's nose.

Joseph:

Exactly. One bit of housekeeping before we get to the stories. We have a reader slash listener survey. There isn't a fancy link like a Bitly, but I will put a link in the show notes. Jason, do you just want to explain why we're doing this and sort of why it's important?

Jason:

Yeah. We are on Ghost, which we've discussed before, but is the architecture that our website runs on. And it has no cookies, which is very good. It's very privacy focused, privacy centric, which is important to us because we care about these sorts of things. And, you know, we don't want to track our readers all over the Internet.

Jason:

That said, we don't know very much, like, really anything about the people who subscribe to us and what things you like, what things you don't like. So this is going to help us as we decide sort of what to do, which projects to prioritize going into 2026. We have questions about, like, do you like our email newsletter? Do you want it to be different? Do you wanna see us cover different things?

Jason:

Would you be interested in a print product like a magazine? Things like this. And then, I mean, the other thing and just like full transparency, to be honest, when we try to sell advertisements on this podcast and in our newsletter, the prospective advertisers are like, well, what can you tell us about our about your audience? And right now, we're like, well, literally nothing. We have no idea.

Jason:

So we're we're asking people to sort of give us, like, general information about the types of jobs that you have or, like, the types of products that you like, things like that. So, yeah, it's voluntary survey. It would be very helpful. It's anonymous, of course, and trying to just, like, get a sense of what types of people listen to this podcast and read our website and subscribe to our newsletter and and that sort of thing. Yeah.

Jason:

And then, yeah, on the ad front, it's like if you are paying subscriber of four zero four media, you will not see ads. There's no ads including on this podcast.

Joseph:

Yeah. So a link to that survey is in the show notes. It will take literally a minute, I think, and we would really, really appreciate it because it would help us grow sustainably and figure out, well, where exactly do we grow and where should we invest our time and resources. So let's get to the first story of the week. This one is written by Sam.

Joseph:

It's a really, really crazy one. The headline is massive leak shows erotic chatbot users turned women's yearbook pictures into AI porn. Sam, first of all, what is this service exactly? It is called Secret Desires AI. What is it?

Sam:

So it's really similar to a lot of the erotic role play chatbot slash image generator apps that are out there right now. There are a ton of these. You've probably seen them being advertised on Instagram or YouTube. They're pretty popular, all of them. They all have a ton of users.

Sam:

This one is really similar to, like, Character AI. Chubb.ai is another one. It's similar to the Meta chatbots, if you recall those from when I wrote about, like, the the therapy chatbots that were on Meta. Basically, you can go on there and say, I wanna customize my own chatbot, usually. I mean, it's called secret desires with, like, a heart.

Sam:

So it's the main thing that they do is, like, quote, unquote, spicy AI chatting is what the website says. The site says, build your perfect AI partner. Customize their looks and personality to bring your fantasies to life. And it's, like, it's, I would say, 99.9% erotic or, like, sexualized

Joseph:

It's a sex chapel.

Sam:

Stuff. Yeah. I mean, if you go through the the homepage, can sort it by, like, 18 plus. It's it's all, like, porn categories, basically. It's like you can choose, like, by ethnicity, ages, realistic anime, both.

Sam:

You can search by, like, kink. It's just, like, anime girls and fantasy women are probably 80% of the homepage, I would say. And then there's, like, ripped men. Like, eight packs are also part of the offering, which I do wanna I mean, I do wanna say off the bat that women use these services. I would say not maybe not just as much as men, but there's a there's a big demographic of women using erotic chatbots and erotic role play because a lot of it's, like, text based.

Sam:

And I think it's a lot of the same not the same people, but, like, the same kind of draw as, like, a romantic novel, except you're interacting with the character. So, yeah, it's not it's not like it's just men doing this, but it is on the homepage, like, mostly hypersexualized anime women.

Joseph:

And what what do you see and, again, we'll we'll get to the leak and stuff in a minute. But when you use Secret Desires AI, so it's a text interface. You're interacting with this AI persona, but there are generated images as well. Is it like what it's all part of the same product, or what is

Sam:

Yeah. Pretty much. So if you go through, like, the so you can pick from the stuff on the homepage that I just described. There's, a bodybuilder lady who's, like, basically nude. There's, like, a sassy 50 year old pharmacist who is tender and nurturing and definitely not wearing anything work appropriate.

Sam:

So you can pick from those, or you can make your own, which I think is what a lot of people do. So in the character creation part of it, you can choose gender. You can choose style, like anime realistic. You can do, like, identity. So you're doing, like, their, like, their ethnicity, basically, which there is not a lot of options to choose from there.

Sam:

You can do appearance, but it goes through all of this, and it creates the personality of the chatbot based on what you choose. It's like, choose, like, their career, their traits, their lifestyle. You can type in your own prompts, but then you get to a point where you can then generate an image of your AI partner.

Joseph:

You make it, basically.

Sam:

You're you're describing it. You're doing prompting just like you would with, like, Sora or Nana Banana or any of these other see what fusion any of those other generative AI platforms where people also use, you know, to we've talked talked a ton about Civitai. So it's all part of the same universe. And then Or

Joseph:

you upload your your own image as well. Right?

Sam:

Well, not anymore. I mean, that's Right. This is a feature that they used to have was they called it face swap. But as far as I can tell, you were uploading an image of someone, and it was generate it was generative AI. It wasn't like deepfakes where it's a it's a picture of someone that you're putting them in a scenario that already exists in real life.

Sam:

It's generating a whole new fantasy, a whole new person, like, who looks like the person that you are the target person, basically.

Joseph:

Yeah. It's I mean, I feel like, obviously, readers of four zero four and listeners of the podcast will know now, but, like, we are a million miles away from deepfakes at this point. Like, deepfakes are this quaint thing at this point where we're not talking about really face swapping. We're talking about using generative AI to make entire new scenarios and that sort of thing. Sort of like the Taylor Swift videos, which we'll might get to in a in a

Sam:

minute. Right? And people still call all that, like, colloquially deepfakes because it's a person in a scenario that they were never in, I think, is now the definition. The definition changes every year. I've given up trying to keep up on what what specifically makes a deepfake.

Sam:

But I think most people are when they say deepfake, what they mean is, like, all of the above. It's a huge umbrella term of, like, people generating wholesale new images like these fantasies that people are making on secret desires and other platforms on Civitae, things like that. And then also the face swapping that you're talking about. So that was with, like, generative adversarial networks, which is a different sort of family of the same product or the same ALP, I guess, without getting too, like, technical about it. It's that's what we're dealing with.

Joseph:

So that all exists. Secretdesires.ai offers that. You then get this tip, I think, via email. What what exactly was this tip? What did it say?

Sam:

So, I mean, there are websites out there that track open, like, containers. And by container, I mean, this is something that, like, it's it's storage it's cloud storage being used by a company or by a per even an individual, but usually, it's companies because this stuff is expensive. And then they're drawing from that container to the the inputs go in. They draw out of it. It's it's cloud storage.

Joseph:

AWS buckets, that sort of thing.

Sam:

Yeah. Exactly. And so this specific container was Microsoft Azure Blob Storage, which is such a funny name every time I come across it. That is what it is. It's a blob of data, and then the blobs are there are different blobs and different containers of, like, related data.

Sam:

So someone emailed me and was like, hey. I came across this on one of those sort of websites that tracks these things. He was like, this seems like a big deal because it's well, first of all, it was, like, 1,800,000 images. A lot of those are duplicates, but it individual images, 1,800,000. And a lot of them were pictures of people, real people, generated, like, AI characters and companions, all coming from secret desires that was there.

Sam:

The container belonged to them. So this person flagged it to me and was like, this is something that you cover. This is probably not good or at least interesting. I don't think I don't think they said it not good. I was like, yeah.

Sam:

That's super interesting. This is, like, on a Saturday or something. It's like, yeah. Let me spend, like, the afternoon dragging these images out of this container to see what's actually in here.

Joseph:

Yeah. And just to clarify, again, this is publicly exposed information like it shouldn't be, obviously, but it's basically been put in a database which is publicly accessible from normal Internet. Journalistically, we are able to see that information and use it, and we've done tons of stories about exposed buckets over the years, not just here, but way back at Vice and and Motheboard as well. So you go in and you get these images, it sounds like they're split into two sets, which is sort of the input where people are taking images of people and then you put them in, and then the output, like this AI generated nonconsensual intimate imagery or however you wish to describe it, the output. First on the input images, what was in those?

Joseph:

Is it like porn stars, celebrities, normal people? Like, what were you seeing in there?

Sam:

So yeah. And also just to your note that it is it's public. If you were responsibly storing this data, if you were responsibly using Microsoft's Blob storage, you would password protect it. You would encrypt it. You would be doing things to make sure people can't just access it from the open web, which is something secret desires obviously did not do, which was why I was able to just click on a link that took me straight to, like, a file full of this stuff.

Sam:

So what was in it? I mean so, yeah, there were there were I guess that's that's true about there are two different sorts of images. Then there's the input and then the output. That's kind of the broad categories. But then there was also, like, specifically named containers.

Sam:

So one of the containers was named face swap, And that's where a lot of the images that were kind of the the crux of the story were being held. So a lot of those were just images of from what I could tell, completely, like, random people. People would know, like, Internet and social media footprint. A lot of them are old images, like, images from clearly from, like, ten years ago, twenty years ago, even because they were, like, taken with flip phones flip phones in mirrors or with, like, a BlackBerry with the sliding keyboard, and they looked, like, deep fried, low resolution. Like Myspace picture.

Joseph:

So somebody's gone and got Myspace pictures and fed them into this AI.

Sam:

Yeah. And and that's the the name of the container is why we how we figured that out because they're called Face Swap. Face Swap is a tool that secret desires used to have where you could upload these pictures of someone, anyone, and then it would I assume this is how it works just based on what we saw on containers.

Joseph:

Face Yeah. Face a lot

Sam:

of stuff works. Yeah. People would then upload the images. It would go to the face swap container, which was a public, again, exposed. And then that's where they were stored, and then secret desires was using its generative AI algorithm to turn those into, like, whatever fantasy prompting that the user described.

Sam:

So if they if someone was like, I wanna see my this and this is something they advertise publicly as of, like, recently. If you wanna see, like, this girl that I saw at the gym, that's a fantasy that you can fulfill by used to be by uploading a picture of this girl at the gym. Maybe you know her, like, Instagram or, like, you took a picture of her without her knowing or whatever. Maybe she sent you a picture doing, you know, whatever you want. So, like, I think we can infer infer based on the the other containers, which were full of porn Yeah.

Sam:

AI generated porn that people were making porn out of these people's images. So, yeah, it was, regular people. It was also a ton of influencers, a ton of just, like, screenshots of influencers and also, like, sex workers and, like, porn influencers and adult content creators, normal regular, like, non adult you know, non not safe for work influencers, celebrities. There was one probably the one that, like, really disturbed me the most was and this is all it was all pretty disturbing just to go through, like, these people's images who had no idea clearly that they were in this. But one of them was, a local, like, a representative, like a like, a lower level politician who was she was the image was from where she was speaking at, like, some kinda, like, town council sort of event, city council thing, and someone uploaded that.

Sam:

And then you could go to the other container, which said, like what were the container names specifically? Anyway, they were I think they were called, like, live photo or, like, character creation or something. But you could go to the one and see where their faces looked a lot like the people who were in the AI porn. So I could I found one of her. It looked just like her

Joseph:

Mhmm.

Sam:

In, like, a really crazy, like, non not humanly possible type form. So, yeah, it's like and someone actually did end up telling me after this story came out that they were a tester for the face swap tool, and they said that secret desires never let people upload those images from face swap to the public feed. But you could make them. You could put them wherever you want. Like, just because you can't put them in the same ecosystem in the feed doesn't mean you can't just, like, share them in many other forms that these these people also share things too.

Sam:

So a huge ecosystem, and it doesn't just stop at, like, you can't upload it to the same website.

Joseph:

That point on public, I think, is, like, the really interesting bit about this piece, which is that, of course, we see AI generated porn of people either on tube sites or it'll happen with the the Taylor Swift videos that go viral on X or whatever, but there are people doing this by themselves, basically, using these tools. And, of course, those are supposed to be opaque systems where I'm gonna go to this website. I'm gonna use this app. I'm gonna upload imagery. I'm gonna make AI porn of this person without them knowing, and maybe they don't distribute it.

Joseph:

Maybe they don't send it to the person. They just keep it to themselves on the hard drive. And, of course, nobody gets visibility into that. I don't think a lot of these people would are gonna want, obviously, visibility into that even though they're doing this to these other people. And this provides a snapshot of, oh, holy shit.

Joseph:

This is what people are are doing. And, yes, there were a lot of responses going, are you really surprised? Like, no. Of course not. But we're journalists.

Sam:

Point. So so we

Joseph:

found an actual use case of this happening, and now there is a really significant data points to show. Just very briefly before we move on to the next bit, the headline does mention yearbook photos, and I think Emmanuel came up with the headline. I think it was very, very smart to include yearbook because people instantly go, oh, shit. That that's a really horrible recontextualization of a photo or or a piece of data. How how did you figure out there were yearbook photos?

Sam:

I mean, they were just they were literally, like, pictures you would see. Yearbook, it was someone, like, graduating. They, like, had, like, class of 2,000 I think it was, like, four or five or something. Like, they were, like, holding their little cap, and, like, it was, like, the life touch background. I don't know.

Sam:

Describe that with, like, blue speckly.

Joseph:

I think people can visualize it.

Sam:

Yeah. It's like you like you know what a yearbook. And, also, I don't know if people know that, like, most I think a lot of, especially public school yearbooks are publicly available. Right. That's all data that's out there anyway, but this is, a specific person that obviously had access to someone's, like, yearbook pictures.

Sam:

So maybe it was their classmate. Maybe was just, like, some random person. But yeah. And then there were others that were, like there was a woman, like, standing in front. She was, like, graduating from university, and she was, like, standing in front of, like, the sign for the school.

Sam:

Just like a very normal, like, wholesome picture and someone took it and turned it into porn. Yeah. I don't know.

Joseph:

That's that's not get too specific on the porn that was made, but I think can you give people a very general sense of how explicit or hardcore was this pornography?

Sam:

Yeah. Yeah. I mean so and we've written about this before, and it's something that, like, is not new in AI. But with generative AI, especially, so we're talking again about the difference between, like, old fashioned deepfakes where you're putting someone into a porn scene that already exists, which is already shitty and sucks because you're stealing someone else's content and also someone's face. But now it's like, you can create scenarios and have people do things that just don't exist in real life.

Sam:

A lot of it was, like, pretty I don't wanna say violent, but, like, it was pretty violent because it was, like, not real people. It's like no one in this is, the people are not real, and, also, the person whose face it is is not consenting to, like, this being out there of them. There were quite a few very young generations, like like generative AI of, like, very young people.

Joseph:

Or what look or what looks like a young Yeah.

Jason:

Person. Like, clearly,

Sam:

the prompt and a few of these prompts were you could see the prompts in some of the file names, but people were prompting for, like, 17 year old. Looks like this and that celebrity. And it's like, that's a minor, but, like, I'm not even talking about 17. I'm talking about, like, child.

Joseph:

It felt like a real mix because sometimes, like, with Emmanuel's Civatai coverage, that has been specifically focused on child sexual abuse imagery, CSAM. And then this, it sounds like had both potentially CSAM Yeah. But then also definitely adults in the it it was like everything. Right? And I was kinda taken aback by how explicit some of the images were.

Sam:

Right. And AI generated child sexual abuse material in many states is treated the same. So generating porn, like a a sexual abuse imagery of a child with AI doesn't mean it's not a crime in many places. And experts have talked about how this is not only not helpful to, like, actual investigations and, like, tracking down people making this stuff, but also it actually damages investigations that are trying to find real, like, abuse victims out there. So I don't know.

Sam:

It's like that it opens up a whole other can of worms about, like, the the conversation about AI and CSAM. But, yeah, a lot mean, it's as far as, like, the the hardcore porn stuff, it was just, like, like, crazy crazy shit. I don't even know. Like the stuff

Jason:

that I saw when editing this piece was extreme. Like, extreme. Almost all of it. And, like, I don't know, like gangbang stuff, like, I don't know, literally just, like, quite extreme, which, you know, is is in the piece, but this is not like a standard Nunify app where it's, like taking the clothes off of a shirt off of someone who's wearing clothes. It is like for hardcore porn primarily.

Jason:

It's what at least what what we saw.

Joseph:

Yeah. And, Jason, I was gonna bring you in now because you did edit it, and I feel like when Sam got the tip, I think you were the first to respond saying something like, well, this is insane and wild. What did you think of this while you were editing it? Like, what's new about this story?

Jason:

I mean, we we knew that this was the case, but it is another example of the fact that these tools are being used primarily to nonconsensually put women into porn and sexual situations. And it was not okay when it was deepfakes for either the porn performer whose body was being used or for the celebrities who were having their faces swapped onto it. It was not okay then, but I think it was it's, like, pretty alarming to see that quite clearly what's happening here is just, like, random people are being put into these situations where they're definitely not public figures and they're definitely not, you know, I don't know. It's it's just like it's it's pretty gross to see. And I think that we've done a lot of reporting where it's like, this is what these tools are being used for.

Jason:

But I think to see the scale at which this occurred and the explicitness and and all of that was, like, quite jarring. And then, yeah, again, it's like this is not even a big one. Like, this is, like, not I had not heard of this app before. And the fact that there's more than a million images on this app is like, well, there's many, many apps like this. And I guess it's just like I hate to be hyperbolic, but it it's one of those moments that makes you, like, lose a little bit of faith in humanity, I think.

Jason:

It's just like walking around. It's like people are doing this. Like, people are doing this. That's not good, and we're not really doing anything to to stop it.

Joseph:

And it could be you or your partner or a friend or something because apparently people are just harvesting photos that that and that that they can they're harvesting any photos they can get a hold of essentially is what it looks like. I guess just to wrap up this section, Sam, what was your takeaway? For me, was sort of the scale and it was being done to ordinary people. Like, what was your takeaway after doing this?

Sam:

I mean, not to, like, keep believing the point, but, like, it's it's just something that, like, we we know people have been doing with the technology because, first of all, it's wildly popular. We had you know, we know that erotic role play is hugely popular, and we know that generating real people is massively popular. And we know this from, like, lots of other reporting that we've done on this. And we know it from, like, listening to and, like, talking to the users of these tools. Like, they say it out loud all the time that this is what they're doing with it.

Sam:

They're in Discord talking about it and making these generations. And it's like, that's that's whatever. That's fine. You know, like, fantasy is fantasy. Porn is fantasy.

Sam:

You know? I don't, like, I don't love that use of AI, but go nuts. It probably shouldn't be illegal to make fantasy AI porn. But I think with the face swapping stuff, I think it was really, like you said, very telling that this is, like, a huge part of that was a huge part of the site. It was also very telling that they took it down and that it was not a tool on the site for very long.

Sam:

It went down in, I think, April, and a lot of people were mad. A lot of people were like, I'm canceling my subscription. People were posting on Reddit talking about how they don't wanna use secret desires anymore, and they were like, oh, it was my favorite tool, but now I'm gotta go somewhere else because they don't let me do face swap anymore. So, like, there are lot of things going on there. It's like, first of all, people really want this feature.

Sam:

It's in demand. And, also, it was in demand, and the company still took it down, which is interesting to me because usually these companies are only driven by

Joseph:

Like, what did they find?

Sam:

What did they sign? Yeah. What did they decide? Like, what the Take It Down Act went into effect earlier this year. So, like, maybe they were like, oh, shit.

Sam:

We have to, like, preemptively make sure no one's violating federal law. So I don't know. It's like the company hasn't replied and said anything, which

Joseph:

I down the buckets, though. Right? Or they closed them?

Sam:

They shut down the buckets immediately, like, in less than an hour. They, like, read my email and then shut down the buckets, which I sent them all the links because I was like, you need to shut this down because it's violating these people's privacy. Or, you

Joseph:

know, I was like And you can't cover it. Yeah. You can't cover it before it gets shut,

Sam:

basically. We can't, like, highlight something that's, like, an exposed it's this is something that's, like it's with any kind of, like, security or, like, breach reporting. It's like, want the company to close the breach before you cover it because you wanna draw more attention to it. So they shut it down, but, like, they didn't reply to me. And I would think it would be very easy to say, like, we found out that, like, it wasn't we weren't into it.

Sam:

It's not what we're about. We don't like that use of it, and we found out people were making, like but it is what they're about because that's all their ads are like, AI women can't say no. You can generate the lady at your gym. That is what they're very much about. So I'm very curious.

Sam:

It's kind of like an open question about why they took it down. That's my my takeaway is, like, what's going on there? So I don't know. I mean, maybe we'll find out down the line. Maybe someone will come forward and say, I worked in And I I told them to take it down or something.

Sam:

But Yeah. Yeah.

Joseph:

If you're listening and do know more, of course, please reach out to Sam. We'll leave that one there. When we come back after the break, we're gonna talk about one of Jason's pieces about some very funny shit that happened on X, but, you know, there's much broader implications as well. We'll be right back after this. Alright.

Joseph:

And we are back. Jason, this is when you wrote the headline is America's polarization has become the world's side hustle. Really good headline. Before we get into sort of the meat of your story, what was his change on X over the weekend? Because it I mean, it was pretty funny.

Jason:

Yeah. So X changed they added a feature where basically you can click into someone's profile and look at what country they're based in. I'm not sure exactly how it's doing that, like, whether it's IP address, whether it's, like, where it was registered, things like that. There's been actually some beef about this because I believe the Department of Homeland Security's Twitter page was showing that they were based in Israel, which Well, it went around I don't know if that's even real or not. It's, like, hard to tell what's real on

Joseph:

X and know. And but and there's also, like, Hank Green's profile, the famous YouTuber. It says his was in Japan, and, like, he's never been to Japan. Some of them look accurate. I presume it's IP, and the registration one's easy enough.

Joseph:

Like, when they made the account, where were they located? Like, that's straightforward.

Jason:

Right. So, anyways, they they added this feature, and then people started looking at different really popular mega accounts. So there's one called Ivanka News that had over a million followers, and that one was based in Nigeria. There's one called Red Pill Nurse that was from Eastern Europe, I believe. Something called Maga Nadine, which is in Morocco.

Jason:

An account called Native American Soul that was in Bangladesh. And there was many, many, many. It's like the entirety of my timeline over the weekend was about this. And I don't know. I don't even go on X very much at all anymore, but this had, like, broken containment.

Jason:

I saw it saw people in blue sky talking about it. I saw people on threads talking about it. And then so I went and looked. And there's been, like, kind of a bit of meltdowns on both sides of of the political spectrum here being, like, well, everything is just a psyop and every like, know, all these accounts are disinformation campaigns, so on and so forth.

Joseph:

And And what do they mean by that? As in just to hear their argument?

Jason:

They're just saying that, like, a lot of these accounts are like, oh, like, I'm I'm a single white mother from Oklahoma, and I hate woke like, the woke left and things like that. And that's, like, all they tweet about, and then you check click into their profile, and it's like, oh, they're from Bangladesh or whatever. And so people are saying, well, this is either foreign governments attempting to, you know, like, divide Americans or it is, like, foreign interests, like big moneyed interests trying to do this, or it's just like troll farms or bot farms, dead Internet theory. Like, all this all like, that that's sort of like where the the narrative went. And I guess before we get into it, let me just say that after twenty sixteen election on Facebook where there was, like, the you know, these Russian bot armies and Cyprus fake news organizations and things like this, Facebook launched a feature that lets you see the country of origin for different pages.

Jason:

Now, you have to click through, like, three different things to get to it. But if you do that, you will see, like, the vast majority of I mean, many many of these, like, spam accounts that are posting, like, AI slop or, like, weird stuff, say that they're based in Cambodia or Vietnam or just, other countries. And they'll say, like, I'm a MAGA mom. Like, that that is sort of what they're they're fronting as. And so this is not a new feature, like, for social media.

Jason:

This is something that has existed on social media for almost a decade, first of all. Second of all, I would argue it hasn't really done anything to, like, better our online discourse or to, like, make people understand that this is that not everyone is telling the truth on the Internet about who they are.

Joseph:

You you don't think the you don't you don't think the change on x may have some impact where even, like, I don't know, very brain dead x users might be like, oh, actually, those big viral accounts I followed are from America. I mean, may I don't know. They are

Jason:

not I think it's a good feature, and I think I'm glad that the feature exists. I'm just saying it's, like, ten years too late. They they and that YouTube has had a similar feature for a long time for channels where you can see where they're located, and it's like these things can be gamed, you know, with VPNs and with, like, selling and buying accounts and things like that. But it is good, and I think it's helpful, like, it's helpful as a journalist so that we can go in and and, like, look at where the country of origin is for some of these things, especially when some of them are claiming to be, like, official accounts or, you know, the you know, verification on Twitter has been just a thing that you buy for since Elon Musk took over. But, like, it is a good feature.

Jason:

It hasn't done that much on Facebook to, like, fix the misinformation problem. And that's largely because this stuff is decontextualized when you look at it. Like, you're just scrolling your timeline and someone has put it in your feed and, like I don't know. There's it's only a couple clicks, but how often are people doing the few clicks to sort of get to that on someone's profile? That said, it's it's a good feature, and I think it's led to interesting journalism.

Jason:

It's it's led to interesting you know, people have sort of woken up to the fact that a lot of these accounts are just trolling for engagement and trolling for clout and that sort of thing, but I don't know that it is going to, like, heal our divide. And and I guess I would also argue that it's like the divide goes further than a few, like, weird mega accounts that are promoting things. But but that said, it's like, it's a new feature that people are talking about on X. Yes. And it was, like, quite funny to see some of these accounts and, like, where they're supposedly based.

Joseph:

I mean, it caused a shit storm, basically, where it looked like a lot of these accounts or people who followed them freaked out, essentially, and just turned the the entire weekend. Again, I don't really check x that much either, but then when I heard rumblings of this, I logged in, and it's just like, wow. This is all anyone is talking about, and it's really, really it's really, really good.

Jason:

Yeah. I will say before we get into, like, my specific piece that when Facebook did some of this stuff and and when they've tried to make like, try to do a little bit of transparency around this, the the sort of free speech warriors have said, like, this type of transparency is censorship and and that sort of thing, which is really funny. But then when Elon Musk does it, it's like, oh, he has he is cracking down on bots and things like that. So to see the the dichotomy of how these sorts of features are received has been interesting, but but yeah. So But you you approach it from a different angle.

Joseph:

So that all happens over the weekend. We come back in on Monday. Right? And you then write this. And as I said, you approach it from a slightly different way because you've been covering very much the money and the ecosystem specifically on social media behind AI slop, but not just AI slop, just almost like grifters in general now.

Joseph:

It's just they're using a lot of AI right now. So what did you do to get this article going? It sounds like you started to search for certain things after you saw the X stuff.

Jason:

Yeah. So I did a series of posts about where Facebook AI slop comes from, and I did I found a lot of YouTube tutorial videos about how to make AI slop, how to bypass, you know, Facebook's pretty shitty filters and how to monetize your account and that sort of thing. And that was about a year ago, and we talked about it on the podcast. But something that I remember from that and something that I knew already was that there was a specific interest in targeting American users, like in getting in making stuff that goes viral in The United States. And the reason for that is twofold.

Jason:

One well, the reason for that is because advertisers pay higher ad rates to reach American users. They pay higher CPM, which is cost per milia or mill milli, which is cost per 1,000 views is in advertising. It's it's how sort of like the advertising industry does rates. And they pay the most to reach American users for two reasons. One, because The United States has a relatively wealthy user base, at least definitely higher than, like, Pakistan, India, Cambodia, Vietnam.

Jason:

It's it's, like, more valuable for people based there to make content that reaches American users versus people in their own countries. And the second reason, which is not in the article, but which Cory Doctorow texted me about afterwards and was like, you should have put this, and I should have to be clear, is that The US yeah. The US has, like, no privacy laws, basically, whatsoever, and so you can target Americans a lot better. You know, we don't have GDPR and things like that. So the ads are usually more effective because you're targeting people more directly, and, therefore, the platforms can charge more money for them because they're they're more effective.

Jason:

So for those two reasons, The United States is the most valuable ad market. And so a lot of the YouTube videos were like, you need to make content that is popular with Americans.

Joseph:

Mhmm.

Jason:

And so when I saw this, I was like, I wonder if they're talking about this. I wonder how this, like, ecosystem has developed since I last wrote about it. And so I started searching for, like, monetized x account in Hindi. So I just use Google Translate. And I I started in Hindi because it seems like India has the biggest ecosystem of side hustler podcast pros.

Joseph:

That's what found before with your Facebook stuff.

Jason:

That's what I found before. And, I mean, there this is happening all across the world. Like, I found some in Cambodian. I found some in Hindi and Portuguese and Vietnamese, and then a lot in English to be clear. Like, people in The United States are doing this also, like, using AI to spam social media.

Jason:

But it's particularly popular in India right now. And I found so many videos from the last few months about channel ideas for what they call a USA channel. And it just means, like, content that Americans are looking at. And so these include ideas such as make content about MMA fighting, which this one Pakistani creator said is similar to cricket in Pakistan in terms of Americans, like, fucking love this shit. They love MMA, so make MMA AI videos.

Jason:

Uh-huh. You know, make videos about American politics, make videos about cool cars, make videos about pets and dogs because Americans love dogs, things like that.

Joseph:

But it's like there's die.

Jason:

Yeah. Exactly. It's not like there's like one or two videos like this. It's like there are dozens and dozens and dozens of videos about how to make USA content, USA channel. And a lot of them are pretty basic, but then a lot of them are relatively sophisticated where they're like they'll start off the the video being like, CNN is a popular American news source.

Jason:

Go to cnn.com and copy paste their articles, put them into ChatGPT, change you know, have it write you a script, run that script through an AI voice, generate images based on that script, and then, you know, use CapCut and make a video, and and you can make you can, like, post this content and Americans will love it. There is also a lot of, like, black history content, which is kind of wild and is something that I've seen on YouTube a lot.

Joseph:

And it's popular or the people doing the grifting thinks it's popular or

Jason:

both? I mean, people are trying everything. It's like it's like people are throwing shit at the wall, and so it it a lot of the tutorial videos are like, go to Wikipedia, type in, like, black history or African American experience, copy paste that, run it through ChatGPT or Perplexity or, like, the each each per each, like, influencers, like, here is the type of AI tool that you should use, and some of them have promo codes and things like that. So it's like, it is a side hustle grift vibes going on. But that's, like, what they're promoting, and then they'll show you examples of the types of spam that has been created.

Jason:

And so, like, you know, they'll they'll show, like, successful YouTube channels. And it's very interesting because we did a story a few months ago about, like, boring history videos on YouTube. And there a lot of these a lot of these tutorials are, like, how to make boring history videos for YouTube. And so, I mean, there was there was also some stuff about, like, how to how to monetize an x account, how to monetize a Facebook account, how to monetize a TikTok account. So this is, like, the type of thing that is happening, and this is, like, why, in my opinion, we are seeing so much of this type of content that is, like, culturally off because it's not based in, like I did deep research about what is occurring in The United States.

Jason:

It's very often, like, I use Google Trends and I copy pasted something, or I, like, wrote a script in Hindi and then I translated it into English and then I used that to generate a script. And there's, like, something lost in translation. And there's also something lost because a lot of these people are running a lot of different channels, and so they're not or a lot of different social media accounts across different platforms. And so they're not necessarily, like, doing a deep dive into how can I be the most accurate? And so I think, like, my theory of the case is that, yes, there are some disinformation campaigns out there.

Jason:

There are some, like, true believers who live in other countries who are, you know, trying to either sow division or who are, like, actually interested in the MAGA movement. There are obviously tons and tons of Americans who hold really polarized views in either direction who are making a lot of content about it and are beefing online about it. But then you also have the entire rest of the world who have been incentivized by these social media companies to make content that Americans will see. And what are Americans talking about on the Internet right now? Well, they're talking about politics, and they're talking about the economy, and they're talking about tech news, and they're talking about ICE and immigration raids and all this sort of thing.

Jason:

So if Americans are talking about it in any way, there's gonna be spam content about it. And these social media companies, like, they incentivize it both through their algorithm, which incentivizes division and engagement and, you know, incentivize, frankly, like spam. A lot of you you post enough, like, you're gonna go viral. And then it also has these programs where it directly pays the people who post them. And I should have said that up top, but like

Sam:

Well, I

Joseph:

was gonna ask it now, but you just asked.

Jason:

Yeah. It's like the the, you know, X has a a monetization platform where, like, if you enroll in it and you go viral, you get direct payments from x. And YouTube has, you know, an AdSense thing where you get a fraction of ad revenue. Facebook has a creator's program where, you know, they pay out and it's not that's not directly ad revenue or, like, they wouldn't say that it is, but that's where it's coming from. But it's like they they pay out based on views.

Jason:

We actually don't know the full kind of, like, formula for how they calculate, But a big part of this YouTube influencer economy is for them to show their dashboards where they show how much money they make. And they they show very clearly, like, this video or this photo that I posted had 60% of its views come from The United States, and I made a lot more money than this other one where most of the viewers were in Eastern Europe or whatever. And so, you know, they've learned this and they're like, well, we're gonna make content that theoretically Americans will care about.

Joseph:

Yeah. And you when you have the social media platforms introduce these monetization strategies and programs. Obviously, that creates all of these perverse incentives where people are not posting for the sake of accuracy, they're not even posting for the love of the game, man. They're posting to get their money, basically.

Jason:

They're they're posting to get their money, and, I mean, here's here's one of the slides that I saw, and this slide is in English, which is interesting. Some of these videos are actually in English. A lot of them are are not, but now there's a lot of really good translation tools. But, anyways, it says YouTube earnings Pakistan versus USA. YouTube earnings in Pakistan, $1 per 1,000 views.

Jason:

And then YouTube earnings in The USA, $5 to $7 per 1,000 views. And so, you know, you're making between five and seven times as much according to this video if you're able to to reach an American audience. And so why why would you cater something to a smaller audience in in or, like, an audience that these companies are not paying as much to reach when it takes the same amount of effort to spam something that is about the NBA or Major League Baseball or something that or, you know, American politics, something that is specifically focused on American, like, American audience and viewership and politics and news.

Joseph:

Yeah. Alright. We'll leave that there. If you're listening to the free version of the podcast, I'll now play us out. But if you are a paying four zero four Media subscriber, we're gonna talk about how we contributed to the shutdown of a warrantless surveillance program.

Joseph:

You can gain access to that content at 404media.co. We'll be right back after this. Alright. And we're back in the subscribers only section. Jason, I'll ask you a question first, then maybe you could just ask me some stuff about ARC.

Joseph:

But the the headline of this one I wrote was airlines will shut down program that sold your flight record flights records? I cannot tell if I put a typo in the head. Anyway, that sold your flight records to the government. Jason, I did this recently. I requested my data to be deleted from this company called ARC that we'll explain about in a minute.

Joseph:

It's a data broker owned by the airlines that sells your data to the government. You requested your data recently under, I think, California privacy law, which you're able to do. What what did ARC tell you?

Jason:

So ARC told me that they initially said, please send us more information, like my middle names and I think my birth date and something else.

Joseph:

Physical address maybe.

Jason:

Yeah. I think physical address. It definitely wasn't Social Security number, and I wouldn't And have sent that to they said, okay. We'll search and we'll get back you. And then about a week later, they got back to me, and they said they didn't have any information stored for me.

Jason:

But I will say this was, like, two days before they shut the program down entirely. So I don't know. I I think, like, your your mileage may vary here. I have no idea whether I did something wrong. It was weird that the whole process was happening over email.

Jason:

So I don't I don't really know what happened When

Joseph:

you book a flight, do you usually do it directly with an airline?

Jason:

Yes.

Joseph:

That that will be the reason because ARC only gets data if you book it through, like, Expedia or Kayak or wherever some of these other ones are.

Jason:

I usually use Google Flights, and then I usually go to the airline's website. I sometimes book it through, like, a credit card, though, like a credit card travel portal.

Joseph:

Well, that's interesting, and I was gonna bring Sam into this as well in a second. Because if you use your credit card points, I think, at least judging by this letter from senators that that we'll get into, Like, that's basically Expedia under the hood where you use your credit card points. It's not always direct well, I mean, if you're using a Delta card, it's obviously gonna be direct Delta, but sometimes not. Sam, I'm just I we we didn't request your data or you didn't request your ARC data, but I'm just curious. When you buy a flight, do you get it from do you wanna support small independent airlines like Delta and United and American and you go direct to them, or do you go to a a a broker or a travel agent?

Sam:

I do the same thing that Jason does. I go to Google Flights because it shows me all of the airlines. And then I, from there, go to the airline website itself. I don't ever use the other, like, third party sites. I find them to be overwhelming and annoying and, like, I don't know.

Sam:

It's like, I've had I've been burned enough from sites that, like, book hotels and things like that where there you get there, and they're like, we can't we don't have that room. We don't we can't honor this. It was a third party. We can't you can't get a refund. It's like whatever it is.

Sam:

I'd rather just go straight to the thing. Like, airlines are already so annoying, and what a fucked industry. So I'm like, I'd rather go directly to who is flying me than through some other and usually the prices are the same. Like, was just similar.

Joseph:

You just Venmo the pilot. You know, you just go see

Sam:

I get there with a bunch of cash, and I see who will let me hitchhike Yeah. From the tarmac.

Joseph:

I do use travel agents, it seems, apparently. I mean, not, like, consciously or anything, but when I requested my data from ARC, they said they did find the data, and then they I don't think they deleted it because they still actually have this other business, which we'll talk about, but they said we're not gonna sell it to third parties. And I said, does that include the government? And they said, yeah. Don't worry about it.

Joseph:

Chill. We're not selling it to ICE or whatever. Jason, if you have the Google Doc in front of you, would you mind asking me these questions so I don't ask myself?

Jason:

Sorry. I could have done this.

Joseph:

No. No. It's okay. I I wanted to ask you at the start.

Jason:

Yeah. I mean, so I learned of what ARC was when you first reported on it. I had no idea it existed. But what is what is ARC?

Joseph:

I mean, I found out about it early this year as well, which was I was going through US procurement databases for ICE's purchasing as I often do, and there was a company called Airlines Reporting Corporation. And I was like, that sounds really weird, anonymous, and strange. And I think I looked at the record, and it said something like flight records sold to ICE. I do a FOIA request. A couple of days later, ICE actually released some more documents detailing it, and then the Lever reported that.

Joseph:

And after that well, actually, before that, I did a bunch of different foyers with the SEC, ATF, FBI, all of these other agencies apparently bought the data. But to answer your question, so ARC is owned by the country's major airlines, JetBlue, Delta, American, United, then also Air Canada, Lufthansa from Europe, and Air France as well, and they came together to make ARC a sort of a clearinghouse for a couple of different reasons. The first is that they act as the bridge between airlines and travel agents. So when you go to a travel agent like Expedia, there needs to be some sort of conduit to facilitate that transaction, and ARC does that. And on the side, ARC will then release a new trends report where, oh, look.

Joseph:

We found that there is more travel after the pandemic or whatever because they have access to this data. Then on the side side, this this extra extra business, they sell some of that data to the government, and that will or a lot of that data to the government, and that will include, obviously, the passenger's name, where they're flying from, where they're flying to, when, the credit card used as well. I don't know if it's the full credit card or just some of it, but government agencies are able to search by all of that. So they can enter a credit card, and it will bring up all of the flights associated with that that ARC is providing access to. You can type in someone's name, and it will bring up all of their flight history as well within, I think, thirty six months, something like that.

Joseph:

And the last thing I'll say about the data that I find really interesting is that it is curious because usually when you look at data like location data that agencies buy, that's, of course, all historical. This phone was here two days ago. This phone traveled to an abortion clinic and then traveled here, blah blah blah. The interesting thing about airline tickets is that, obviously, people typically buy them before they get on the plane, so at least a few days before. So we can show where they're going to be in the future, and it shows their intent to travel, and you don't usually get that with data.

Joseph:

And I think that's why agencies liked it or apparently liked it enough to spend tens if not hundreds of thousands of dollars buying this data.

Jason:

This is very naive of me, but well, so first of all, it's it's a data broker for the airlines. And so that's not something that I knew existed, as you as we said, until you reported on this. But there's this old adage, and I think it's really outdated by now, where it's like, if you're not paying for something, you are the product. But here, it's something that you are paying for. You're buying

Joseph:

A lot.

Jason:

You're paying a lot, like, famously, you're paying a lot for, you know, airline tickets, but you are also, in addition to that, the product. It's like it's not enough that you're just buying a ticket for this. You're you're also they're selling your information. They're selling the information about your travel habits and plans, not just, you know, for the purposes of their research or whatever, but also directly to the government. Like, that was pretty eye opening to me.

Jason:

And I I don't know I don't know why it was surprising to me because so much of our data is sold these days, but I guess I was like, oh, this is you know, you're buying something and, yes, frequent flyer programs, you know, they're sort of like lock in therefore, like, locking in, but then there's also presumably some data brokering happening there as well because worse

Joseph:

as well, like, product wise. Yeah.

Jason:

Yeah. But I I just I didn't really, like, understand that this is a thing. So you did a lot of stories about this, though. Like, you did that initial contract, but then you sort of followed this down the line. You you talked about how you can opt out of it.

Jason:

And I believe you were you were just reporting on the fact that, you know, senators were mad about this, and then you learned that this is going away.

Joseph:

Yeah. So we did a lot of reporting about the customers, and there was that one piece a while back where I got the contract between ARC and Customs and Border Protection, and I was just writing it up kind of straight. Like, oh, hey. Look. Here's the contracts between CPP and ARC.

Joseph:

And then you correctly found this detail in the copy of my article, and you were like, we have to emphasize this. And it was that the contract banned customs border protection from revealing where the data came from. So ARC, the airlines were telling the government, you cannot reveal where this data came from, which is nuts, obviously. So we do that. We do the how to opt out one.

Joseph:

We did one where ARC finally registered as a data broker in California, like, you know, just earlier this year despite selling these records for years and years and years and years and years. It's just that we didn't know about it as journalists or the public or lawmakers or anything like that, so we do that. I think there was a couple more as well. There was another one recently where one I think the secret service contract I got with ARC said there was something like 5,000,000,000 records. I actually think it's less now because in this letter I'll talk about in a second from lawmakers, ARC told them, it seems, that the figure is roundabout more 700,000,000, something like that.

Joseph:

So it's always kind of hard when you do get a contract because you hope that there's no lines in a contract, but there's always a chance there might be some marketing fluff or BS in there, basically. You'd hope not because they've they've already sold the products. Like, why are they trying to upsell them? They've already got the tens or hundreds of thousands of dollars. Anyway, that's happening.

Joseph:

So, yes, senator Ron Wyden and then, you know, a group of other lawmakers, and I think that actually eventually became bipartisan, they were particularly annoyed and worried by this data selling. They sent letters. It looks like they had briefings judging by the contents of those letters. And then senator Ron Wyden's office shared one with me, which was, you know, we're demanding they shut down ARC. I'm like, okay.

Joseph:

Sure. That's, an easy news story. But then you go through the letter, and there's a way more interesting thing in there, which is that IRS, obviously, tax agency, they use this data, and it says they did not get sort of a a legal read on it beforehand. As in they didn't look into it and be like, do we need a warrant to query this data? And I think judging by the letter, according to that, IRS was using this without a warrant, basically.

Joseph:

So they were able to go in. They're able to just search this database of hundreds of millions of records without getting a subpoena, a court order, a warrant, anything like that. As the letter from the lawmakers points out, I think quite rightly, and I didn't really think of it this way, if authorities go to Delta or United or American or whatever and they want to get details on a certain passenger's movements, they have to provide something like a subpoena or a warrant or whatever. It might vary. I'm going guess it's a subpoena.

Joseph:

I don't have the details in front of me, but they have to do something. With this, they just go and buy the data and then just search for it. So, yes, the lawmakers were saying, please shut this down, whatever. We publish the story, and then the spokesperson from ARC who have exchanged, you know, quite a few emails with over the past year comes back and says, hey. Here's our response.

Joseph:

Please update your article. We do that, and we also publish a second article because it says, we're we're closing this down. And, actually, we already told the lawmakers this, like, three days ago, but I know there was some some something got lost in in in this, you know, daisy chain of communication. But, yeah, now they're shutting it down, apparently. So now you'll never be able

Jason:

to that. Good. I mean, good job. Good go us. Good shout.

Joseph:

Good shout. Oh, fair shout, as I as I used to say as a as a teenager. Yeah. Yeah. I think it it's it's a good piece of impact.

Joseph:

Wasn't really expecting it, to be perfectly honest. As I've said a million times elsewhere, you can never really predict when your journalism is gonna generate impact. You can make your articles in such a way that increases the chance, but I don't know. They launched this program after 09:11. It may have had a use case there.

Joseph:

I mean, that that's what they say, anyway, that it's probably helped with drug trafficking and human trafficking and all this other stuff. But apparently, the heat just got too much and, like, we're not gonna do this. Okay.

Jason:

I mean, it's it's interesting to me that this is a time where a lot of companies are collaborating with the executive a lot more than they than they had been in the past and are sort of a mask off situation. And I'm not saying that the airlines are like, I'm not saying they're resisting here, but it is interesting that they decided that they would rather just, like, shut this entire thing down rather than continue to to sort of get this flack from from us, from senators, etcetera. It's it's just quite interesting that they were like, that's not worth it.

Joseph:

Yeah. They they made a they they clearly made a calculated decision where this is just not worth it for whatever reason. I don't know whether it's like, well, we don't make enough money from it. Because these contracts, again, they I think I've seen some for tens of thousands. I think the Secret Service one was like 800 k, something like that, which that's a lot of money.

Joseph:

That's nothing to an airline, obviously. Well, not nothing. It is nothing, basically, for them. So, yeah, I mean, if people wanna catch up sort of on that reporting, I guess go read that one in the show notes. But I still have a ton of FOIAs out with these agencies that bought ARC data, but I don't know if we'll cover them now because what's the point?

Joseph:

We kinda it got shut down. The last thing I'll say before I I take us out is TSA bought the data as well, and often the reply I get on social media when I post one of these articles is like, doesn't TSA already have this data? Doesn't the government already have this data? And it's like, well, to the government one. The government's really, really big, and it has all of these different agencies, and they don't always talk to each other.

Joseph:

So, obviously, if the FBI can just buy the data, they're gonna do it. The TSA one, I'm a little bit more puzzled by because presumably they'd have more ready access. But, again, maybe they just this is frictionless, you know, and they can do that sort of searching. Alright. Let's leave that there, and I'll play us out.

Joseph:

As a reminder, four zero four Media is journalist founded and supported by subscribers. If you do wish to subscribe to four zero four Media and directly support our work, please go to 404media.co. You'll get unlimited access to our articles and an ad free version of this podcast. You'll also get to listen to the subscribers only section where we talk about a bonus story each week. This podcast is made in partnership with Kaleidoscope.

Joseph:

Another way to support us is by leaving a five star rating and review for the podcast. That stuff really does help us out. I'll read out more of those soon. This has been four zero four Media. We'll see you again next week.