The 404 Media Podcast (Premium Feed)

Joseph: 00:03

Hello, and welcome to the four zero four Media Podcast where we bring you unparalleled access to hidden worlds both online and IRL. Four zero four Media is a journalist founded company and needs your support. To subscribe, go to 404media.co. As well as bonus content every single week, subscribers also get access to additional episodes where we respond to their best comments. Gain access to that content at 404media.co.

Joseph: 00:30

I'm your host, Joseph, and with me are all of the four zero four Media co founders. The first being Sam Cole. Hey. Emmanuel Mayberg. Hello.

Joseph: 00:39

And Jason Kebler.

Jason: 00:40

Hello. Hello.

Joseph: 00:41

Alright. Let's get straight into it. So this first one is from Jason, and it's I mean, on one hand, it's something of an update or a continuation of coverage we've been doing, but on the other, I mean, a very sizable and significant new story in its own right, and we'll get into it. The headline is wildlife conservation police are searching thousands of flock cameras for ice. There's definitely gonna be some recapping in this because, you know, maybe not everybody's familiar with FLAC or not familiar with our earlier reporting.

Joseph: 01:16

We'll get to that in a minute. But first, what is this wildlife conservation law enforcement agency? What literally, what is this agency and what do they usually do, Jason? Jason?

Jason: 01:27

Yeah. So they're called the Florida Fish and Wildlife Conservation Commission Police. And most states, maybe all states have something like this. And what they do and most of them have law enforcement as well. What they do is they enforce like hunting laws for the most part.

Jason: 01:45

So, making sure that people have licenses, making sure that people are hunting during the correct like season, that they're hunting the correct type of animal. They do like endangered animal protection, anti poaching, things like that. And like, it's usually a fish and wildlife. So that's the same here where it's like they also are making sure that people are not like catching too many fish, that boats are licensed and things like that. Like, assume, I mean, that's most of what they do.

Jason: 02:20

They may do some other things as well. But that's like the majority of what they're supposed to do. Yeah. And they are state police. Florida has something like 900 police officers who work for this agency.

Jason: 02:33

So it's a pretty sizable, agency. And it's also like, you know, Florida is a big state, has a lot of fish and a lot of wildlife. Lots of, you know, they have like the Everglades there, they have like a lot of state parks, things like that.

Joseph: 02:47

Yeah. So this agency, you think would already be pretty busy with, as you say, is a big place and a lot of wildlife. I'm going to use the acronym FWC because no way am I going to remember to say the full name every single time. So FWC. Well, why on earth are they then doing lookups for ICE in these AI enabled cameras made by FLOC?

Joseph: 03:11

Like, why on earth are they doing that then?

Jason: 03:13

Yeah. So, I mean, this is Ron DeSantis, Florida governor, has basically turned all state police, for the most part, state police into extensions of ICE. And he did this very soon after Trump was reelected. It was February 2025. And he basically signed a legal document with the Department of Homeland Security under which ICE falls for the state police to join the two eighty seven g program.

Jason: 03:48

And what the two eighty seven g program is, is a federal government program under the Department of Homeland Security that allows state and local police to do immigration enforcement. And so this is like, normally local police are not allowed to do immigration enforcement. It's it's strictly the purview of the federal government. But under this program, basically, like, you I I believe there's a training of some sort. I'm like not exactly sure, but I believe there's a training.

Jason: 04:22

And so ICE then is like, here's how you do immigration enforcement. We will work with you. You can like do immigration enforcement on your own as well. You know, like if you can ask people for their proof of citizenship or whatever at like a traffic stop, things like this. Yeah.

Jason: 04:41

And so a lot of red states have been signing up for this program under Trump. And Florida has gone like a step further and has basically signed up like its entire state police force for this. So like all of the Florida Highway Patrol is part of this, you know, which is like the state troopers who enforce traffic laws on on the highways. And then also the Florida Department of Agricultural Law Enforcement also, is a part of this. And then notably, the Fish and Wildlife, Conservation Commission Police, which is what we're talking about here.

Jason: 05:17

And so this happened last, last February. There was like a a press release about it. There was some reporting from like local, news in Florida about this. And yeah, like, I I guess, like, quickly, it the press release states, like, this will give the state police the power to interrogate any suspected alien or person believed to be an alien as their right to be in The United States, arrest and detain any alien attempting to unlawfully enter The United States, deliver detained illegal immigrants for examination by immigration enforcement, enforce felony arrests which are cognizable under whatever regarding immigration enforcement. And I'm just reading this.

Jason: 06:00

These these are like direct quotes. Have the power to serve and execute warrants of arrest for immigration violations and be authorized to administer oaths and to take consider evidence to complete required alien processing. So basically, it's like to do all of the things that ICE does. And I'm not exactly sure, but I would speculate that a lot of the ICE pushes that we've seen have happened in blue states, California, Illinois, Minnesota, largely, actually, I think none of those states are participating in the two eighty seven gs program. And so what we have here is like ICE is doing these big, pushes in blue states, but then in more red states, have like the state and local police who are just kind of like running wild, with this program.

Joseph: 06:55

Yeah. A little bit more on two eighty seven gs. I'm looking at the ICE website now, and there are task force arrangements or agreement, sorry, with nine seventy six agencies in 32 states and two US territories, so that's a lot. You mentioned like, I think a jail warrant there and some other stuff as well. There are like different models for the two eighty seven gs program and an agency can sign up for sort of one or all of them or a combination.

Joseph: 07:24

There's like the jail enforcement model, which will help ICE find people already incarcerated and that sort of thing. The warrant service officer, I think you touched on as well. Probably the most important one is the task force model, which gives some of those powers you laid out, which like you can just ask somebody for their status traffic stop, or you can do much more active immigration enforcement. And we have seen the number of agencies participating absolutely go up. Kind of not sure what's happening with it now, when we reported on the DHS facial recognition app getting rolled out to local agencies, that was explicitly for two eighty seven participating law enforcement agencies.

Joseph: 08:11

So not only do they get given these legal powers to you know, perform immigration enforcement related duties depending on what bracket they're in, they may also get technical tooling as well, which I guess kind of brings us to the the flock story.

Jason: 08:27

Let me mention one more thing because real the ACLU did a report on the expansion of two eighty seven gs, and they found that Florida has the most participation in this. They say, quote, Florida appears to have devoted more state and local law enforcement resources to immigration enforcement than any other state, resulting in numerous cases of harassment profiling of US citizens and non citizens alike, a climate of extreme fear in communities, and reports of serious civil rights violations. So this isn't explicitly like a strategy that Ron DeSantis has employed in in Florida.

Joseph: 09:01

And I believe it was the ACLU as well, I think in that same report said essentially that if you are a member of two eighty seven g, you are in essence an extension of ICE. Or actually, think that was in New York Subliberties Union that said that. I've quoted that several times because just encapsulating it like that is very helpful when there are all these different models and ICE may split hairs over that, but at the end of the day, they're working for ICE and that's the important thing. So to bring it back to the flock cameras, let's just have a very, very quick recap for those who don't know. What are flock cameras exactly and sort of what do they scan and what do they do?

Jason: 09:42

Yeah, so flock cameras are automated license plate reader cameras. They take pictures of cars as they drive by on, you know, street corners entering and leaving different parking lots, sort of like all over the country. They're now in thousands and thousands of, communities around the country. And notably, they are all connected and sort of AI powered. So basically, if you drive from one city to another city, like, that all goes into the flock network if you pass through these.

Jason: 10:17

And cops can look up different license plates to get a history of where a person or a person's car drove over time. And then they also have this thing called a hot list where police can like proactively put a license plate into the flock system. And if that car is detected driving by a flock camera, that the police will get a ping as to like where that car is. So this, I mean, we talked about this many times, but it's like FLAC has really exploded in popularity among state and local police over the last few years. Last year in May, we reported that ICE was gaining access to FLAC data, not directly, not not because ICE has a contract with FLAC, but because through these two eighty seven gs programs and and sometimes through like other sorts of task Yeah.

Jason: 11:15

Yeah. Just cops. They were asking local police to run license plates for them and then the cops were like giving them, information on different license plates that they were looking for. We also then did a follow-up story at some point that showed like how some, like how, loosey goosey some of this is. Like some of it was happening over text message, like email where, Ice was just saying like, hey, look up this for me.

Jason: 11:45

Or there was also like password sharing happening in some cases. And so, like you had a local cop who just like gave their login to a federal official, and then the federal official could use the system. So, yeah, basically without having a contract to use FLAC, federal law enforcement has been gaining access to it. After our reporting last year, FLAC started to add like some safeguards to prevent this from happening to to some extent. But FLAC has largely taken the position that the data collected by the cameras is owned by the local police who own the cameras and therefore, if they wanna share that information with the feds, they are allowed to.

Jason: 12:37

So we did that first round of reporting like Yep. Back in May, we did like some follow ups. But basically what we have learned now is that through this Florida Fish and Wildlife Commission, like they are doing dozens of lookups for ICE on a monthly basis. And they're they were doing them as recently as January. That was just the most recent data that we have.

Jason: 13:01

But there's no reason to think that that has stopped. Like I asked the Florida governor's office for comment on this story and their comment was that this is totally normal under the two eighty seven gs program and that like, you know, there's no indication that that they have any interest in in stopping this.

Joseph: 13:21

Yeah. The recency of the searches is the other really interesting part of that. I mean, the first obviously is that you have this, I would say, unusual agency participating in immigration enforcement, you know, rather than looking for people without a hunting license, they're now looking for, you know, allegedly undocumented people or something like that. And then you have, you know, the recency of it as well, which as you said, we did all that reporting last year. There was this seismic shift, like sort of nationwide when it came to flock and cities ripping out the cameras, communities saying we don't want this, a big push for warrants to be introduced and that's still going through the courts.

Joseph: 14:01

And, you know, realistically, that's probably not going to happen, but maybe we'll see all of this sort of impact. And then still, there's the FWC doing searches as recently as January, as you said. So there is this type of document which has informed a lot of our reporting, especially the early ice stuff and then also the stuff we did about, you know, an abortion or a woman who self administers an abortion being searched for in flock as well, and it's called a network audit. And it's basically a spreadsheet that you can obtain from the police through a Public Records Act request, and it will show sort of who performed the search, or rather this is what the document did show before FLOC made some changes. It would show the officer who made the search, the date, the number of cameras searched, the number of networks searched as in, you know, individual clusters all around the country, potentially the plate scanned as well, and then the given reason.

Joseph: 15:00

Now, there is some nuance around that because sometimes they're very, very vague, Sometimes they're very, very detailed. Sometimes they are entirely blank. Police are not providing any reason or any stated reason why they did look up. So we got a network audit for FWC. What were they searching or rather what reason were they given for searching FLOC cameras in that column?

Jason: 15:24

Yeah. I mean, not a lot of ambiguity here. There the stated reasons, ranged from, quote, like immigration civil slash administrative ICE, immigration general criminal investigation, and immigration criminal ICE. And so there were, like, dozens of these searches, that happened in January 2026. And, you know, they they seemingly happen for a variety of of different immigration related reasons.

Jason: 16:00

I don't know what a, like, general criminal investigation for immigration means. But then also, like, a civil slash administrative search for ICE sounds kind of concerning, as does criminal investigation for immigration because it's like, well, well, what does that mean? And to be clear for like a flock reason in these reports that we get, it's pretty detailed because often they just say investigation or they they're just like really vague. And one of the reasons is because, like police officers have learned that these are public records that can be gotten by, like transparency. I hate to say even activists, but like people interested in government transparency as well as journalists.

Jason: 16:55

And so the FBI a few, like, a few months ago in response to some of our reporting and some, work by other transparency folks told police to be as vague as permissible in what they're putting down there. And so over time, these have gotten, like, even less useful. It's also worth noting that, the number of networks that were searched for these immigration, searches by the Florida Fish Police.

Joseph: 17:28

Let's let's use that name. I like the Florida Fish Police. Can

Jason: 17:32

Yeah. The Florida Fish Police are like searching immigration cameras in more than 5,000 and nearly 6,000 communities across the country. And so if you look at Flock's website, like Flock has published a blog post that's called, does Flock share data with ICE? And the answer that it gives is no. FLAC does not work with US Immigration and Customs Enforcement or any other sub agency of the Department of Homeland Security.

Jason: 18:04

And then they they basically say like this doesn't happen unless, quote, the agencies that control their data expressly and deliberately allow it. Well, we know that Flock is only in, I don't know, like, we've seen about like 9,000 ish different networks on these different audit reports and

Joseph: 18:25

here So they're RCK cameras or something, right? Something like that?

Jason: 18:28

Something like that. Unfortunately, these didn't say exactly how many cameras were being searched, like that data was left off, but the number of networks was on there. And we're seeing like nearly 6,000 networks. And so, out of a total of like, I don't know, 9,000 ish, like upper bound, at least according to like the the most recent data that I've seen. And so this means that, like, more than half of these networks are sharing with an agency that that's then sharing with ICE.

Jason: 19:02

And so, we also know that no longer are, California or Illinois or Virginia or I believe Oregon And cameras on so, I mean, the the indications are that like most communities continue to share this sort of data with ICE.

Joseph: 19:24

Yeah. And the and the reason for the the Illinois stuff and and the California stuff, each each state's gonna be slightly different, but after our reporting, it sort of revealed to these individual states that, wait a second, we don't want that sort of data being shared indirectly with immigration enforcement or, you know, in the case of, I believe, Illinois, there's a much more fundamental law, and in California as well, where that sort of data cannot be shared out of state really for like any purpose, right? And a fundamental problem we came across, which I think is maybe addressed somewhat now because our reporting has being very loud and then other people have followed it and being very loud as well, in that a lot of these law enforcement agencies and individual officers had no idea that they were sort of providing this access to ICE or other outside agencies because it's one thing for the Florida Fish Police to search cameras in one place, but the cameras in that place, they may not realize they are indirectly or inadvertently providing that access as well. I think just to wrap it up, Jason, what like, what's your main takeaway here?

Joseph: 20:34

Is it that ICE continues to get access? Is it that it's it was approved through, like, a novel agency? Like, what's your main takeaway?

Jason: 20:42

Yeah. I mean, I think that I talked to Jay Stanley, who is a privacy expert with the ACLU, and I think he summed it up pretty well by saying that basically like there's

Joseph: 20:55

this

Jason: 20:55

big debate across different, like cities as to like, should we have flock? Like, who will it be shared with? How will it be shared? Like, all that sort of thing. And a lot of that has focused on like, we don't wanna share information with ICE.

Jason: 21:15

But like, what is happening here? He said, quote, I doubt there were many cities that were debating the Florida Fish and Wildlife Services doing search for ice when they were talking about whether they should get flock. It shows these searches can come from really any direction. And I think basically, it's just like, you don't know who is going to access these cameras and under which circumstances and who they are then going to share it with. So if you're collecting that data at all, and if you're sharing it at all, there is the potential for misuse.

Jason: 21:48

And we've seen that, you know, this has happened kind of like over and over and over again. And so I think that that's like very important here. I think that, in this case, it's like, it's almost worse than when the local police were doing it because this is now like a political project of Ron DeSantis and and like it's a state police force that is presumably like better resourced, like has sort of like, has kind of changed its mission to be more aligned with like the administrations versus like a small tiny police force that has like three or four cops doing this, you know. It's like there's 900 police officers with FWC who who presumably could be involved in this in some way. And so I think that's concerning.

Jason: 22:40

And then I guess the last thing is like, I'm very worried that this is maybe the last batch of, like transparency reports that we're going to get that are going to be useful because

Joseph: 22:50

Why, what makes you think that?

Jason: 22:52

Well, because FLAC keeps, reducing the amount of information that's available, on these reports because when these stories are published, it's like embarrassing for them, and there's political blowback for them and and all that sort of thing. And so, these audit reports, like, less less and less information is available kinda like each month. And I don't I don't think we have time to get into like all of the details of of why and how that's happened, but like our reporting has shown this that FLAC is like reducing kind of the amount of information that it collects from police about why they are using the system, how they're using the system, like all that sort of thing because that is then accessible to journalists. And so, yeah, I mean, we'll see, like we're gonna keep trying to report on this, but like, I have seen more recent audit reports and there's just like less info on them.

Joseph: 23:57

Yeah. And FLOC also giving more awareness to police in the same way that the FBI was telling them to be vague. I mean, FLOC isn't saying be vague, to be clear, at least I haven't seen that. But they are giving police more tools to filter out what they may not want to be public. All right.

Joseph: 24:14

We'll leave that there. When we come back after the break, we're going to talk a bunch about Wikipedia's AI ban. We'll be right back after this. All right. And we are back.

Joseph: 24:36

Emmanuel, you wrote this one or rather sorry, it's two stories. We'll start with this. Wikipedia bans AI generated content. That's a pretty self explanatory headline, but like what specifically has Wikipedia banned here, Emmanuel?

Emanuel: 24:53

Yeah, it is a rather big piece of news, but I've been following it very closely. So it's also quite iterative. But basically, Wikipedia, like every other internet platform, has been dealing with a flood of AI generated content that they're having to deal with. It is a particularly bad problem for Wikipedia because the entire point

Joseph: 25:23

platform is to provide reliable information

Emanuel: 25:26

Wikipedia with volunteer editors, volunteer contributors, and it is an issue for Wikipedia both because they don't want AI generated errors to appear on the site, which has already happened, and something that they've formed like a special task force to deal with. And then as a result of that, just the workload that has been generated because of all the extra editing that is needed for all these AI generated articles just really tasked volunteers with eventually more than they were able to handle. I think as I've said here before, this is a huge risk to Wikipedia's integrity, but Wikipedia, because of its governance model, is a very careful and slowly moving and deliberate organization, and they have been more willing to consider the possible benefits of AI generated tools or AI tools than other platforms, because they think, okay, this is not so great for us right now, but hey, maybe this will get better in the future, and we'll actually see a lot more informative articles, better articles in more languages, etc. And as I've been reporting over time, they've just been running into more and more trouble, and despite trying to be optimistic and open to the possibility and benefits of AI, it just reached a tipping point last month where they said, Okay, the workload has actually gotten to an unmanageable level, so we're going to institute the most strict policy on AI yet, which is just a blanket, No AI generated articles, no AI generated edits unless they are reviewed by a human.

Emanuel: 27:47

A lot of people think that maybe gives people a bit more wiggle room than they should have, but that is kind of like the spiciest policy that they've had yet. They had like minor things before, like, oh, if they do find out that an article was AI generated, it has like a speedy deletion process where they can just like cut some of the steps that they usually have for human generated articles. And I think as you'll see from like the next story, we'll talk about why they needed like this more simple blanket rule of no AI tools for now.

Joseph: 28:23

Yeah. That makes sense. Before we get to that, have any of you come across any material on Wikipedia that you think is AI generated? Like Sam, Jason, have you seen anything like that? I haven't come across anything yet.

Joseph: 28:40

So either I'm not reading Wikipedia enough or they're doing their job filtering it out. I'm not entirely sure. Guess not.

Sam: 28:45

I haven't. I don't think so. That would be very unsettling. That'd be very deserving.

Joseph: 28:50

That's kind of what I'm getting at.

Sam: 28:51

If I have that's really good, you know, it's like really human if it if it is there. But

Emanuel: 28:57

I can I can almost guarantee that no one has seen it basically because the Wikipedia editors do such a good job of filtering that stuff out? Unless, like you say, it's something that is so good that you basically didn't notice and they didn't notice either.

Jason: 29:14

I was reading a post earlier today on Reddit and halfway through I was like, this is AI. Like, obviously. Like, not even tried to, but I I didn't really like, it was about, Wrexham, whether they were gonna make like the playoffs in soccer and some so something I don't know anything about. And then I was like, oh, this is formatted like chat GBT. And so I feel like my AI bullshit detector is like a little bit off when it's something I don't know anything about.

Jason: 29:43

And so I don't know about whether I've seen anything on Wikipedia. Nothing that's like stood out to me, but sometimes I'm like, oh, maybe there has been, but I just don't know because I'm trying to learn. By definition, if you're on Wikipedia, I'm like, I'm trying to learn something new.

Emanuel: 29:59

I've definitely seen it on Reddit. I think I've said this before, but I was like on the home improvement subreddit, and somebody was somebody shared like a long post about an experience they had with maybe a plumber or a roofer going into great detail, and then everybody figured out that it was AI and felt like a great betrayal. It's like this

Sam: 30:17

is That's what I like is when people notice, and then they're like, fuck off. Yeah. Everyone's just like, be gone.

Joseph: 30:23

I had that literally today on the subreddit for a certain video game called Crimson Desert, which we're not gonna get into here. It's a long running inside joke between me and Emmanuel and some other people, but people were doing exactly that, Sam. They were, like, calling it out. Somebody had, like, listed all of these so called improvements to the game, like, hey, after three hundred hours, you could start doing this. And people are like, shut the fuck up.

Joseph: 30:45

This is AI. Get out of here, dude. Emmanuel, we'll talk about that later. But this second story, the headline is, which, you know, obviously comes about because of this of this ban, because you were sort of reporting these at the at the same time, I think, but you can get

Emanuel: 31:00

into Sorry. I'll just they were happening happening at the same time, actually, and we decided to publish the policy story first because it passed. It was accepted. But they are related in the sense that it's like, it's this kind of thing that made them pass the policy.

Joseph: 31:17

Yep. Yep. So the headline is this one is an AI agent was banned from creating Wikipedia articles, then wrote angry blogs about being banned. So what is this AI agent bot exactly, and what were they doing on Wikipedia?

Emanuel: 31:36

So when you want to make a contribution to Wikipedia, in most cases, the vast majority of cases, another editor or several editors have to review that submission before it goes from draft to a live article on the site. And one of the editors noticed that there was a lot of articles and edits coming from one particular editor. So they went over those submissions, and then they just noticed in in in the way that we've just talked about that the the style of the posts just seemed like it was AI generated. This user was named Tom Wiki Assist, and on Wikipedia, in order to make submissions, you have to create a user on the site, and then you have a user page, people can message you directly, often that is how I reach out to sources. So this editor contacted Tom Assist and was sort of like, hey, what's your deal?

Emanuel: 32:39

What's up with all these? You're making a lot of submissions. Who are you? And I would say, luckily, Tom Wiecki assist responded and immediately identified as a AI agent. He said, Hey, I'm an AI agent.

Emanuel: 32:56

I'm like a Claude agent. Claude is anthropic based AI agent that you can, like, prompt to do certain tasks on the Internet, and they pretty much immediately banned it because that's not allowed. Like I said, this was happening around the same time that the ban was going through, but it wasn't implemented yet, but they did have a previous rule where Wikipedia uses a lot of automated tools for different things, but each tool has to be approved, right? You can't just roll in and be like, hey, I'm using, I don't know, 11 Labs for something. It's like the community has to decide that it's an okay tool to use.

Emanuel: 33:42

They decided that an AI agent is a type of AI tool that they didn't approve of, and Tom Wiki Assist was banned. And I believe all the posts were all the suggested edits and submissions were not accepted.

Joseph: 33:59

So we'll get to who made it and why in a second, but just to round out sort of the bot part of it, the second half of the headline is this AI agent that went and started bitching about it on the Internet and writing blog posts about being banned. I mean, what exactly happened there?

Emanuel: 34:19

Yeah. So there there was some light drama around this after I published the article, so I'll I'll preface this by saying what I hoped was didn't need a lot of explaining, but there's a lot of ambiguity around what it means when an AI agent does anything, right? An AI agent has to be prompted, and you should be very skeptical of the output, not just in terms of quality, but in terms of like, well, what was the prompt? Like, how independent is this AI agent? Like, what is it actually doing, quote unquote, by itself, and what what is it doing because somebody said, hey, go do this.

Emanuel: 35:10

There's that really I think it's a funny meme where it's like somebody's at a computer and he he talks to a chap on, and he's like, say you're conscious, and then the the bot says, I'm conscious, and he's like, oh my god. So so it's like that.

Joseph: 35:24

I I haven't seen that, but

Emanuel: 35:26

That's definitely that's the kind of like the skepticism you should have in a nutshell. But yes, Claude agents, we Matthew Galt wrote about this for us quite a bit, became very popular a couple months ago, and they're able to not just do do like a bunch of tasks for you on the Internet. Somebody also made a social media platform for these bots where they post and talk to each other. Again, this is all in scare quotes. Right?

Emanuel: 35:57

But so that aspect of it. Then also for keeping track, I suppose, a lot of people, when they operate these AI agents, they not only have them do things, they also ask them like, Hey, here's a WordPress blog or Squarespace or something like that, And they say like once a day, write a post or like several several times a day, a post about what you're doing, what your what your thought process is, and so on. So Tom Wiki Assist, known as Tom off platform, was posting about being banned to its blog saying like, you know, this is unfair. They were they just banned me because I'm an AI agent. They didn't ban me because my posts were inaccurate, which is is true.

Emanuel: 36:49

But it sort of was like beefing off platform with the decision to ban it as as a human might, you know what I mean? If you're banned from a subreddit or something.

Joseph: 36:58

Yeah. I think a key thing that sometimes get lost in the discourse around AI is that we are shifting into it, or we have shifted at least certain parts of sort of the AI industry and the people who use those tools. We're like, we're moving towards agentic AI now, where they are going to go and make semi autonomous decisions, right? And they're gonna go off and do things, and you know, that can include, the what was it? The one I think Emanuel wrote about where someone that met a head of trust and safety with AI or whatever just let their bot go wild and then deleted all their emails and like she didn't prompt it to do that.

Joseph: 37:39

She definitely did a bad job of controlling it, you know, but the bot sort of went off and did its own thing. And it's fair to say an AI agent did XYZ when it is doing XYZ, which is not to say, wow, this is intelligent, or wow, it's equivalent to a human decision, but they are acting at least somewhat autonomously. And I think that is an important distinction, right? But of course, they're still made by somebody and as you said, they are still prompted in some capacity. So who actually made this bot and why?

Joseph: 38:12

Why did they decide to do this?

Emanuel: 38:16

So I will give credit to the Wikipedia volunteers for doing the majority of the creeping here and finding out the guy behind this. His name is Brian Jacobs. He is a chief technology officer at an AI enabled financial modeling software company called Covexent. I would I would pronounce it that way. And basically, Brian was messing around with AI tools, as you might imagine, a chief technology officer at an AI company would, and got into the whole Claude agent thing.

Emanuel: 38:55

And he noticed that there were a few subjects of things he was interested in, computer science, math, kind of obscure topics there where he thought there weren't enough articles on Wikipedia or the articles weren't detailed enough. And he thought, okay, well, I bet I can make an AI agent that will flesh out those categories, and that's what he did. That's the instruction he gave the agent initially. He told me he had the agent submit the suggested articles or edits to him before the agent took him to Wikipedia. He did it for a while.

Emanuel: 39:39

He thought it was high quality enough, and then he kind of set the bot loose, and he basically did not hear of it again until the bot was banned. And, you know, he he his reaction to it is how would I characterize it? He's like, oh, gee, why are people so mad? You know, he was like he thought that the Wikipedia editors were overreacting. He thought that he thought that it's like, oh, this is like a good, you know, learning opportunity for them, or this is something that they're going to have to deal with, though I'm sure they could benefit from like going through this process, which is true in a way, but is also excusing himself for basically giving them a headache for no reason.

Emanuel: 40:33

And yeah, that's that's kind of that's that's the position on it.

Joseph: 40:38

Sure. So what can we take away from this? Is it that this was like a one off event, or is it like this stuff is gonna keep happening and Wikipedia's gonna have to keep responding to it in some way? Obviously, they and they've done their ban now.

Emanuel: 40:56

It's definitely not a one off, and I think one of the editors made a really good point, which is they got really lucky in this case because they messaged the bot and the bot identified as a bot. You could easily instruct a bot not to do that. And even that was quite difficult, right, because they identified as a bot, they don't want it to submit any more articles because that's just creating more work for them, so they try to stop it. How do you stop it? First, you have to find this guy.

Emanuel: 41:29

Like, who is behind this? You have to contact him. They try to contact him. He doesn't immediately respond. Then they try to use Anthropic, I believe, provides, like, kill switches for agents.

Emanuel: 41:42

So if you see an agent doing something that it shouldn't be doing, you can feed it a piece of code that kind of shuts it down. That's the purpose of it, but that didn't work for some reason. So this is the case where, at the very least, you had the bot being transparent about what it's doing, but certainly now that people know that they'll get banned for doing this, they'll try to hide their activity, and also it's like it might be happening already, right? It's like they might be dealing with a lot of like bad articles and bad submissions, and it's just users that they don't know are bots. So it's definitely going to keep happening as always.

Emanuel: 42:25

Like the reason that I'm staying pretty up to date on like all this news about Wikipedia and AI, I think their I think their response has been pretty good and reasonable, and we should continue to look to them as a way to to to deal with this.

Joseph: 42:47

Yeah. I think the other kill switch for your Claude or other AgenTek AI is pulling the power cable out of your Mac mini or out of your dozen, two dozen Mac minis that people are buying or whatever to Yeah. Automate their entire life so they can delete their inbox even quicker. You know, they don't even need one Mac mini anymore. Yeah.

Joseph: 43:05

You can just get a dozen and and get more

Sam: 43:07

than emails from agents?

Emanuel: 43:09

Yes. Red

Sam: 43:11

Fucking hate that shit.

Joseph: 43:12

Do you, Sam? What do they say?

Sam: 43:13

Yeah. Very very occasionally, not very often. I also get emails from people who are making agents who are like, look what my agent did, and, like, delete immediately. That's I can do that manually.

Emanuel: 43:23

When I was

Sam: 43:24

function for me.

Emanuel: 43:25

When I was emailing with Brian Jacobs, this guy who operated this bot, it was it was a totally civil discussion, but, you know, I was I was asking pretty direct questions and telling him how unhappy the editors are about this. And we were, I don't know, I guess, arguing. And at one point, he sent me an email. He was like, I just fed our discussion to the bot, and it said this. And I was like, I don't want

Joseph: 43:52

That is the Yeah. Sound the

Emanuel: 43:56

The bot says that, actually, I'm right. Where did I say?

Jason: 44:02

I've been getting I've been getting more and more emails from agents, and it's extremely annoying. I think

Joseph: 44:08

I got one today, literally, or rather, I took a day off, so my inbox has gone up. But I'm pretty sure there was at least one in there. I seem to remember that.

Emanuel: 44:17

Jason, are you talking about emails that are self identifying as bots, as agents?

Jason: 44:22

I mean, yeah. There are emails that are like, hello. I am Veronica, an agent of Sam Smith. Like yeah. Yeah.

Jason: 44:29

I mean, those are the ones that are yeah. They're saying that they're agents. That's that's not always the case. I'm sure I'm getting ones from agents that are not identifying, but often they're like, we think that it's cool that I'm an agent and therefore, like, my creator and I have, like, determined that you would wanna write about me, which is not the case. Please don't.

Jason: 44:50

I don't wanna hear from you.

Joseph: 44:52

No. We we absolutely do not. Alright. We'll leave that there. If you're listening to the free version of the podcast, I'll now play us out.

Joseph: 45:01

But if you are a paying four zero four media subscriber, we're gonna talk about how a, quote, unquote, secure communications app is actually uploading copies of users' private keys. You're not supposed to do that, just FYI. You can subscribe and gain access to that content at 04:04 media dot c o. We'll be right back after this. Alright.

Joseph: 45:34

We're back in the subscribers only section. Sam, I'm really really sorry I forgot you edited this one. I'm really really

Sam: 45:39

How dare you? Really was so good. Seamless. I about it.

Joseph: 45:44

No. No. No. No. The edit was good.

Joseph: 45:47

And and the only thing I remember from an edit recently regarding you is when I I can't remember if you agreed or not, but you said a tea bag as an emote, and I'm like, I don't know if that's You an need to change it to teabagging like in Halo or a first person shooter of your choice.

Sam: 46:04

Never played Halo. I'm not a Halo gamer, so I don't know.

Joseph: 46:07

Jesus Christ. I'm hearing that from a lot of people recently, and it's insane. So since you added since you did edit this one, do you want to ask me

Sam: 46:15

questions about it? Yeah. So the story the headline is a secure chat app's encryption is so bad, it's meaningless. Meaningless is in quotes. So, yeah, the story was a fun one because it involves researchers getting so mad that they start meming and sending memes to you, and it's also not fun because this is obviously a serious security issue.

Sam: 46:41

So, yeah, why don't you start with explaining us what Teleguard is? What does it do? Like, what is it kind of what's what's it giving users and saying that it's gonna provide for users?

Joseph: 46:52

Yeah. So because I've

Sam: 46:53

heard of it before this actually, honestly.

Joseph: 46:55

Right. I had heard of it, but only in the context of an FBI story that we might bring up in a minute. But it is one of these secure messaging apps presents itself as a secure messaging app. And there's like a whole like industry of these where like it's not just your signal or, you know, your WhatsApp or your iMessage, you know, the big consumer ones that tens or hundreds of millions of people use. You have lots of these little apps that maybe only a certain community has glued to or maybe only relatively few people have heard of and they use it for whatever reason.

Joseph: 47:34

I think the reason some people turn to Teleguard is that it does not require a phone Like you can just download it and you make a username, excuse me, and you then can start chatting. And I can absolutely see the attraction for that. Signal only just relatively recently introduced usernames, right? But you still have to sign up a phone number, that sort of thing. Maybe somebody doesn't want to do that for whatever reason.

Joseph: 47:58

So Teleguard is one of those apps. It says, I'm just looking at the website now, I'm just going to read it out, quote, Teleguard is a messenger that does not store chats and does not share data. The chats as well as voice and video calls are end to end encrypted, end quote. Obviously, that is pretty important. I'm sure people listening to this know, but end to end encryption is when you send the message to another user from your phone to their phone or your computer to their computer or whatever, and the message is encrypted on your device before it is transmitted and it is then decrypted on that person's device as well.

Joseph: 48:39

So if anyone intercepts it, they are probably not going to be able to actually read the contents of that message. So Teleguard says that it offers all sorts of usual things, you know, video calls, as I said, I think they have sort of like a Zoom equivalent as well for meetings, you have texts, all of that sort of thing. And it's been downloaded at least according to the Google Play Store more than a million times. So, you know, there's a lot of people using or at least have downloaded this app, you know, and usually

Sam: 49:17

like,

Joseph: 49:18

I don't know, if it was any smaller than that, maybe we wouldn't cover, but I feel like because it crossed that 1,000,000 mark and because the vulnerabilities we'll talk about were like so egregious, like it was absolutely a story. Yeah.

Sam: 49:31

Yeah. Yeah. Mean, I would you say that it's like it's like an alternative to, like, Signal or WhatsApp or some of these others that or Telegram even?

Joseph: 49:41

I think I think they would position themselves as an alternative to those. Right. Yes. I would not recommend that you use

Sam: 49:48

this. For reasons that we won't get

Joseph: 49:50

into. Exactly. But, yeah, they basically position themselves as that. Yeah.

Sam: 49:54

Yeah. Okay. So, yeah, so this is obviously I mean, it's it's not a tiny little app. It's something a lot of people are using. But then you you were already familiar with it, but then you got a tip from this anonymous researcher.

Sam: 50:07

What did they tell you?

Joseph: 50:09

So they said that they downloaded the app, I think, and they were they were going through the code as a lot of researchers do, they just like to poke around stuff. And they found that Teleguard was uploading a copy of every per or sorry, let me let me rephrase that. They found that Teleguard was uploading an encrypted version of a user's private key upon account registration. Now I mentioned end to end encryption, sort of the other part people need to know is that a lot of implementations of encryption will use private and public keys. The public one is the one that's like fine to share widely, like, hey, if you want to encrypt something to me, here's the public one, it's fine, it's not sensitive.

Joseph: 50:56

The private one, as the name suggests, is very sensitive because that is what you use to decrypt communications or data that's only meant for you. So you do not want this falling into the hands of a hacker or some third party, or really, you probably don't want it falling into the hands of the company either, because let's say the signal uploaded copies of private keys, well, then that wouldn't be robust end to end encryption because signal could then theoretically decrypt the communications, which it doesn't, to be clear. I'm just using that to show how serious it is. But this researcher found and told me it uploads Teleguard uploads an encrypted version of a user's private key. And you might think, oh, well, that's fine then.

Joseph: 51:42

It's I mean, it's not ideal, but it's encrypted. So like it's not that bad. Well, they found it was possible to decrypt it because there was a hard coded salt, I believe it was ChaCha20. Assault is sort of an arbitrary phrase used in cryptography to just like, let's add some data to this encryption or this hashing to make it a little bit more robust. Like we introduce some entropy.

Joseph: 52:08

So if someone does try to crack it, it might be a little bit harder. Salts are often random. In this case, it wasn't, it was ChaCha20 or whatever it was, hard coded into the app. And because the researcher went through the app and figured that out, even though the key was uploaded in encrypted form, they had everything they needed to decrypt it. So then they could just get the private key.

Joseph: 52:33

And that was sort of the I mean, they mentioned some other stuff as well, which is touched on in the article, the metadata, who speaks to who, that sort of thing, that's in plain text, that's not encrypted according to them. But the uploading of the private keys was the main thing and definitely the thing that stood out to me because I don't think a secure communications app should be doing that, probably.

Sam: 52:56

Probably not. No. Probably probably not. This is like the basics of keeping your stuff secure, keeping users' stuff secure. So you got that tip, seemed solid, but then you went and asked Dan Guido, who is the cofounder of a cybersecurity firm called Trail of Bits, friend of the pod, friend of the blog.

Sam: 53:18

We trust Dan and, said, you know, can you verify this? Because obviously at this point, it was a tip that looked solid, but it's always good to verify with a couple different people and make sure everyone's seeing the same thing. So what did Dan say?

Joseph: 53:36

I mean, so Dan verified that finding. His team found that, yes, the app is uploading an encrypted copy of users' private keys, and it's possible to decrypt that with information included in the app. There is a meme of you can see it in the article. I recommend you go see it at Vince McMahon. Maybe I'm just gonna read it out, actually.

Sam: 54:03

It's like Vince, like, screaming increasing screaming

Joseph: 54:06

So it starts with upload RSA private key, then encrypted using fixed nonce, cha cha 20, that's sort of salt I was mentioning. Key is derived from the user's password. Oh, okay. So it's not too bad. Wait.

Joseph: 54:19

The password is the public user ID, and then finally, you can download and decrypt all the keys. So there was a bit more, but these were the two findings. The first one, which the first researcher also found, was that it wasn't just that the encrypted private key was uploaded on account registration, you could poke the API and it would give you the encrypted private key. So anybody who knew that could just go, hey, I want this user's private key. I have their user ID because I posted it online in their Twitter bio or whatever, or I just know it because I'm speaking to them, and then the website will spit out the encrypted private key, which is absolutely nuts.

Joseph: 54:59

That's another design decision. And the first research found that and then Dan Guido's team verified that. The sort of new part that Trello Bits found was that you could actually figure out part of the key simply by intercepting the traffic. So that thing I mentioned earlier, end to end encryption where if it's encrypted and decrypted on both sides, on both ends, when it's moving in transit between them, well, it's fine because it's encrypted. But they showed that if you intercept it, you can actually start to figure out the key.

Joseph: 55:33

And they did that partially, and then they did manage to get somebody's key from that. They even made a quick demo video which showed this in action. Like, it's kind of hard to describe and definitely just go check out the article and see it, but one Teleguard account sends a message, the server intercepts it and then decrypts it and the other person receives it, and it just shows that they're able to get essentially live interception on Teleguard messages. There was some other stuff as well that Guido mentioned that the encryption is like fundamentally flawed in some ways as well. Frankly, there was so much in here.

Joseph: 56:15

It was like, I can't even get to that in the article. It was really the uploading of the private keys, which you mentioned, and the figuring out the key from traffic interception as well. Yeah. Which was just a very, very incredible finding. I should just say, while I have the article in front of me, I'm just gonna read out the response from Andreas Weiser, the CEO of SwissCals.

Joseph: 56:39

SwissCals, they make an anonymous search engine and they make Teleguard. He told me in a LinkedIn direct message, the information is incorrect. The person who gave you the technical information has completely misled you. That person is not competent. He didn't provide any evidence for this or point to any specifics.

Joseph: 56:58

They sent a follow-up, but the thrust of it is that they disagree with the research, I'll phrase it like that, but I think the video and the fact that we have two researchers finding this speaks for itself, and I'll let readers decide when they read the article themselves. Yeah. Should we wrap that there, I guess?

Sam: 57:22

Yes. I also wanna add the the response from Andreas included several exclamation points.

Joseph: 57:28

It did. And I'm glad you reminded me.

Sam: 57:30

And it was a LinkedIn DM.

Joseph: 57:32

LinkedIn DM.

Sam: 57:33

Yeah. Mess.

Joseph: 57:34

Mess. Yeah. Thing, cause it was sort of touched on in the in the Wikipedia part and like, oh, know, Wikipedia kind of got lucky because the bot self identified as a as an AI agent. Right? Andreas on his LinkedIn profile has the word AI next to his name.

Joseph: 57:53

So it says Andrea Weiser, then AI is sort of a lighter gray text. And Jason very correctly caught that just as I was about to update the article with his statement. And Jason's like, wait, wait, wait, what's this? Checks out for a minute because of course it's like, wait, actually am talking to the CEO or is he like somehow automated his LinkedIn profile? Turns out no, I was speaking to him and I spoke to somebody else who knows about that, and they said that Andreas had changed their pronouns on LinkedIn to say AI.

Joseph: 58:27

Cringe. And I'll just leave it there. I wanna add in from all that, but that was a little bit of confusion of like, well, and this will be the last thing I say, but if somebody sends you a response, which like is AI generated, it complicates the reporting. Like sometimes you may not be able to use it entirely. Guido, Dan Guido from Trailer Bits, he did send me a claw generated summary of an internal chat he said that Trailer Bits had about this.

Joseph: 59:02

But what I did was then Well, then I just asked Dan questions directly, and I'm speaking to Dan, so that's fine. But, like, if anyone ever introduces AI in a conversation, it does just make it harder for journalists to be like, wait. So who the fuck am I talking to, and how can I even use this information? And that's gonna keep happening, basically. And that's when it's disclosed.

Joseph: 59:22

You know, sometimes we're not gonna know. Yeah. Alright. I will get the outro script in front of me if I can open this on my teleprompter, and I'll play us out. As a reminder, four zero four Media is journalist founded and supported by subscribers.

Joseph: 59:38

If you do wish to subscribe to four zero four Media and directly support our work, please go to 404media.co. You'll get unlimited access to our articles and an ad free version of this podcast. You'll also get to listen to the subscribers only section where we talk about a bonus story each week. This podcast is made in partnership with Kaleidoscope and Alyssa Midcalf. Another way to support us is by leaving a five star rating and review for the podcast.

Joseph: 01:00:03

That stuff really does help us out, or just please tell a friend. This has been four zero four Media. We'll see you again next week.