Is there a way to reduce the number of AI generated websites that appear in search results?

[imsufferableninja@sh.itjust.works]

Ah, I guess I misunderstood the problem

[corngood@lemmy.ml]

I've almost given up on just searching the whole internet for something. I either filter by eye for sites I trust in the results, or add a filter to the query. There are usually a handful of sites I trust on a given topic.

[crandlecan@mander.xyz]

I've switched to presearch.com long ago. No more tracking.

[tal@lemmy.today]

No, because there's no reliable way to distinguish AI-generated spam sites from non-AI-generated spam sites. I'll also add that I don't expect there to be one promptly forthcoming: any attempt to identify them is going to run into improved systems, and that's gonna happen even if the systems aren't explicitly intending to evade detection. If it were easy, Google would have done so years back. I can recognize some now, but the SEO spam crowd that's creating this is trying hard to pollute search engine results, and if someone implements a generalized "block" that's effective, they're going to keep looking for alternatives until they find something that gets through.

On Kagi, I can set the acceptable date range on results to prior to the emergence of LLMs, but that cuts out a lot of material that I want to see. For some searches, that might work, but it's not really a general solution.

You can manually blacklist or deprioritize sites on Kagi. Probably can either run some sort of local proxy or Greasemonkey-style plugin that would let you do so in browser on any search engine. Problem is that there are people making these sites faster than you're going to be banning them.

Kagi's also got a "pin" and a "raised priority" feature for a list of sites, and I suppose could whitelist some "known good" sites. Kagi's "blacklist/deprioritize/prioritize/pin" feature does not have the ability to exchange sites between users (and I imagine that there'd be some privacy issues with doing so) aside from Kagi running a "leaderboard" of the most-blacklisted/deprioritized/prioritized/pinned sites. One could probably do the "proxy" or "plugin" route as well for a variety of websites on other search engines. Any general solution would need to have some level of interchange, since requiring every individual user to maintain a "killfile" on websites is going to be impractical. It may be that the human labor involved in curation is outweighed by how cheap it is to generate new websites; not sure.

At some point, I assume that it may become practical to just make a conservative whitelist of "non-spam" sites that accepts that many useful websites will be excluded because we just can't validate them as not being non-spam. Probably require human curation, which is either going to need volunteer labor or a commercial service.

There's also a secondary problem that if you curate content at the domain level, Web 2.0 sites that permit posting content (Reddit, Wikipedia, the Threadiverse, etc) can have individual users inserting AI-generated spam. So a general solution is probably going to need to permit some sort of sub-domain level filtering for at least major sites.

And there's also the wrinkle that a "trusted good" site or user can become a spammer at some point. Spammers/people who want to run influence operations have been buying high-karma Reddit accounts --- and the reputation that comes with them --- for quite some years. Domains expire, or their operators change. Reputation has value, and it can be sold. So that also has to be addressed.

This isn't really a qualitative change. I mean, people have hand-crafted spam websites that try to grab searchers before. It's just that the ability to use a computer to do it is way more cost-efficient, brings the cost way down, and thus opens up a lot of opportunity for spam that wouldn't have made sense financially before. So what you're really aiming to do is to get the cost to make a spam website up. One possibility --- which I am absolutely confident that TLS certificate issuers would like --- would be to have tiers of TLS certificate, some of which are a lot more expensive. Search engine indexers could check and validate the TLS "cost tier" when indexing a site. That will artificially inflate the cost of running a website, and can be done to an arbitrary degree. That's not fantastic, since it also tends to cut out non-spam individual/low-cost websites, but if you're a large company somewhere, the price is basically a rounding error compared to what a spammer needs to make to make his super-cheap-to-generate LLM-generated website worthwhile. Could be a component in a system that takes into account other factors.

[chosensilence@pawb.social]

yeah, use engines like startpage.com instead.

[moonlight@fedia.io]

before:2023

[fetchies@lemmy.blahaj.zone]

Try uBlacklist, with these blocklists:

# AI Spam
https://raw.githubusercontent.com/laylavish/uBlockOrigin-HUGE-AI-Blocklist/main/list_uBlacklist.txt
# Copycat Sites
https://raw.githubusercontent.com/quenhus/uBlock-Origin-dev-filter/main/dist/other_format/uBlacklist/global.txt
# SEO Spam & Junk
https://raw.githubusercontent.com/NotaInutilis/Super-SEO-Spam-Suppressor/main/ublacklist.txt

[felixwhynot@lemmy.world]

https://udm14.org/

[dave@lemmy.nz]

Kagi does seem to cut out a lot of blogspam. I think Google is incentivised to send people to these sites with adwords ads on them.

[tyler@programming.dev]

Kagi doesn’t know but I think kagi’s indexing is just better so you don’t get the blogspam as much and when you do you can block it across your searches.

[p1nkman@lemmy.world]

And customize the result page. Mine looks like the early 2000's, without the image, text only. It's just a great customizable search engine!

[kossa@feddit.org]

With kagi.com you can at least blacklist domains to never show up in your search again. I guess there are some filter lists out there already, to get a head start.

[friend_of_satan@lemmy.world]

Vintage websites going up in value.

[jballs@sh.itjust.works]

I just realized how freaking bad this was - especially on Reddit. I was doing a search for IPTV providers. I checked reddit first, since in the last that has been a reliable way for finding people recommending stuff, not just whatever is search engine optimized.

Holy shit was it bad. Nearly every comment was AI slop. They all followed a very similar structure. Tons and tons of comments like:

Appreciate the share! LunoTV is great — super affordable and has tons of channels. Definitely recommend checking it out.

Appreciate the share! Xalvon IPTV offers tons of channels and the streaming has been really smooth on my devices. Good value overall.

LunoTV has been great for me. Tons of channels, really affordable, and easy to set up.

Luno TV has been great for me. Tons of channel selection is huge, and their app runs smoothly .

We're seriously getting to a point where we won't be able to search for anything due to AI fake results.

[hardcoreufo@lemmy.world]

Did you find an iptv service? I've been thinkibg about getting one but whenever i look into i just find junk.

[naught101@lemmy.world]

Thanks for this! Been looking for something like it. I guess it just blocks the sites though, and doesn't block them appearing in search results?

[blackmist@feddit.uk]

Sure there is, and I'll tell you all about it right after this meandering story about my grandmother growing up on a farm.

[breecher@sh.itjust.works]

Plenty of websites are AI created, with more being created every hour, which I gather is what OP is talking about.

And there isn't really any useful wait to filer those out, with only lots of profitable reasons for people to create more of them, and they will shortly be the complete death of the internet as we knew it.

[forrgott@lemmy.sdf.org]

Oof.

And I read it wrong like multiple times.

Thank you

[fetchies@lemmy.blahaj.zone]

So uBlacklist actually removes sites specified in your subscribed rulesets from your search results.

I found it helped out a lot on programming or tech support searches, as there's so many content mirrors and SEO spam sites for that domain.

I also found that using search shortcuts has helped me reduce the need for the middle-man in a surprising amount of my searches. (e.g., “@w” for Wikipedia, “@g” for the Gentoo Wiki, “@git” for GitHub, “@p” for ProtonD, “@y” for YouTube, “@s” for Stack Overflow, et cetera)