FediDB has stoped crawling until they get robots.txt support
-
-
It's not about the impact it's about consent.
-
lol FediDB isn't a crawler, though. It makes API calls.
-
-
Because of AI bots ignoring robots.txt (especially when you don't explicitly mention their user-agent and rather use a * wildcard) more and more people are implementing exactly that and I wouldn't be surprised if that is what triggered the need to implement robots.txt support for FediDB.
-
-
Why invent implied consent when complicit consent has been the standard in robots.txt for ages now?
-
-
ANY SITE THIS SOFTWARE IS APPLIED TO WILL LIKELY DISAPPEAR FROM ALL SEARCH RESULTS.
I’m sold
-
-
-
You can consent to a federation interface without consenting to having a bot crawl all your endpoints.
Just because something is available on the internet it doesn't mean all uses are legitimate - this is effectively the same problem as AI training with stolen content.
-
Yes. I wholeheartedly agree. Not every use is legitimate. But I'd really need to know what exactly happeded and the whole story to judge here. I'd say if it were a proper crawler, they'd need to read the robots.txt. That's accepted consensus. But is that what's happened here?
And I mean the whole thing with consensus and arbitrary use cases is just complicated. I have a website, and a Fediverse instance. Now you visit it. Is this legitimate? We'd need to factor in why I put it there. And what you're doing with that information. If it's my blog, it's obviously there for you to read it... Or is it!? But that's implied consent. I'd argue this is how the internet works. And most of the times it's super easy to tell what's right an what is wrong. But sometimes it isn't.
-
Robots.txt started I'm 1994.
It's been a consensus for decades.
Why throw it out and replace it with imied consent to scrape?
That's why I said legally there's nothing they can do. If people want to scrape it they can and will.
This is strictly about consent. Just because you can doesn't mean you should yes?
-
I just think you're making it way more simple than it is... Why not implement 20 other standards that have been around for 30 years? Why not make software perfect and without issues? Why not anticipate what other people will do with your public API endpoints in the future?
There could be many reasons. They forgot, they didn't bother, they didn't consider themselves to be the same as a commercial Google or Yandex crawler... That's why I keep pushing for information and refuse to give a simple answer. Could be an honest mistake. Could be honest and correct to do it and the other side is wrong, since it's not a crawler alike Google or the AI copyright thieves... Could be done maliciously. In my opinion, it's likely that it hadn't been an issue before, the situation changed and now this needs a solution. And we're getting one. Seems at least FediDB took it offline and they're working on robots.txt support. They did not refuse to do it. So it's fine. And I can't comment on why it hadn't been in place. I'm not involved with that project and the history of it's development.
-
Maybe the definition of the term "crawler" has changed but crawling used to mean downloading a web page, parsing the links and then downloading all those links, parsing those pages, etc etc until the whole site has been downloaded. If there were links going to other sites found in that corpus then the same process repeats for those. Obviously this could cause heavy load, hence robots.txt.
Fedidb isn't doing anything like that so I'm a bit bemused by this whole thing.
-
It's been a consensus for decades
Let's see about that.
Wikipedia lists http://www.robotstxt.org as the official homepage of robots.txt and the "Robots Exclusion Protocol". In the FAQ at http://www.robotstxt.org/faq.html the first entry is "What is a WWW robot?" http://www.robotstxt.org/faq/what.html. It says:
A robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced.
That's not FediDB. That's not even nodeinfo.
-
https://lemmyverse.net/ still crawling, baby.
-
From your own wiki link
robots.txt is the filename used for implementing the Robots Exclusion Protocol, a standard used by websites to indicate to visiting web crawlers and other web robots which portions of the website they are allowed to visit.
How is f3didn not an "other web robot"?
-
stoped
Well, they needed to stope. Stope, I said. Lest thy carriage spede into the crosseth-rhodes.
