What's the best paid search engine?
-
I feel like there are probably some ad based search engines which are privacy and service oriented, but in general even for those there remains a misalignment problem. Hence if I don’t want to be a product now or in the future, what good search engines are there that I can pay for?
I didn't know there were paid search engines. Could just list ANY of them for us?
-
I feel like there are probably some ad based search engines which are privacy and service oriented, but in general even for those there remains a misalignment problem. Hence if I don’t want to be a product now or in the future, what good search engines are there that I can pay for?
Kagi is my absolute favorite.
-
I didn't know there were paid search engines. Could just list ANY of them for us?
I think I only know of Kagi. Are there any more?
-
Kagi is my absolute favorite.
Ok cool I saw this one— is it really like the old internet?
-
Ok cool I saw this one— is it really like the old internet?
I also recommend Kagi. I wouldn’t say it’s like the “old internet” but it works and it has amazing features that every search engine should have.
-
I think I only know of Kagi. Are there any more?
I think qwant is one? Or maybe it’s just European and not paid.
-
I feel like there are probably some ad based search engines which are privacy and service oriented, but in general even for those there remains a misalignment problem. Hence if I don’t want to be a product now or in the future, what good search engines are there that I can pay for?
You can pay me and I'll setup a searxng instance for you if you want.
Or just use the free ones, mine is here
Kagi is the only paid search ive heard of but its more of a AI company that just happened to have a search engine as their most successful product. Consider reading this before switching to them: https://d-shoot.net/kagi.html
-
I feel like there are probably some ad based search engines which are privacy and service oriented, but in general even for those there remains a misalignment problem. Hence if I don’t want to be a product now or in the future, what good search engines are there that I can pay for?
I'm using MetaGer and it's pretty cool. They also have a nice OpenStreetMap interface maps.metager.de.
-
I feel like there are probably some ad based search engines which are privacy and service oriented, but in general even for those there remains a misalignment problem. Hence if I don’t want to be a product now or in the future, what good search engines are there that I can pay for?
There are only 2 relevant web crawlers; Google's and Microsoft's. All queries from every search engine goes through these two crawlers either directly or through a middle layer of obfuscation.
The issue is that the internet is too large to index. This has been a known emerging issue for a long time. This is the real reason search sucks. It is not deterministic because it cannot be, but therein lies the issue. Without deterministic unbiased information, democracy is dead. And so search sucks. No one has been able to find a solution for efficient access to enormous databases like this except through the methodologies behind AI. At least not for real time search queries.
-
Ok cool I saw this one— is it really like the old internet?
wrote last edited by [email protected]I use Kagi. It's not especially like the Internet at some long-ago date, other than maybe than in that they run a search engine for Usenet archives.
I use Kagi for privacy reasons. They run a no-log, no-profile, no-ads setup, which is what I'd been wanting for some time.
If you're looking for some kind of retro Internet experience, maybe grab a gopher client and look at gopher servers, or an IRC client, telnet to a MUD, use an FTP client to connect to an FTP server. All of those still have active servers and haven't changed much, so it'll still give a comparable feel.
-
There are only 2 relevant web crawlers; Google's and Microsoft's. All queries from every search engine goes through these two crawlers either directly or through a middle layer of obfuscation.
The issue is that the internet is too large to index. This has been a known emerging issue for a long time. This is the real reason search sucks. It is not deterministic because it cannot be, but therein lies the issue. Without deterministic unbiased information, democracy is dead. And so search sucks. No one has been able to find a solution for efficient access to enormous databases like this except through the methodologies behind AI. At least not for real time search queries.
I once read that running a search crawler costs upwards of a billion dollars a year. Anyone other than Microsoft or Google running their own search index are either not getting a wide spread of the internet or they are using their own index to supplement Google or Bing results.
-
I think qwant is one? Or maybe it’s just European and not paid.
It's French and free.
-
I feel like there are probably some ad based search engines which are privacy and service oriented, but in general even for those there remains a misalignment problem. Hence if I don’t want to be a product now or in the future, what good search engines are there that I can pay for?
https://marginalia-search.com/ is pretty great for a different approach to search. While not exactly what you were asking for since its free, you can definitely pay via donations if you find it valuable.
-
There are only 2 relevant web crawlers; Google's and Microsoft's. All queries from every search engine goes through these two crawlers either directly or through a middle layer of obfuscation.
The issue is that the internet is too large to index. This has been a known emerging issue for a long time. This is the real reason search sucks. It is not deterministic because it cannot be, but therein lies the issue. Without deterministic unbiased information, democracy is dead. And so search sucks. No one has been able to find a solution for efficient access to enormous databases like this except through the methodologies behind AI. At least not for real time search queries.
The issue is that the internet is too large to index.
It's really not. At least, not yet. It's a large part of why it isn't done, but it's not the only one, and I'd argue, not even the main reason it isn't really done.
A complete crawl with meta data of the internet in 2025 is only 424TiB. For comparison, my 1000$ home setup can handle about a tenth of that(in storage at least). The hardware to maintain a single database of the internet with metadata could cost under $100,000, easily.
Dave, your comment about it costing a billion to run Bing or Google might be true, but it is completely unrelated to the realities of running a small search engine and has everything to do with the fact that they are Google and Microsoft products respectively.
The real issue isn't the physical size of the internet, it's much more likely to be the complexity of making a search algorithm that can compete with the 75 billion seo market that wxists to break search engines.
-
I once read that running a search crawler costs upwards of a billion dollars a year. Anyone other than Microsoft or Google running their own search index are either not getting a wide spread of the internet or they are using their own index to supplement Google or Bing results.
That's like saying that it's impossible to run a car manufacturing company without 100 billion because that's how much Ford spends on their car manufacturing processes. It makes no sense.
Yes, making an original search engine is hard, just like making trucks is. But that doesn't mean that running either one requires billions of dollars to do.
Common crawl is a nonprofit that regularly shares free copies of every internet page with metadata, and it damn well doesn't take billions to do it either. https://commoncrawl.org/
-
I feel like there are probably some ad based search engines which are privacy and service oriented, but in general even for those there remains a misalignment problem. Hence if I don’t want to be a product now or in the future, what good search engines are there that I can pay for?
ALL HAIL KAGI
-
Original comment said in good faith, but from sketchy long term memory of stuff I've come across. It seems like it was in a Lex Friedman or similar podcast at some point, but from some time in the last 3-10 years. I may have conflated or misunderstood, as I am not experienced with such complexity. I seem to recall it coming up around the time several astronomers were speaking publicly about issues with processing large amounts of data and soliciting solutions. I just recall wondering why search started to suck around 2017, and putting the pieces together when I heard this. Now, in retrospect, it seems much of the changes were also adversarial for rival AI training after the Transformers paper. At least, looking at how search results are salted now, and the way images are selected for search is absolutely adversarial for AI training datasets... but that is all I know, and should be taken as friendly neighborhood water cooler talk, always with the best of intentions.
-
Original comment said in good faith, but from sketchy long term memory of stuff I've come across. It seems like it was in a Lex Friedman or similar podcast at some point, but from some time in the last 3-10 years. I may have conflated or misunderstood, as I am not experienced with such complexity. I seem to recall it coming up around the time several astronomers were speaking publicly about issues with processing large amounts of data and soliciting solutions. I just recall wondering why search started to suck around 2017, and putting the pieces together when I heard this. Now, in retrospect, it seems much of the changes were also adversarial for rival AI training after the Transformers paper. At least, looking at how search results are salted now, and the way images are selected for search is absolutely adversarial for AI training datasets... but that is all I know, and should be taken as friendly neighborhood water cooler talk, always with the best of intentions.
I think most startup search engines use Google/bing because it's free/way cheaper than running their own database, not because it's impossible. It also likely sidesteps a lot of the seo bullshit simply because Google/bing have more experience working around it
So like, short term/small size its cheaper and straight up easier to piggyback off of the big two companies, rather than manage your own data set. Long term, if you get popular enough to be noticed, I expect that the seo business would wreck any selfhosting search engine startup company's results pretty regularly.
-
wrote last edited by [email protected]
That website claims they add 3-5 billion pages a month. Google is doing that in a day or three, as recency of information is very important in search. Plus that site claims 100 billion pages to Googles 400 billion. It's still an impressive project.
Size isn't everything, so the real question is: what search site uses only the common crawl index and has results on par with bing or google?
-
That website claims they add 3-5 billion pages a month. Google is doing that in a day or three, as recency of information is very important in search. Plus that site claims 100 billion pages to Googles 400 billion. It's still an impressive project.
Size isn't everything, so the real question is: what search site uses only the common crawl index and has results on par with bing or google?
Size isn't everything, so the real question is: what search site uses only the common crawl index and has results on par with bing or google?
None of them. At least, none that I'm aware of. I just don't think that direct expenses are the reason that there are are only two major web search tools. I also don't think Google and bing are good examples to point at when estimating the cost of running a complete search engine.
If you read all of your article, the author notes that while Google has index of about 400 billion, the internet archives index is actually bigger at around 865 billion.
The internet archive has an operating cost of about 33m/year. I think that is a much more reasonable example to point to and say "running a complete search engine would have a similar price as that".
Also, very neat article btw. I would have never guessed that googles search index count has been shrinking for the past little bit. Or that Google actively culls results from their database that it thinks people won't ever want to see.
