What's the best paid search engine?
-
I'm not disputing that you might be right, but the internet archive runs a very different service. Mainly that Google needs to continuously prune their 400 billion page index because of link rot. The Internet Archive has the opposite aim, they are preserving sites that no longer exist.
I'm also not sure they even crawl. Do sites get added on user request? When looking at a medium popularity page, you see it only has a couple of scrapes a year.
None of them. At least, none that I'm aware of. I just don't think that direct expenses are the reason that there are are only two major web search tools. I also don't think Google and bing are good examples to point at when estimating the cost of running a complete search engine.
I would suggest direct expenses are the barrier, but perhaps crawling is not the main expense. I would be interested to know any speculations you have outside of expenses that cause a barrier?
When I said 'direct expenses' I mostly meant the cost of owning / running a database of internet pages and metadata comprehensive enough to be considered part of a 'fully featured search engine'. There's also the other half; the compute required to create that metadata, as well as obtain it, but at most I would guess that those would be equal in cost to just having the space for a database of all the internet pages (scaling up after that based on how many users you need to support). In short, a scaled down web engine that had access to every page on the internet that people would want to find could cost as low as 100,000$ for a first time purchase for the hardware.
The internet archive does in fact have their own web crawler they use. They also do sites upon request as well; i've had my personal website on there for almost two decades now, specifically at my request.
They also have a full-featured search function available for anyone on their website at archive.org. This is why I say they're a reasonable price comparison for a full-featured search engine. They may spend more on storage and less on metadata compute than a theoretical smaller search engine, but at the end of the day, that's just a re-balancing of the cost, not a completely new and more excessive cost.
I think direct expenses; the cost of owning and maintaining an internet index database, are definitely significant enough that the completely free access that google gives to anyone who wants it, are way more than any single private entity or company is able to support just because they want to have it. I don't think it would be anywhere even close to a billion dollars though.
I think the hardest part of having a internet index database would be the knowledge required to create and maintain it, especially under the hostile forces that are the 75 billion dollar seo industry. If a selfhosted search engine became big enough that the seo industry started trying to break it, I don't think that company would survive for very long at all.
Google is losing that battle, like, almost completely. What hope would a small startup style company have of battling it and staying financially solvent, especially if they're trying to be different from google and bing and actually showing results without the pressure of advertisers breathing down their necks?
I think the hardware side of a search engine is solvable with silicon valley startup level of funding. I think it's impossible for anyone in the current day and age to make that sort of project solvent while keeping the user (instead of the advertiser) as the main customer. For anyone else who can't get those funds, or don't actually want to do a results-oriented search engine, they can just mooch of off google and bing for free.
-
Kagi is my absolute favorite.
Sounding off as another kagi user. It's great. If the company doesn't do anything extra stupid or come out pro-techno fascist they'll have my business for life.
If for no other reason than to decrease my digital footprint, get rid of ads, and pay billionaires less.
-
I think you'd be right that the direct cost of running the crawler and index would not be the issue. But fighting SEO to keep your results decent is probably a cost that dwarfs the basic technical cost of running the crawler and index.
And you'd need a technical security team on top of things as link farms aren't your only risk, I'm sure there are countless ways to manipulate the algorithm to put your site on top that Google probably have multiple teams working on fighting it full time.
Many of these things would likely not be a problem for a startup, though. No one is paying SEO firms big money to get into a search index no one has heard of and hardly anyone uses, so these costs probably grow exponentially over time as you become more well known.
-
I think you'd be right that the direct cost of running the crawler and index would not be the issue. But fighting SEO to keep your results decent is probably a cost that dwarfs the basic technical cost of running the crawler and index.
And you'd need a technical security team on top of things as link farms aren't your only risk, I'm sure there are countless ways to manipulate the algorithm to put your site on top that Google probably have multiple teams working on fighting it full time.
Many of these things would likely not be a problem for a startup, though. No one is paying SEO firms big money to get into a search index no one has heard of and hardly anyone uses, so these costs probably grow exponentially over time as you become more well known.
Yeah, and on the smaller / earlier side of a theoretical search engine company, google offers their api for free. I think this is actually another one of the biggest contributors to why nobody has tried to make a new search engine with their own index. Why waste hundreds of thousands of dollars in hardware, and even more on personnel costs, when you can just have google do it for you instead?
-
Yes offering everything for free to prevent competition has been a surprisingly effective strategy for Google.
-
You can pay me and I'll setup a searxng instance for you if you want.
Or just use the free ones, mine is here
Kagi is the only paid search ive heard of but its more of a AI company that just happened to have a search engine as their most successful product. Consider reading this before switching to them: https://d-shoot.net/kagi.html
Wow! Your search engine is fast. Nice layout, simple, clean. I really like it!
So we can use your magic?
-
ALL HAIL KAGI
Everyone should be aware of the fact that Kagi supports Russia by buying Yandex index API. They excuse themselves with neutrality. Well, DuckDuckGo for some reason took the L and dropped them after the Ukraine invasion, so make your own conclusions here. I think it's easy to come up with all sorts of justifications for your actions and I cancelled my sub a couple of months back. They do have a decent product, though but it's also pretty expensive.
-
Everyone should be aware of the fact that Kagi supports Russia by buying Yandex index API. They excuse themselves with neutrality. Well, DuckDuckGo for some reason took the L and dropped them after the Ukraine invasion, so make your own conclusions here. I think it's easy to come up with all sorts of justifications for your actions and I cancelled my sub a couple of months back. They do have a decent product, though but it's also pretty expensive.
Bummer. Yes it is kind of spendy for what you get . 5 dollars doesnt cover my searches for a month.
That really sucks because kagi was so good at finding content that wasn't typical ad spam and slop. Oh well.
-
It's French and free.
By free, I assume you mean that you pay with attention for advertisements? Or is it donationware or something else?
-
I feel like there are probably some ad based search engines which are privacy and service oriented, but in general even for those there remains a misalignment problem. Hence if I don’t want to be a product now or in the future, what good search engines are there that I can pay for?
Non-targeted ads and funding from investors, as far as I can tell.
-
Everyone should be aware of the fact that Kagi supports Russia by buying Yandex index API. They excuse themselves with neutrality. Well, DuckDuckGo for some reason took the L and dropped them after the Ukraine invasion, so make your own conclusions here. I think it's easy to come up with all sorts of justifications for your actions and I cancelled my sub a couple of months back. They do have a decent product, though but it's also pretty expensive.
What a way to morally stand up to one evil while supporting a greater evil. Google and Microsoft are complicit in the Israeli genocide, like they're not just American companies, they provide resources, infrastructure, and intellectual support for the genocide and apartheid systems. But you have no problem with DDG using them, right?
Just be consistent.
-
What a way to morally stand up to one evil while supporting a greater evil. Google and Microsoft are complicit in the Israeli genocide, like they're not just American companies, they provide resources, infrastructure, and intellectual support for the genocide and apartheid systems. But you have no problem with DDG using them, right?
Just be consistent.
And what do you think Kagi uses? They rely heavily on the same Bing and Google. If you played around with them you will see most of their searches are identical to what you would get if you did a bing search and then follow up with a google search, you're not getting much more.
The fact of the matter is that there are only 2 real choices of search engines. Google and Bing. There's a massive gap between them and every other independent search index. -
And what do you think Kagi uses? They rely heavily on the same Bing and Google. If you played around with them you will see most of their searches are identical to what you would get if you did a bing search and then follow up with a google search, you're not getting much more.
The fact of the matter is that there are only 2 real choices of search engines. Google and Bing. There's a massive gap between them and every other independent search index.wrote last edited by [email protected]I didn't push for Kagi... You're spilling out my point though
I'm just saying that saying "use DDG because it doesn't use a Russian company's search engine" while ignoring that DDG uses objectively more evil companies is stupid.
-
Bummer. Yes it is kind of spendy for what you get . 5 dollars doesnt cover my searches for a month.
That really sucks because kagi was so good at finding content that wasn't typical ad spam and slop. Oh well.
I do some dev and linux stuff and I still found it a bit hard to justify the cost. Like it's not a bad deal, however i did find myself going through days not really using it properly and just doing simple searches like "product x documentation" or just "site:" searches that i can do with simply DDG or Ecosia (Google). I think it's cuz a lot of the internet is converging into larger sites rather than being scattered all over and i do find myself relying a lot more on LLM's. If you actually test out their search results you will get something like 97% there if you do a Google search and a Bing search to follow up or just use searx.
What has helped me completely stop using them is finding sites with valuable information for my use case and creating a manual "lens" something similar to what Kagi offers. It's really simple and works better than theirs. I have this pasted into my obsidian note for linux (example):
site:archlinux.org OR site:endeavouros.com OR site:reddit.com/r/arch OR site:reddit.com/r/archlinux OR site:reddit.com/r/endeavouros OR site:manjaro.org OR site:reddit.com/r/hyprland OR site:hypr.land
Everytime i search for anything arch linux related so i paste this in and enter the search phrase at the start.
I use t3.chat for my LLM needs (you get way more for the price that you pay, including premium models) so I don't need their LLM's. I havent found LLM web
search to work well for my use case so don't need that.I use DDG as my daily driver and then switch to Ecosia (google wrapper) if i cant find stuff. DDG is a nice search engine overall, fills in that "gap" left by Kagi.
I use zen browser and have created search engine keywords like @ecosia, @ddg, @wolfram so i can CTRL + T and "@ecosia search term" easily.
Overall, I feel like they're not offering anything groundbreaking but their whole package is nice, i see the appeal especially if you browse on mobile. Whether you find them useful or not will depend on your use case. For those 10$ + tax I personally don't. The Russia support is disgusting though.
