Searching through massive files
-
I have aquired several very large files. Specifically, CVSs of 100+ GB.
I want to search for text in these files faster than manually running grep.
To do this, I need to index the files right? Would something like Aleph be good for this? It seems like the right tool...
https://github.com/alephdata/aleph
Any other tools for doing this?
-
I have aquired several very large files. Specifically, CVSs of 100+ GB.
I want to search for text in these files faster than manually running grep.
To do this, I need to index the files right? Would something like Aleph be good for this? It seems like the right tool...
https://github.com/alephdata/aleph
Any other tools for doing this?
No idea about aleph. I've used Solr for that (solr.apache.org), thus my username, but maybe that is considered old school by now.
-
I have aquired several very large files. Specifically, CVSs of 100+ GB.
I want to search for text in these files faster than manually running grep.
To do this, I need to index the files right? Would something like Aleph be good for this? It seems like the right tool...
https://github.com/alephdata/aleph
Any other tools for doing this?
wrote last edited by [email protected]Are you looking for specific values in some field in this table, or substrings in that field?
If specific values, I'd probably import the CSV file into a database with an column indexed on the value you care about.
-
I have aquired several very large files. Specifically, CVSs of 100+ GB.
I want to search for text in these files faster than manually running grep.
To do this, I need to index the files right? Would something like Aleph be good for this? It seems like the right tool...
https://github.com/alephdata/aleph
Any other tools for doing this?
wrote last edited by [email protected]Really depends on what data it is and whether you want to search it regularly or just as a one time thing.
You could load them into an rdbms (MySQL/Postgres) and have it handle the indexing, or use python tools to process the files. Something like elasticsearch could work too.
If it's just a one time thing grep is probably fine tho.
Aleph could work as well but I have no experience with it.
I guess it depends on how much time you want to invest in setting something up versus how much time you'd lose waiting for grep to finish (if you only need to search a certain column, you can create an index with just that column using awk, search that index file, then extract the full line from the source file based on that result, but at that point you're basically creating a new database engine).
-
No idea about aleph. I've used Solr for that (solr.apache.org), thus my username, but maybe that is considered old school by now.
Elasticsearch should work too
-
I have aquired several very large files. Specifically, CVSs of 100+ GB.
I want to search for text in these files faster than manually running grep.
To do this, I need to index the files right? Would something like Aleph be good for this? It seems like the right tool...
https://github.com/alephdata/aleph
Any other tools for doing this?
Done this with massive log files. Used perl and regex. That's basically what the language was built for.
But with CSVs? I'd throw them in a db with an index.
-
Are you looking for specific values in some field in this table, or substrings in that field?
If specific values, I'd probably import the CSV file into a database with an column indexed on the value you care about.
Many (most?) databases these days support some sort of full text search.
-
I have aquired several very large files. Specifically, CVSs of 100+ GB.
I want to search for text in these files faster than manually running grep.
To do this, I need to index the files right? Would something like Aleph be good for this? It seems like the right tool...
https://github.com/alephdata/aleph
Any other tools for doing this?
I've used java Scanner objects to do this extremely efficiently with minimal memory required even with multiple parallel searches. Indexing is only necessary if you want to search for information many times and don't know what exactly the search will be. For one time searches, it's not going to be useful. Grep honestly is going to be faster and more efficient for most one time searches.
The initial indexing or searching of the files will be bottlenecked by the speed of the disk the files are on, no matter what you do. It only helps to index because you can move future searches to faster memory.
So it greatly depends on what and how often you need to search and the tradeoff is memory usage, but only for multiple searches of data you choose to index from the files in the first pass.
