Would SQLite for Python be helpful for a file this size?

Question

I have a file of around 60 million lines. I am trying to constantly query the file to find information for a list of names. Each line in the file contains a name followed by relevant information. I tried to build a dictionary but the file was too big.

Many suggesting SQLite for Python. Would that be helpful for a file this size?

Answer 1

you can use Linux Power tools. Awk can do your work very easily.

Suppose your file looks like this

name1 a b c

name2 e d r t

name3 x y

and you want to find info of name2. then using this command you can find

awk -F" " -v name=name2 '{if(name==$1){$1=""; print $0}}' filename

it will output only info of name2 like this

e d r t

you can pass multiple name and manage your logic