Building large Linear Program

Question

I want to build an LP whose parameters are given by, between 5 and 10, 25,000,000 to 50,000,000 row .csv files (approx 500mb to 1Gb each).

My model is currently coded in AMPL, and reads in the parameter values directly from the .csv files. The Windows XP with 1 Gb RAM I am using runs out of memory trying to build a model based on data from only one 500mb .csv

My question: Is there a way to manage my data so that I can build the LP using less memory?

I appreciate all feedback from anyone with experience building massive LP.

Answer 1

It is hard to see that you will ever be able to load and solve such large problems where the .csv file alone is 500 MB or more, if you only have 1 GB RAM on your computer.

If it is not an option to add considerably more RAM memory, you will need to analyze your LP problem to see if it can be separated into smaller independent parts. For example, if you have a problem with 10,000 variables and 10,000,000 rows, perhaps it is possible to break the main problem up into say 100 independent sub-problems with 100 variables and 100,000 rows each?

Here is a link to an albeit dated book chapter that discusses separation of a large LP problem into manageable sub-problems.