Printing arraylist into output file?

Question

FOLLOW UP QUESTION HERE: Java: Printing Arraylist to Output File?

For some reason, it takes an extraordinary amount of time for the ArrayLists to be printed to the output file, usually 20-30 minutes. However, this only happens with the sort methods or the filterTitle or filterArtist methods (methods that concern String inputs). When I run filterRank or filterYear, it runs perfectly fine.

When I print the song2 ArrayList directly from the filter methods, the only thing that is printed is [], which means the ArrayList is empty, but it shouldn't be? And the filterRank and filterYear methods still work regardless of this. Somehow I think it's related, though.

Input file can be found here: http://staff.rentonschools.us/hhs/ap-comp-science/projects/download/agazillionsongs.txt?id=223098

Example sort method:

public void sortYear() {
   Collections.sort(songs2, SongComparator.byYear()); // now we have a sorted list
   System.out.println(songs2);
}

Example filter method (for strings)

public void filterArtist(String s) {
   int n = 0;
   if (n == 0) {
      System.out.println("Program is processing.");
      n++;
      for (Song song1 : songs2) {
         if ((!(((song1.artist).contains(s))))) {
            itemsToRemove.add(song1);
         }
      }
      songs2.removeAll(itemsToRemove);
      itemsToRemove.clear();
   }
   System.out.println(songs2);
}

Example filter method (for ints)

public void filterRank(Range r) {
   int n = 0;
   if (n == 0) {
      System.out.println("Program is processing.");
      n++;
      for (Song song1 : songs2) {
         if (song1.rank > (r.getMax()) || (song1.rank) < (r.getMin())) {
            itemsToRemove.add(song1);
         }
      }
      songs2.removeAll(itemsToRemove);
      itemsToRemove.clear();
   }
   System.out.println(songs2);
}

Code in main class for printing/output file

while (input.hasNextLine()) {
   int n = 0;
   SongCollection collection = new SongCollection(songs);
   String inputType = input.nextLine();
   String delims = "[ ]";
   String[] tokens = inputType.split(delims);
   for (int i = 0; i < tokens.length; i++) {
      n = 0;
      if (n == 0) {
         if ((tokens[i]).contains("year:")) {
            collection.filterYear(Range.parse(tokens[i]));
            n = 1;
         }// end of year loop
         if ((tokens[i]).contains("rank:")) {
            collection.filterRank(Range.parse(tokens[i]));
            n = 1;
         }// end of rank
         if ((tokens[i]).contains("artist:")) {
            collection.filterArtist(tokens[i]);
            n = 1;
         }// end of artist
         if ((tokens[i]).contains("title:")) {
            collection.filterTitle(tokens[i]);
            n = 1;
         }// end of title
         if ((tokens[i]).contains("sort:")) {
               if ((tokens[i]).contains("title")) {
                  collection.sortTitle();
                  n = 1;
               }// end of sort title
               if ((tokens[i]).contains("artist")) {
                  collection.sortArtist();
                  n = 1;
               }// end of sort artist
               if ((tokens[i]).contains("rank")) {
                  collection.sortRank();
                  n = 1;
               }// end of sort rank
               if ((tokens[i]).contains("year")) {
                  collection.sortYear();
                  n = 1;
               }// end of sort year
         }//end of sort
      }// end of for loop

   }// end of input.hasNextline loop
   final PrintStream console = System.out; //saves original System.out
   File outputFile = new File("output.txt"); //output file
   PrintStream out = new PrintStream(new FileOutputStream(outputFile)); //new FileOutputStream
   System.setOut(out); //changes where data will be printed
   System.out.println(collection.toString());

   System.setOut(console); //changes output to print back to console
   Scanner outputFileScanner = new Scanner(outputFile); //inputs data from file
   while ((outputFileScanner.hasNextLine())) { //while the file still has data
      System.out.println(outputFileScanner.nextLine()); //print
   }
   outputFileScanner.close();
   out.close();
}

Full code for compilation:

   import java.io.*;
   import java.io.File;
   import java.io.FileNotFoundException;
   import java.util.*;
   import java.util.Comparator;
   import java.util.Scanner;
   import java.util.StringTokenizer;

public class GazillionSongs {
   public static void main(String[] args) throws FileNotFoundException, IOException {
      System.out.println("Welcome to Java Song Collection!"); // greets the user
      System.out
      .println("This program sorts and filters large databases of popular songs."); // explains purpose of program
      System.out
      .println("This program is able to filter and sort by year, artist, title and rank.");
      System.out
      .println("Please enter a file that contains a database you wish to filter or sort. (i.e, alistofsongs.txt)"); // sample file = agazillionsongs.txt
      Scanner fileInput = new Scanner(System.in); //Scanner which accepts filename
      String filename = fileInput.nextLine();

      File f = new File(filename); //creates file from input
      /*error check for file here*/
      Scanner fileScanner = new Scanner(f); //inputs data from file

      ArrayList<Song> songs = new ArrayList<Song>();
      while ((fileScanner.hasNextLine())) {
         songs.add(Song.parse(fileScanner.nextLine()));
      }

      System.out
      .println("Please select which commands you would like to use for the program.");
      System.out
      .println("Please format your command like the following example: year:<year(s)> rank:<rank(s)> artist:<artist> title:<title> sortBy:<field>");
      System.out.println();
      System.out.println("You may pick any number of commands you want.");
      System.out
      .println("For years and rank, you may select a range of years or ranks.");
      System.out
      .println("For artists and titles, you may enter a partial name or title.");
      System.out.println("i.e, year:1983 rank:1");
      Scanner input = new Scanner(System.in);

      while (input.hasNextLine()) {
         int n = 0;
         SongCollection collection = new SongCollection(songs);
         String inputType = input.nextLine();
         String delims = "[ ]";
         String[] tokens = inputType.split(delims);
         for (int i = 0; i < tokens.length; i++) {
            n = 0;
            if (n == 0) {
               if ((tokens[i]).contains("year:")) {
                  collection.filterYear(Range.parse(tokens[i]));
                  n = 1;
               }// end of year loop
               if ((tokens[i]).contains("rank:")) {
                  collection.filterRank(Range.parse(tokens[i]));
                  n = 1;
               }// end of rank
               if ((tokens[i]).contains("artist:")) {
                  collection.filterArtist(tokens[i]);
                  n = 1;
               }// end of artist
               if ((tokens[i]).contains("title:")) {
                  collection.filterTitle(tokens[i]);
                  n = 1;
               }// end of title
               if ((tokens[i]).contains("sort:")) {
                     if ((tokens[i]).contains("title")) {
                        collection.sortTitle();
                        n = 1;
                     }// end of sort title
                     if ((tokens[i]).contains("artist")) {
                        collection.sortArtist();
                        n = 1;
                     }// end of sort artist
                     if ((tokens[i]).contains("rank")) {
                        collection.sortRank();
                        n = 1;
                     }// end of sort rank
                     if ((tokens[i]).contains("year")) {
                        collection.sortYear();
                        n = 1;
                     }// end of sort year
               }//end of sort
            }// end of for loop

         }// end of input.hasNextline loop
         final PrintStream console = System.out; //saves original System.out
         File outputFile = new File("output.txt"); //output file
         PrintStream out = new PrintStream(new FileOutputStream(outputFile)); //new FileOutputStream
         System.setOut(out); //changes where data will be printed
         System.out.println(collection.toString());

         System.setOut(console); //changes output to print back to console
         Scanner outputFileScanner = new Scanner(outputFile); //inputs data from file
         while ((outputFileScanner.hasNextLine())) { //while the file still has data
            System.out.println(outputFileScanner.nextLine()); //print
         }
         outputFileScanner.close();
         out.close();
      }
   }// end of main
}// end of class

class Song{
   public enum Order {Year, Rank, Title, Artist}
   public int year;
   public int rank;
   public String artist;
   public String title;

   public static Song parse(String s) {
      Song instance = new Song();
      StringTokenizer tokenizer = new StringTokenizer(s, "\t");
      instance.year = Integer.parseInt(tokenizer.nextToken());
      instance.rank = Integer.parseInt(tokenizer.nextToken());
      instance.artist = (tokenizer.nextToken());
      instance.title = (tokenizer.nextToken());
      return instance;
   }

   public int getYear() {
      return year;
   }

   public int getRank() {
      return rank;
   }

   public String getArtist() {
      return artist;
   }

   public String getTitle() {
      return title;

   }

   public String toString() {
      String output = "\n\nYear = " + year + "\nRank = " + rank + "\nArtist = "
            + artist + "\nTitle = " + title;
      return output;
   }

}
class Range {
   private int min;
   private int max;

   public Range() {
      System.out.println("Please wait.");
   }

   public static Range parse(String s) {
      Range instance = new Range(); // instance is created here so object
                              // variables may be accessed
      String field; // String to contain deleted part of user input
      StringTokenizer tokenizer = new StringTokenizer(s, "-");
      StringTokenizer tokenizer2 = new StringTokenizer(s, ":");// for separating "field:" from the
                                                   // other part of the String
      if (s.contains(":")) { // this deletes the "field:" of the user input so
                        // it does not interfere with the parsing
         field = (tokenizer2.nextToken());
         s = s.replace(field, "");
         s = s.replace(":", "");
      }
      if (s.contains("-")) {
         instance.min = Integer.parseInt(tokenizer.nextToken());
         instance.max = Integer.parseInt(tokenizer.nextToken());

      } else if (!(s.contains("-"))) {
         {
            instance.min = Integer.parseInt(s);
            instance.max = Integer.parseInt(s);
         }
      }
      System.out.println("Range max = " + instance.max);
      System.out.println("Range min = " + instance.min);
      return instance;
   }

   public boolean contains(int n) {
      if (n > min && n < max) { //if the number is contained in the range, method returns true.
         return true;
      } else if (n == min && n == max) {
         return true;
      } else {
         return false;
      }
   }

   public int getMin() {
      return min;
   }

   public int getMax() {
      return max;
   }
}
class SongCollection {
   ArrayList<Song> songs2;
   ArrayList<Song> itemsToRemove = new ArrayList<Song>(); // second collection
                                             // for items to
                                             // remove
   public SongCollection(ArrayList<Song> songs) { // constructor for SongCollection
      System.out.println("Test");
      this.songs2 = songs;
      }
   public void filterYear(Range r) {
      int n = 0;
      if (n == 0) {
         System.out.println("Program is processing.");
         n++;
         for (Song song1 : songs2) {
            if (song1.year > (r.getMax()) || (song1.year) < (r.getMin())) {
               itemsToRemove.add(song1);
            }
         }
         songs2.removeAll(itemsToRemove);
         itemsToRemove.clear();
      }
      System.out.println(songs2);
   }

   public void filterRank(Range r) {
      int n = 0;
      if (n == 0) {
         System.out.println("Program is processing.");
         n++;
         for (Song song1 : songs2) {
            if (song1.rank > (r.getMax()) || (song1.rank) < (r.getMin())) {
               itemsToRemove.add(song1);
            }
         }
         songs2.removeAll(itemsToRemove);
         itemsToRemove.clear();
      }
      System.out.println(songs2);
   }

   public void filterArtist(String s) {
      int n = 0;
      if (n == 0) {
         System.out.println("Program is processing.");
         n++;
         for (Song song1 : songs2) {
            if ((!(((song1.artist).contains(s))))) {
               itemsToRemove.add(song1);
            }
         }
         songs2.removeAll(itemsToRemove);
         itemsToRemove.clear();
      }
      System.out.println(songs2);
   }

   public void filterTitle(String s) {
      int n = 0;
      if (n == 0) {
         System.out.println("Program is processing.");
         n++;
         for (Song song1 : songs2) {
            if ((!(((song1.title).contains(s))))) {
            itemsToRemove.add(song1);
            }
         }
         songs2.removeAll(itemsToRemove);
         itemsToRemove.clear();
      }
      System.out.println(songs2);
   }

   public void sortTitle() {
        Collections.sort(songs2, SongComparator.byTitle()); // now we have a sorted list
        System.out.println(songs2);
      }
   public void sortRank() {
        Collections.sort(songs2, SongComparator.byRank()); // now we have a sorted list
        System.out.println(songs2);
      }
   public void sortArtist() {
        Collections.sort(songs2, SongComparator.byArtist()); // now we have a sorted list
        System.out.println(songs2);
      }
   public void sortYear() {
        Collections.sort(songs2, SongComparator.byYear()); // now we have a sorted list
        System.out.println(songs2);
      }
   public String toString() {
      String result = "";
      for (int i = 0; i < songs2.size(); i++) {
         result += " " + songs2.get(i);
      }

      return result;

   }
}
class SongComparator implements Comparator<Song> {
   public enum Order{
      YEAR_SORT, RANK_SORT, ARTIST_SORT, TITLE_SORT
   }
   private Order sortingBy;
   public SongComparator(Order sortingBy){
      this.sortingBy = sortingBy;
   }
   public static SongComparator byTitle() {
       return new SongComparator(SongComparator.Order.TITLE_SORT);
   }
   public static SongComparator byYear() {
       return new SongComparator(SongComparator.Order.YEAR_SORT);
   }
   public static SongComparator byArtist() {
       return new SongComparator(SongComparator.Order.ARTIST_SORT);
   }
   public static SongComparator byRank() {
       return new SongComparator(SongComparator.Order.RANK_SORT);
   }

   @Override
   public int compare(Song song1, Song song2) {
      switch (sortingBy) {
      case YEAR_SORT:
         return Integer.compare(song1.year, song2.year);
      case RANK_SORT:
         return Integer.compare(song1.rank, song2.rank);
      case ARTIST_SORT:
         return song1.artist.compareTo(song2.artist);
      case TITLE_SORT:
         return song1.title.compareTo(song2.title);
      }
      throw new RuntimeException(
            "Practically unreachable code, can't be thrown");
   }

}

Answer 1

With almost 37,000 songs in your text-file database, and a significant number of ways for users to access it (by year, year-range, rank, rank-range, ...), including the possibility of sorting, I think it's almost required that you pre-process your song-database with a one-time-only process--once every time the database is changed--to optimize your data before ever giving your users the chance to query against it.

(I am making the assumption here that you can't use a real database, which would be the ideal solution)

The first thing I would recommend is to assign a unique key (ID) to each song, starting with 1, which, in code, should be represented with a long. Make this the first column in your database. For example, call it agazillionsongs_with_id.txt:

1 2008  50 Ashley Tisdale He Said, She Said
2 2008  123   Taylor Swift   Teardrops On My Guitar
3 2008  233   Finger Eleven  Paralyzer
4 2008  258   Paramore Misery Business
...
470  2007  251   Hannah Montana True Friend
471  2006  1  Beyonce  Irreplaceable
...

Now create additional sub-tables (indexes), in separate text files, each of which simply refer to the song's key. The simplest one is a year-index, which could be stored in a file called agazillionsongs_sub_year.txt:

2008 1
2008 2
2008 3
2008 4
...
2007 470
...
2006 471
...

Alternatively, you could store each year in this form (I prefer this format)

2008 1, 2, 3, 4, ...
2007 470, ...
2006 471, ...

Either way, this could be represented in code as a yearMap object, which is a Map<Integer,List<Song>>, where the value is a reference to the appropriate Song object. Once these maps are created, outputting them to file is easy.

Do this also with rank: agazillionsongs_sub_rank.txt / rankMap / Map<Integer,List<Song>>

Indexing by name and title is trickier--do you index the whole name, literally or with some concept of "fuzziness", or only the beginning, ...? It's a hard but important concept.

The farther you can take this idea, the more ways you slice-and-dice your data, the faster your users can query the database. This is because the need for reading through the full song-database each time, putting every line into a Song object, is eliminated.

Instead, this pre-processing allows you to know exactly which rows in the database need to be retrieved. So you can ignore all other lines and discard them.

I hope this helps you.