Algorithm comparison in java: test program does not work

Question 1

Your implementation of Partition is not correct. Consider the small test below (I made Partition static for my convenience).

Both while loops won't be executed, because A[i] == A[j] == pivot. Moreover, i<j, so the two elements will be swapped, resulting in exactly the same array. Therefore, the outer while loop becomes infinite.

The same problem occurs for any array for which the first and last element are the same.

public class Test {
    public static void main(String[] args) {
        int[] A = {1, 1};
        Partition(A, 0, 1);
    }

    private static int Partition(int[] A, int left, int right){
        int pivot = A[right];
        int i = left;
        int j = right;

        while (true) {
            while ( (A[j] > pivot)) {
                j--;
            }
            while ((A[i] < pivot)) {
                i++;
            }
            if (i < j){
                int swap = A[j];
                A[j] = A[i];
                A[i] = swap;
            }else{
                return i;
            }
        }
    }
}

Question 2

Have you tried increasing memory settings for your code to run in eclipse. You may find this Setting memory of Java programs that runs from Eclipse helpful.

Question 3

Some tips / possible solution?:

I haven't read your implementation of QuickSort or HybridSort but I am assuming they are correct.

If you are comparing the performance of two algorithms you should most definitely compare their performance to indentical inputs. Currently you are generating two random arrys (albeit of the same size). This isn't necessarily going to be an accurate test as I can easily find a test case where one algorithm will outperform the other if the random generator is out to troll you.
Your logic for comparing the two algorithms is a bit weird and incorrect according to me. Why do you compare the lengths of the strings of the times? according to your logic 1 is the same as 9 and 1,000,000,000 is the same as 9,999,999,999 which is clearly incorrect. One algorithm is almost 10 times faster than the other.
Moreover, one reason for no output might be the reason that you are only outputing when hybridsort is better than quicksort and not the other way around. I am sure there are other reasons as well but this could be one easily noticable reason (if your implementations are incorrect).

I do notice that you close your outputstream which is good as that is a very common reason why there is no output. You should however, close steams in the finally section of the try-catch as then they are guaranteed to close. You could be getting an IOException and in your case this would also not close the outputsteam and consequently lead to no ouput in your file.

Here is a sample structure that I would follow for doing any comparitive testing. It is easy to read and easy to debug with enough output for you to figure out which algorithm performs better. This is merely a suggestion.

import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.PrintWriter;
import java.util.Random;

public class Tester {

  private static int[] initArray(int size) {
    Random rand = new Random();
    int[] arr = new int[size];
    for (int i = 0; i < arr.length; i++) {
      arr[i] = rand.nextInt();
    }
    return arr;
  }

  public static void main(String[] args) {
    final int MAX_ITERATIONS = 25;
    final int INITIAL_ARRAY_SIZE = 10;
    final int MAX_ARRAY_SIZE = 4500;
    final int ARRAY_SIZE_INCREMENT = 100;

    long start;
    int[] score = null;

    PrintWriter out = null;
    try {
      out = new PrintWriter(new FileOutputStream("out.txt"));
      for (int arraySize = INITIAL_ARRAY_SIZE; arraySize <= MAX_ARRAY_SIZE; arraySize += ARRAY_SIZE_INCREMENT) {
        // score[0] is for quickSort and score[1] is for hybridSort
        score = new int[2];
        for (int iteration = 0; iteration < MAX_ITERATIONS; iteration++) {
          int[] testArray = initArray(arraySize);
          int[] testArrayCopy = new int[arraySize];
          System.arraycopy(testArray, 0, testArrayCopy, 0, arraySize);

          start = System.nanoTime();
          // Do quicksort here using testArray
          long qSortfinish = System.nanoTime() - start;

          System.arraycopy(testArray, 0, testArrayCopy, 0, arraySize);
          start = System.nanoTime();
          // Do hybridsort here using testArrayCopy
          long hybridSortfinish = System.nanoTime() - start;

          // Keep score
          if (qSortfinish < hybridSortfinish)
            score[0]++;
          else if (qSortfinish > hybridSortfinish) {
            score[1]++;
          } else {
            score[0]++;
            score[1]++;
          }
        }
        out.println("Array Size: " + arraySize + " QuickSort: " + score[0] + " HybridSort: " + score[1]);
      }
    } catch (FileNotFoundException e) {
      e.printStackTrace();
    } finally {
      if (out != null)
        out.close();
    }
  }
}