Knime too slow - performance

Question

I just started to use KNIME and it suppose managed a huge mount of data, but isn't, it's slow and often not response. I'll manage more data than that I'm using now, What am I doing wrong?. I set in my configuration file "knime.ini":

-XX:MaxPermSize=1024m
-Xmx2048m

I also read data from a database node (millions of rows) but I can't limit it by SQL (I don't really mind, I need this data).

SELECT * FROM foo LIMIT 1000

error:

WARN     Database Reader     com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'LIMIT 0' at line 1

Answer 1

see http://tech.knime.org/forum/knime-users/knime-performance-reading-from-a-database for the rest of this discussion and solutions...

Answer 2

I had the same issue... and was able to solve it really simply, KNIME has a KNIME.ini file, this one is like the paramethers KNIME uses to execute...

The real issue is that JBDC driver is set for 10 Fetch Size. By default, when Oracle JDBC runs a query, it retrieves a result set of 10 rows  at a time from the database cursor. This is the default Oracle row fetch size value... so whenever you are reading database you will have a big pain waiting to retrieve all the lines.

The fix is simply, go to the folder where KNIME is installed, look for the file KNIME.ini, open it and then add the following sentences to the bottom, it will override the defauld JBDC fetching, and then you will get the data in literally seconds.

-Dknime.database.fetchsize=50000 -Dknime.url.timeout=9000

Hope this helps :slight_smile:

Answer 3

I'm not sure if your question is about the performance problem or the SQL problem.

For the former, I had the same issue and only found a solution when I started searching for Eclipse performance fixes rather than KNIME performance fixes. It's true that increasing the Java heap size is a good thing to do, but my performance problem (and perhaps yours) was caused by something bad going on in the saved workspace metadata. Solution: Delete contents of the knime/workspace/.metadata directory.

As for the latter, not sure why you're getting that error; maybe try adding a semicolon at the end of the SQL statement.