How to avoid manually writing/managing SQL

Question

My team and I are rapidly developing an Webapp backed by an Oracle DB. We use maven's plugin flyway to manage our db creation and population from INSERT SQL scripts. Typically we add 3-4 tables per sprint and / or modify the existing tables structure.

We model the schema in an external tool that generates the schema including the constraints and run this in first followed by the SQL INSERTs to ensure the integrity of all the data.

We spend too much time managing the changes to the SQL to cover the new tables - by this I mean adding the extra column data to the existing SQL INSERT statements not to mention the manual creation of the new SQL INSERT data particularly when they reference a foreign key.

Surely there is another way, maybe maintaining raw data in Excel and passing this through a parser to the DB. Has anyone any ideas?

10 tables so far and up to 1000 SQL statements, DB is not live so we tear it down on every build.

Thanks

Edit: The inserted data is static reference data the platform depends on to function - menus etc. The architecture is Tomcat, JSF, Spring, JPA, Oracle

Answer 1

Please store your raw data in tables in the database - hey! why on earth do you want to use Excel for this? You have Oracle Database - the best tool for the job!

Load your unpolished data using SQL*Loader or external tables into regular tables in the database. From there you have SQL - the most powerful rdbms tool to manipulate your data. NEVER do slow by slow inserts. (1000 sql statements). Please do CTAS. Add/enable the constraints AFTER you have loaded all the data.

create table t as select * from raw_data; 

or 

insert into t (x,y,z) select x,y,z from raw_data;

Using this method, you can bypass the SQL engine and do direct inserts (direct path load). This can even be done in parallel to make your data go into the database superfast!

Do all of your data manipulation in SQL or PLSQL. (Not in the application)

• Please invest time learning the Oracle Database. It is full of features for you to use!
• Don't just use it like a datadump (a place where you store your data). Create packages - interfaces to your application - your API to the database.
• Don't just throw around thousands of statements compiled into your application. It will get messy.
• Build your business logic inside the database PLSQL - use your application for presentation.

Best of luck!

Answer 2

Alternatively, you also have the option to implement a Java migration. It could read whatever input data you have (Excel, csv, ...) and do the proper inserts.