WSO2 - Table created using Analytic Script Invisible in Gadget Generation Tool

Question

My use case: pushes data from a stream configured in the ESB to BAM and create a report using “Gadget Generation Tool”

Publishing the stream from ESB to BAM after adding an agent to the proxy service worked fine.

From the stream I created a table using the Analytics->Add screen and the table seems to persist as I am able to do a select and see results from the same screen.

Now I am trying to generate a Dashboard using the Gadget Generation Tool but the table is not available, though the jdbc connection is working fine but the table is nowhere:

Script for Analytic Table run from Analytics->Add screen

CREATE EXTERNAL TABLE IF NOT EXISTS CREDITTABLE(creditkey STRING, creditFlag STRING, version STRING)
  STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler'
  WITH SERDEPROPERTIES ( "cassandra.host" = "127.0.0.1" , 
    cassandra.port" = "9163" , "cassandra.ks.name" = "EVENT_KS" ,
    "cassandra.ks.username" = "admin" ,
    "cassandra.ks.password" = "admin" ,
    "cassandra.cf.name" = "firstStream" ,
    "cassandra.columns.mapping" = ":key,payload_k1-constant, Version" );

Tried looking for table in following databases:

jdbc:h2:repository/database/WSO2CARBON_DB;AUTO_SERVER=TRUE
jdbc:h2:repository/database/metastore_db;AUTO_SERVER=TRUE
jdbc:h2:repository/database/samples/BAM_STATS_DB;AUTO_SERVER=TRUE

Have not done any custom db configurations.

Answer 1

Why did I not see the table in Gadget Generation tool?

The table I have created using the Hive script is a Casandra Distributed database table and the reference I gave in the Gadget generation tool while looking up for the table were from the h2 RDBMS database table.

Below are the references to the h2 RDBMS databse which comes out of box with WSO2

jdbc:h2:repository/database/WSO2CARBON_DB;AUTO_SERVER=TRUE
jdbc:h2:repository/database/metastore_db;AUTO_SERVER=TRUE
jdbc:h2:repository/database/samples/BAM_STATS_DB;AUTO_SERVER=TRUE

Resolution ----- How to get tables listed in the Gadget Generation tool?

To get the tables listed in the Gadget Generation tool you have to extensively use the Hive Script to complete the following 3 steps:

1. Create a Hive table reference for the Casandra data stream to which data is pushed from ESB in my case.

   CREATE EXTERNAL TABLE IF NOT EXISTS CREDITTABLE( payload_creditkey STRING, payload_creditFlag STRING, payload_version STRING) STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler' WITH SERDEPROPERTIES ( "cassandra.host" = "127.0.0.1" , "cassandra.port" = "9163" , "cassandra.ks.name" = "EVENT_KS" , "cassandra.ks.username" = "admin" , "cassandra.ks.password" = "admin" , "cassandra.cf.name" = "firstStream" , "cassandra.columns.mapping" = ":key,payload_k1-constant, Version" );

2. Using Hive script create a H2 RDBMS script and reference to which I would be copying my data from the Casandra stream.

   CREATE EXTERNAL TABLE IF NOT EXISTS CREDITTABLEh2summary( creditFlg STRING, verSion STRING ) STORED BY 'org.wso2.carbon.hadoop.hive.jdbc.storage.JDBCStorageHandler' TBLPROPERTIES ( 'mapred.jdbc.driver.class' = 'org.h2.Driver' , 'mapred.jdbc.url' = 'jdbc:h2:C:/wso2bam-2.2.0/repository/samples/database/BAM_STATS_DB' , 'mapred.jdbc.username' = 'wso2carbon' , 'mapred.jdbc.password' = 'wso2carbon' , 'hive.jdbc.update.on.duplicate' = 'true' , 'hive.jdbc.primary.key.fields' = 'creditFlg' , 'hive.jdbc.table.create.query' = 'CREATE TABLE CREDITTABLE_newh2(creditFlg VARCHAR(100), version VARCHAR(100))' );

3. Write a Hive query using which data would be copied from Casandra to H2[RDBMS]

   insert overwrite table CREDITTABLEh2summary select a.payload_creditFlag,a.payload_version from CREDITTABLE a;

On doing this I was able to see the table in the Gadget Generation tool however I also had to chage the referenc to the H2 Database to absolute in the JDBC URL value that I passed.

Observation:

Was wondering if the Gadget generation tool can directly point to the Casandra Stream without having to copy the tables to a RDBMS database.

Answer 2

Did you try jdbc:h2:repository/database/samples/WSO2CARBON_DB;AUTO_SERVER=TRUE? Also, what you have pasted is the Cassandra Storage Definition, probably used for getting the input, not persisting the output. If you give the full hive query, that would help to figure out the problem more.