SymmetricDS how to filter rows during a transformation

Question

I am using SymmetricDS with many store nodes and a corp node.

I have to do a transformation sync between a table in the store node to a table in the corp node and successfully configured a transformation between them using sym_transform_table and sym_transform_column . Now all the rows of the source table is transformed or copied to the target table in the corp node. I want to prevent or filter some rows from the source table from moving to the target table. Any idea to achieve this.

Practical Scenario: Customer with a certain age (a subset of rows in the customer table) in a store have to be copy transformed to a master_customer table in corp node which have different column names.

Answer 1

Routers are used for filtering data. Instead of a default router use a lookup, subselect, or BSH router.

Lookup will be the fastest but requires creating a separate lookup table in your DB to reference which ages can be sent and which ones can't.

BSH will take a little bit longer but can handle logic like if (age < 65 && age >= 18).

Subselect router will take the longest since it will have to query the DB each time a record needs to be routed. This query can select a value from the DB to compare against or just be the same type of logic you could have put in the BSH router expression.

If your filtering depends on a value after your transform, make sure the transform occurs at the source node and not the target (TRANSFORM_POINT='EXTRACT'). If you want to filter before the transform, make sure the transform takes place at the target (TRANSFORM_POINT='LOAD'). If it doesn't matter, you probably want the transforms to happen at EXTRACT to offload the processing work from the server.

Answer 2

Take a look at load filters.

insert into sym_load_filter
    (LOAD_FILTER_ID, LOAD_FILTER_TYPE, SOURCE_NODE_GROUP_ID,
    TARGET_NODE_GROUP_ID, TARGET_CATALOG_NAME, TARGET_SCHEMA_NAME,
    TARGET_TABLE_NAME, FILTER_ON_UPDATE, FILTER_ON_INSERT, FILTER_ON_DELETE,
    BEFORE_WRITE_SCRIPT, AFTER_WRITE_SCRIPT, BATCH_COMPLETE_SCRIPT,
    BATCH_COMMIT_SCRIPT, BATCH_ROLLBACK_SCRIPT, HANDLE_ERROR_SCRIPT,
    CREATE_TIME, LAST_UPDATE_BY, LAST_UPDATE_TIME, LOAD_FILTER_ORDER,
    FAIL_ON_ERROR) values
    ('SampleFilter','BSH','Client','Server',NULL,NULL,
    'ITEM_SELLING_PRICE',1,1,1,'
    if (OLD_COST > COST) {
            // row will not be loaded
            return false
    }
            // row will be loaded
            return true
    }
    ',
    null,null,null,null,null,
    sysdate,'Documentaion',sysdate,1,1);