Is there a way to make ExecuteSQL processor output in multiple flowfiles?

AnuragAnurag
in Apache-Nifi

We Have Join multiple tables using NiFi. The datasource can be MySQL or RedShift maybe something else in future. Currently, we are using ExecuteSQL processor but output is in a Single flowfile. Now, for terabyte of data, this may not be suitable. So we tried generateTableFetch but this does not have join option.

  1. do we have any alternative for ExecuteSQL processor?
  2. Is there a way to make ExecuteSQL processor output in multiple flowfiles? Currently I can split the output of ExecuteSQL using SplitAvro processor. But I want ExecuteSQL itself splitting the output
  3. GenerateTableFetch generates SQL queries based on offset. Will this slows down the process when the dataset becomes larger?
Tagged:
Share on Google+
Sign In or Register to comment.