Fastest way to clear flowfile data
Have a workflow in which we are getting json files as a response of rest api. We are getting approximately 80k files in a session. Total size of all the files is 17GB. We need to save each file to file system. At the end of the process we have to wait for all the files to be present before sending success message.
Once we save the file in File System, we are calling notify+wait. But we dont need 15 gb data in flowfile anymore. So to release some space, can we use replaceText or ModifyByte to clear content. So that notify+wait runs smoothly. Total wait for this process is 4.30 hrs.
But process is taking too long in both (replaceText or ModifyByte) case.
Can Anyone please suggest the fastest way to clear flowfile data.
we do not need any attributes too. so is there a way I can abandon old flowfile and generate kb flowfile, midway?
what We want is like generateflowfile, but in middle, so for each of my existing flowfile, i can drop old one, and generate blank flowfile for notify and wait.