Building Resilent Services with ColdBox and Apache Nifi

Building Resilent Services with ColdBox and Apache Nifi

I had a hard time deciding the best title for this post. In the end, I went with the above title. I deal with a few 3rd parties at work and one of my main tasks are calling API endpoints and retrieving transaction data for processing. The number of API calls per vendor vary but there is one constant; the API calls need to complete successfully or try again if not.

It took a while, but I finally have a solution.

There are two main things required to make this work. 1) if the API does not complete successfully we need to set the status code accordingly, and 2) Nifi needs to handle unsuccessful requests and try them again.

The first step of the process is setting up the API in such a way that cfhttp will throw an exception if it encounters an error. Using thowOnError with cfhttp, the request will throw an exception that can be caught with try/catch if it is not successful.

   try{
    cfhttp(url = reportUrl, timeout = "45", throwonerror = "true");
    if(listToArray(trim(cfhttp.fileContent), chr(10)).len() > 5){
     // do stuff
    }
    else{
     // no data, do other stuff
    }
   }
   catch(any e){
    // errored out
    data['meta_data']['code'] = 500;
    data['meta_data']['message'] = e.message;
    data['meta_data']['status'] = "error";
    data['meta_data']['type'] = "error";
   }

The next step is setting up the Nifi process group. This flow represents one request:

GenerateFlowFile is cron scheduled to generate a FlowFile daily at 12:01am. The success relationship of GenerateFlowFile is flows into InvokeHTTP.

InvokeHTTP Processor Configuration

InvokeHTTP calls ColdFusion/ColdBox to process the API request. If the request is a Failure (the InvokeHTTP processor does not get a response before the request times out) or Retry (the API returns a 500 status code) the FlowFile is routed to the RetryFlowFile processor.

Next the RetryFlowFile processor needs to be configured:

Configure RetryFlowFile Settings Processor
Configure RetryFlowFile Properties

RetryFlowFile is configured to wait 30 seconds (the penalty duration) and then route the FlowFile to the retry relationship which points back to the InvokeHTTP processor to try it again. I also changed the Maximum Retry property to allow up to 10 retries.

If the moon and stars align, hopefully at least one request in 11 (the initial request and up to 10 retries) will complete. After the last retry it will just fail silently.

Good information on handling retries with Nifi is either very hard to find or my google-fu skills are getting rusty. At any rate, I would be very interested to hear how you are handling retries with Nifi.