In a typical high volume integration scenario most solutions take a scheduled script approach or SuiteTalk, which works fine. But, in one of the benchmark I discovered that RESTlet could achieve much better performance than that of Scheduled scripts or SuiteTalk. Read below to understand how. For this benchmark I considered a CSV file of about 8k cash sale transactions.The initial integration was performed by SuiteTalk which took about 8-10 hours of time, using Java Client, even though SuiteCloud Plus license parallel end points were used using Java Threads. I thought of using SuiteScripts, as SuiteScripts seems to be more native to NetSuite infrastructure and could save HTTPS and SOAP overhead. In this approach, I split the CSV files into chunks of 5MB using Java client and then uploaded the multiple CSV files into file cabinet using RESTlet 2.0.
Map-Reduce had an advantage of utilising all the 5 scheduled script queues of SuiteCloud Plus license and perform tasks in parallel. Once, file chunks were uploaded, map-reduce was kicked off, and script was designed in a way that each map phase picks up a separate file. This gave a speed of 30 records per minute, certainly there were signs of improvement, but, now, another problem turned up - Script started showing Instruction Count Limit Exceeded Error. This was because Map-Reduce runs multiple elements in the same execution context rather than loading different context for each map element, and same script code was running multiple times for large number of input data.
eg: if you return 20 files in input phase and shuffle phase distributes 4 files to each of 5 queues, then the 4 files in each queue will be in 5 execution context rather than 20 i.e. suitescript governance of each of 4 files set will be considered as one.RESTlets allows 10 parallel hits per user session. This has an added advantage that you can run scripts in parallel without having to go for costly SuiteCloud Plus License.
In this approach input CSV file was read in chunks asynchronously in Java, such that entire file was never into memory. Once, a chunk of CSV was read, it was transformed into JSON records from CSV data, and then it was asynchronously sent to RESTLet using HTTPS request with each request containing 8 JSON records. Since, HTTPS request were asynchronous, in the single thread we could send 9 concurrent requests to RESTlet. This approach gave us the speed of about 66 records per minute, which was much more faster than previous approaches and the entire job took only 2 hours for 8k records.
- Low memory usage:
File is read asynchronously in chunks and entire file is never loaded into memory.
- The asynchronous vert.x library of Java utilizes ideal time of CPU in parsing CSV data into JSON, while HTTPS requests are being made and file’s data chunks are being read, than doing context switching amongst threads, this ensured that Java client utilises the time and resources efficiently, and with single thread I was able to make 9 concurrent requests efficiently.
- Using concurrent sessions of RESTlet is useful to create records in parallel and at the same time saves the cost of costly SuiteCloud Plus license.
- As stated by NetSuite, SuiteScript 2.0 has better performance than 1.0.