Creating a thread for file transfer

I am creating an application in Java that runs at scheduled intervals and it transfer files from one server to another server.

For SFTP I'm using jSch, and my server and file details came from Database.

My code is working fine but its performance is not good because I'm using too many loops in my code.

Is there any way to increase performance of my code?

```
public class FileTransferThread implements Runnable {

public static final Logger log = Logger.getLogger(FileTransferThread.class.getName());
private Session hibernateSession_source;
private Session hibernateSession_destination;
private List ruleObjList = new ArrayList<>();
private Map> filesMap = new HashMap<>();
private int i = 1;

@Override
public void run() {
try {
hibernateSession_destination = HibernateUtilReports.INSTANCE.getSession();
// Getting Active rules from (nr_rec_backup_rule)
Criteria ruleCriteria = hibernateSession_destination.createCriteria(nr_rec_backup_rule.class);
ruleCriteria.add(Restrictions.eq("status", "active"));
List list = ruleCriteria.list();
for (Object object : list) {
nr_rec_backup_rule ruleObj = (nr_rec_backup_rule) object;
ruleObjList.add(ruleObj);
}

System.out.println("List of Rule Objs : " + ruleObjList);
getTargetServerAuthentication();

} catch (Exception e) {
log.error("SQL ERROR ======== ", e);
} finally {
hibernateSession_destination.flush();
hibernateSession_destination.close();
hibernateSession_source.flush();
hibernateSession_source.close();
}
}

private void getTargetServerAuthentication() throws Exception {
if (ruleObjList.size() > 0) {

JSch jsch = new JSch();
hibernateSession_source = HibernateUtilSpice.INSTANCE.getSession();

for (nr_rec_backup_rule ruleObj : ruleObjList) {

//getting authentication details for backupserver from table "contaque_servers"

At face value it appears that there can be only one place where the major bottleneck is: the actual file transfer. Your code does the following:

• builds up a bunch of source files to copy

• creates a 'target' destination for the file copy

• goes through each source

• for each source, it 'downloads' the files one at a time

• as it downloads each file, it uploads it to the target.

While this whole task may be running in a separate thread, it is by no means multi-threaded.

The probable bottleneck here is the amount of CPU time required to decrypt the data from the source, and re-encrypt it to the destination.

It is likely, also, that close behind the CPU bottleneck (perhaps even in front of it) is the network transfer speeds you can get in a single socket connection.

I would suggest four things to do, and possibly a combination of them:

• try to set up a system where you can sfp direct from the source to the destination without needing to process the file in between. You have ssh access to them both, so it should not be that hard to create a script on the source, and run that script with some parameters that copies the file to the destination.

• Use BlowFish encryption algorithm for the transfer. It is rumoured that blowfish is faster than the other algorithms, and, by the sounds of it, it should be fine for your use case.

• Wrap the InputStream you get from jsch in a BufferedInputStream

• spread the load of the decrypt/encrypt on multiple threads.

The most effective option will be 1, but the most fun to write will be 4....

Something like:

-
create a method that takes the details required to copy a single file....

public Boolean copyFile(Session source, Session target, String sourcefile, String targetfile) throws IOException {
     // connect to the source
     .....
     // connect to the target
     .....
     // get a BufferedInputStream on the source
     .....
     // copy the stream to the target
     .....
     return Boolean.TRUE; // success.
}

-
instead of populating a filesMap Map, do something like:

ExecutorService threadpool = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors());
List> transfers = new ArrayList<>();
....
    final Session source = ......;
    final Session target = ......;
    final String sourcefile = ....;
    final String targetfile = ....;
    transfers.add(threadpool.submit(new Callable() {
        public Boolean call() throws IOException {
            return copyFile(source, target, sourcefile, targetfile);
        }
    });

....
// all copy actions are submitted now... so we wait for the threadpool.
threadpool.shutdown(); // orderly shutdown, all tasks are completed.
for (Future fut : transfers) {
    try {
        fut.get();
    } catch (Exception ioe) {
        LOGGER.warn("Unable to transfer file: " + ioe.getMessage(), ioe);
    }
}
// all copies have been attempted, in parallel.

StackExchange Code Review Q#41041, answer score: 6