Waarp is an open source project with multiple subelements. There are related to each other, and they improve the development by being in several separated modules.
All are Open Source projects and fully in Java. The global license is GPL V3.
The new version is v3.5.0 and starting from v3.2, all modules are assemble into one single project with sub-modules (see https://github.com/waarp/Waarp-All).
From v3.3.3 version, one version fully compiled and assembled with Java 8 dependencies is available too in one jar (or dependency through its pom) (see https://github.com/waarp/Waarp-All-Jdk8).
Waarp is a project that provides among other an open source massive file transfer monitor in Java. Its goal is to unify several protocols (FTP, FTPS, SSH, HTTP and proprietary protocols) in an efficient and secured way. Its purpose is to enable bridging between several protocols and to enable dynamic pre or post action on transfer or other commands. Currently FTP(S) and efficient and secure R66 protocols are implemented. HTTP is supported but not yet in production release.
This project was developed initially for the French Ministry of Finance. The 3 main components (originally named GoldenGate, GoldenGate FTP and OpenR66) are in production in the Ministry since end of 2007. It is also in production in the French Gendarmerie Nationale since 2012 (see Crawd User). It is an Open Source project based on an independent community.
The company named Waarp propose professional services (contact through info 'at' waarp.fr). In addition to professional services, Waarp company is able to propose extensions to Waarp in order to centralize all configurations and events for a large installation, to provide easiness of exploitation, monitoring and therefore for the production. This software is based on Waarp standard API, with no proprietary or backdoor access. We commit to keep Waarp to be a full functional Open Source solution.
Waarp is composed of several parts, most of them could be used separately without the full package:
an open FTP(S) Server with an extension to serve as a gateway for R66: it can be used in a stand alone version to fit your needs.
an open proprietary file transfer monitor, R66 it can be used in a stand alone version to fit your needs.
a kernel for common actions (Waarp) that could be taken when merging any part of the above.
The start idea was to be able to receive through several protocols some files (or not files but data) and to transmit them into a system not directly accessible, but without breaking the global vision of the data transfer (where it is, what status, and so on...).
A standard FTP server (or SFTP or FTPS) do not allow to make post action (and of course neither pre action) when a file transfer occurs. So the reason of this new project to be able to imply business actions that can be whatever the need when an event occurs in the transfer protocol.
For instance, when a file is to be transmitted through an FTP service, perhaps some tests should be done to see if the next business step (another application) is ready to accept transfers. When a file is transmitted, an action can occur like for instance checking of integrity (business vision) or retransmission to another business partner (using eventually a different protocol). Also one possibly wants to follow any transfers (or other actions defined in the protocols) and to log them somewhere in its own format, either to just be able to look at what happened or even to restart some actions to correct wrong situation in production.
To be able to do all of that, we decided to implement the most common used file transfer protocols (FTP, HTTP) in such a way they can fulfill those behaviours. FTP was the first step. HTTP for now is simply a bunch of JSP and classes attached to this JSP. FTPS (implicit or explicit) is now supported.
Of course, if they can do all the stuff we present before, they can also do simple thing like standard FTP server.
Well, all of those protocols suffers from pitfalls like:
When a file is transferred, even if it is by block (FTP BLOCK mode), it does not have any control on each block transferred. So if something goes wrong on the network, it could happened that the sender or the receiver will not see what happens. The proprietary protocol will implement such a check on each transferred block.
When a file transferred ends up abnormally, it does not have true facility to enable to restart from where it falls. Even the FTP REST command is not enough since who knows at which step the error occurs. And asking for the size of the remote file could be not enough. The proprietary protocol will implement such a restart facility on failed transfer.
In production context, if you have to send many files, how do you follow those transfers and check them? Some FTP clients does such thing, however they generally don't have the hability to enable planned production. The proprietary protocol will implement such control, planning, and adding some functionalities like limiting (or not) the number of concurrent file transfers.
Of course, all of the behaviours we described before for GoldenGate will be implemented in this proprietary protocol.