World of Waarp

Open Source

Waarp is an open source project with multiple subelements. There are related to each other, and they improve the developement by being in several separated modules.

All are Open Source projects and fully in Java. The global license is GPL V3.

 

Production purpose

Waarp is a project that provides among other an open source massive file transfer monitor in Java. Its goal is to unify several protocols (FTP, FTPS, SSH, HTTP and proprietary protocols) in an efficient and secured way. Its purpose is to enable bridging between several protocols and to enable dynamic pre or post action on transfer or other commands. Currently FTP(S) and efficient and secure R66 protocols are implemented. HTTP is supported but not yet in production release.

 

This project was developped initially for the French Ministery of Finance. The 3 main components (originally named GoldenGate, GoldenGate FTP and OpenR66) are in production in the Ministery since end of 2007. It is also in production in the French Gendarmerie Nationale since 2012 (see Crawd User). It is an Open Source project based on an independent community.

 

Professional services

The company named Waarp propose professional services (contact through info 'at' waarp.it). In addition to professional services, Waarp company is able to propose extensions to Waarp in order to centralize all configurations and events for a large installation, to provide easiness of exploitation, monitoring and therefore for the production. This software is based on Waarp standard API, with no proprietary or backdoor access. We commit to keep Waarp to be a full functional Open Source solution.

 

waarp

 

What is Waarp?

Waarp is composed of several parts, most of them could be used separately without the full package:

  • an open FTP(S) Server with an extension to serve as a gateway for R66: it can be used in a stand alone version to fit your needs.

  • an open HTTP Server but only intend to be used within a Waarp gateway: this component is not yet publicly available.
  • an open proprietary file transfer monitor, R66 it can be used in a stand alone version to fit your needs.

  • a kernel for common actions (Waarp) that could be taken when merging any part of the above.

 

From where comes the idea of Waarp?

The start idea was to be able to receive through several protocols some files (or not files but data) and to transmit them into a system not directly accessible, but without breaking the global vision of the data transfer (where it is, what status, and so on...).

 

A standard FTP server (or SFTP or FTPS) do not allow to make post action (and of course neither pre action) when a file transfer occurs. So the reason of this new project to be able to imply business actions that can be whatever the need when an event occurs in the transfer protocol.

For instance, when a file is to be transmitted through an FTP service, perhaps some tests should be done to see if the next business step (another application) is ready to accept transfers. When a file is transmitted, an action can occur like for instance checking of integrity (business vision) or retransmission to another business partner (using eventually a different protocol). Also one possibly wants to follow any transfers (or other actions defined in the protocols) and to log them somewhere in its own format, either to just be able to look at what happened or even to restart some actions to correct wrong situation in production.

 

To be able to do all of that, we decided to implement the most common used file transfer protocols (FTP, HTTP) in such a way they can fulfill those behaviours. FTP was the first step. HTTP for now is simply a bunch of JSP and classes attached to this JSP. FTPS (implicit or explicit) is now supported.

Of course, if they can do all the stuff we present before, they can also do simple thing like standard FTP server.

 

Why a proprietary protocol therefore ?

Well, all of those protocols suffers from pitfalls like:

  • When a file is transfered, even if it is by block (FTP BLOCK mode), it does not have any control on each block transfered. So if something goes wrong on the network, it could happened that the sender or the receiver will not see what happens. The proprietary protocol will implement such a check on each transfered block.

  • When a file transfered ends up abnormally, it does not have true facility to enable to restart from where it falls. Even the FTP REST command is not enough since who knows at which step the error occurs. And asking for the size of the remote file could be not enough. The proprietary protocol will implement such a restart facility on failed transfer.

  • In production context, if you have to send many files, how do you follow those transfers and check them? Some FTP clients does such thing, however they generally don't have the hability to enable planned production. The proprietary protocol will implement such control, planning, and adding some functionalities like limitating (or not) the number of concurrent file transfers.

  • Of course, all of the behaviours we described before for GoldenGate will be implemented in this proprietary protocol.

What kind of proprietary protocol is it?

  • It is a proprietary protocol that mimic the CFT PESIT one plus some extension, including in the protocol itself (block checking and restarting), but also to enable specific functionalities like MD5 block control (optional), restart possibilities, duplexing network connection, SSL support, database support, ...
  • Note that Waarp protocol is not compatible with PESIT, but there are some facilities to setup a PESIT/Waarp gateway.
  • Finally Waarp protocol is proprietary in the sens it does not follow any standard ISO or whatever protocol, but it is open and freely implementable since it is documented and open source.