Waarp R66-Protocol

We describe here the logic of the protocol. It has been designed to be efficient, secured and to fullfill specific requirements of MFT and integration in IT.

 

A simplified picture is shown here:

openr66-protocol

First we describe the different value that can be found.

  • A Request (DbTaskRunner) can have several UpdatedInfo status:
    • UNKNOWN : no particular information on it.
    • NOTUPDATED : in used by other database object when they are taken into account.
    • INTERRUPTED : a request is interrupted but can be rescheduled.
    • TOSUBMIT : a request is proposed to be submited by the Commander (or other database object are supposed to be taken into account).
    • INERROR : a request is in error and can not be submited by the Commander until its status is changed explicitely.
    • RUNNING : a request is currently running.
    • DONE : a request is over and fully done.
  • A Request can have several Step values:
    • NOTASK : the request has never started.
    • PRETASK : the request is currently in Pre transfer step.
    • TRANSFERTASK : the request is currently in transfer step.
    • POSTTASK : the request is currently in Post transfer step (valid transfer only).
    • ALLDONETASK : the request is fully finished (UpdatedInfo is in DONE status too).
    • ERRORTASK : the request is currently in the Error step while an error occurs (either in PRE, TRANSFER or POST step).
  • A Request has two Step values:
    • GlobalStep : the current request step value
    • GlobalLastStep : this is the last valid request step value. When GlobalStep is in ERROR step, GlobalLastStep says in which step it was before entering in error. This information is used to enable restart of the Request from this valid last step.
  • Each Step values and UpdatedInfo has an ErrorCode detailed information:
    • InitOk : stands for correct initialization of the Request (startup and authentication)
    • PreProcessingOk, TransferOk, PostProcessingOk : stand for correct ending of the specified step
    • CompleteOk : stands for all action are correct (ALLDONE for step and DONE for UpdatedInfo)
    • Running : stands for current Step is in Running status.
    • StoppedTransfer, CanceledTransfer : stand for a Request where an action stopped or canceled the given Request. A Stopped Request can be restart from the current status. A Canceled Request starts from the beginning of the current step. For instance for the Transfer step, Stopped will imply that restart is from the current valid transferred block, while Cancel will imply to restart from the very beginning of the transfer (first block).
    • QueryAlreadyFinished : stands for special code where the Request is in fact remotely already finished and so can be finished locally.
    • Other codes specifies different kinds of error (NotKnownHost, Shutdown, RemoteError, ...): 
      • ConnectionImpossible : connection is impossible (local or remote)
      • BadAuthent : authentication is in error (local or remote)
      • ExternalOp : external operation is in error (pre, post or error actions)
      • TransferError : transfer is in error (for instance, uncorrect block)
      • MD5Error : transfer is in error due to a bad MD5 checking (in MD5 mode)
      • Disconnection : the network was disconnected
      • RemoteShutdown : the remote host is in shutdown
      • Shutdown : the local host is in shutdown
      • FinalOp : a final operation (not an external operation) is in error (for instance the finalization of the received file)
      • Unimplemented : the requested command is unimplemented (for instance, a task in a rule is unknown)
      • RemoteError : a remote error was received and stops the transfer
      • Internal : an internal error arrives
      • Warning : an external execution (pre or post operation) is executed well but with a warning (code 1)
      • Unknown : an unknown error arrives
      • QueryStillRunning : a required transfer during a restart is still in transfer
      • NotKnownHost : the remote host is unknown
      • QueryRemotelyUnknown : a required transfer during a restart or other commands should exist but is not identified on the remote host (for instance after a purge of log)
      • FileNotFound : the file is not found
      • CommandNotFound : the Command is not found 
      • PassThroughMode : the requested transfer (restart) is in PassThrough mode and is not compatible with the required action
      • IncorrectCommand : the command passed to the server is incorrect and cannot continue
      • FileNotAllowed : the File requested is not allowed (due to a path not allowed for instance)
      • SizeNotAllowed : the file size is not allowed (either because the size is out of the limit or because the working/recv directories cannot host this file due to insufficient space available)

A request of transfer follows a sequential logic:

 

Startup

Requester:

  1. The requester checks first if the given partner name is an alias, and if so, replaces it by the real host id.

  2. The request is registered in the database with a TOSUBMIT status.

  3. A Request that was in a INTERRUPTED status is changed in a TOSUBMIT status.

  4. The Commander get some requests with the TOSUBMIT status and makes them as RUNNING status.

  5. The Commander submits those requests as separates ClientRunners.

  6. The ClientRunner first checks that the given Request is not a "Self Requested" request, meaning that only requester host can execute a ClientRunner, except if this request was in the POSTTASK step so that Requested Host can finalize the request.

  7. The ClientRunner gets the remote requested Host address and tries to open the connection. If a network conection with the given requested host is already opened, this network connection is reused by the new ClientRunner.

  8. The connection can use SSL support (different port than non SSL). This is an option of transfer. This option is selected while selecting the Host ID for SSL support for the remote Requested Host. This option is CPU and Memory consuming.

Requester and Requested:

  1. Once the network connection is found, a private connection (in memory connection) is opened to enable the multiplexing of this request with other requests on the same network connection. This private connection is attached to a new LocalChannelReference which references the Request, the session, the remote and private connections. A valid LocalChannelReference contains two private connection Ids, one for the local private connection, and one for the remote private connection.

    1. A Startup Message is sent to the local private connection to initiate it.

    2. The same Startup Message is sent on the remote private side (when connection occurs) to initiate the relation between them and to instantiate the same LocalChannelReference on the remote host.

Authentication

Requester:

  1. Once the LocalChannelReference is OK, the Requester host sends an Authent Message in order to authenticate this host.

Requested:

  1. The Requested host sends back its own Authent Message too.

Request

Requester:

  1. Once authenticated, the Requester Host sends the Request Message.

Requested:

  1. The Requested Host check if the authentication and the request are compatibles, check some specific options on the request itself (start, restart, …), the status of the file if it is the sender, then it runs its Pre Task step

    • The Request is in PreProcessingOK as status.

Pre-Tasks

Requested:

  1. If the Pre Task step is ok, it sends back the validated Request Message to the Requester Host.

    • The Request is in InitOk as detailed information on requester side.

Requester:

  1. The Requester Host check if the authentication and the request are compatibles, check some specific options on the request itself (start, restart, …), the status of the file if it is the sender.

    • The Request is in InitOk.

  1. The validated Request is now running the Pre Task step.

    • Once finished, the Request is in PreProcessingOK status.

During pre-start, some preconditions could change (file name, file size, transfer information): if so, a special message is send in order to change the remote corresponding information.

 

Data Transfer

Sender:

  1. Now the transmission can start. From now on, this is the sender that leads the communication, no more the requester (it could be the same host however, depending on the way of transfer). The sender (which could be either the Requested or the Requester host) launch its own RetrieveRunner. This RetrieveRunner sends to the other host all DataBlock Messages.

  2. Each DataBlock Message can include a Hash control (MD5 or SHA or other) of the packet in it (option of transfer). This option is not mandatory and is CPU consuming.

  3. Once all DataBlock are sent, the RetrieveRunner sends an EndTransfer Message to the receiver host.

Receiver:

  1. For each DataBlock, the receiver appends this new block, checking the order of the block, and optionnaly the hash control.
  2. Once all blocks are received, the EndTransfer Message indicates the end of transmission from the Sender.

Post-Actions

Receiver:

  1. The receiver host executes the Post actions.

    • The Request is in PostProcesseingOk status.

  1. The receiver host sends back the validated EndTransfer Message.

    • The Request is in TransferOk status on sender side.

Sender:

  1. The Request is now on Finalize way. Sender host executes the Post actions.

    • The Request is in PostProcesseingOk status.

End Request

Sender:

  1. Once the PostProcessing is over, the RetrieveRunner (Sender) sends to the remote host a EndRequest Message.

Receiver:

  1. The remote host sends back the validated EndRequest Message.

Receiver and Sender:

  1. The Request is now totally finished and its status is CompleteOk ALLDONE.

 

At each step, an error can occurs and will stop the request, setting its UpdatedInfo to INERROR or INTERRUPTED status. The GlobalStep could be in ERROR status if the ERROR step action is run.