Waarp R66-Command

Waarp R66 is mainly a set of commands,from the server, to the clients, through utilities.

In the following, commands are in fact Class in Java. To access them from shell, follow the link CommandLine.

 

org.waarp.openr66.server package:

  • R66Server

It launches the Waarp R66 Server using as unique argument the full XML configuration file. It loads the rest of the configuration (or update its configuration) from the database specified in the XML configuration file.

By default, three services are opened:

  • Non SSL file transfer monitor
  • SSL file transfer monitor
  • HTTP file transfer observer

By default, 5 subdirectories (under the main home) are defined:

  • Config directory where configurations files are stored
  • Archive directory where archive transfer can be put (using post actions) or where transfer logs are placed
  • Input directory where received files are stored (except if a post action moves it)
  • Output directory where to be sent files are stored (or linked)
  • Working directory where currently received files are temporarily placed

Utilities:

  • ServerInitDatabase

The first step in installation of an Waarp R66 Server or heavy client is to construct the XML configuration files and to initiate the database that support file transfers and configurations. This function is an helper for database initialisation.

  • The first argument is the JDBC database connection XML file.

From there any argument are optional. Any existing data will be replaced. No data are deleted if not replaced.

  • '-initdb' : It will try to create the database tables and to initiate the sequence (unique Id by host of runners from runner table).
  • '-loadBusiness businessConfig' : It will load the business configuration into the database HostConfig table.
  • '-loadAlias aliasConfig' : It will load the aliases configuration into the database HostConfig table.
  • '-loadRoles rolesConfig' : It will load the roles configuration into the database HostConfig table.
  • '-dir directory' : this argument is the directory from where to load rules into the database using the XML rule files present in this directory (rules table).
  • '-auth hostConfig' : this argument is to load the host authentication XML file into the database (hosts table).
  • '-limit limitConfig' : this argument is to load the bandwidth XML file into the database (configuration table).
  •  '-upgradeDb': this argument allow to try to upgrade the database schema (from version 2.4.17).

From that point, the Waarp R66 server (R66Server) can be started.

  • ServerExportConfiguration

One of the ideas of Waarp R66 is the ability to centralized some information asynchronously into a centralized database. This centralization can be in two ways:

  • Import new rules, host authentication or bandwidth limitation: this can be done using the ServerInitDatabase after some file transfers from a post operation which could execute this utility.
  • Export rules, hosts authentication and actual runners (done or not): this is the goal of this utility. It takes two arguments:
    • The client configuration XML file
    • The directory where to export files in XML format

It will export Rules, all present Runners, Host Authentications.

 

  • ConfigExport

This tool enables to ask the server remotely to export Rules and Hosts configuration into one file each.

It could be used before a retrieve of the configuration in a central repository.

Options are:

  • The first argument is the client XML configuration file
  • -hosts : export all Hosts configuration
  • -rules : export all Rules configuration
  • -business : export all Business configuration
  • -alias : export all Alias configuration
  • -role : export all Roles configuration
  • -host host : try to run this command over host (if allowed)
  • ConfigImport

This tool enables to ask the server remotely to import Rules and Hosts configuration from one file each.

It could be used before a upload the configuration from a central repository after transmission.

Options are:

  • The first argument is the client XML configuration file
  • -hosts : import all Hosts configuration
  • -purgehosts : purge all hosts configuration before importing new configuration
  • -rules : import all Rules configuration
  • -purgerules : purge all rules configuration before importing new configuration
  • -business file (if compatible)
  • -alias file (if compatible)
  • -roles file (if compatible)
  • -purgebusiness (if compatible)
  • -purgealias (if compatible)
  • -purgeroles (if compatible)
  • -hostid fileTransferId (if compatible, from a previous file transfer)
  • -ruleid fileTransferId (if compatible, from a previous file transfer)
  • -businessid fileTransferId (if compatible, from a previous file transfer)
  • -aliasid fileTransferId (if compatible, from a previous file transfer)
  • -roleid fileTransferId (if compatible, from a previous file transfer)
  • -host host (optional)
  • ServerShutdown

The Waarp R66 Server can be shutdown mainly in two ways:

  • Sending a -SIGTERM signal (or under Unix a -SIGUSR1 signal) to the JVM process hosting the OpenR66 Server.
  • The prefered way is to use the HTTPS Administrator interface.
  • Another prefered way is to use this utility which sends a shutdown request through network but using a key shared physically (adminkey). This utility takes the same XML configuration file than the server (where the admin key is refered but not stored in any database) and uses the SSL service.
  • If one wants to used the non SSL service, use the option -nossl
  • This command allows also to "block" or "unblock" new requests while leaving currently running requests to finish normally. To block, add the argument '-block', to unblock add the argument '-unblock'.

  • LogExport

If one wants to centralized in asynchronous way the transfer logs, this utility is made for it. Its purpose is to export logs into the archive directory and to eventually purge them from database. It sends a local request to the server, which really export the logs into a file. No file are transfered, they are just export to a file in the global archive directory. To use it, two main ways can be achieved:

  1. The local server runs this tool refering the OpenR66 Server using the client configuration XML file and then running a log request.
  2. Directly runs a transfer request where the rule execute at pre processing this utility (again either in receive or send mode according to the initiate server).
  • The First argument is the Client Configuration (eventually without database access).
  • The rest of arguments can be:
    • '-clean' option: Change all UpdatedInfo to Done where GlobalLastTask is ALLDONETASK and status is CompleteOk (sometimes some runners can be done but UpdatedInfo could be erroneous - no impact but clean function -).
    • '-purge' option: This option removes all ALLDONETASK from those that will be exported.
    • '-start' option: This option specifies the low limit to select runners from start runner time. If not specified, there is no low limit.
    • '-stop' option: This option specifies the upper limit to select runners from start runner time. If not specified, there is no upper limit.
  • LogExtendedExport 

This command is the same as LogExport but with more options

  • The First argument is the Client Configuration (eventually without database access).

  • The rest of arguments can be:

    • '-clean' option: Change all UpdatedInfo to Done where GlobalLastTask is ALLDONETASK and status is CompleteOk (sometimes some runners can be done but UpdatedInfo could be erroneous - no impact but clean function -).

    • '-purge' option: This option removes all ALLDONETASK from those that will be exported.

    • '-start' option: This option specifies the low limit to select runners from start runner time. If not specified, there is no low limit.

    • '-stop' option: This option specifies the upper limit to select runners from start runner time. If not specified, there is no upper limit.

    • -startid id: This option specifies the low limit for Transfer Id

    • -stopid id: This option specifies the upper limit for Transfer Id

    • -rule rule: This option specifies that logs will only concern this rule

    • -request host: This option specifies that logs will only concern this host

    • -pending: This option specifies that logs will only concern pending request

    • -transfer: This option specifies that logs will only concern in transfer request

    • -done: This option specifies that logs will only concern done (finished) request

    • -error: This option specifies that logs will only concern in error request

    • -host host: This options specifies a specific server to contact

      • Additional option: -ruleDownload rule : if set, will try to download the exported logs

      • Additional option: -import : if set, will try to import exported logs using ruleDownload

  • LogImport

    • The First argument is the Client Configuration with database support 

    • This command will import the XML logs into the database

  • ChangeBandwidthLimits

This tool enables to change dynamicaly the bandwidth limitations (only in memory, not in database). To use it, it should be used locally with the admin account.

  • The First argument is the Client Configuration (eventually without database access).

  • The rest of arguments can be:

    • '-wglob' option: Write Global limitation in Bytes by Second (minimum 1024 so 1KBs).

    • '-rglob' option: Read Global limitation in Bytes by Second (minimum 1024 so 1KBs).

    • '-wsess' option: Write Session limitation in Bytes by Second (minimum 1024 so 1KBs).

    • '-rsess' option: Read Session limitation in Bytes by Second (minimum 1024 so 1KBs).

org.waarp.openr66.client package:

 

All commands accept special extra arguments to format the output as desired:

  • -csv : output will be as one line for the title, one line for the data, all fields separated by ';'

  • -property : output will be one value per line, as name=value

  • -xml : output will be in XML format

  • -json : output will be in JSON format (default)

  • -quiet : no output will be done (only logging)

  • In addition, for success result, if -logWarn is set (default), the information will be logged using WARN level, while if -notlogWarn is set, the information will be logged using INFO level.

Wildcard characters are allowed ('*', '?') in submit or direct requests, with some point of attentions:

  • When using wildcard characters combined with DirectTransfer or SubmitTransfer, the command is in error if the result gives multiple files.
    • To enable multiple files resolution, use MultipleDirectTransfer or MultipleSubmitTransfer.

      • Note that on MultipleSubmitTransfer, if the request is a "RECV" request, you can specify the option "-cient" which allows the MultipleSubmitTransfer to run a RequestInformation first to the remote partner in order to get the list of remote files.

  • In some special cases, wildcard characters are badly interpreted (at Shell first level of interpretation), for instance as "*" , "*.*"  or "*xx" . In particular, the shell might replace immediately the value, which is not the desired result.

    • In order to allow a "non-interpreted" wildcard character, Waarp allows to use the char '§' in place of '*', so for the previous examples giving "%", "%.%" , "%xx". You can of course still continue to use '*' and '?'.

 

  • SubmitTransfer

To transfer a file, two main methods exist. This one is a submission request, so an asynchronous operation since once the request is submitted, the client returns without waiting for the end of the operation.

It takes the following argument:

  • The client XML configuration file as first argument, the one including database access
  • At least 3 other arguments are necessary:
    • '-to' option: specifies the remote Host Id (either the Id for SSL or not ).
    • '-file' option: specifies the file to transfer (either in receive or send mode).
    • '-rule' option: specifies the rule to apply (which specifies the transfer mode, the pre, post or error operations, ...).
  • Or at least 2 other arguments are necessary:

    • '-to' option: specifies the remote Host Id (either the Id for SSL or not ).

    • '-id' option: specifies the Id of a previous transfer (stopped or in error).

  • Other options are:
    • '-info' option: specifies the optional information that is send at the same time with the transfer request (extra information that could be needed by the remote host).
    • '-md5' option: specifies that each block transfer will be checked with a MD5 key. If the rule used is already in MD5 mode, this option will change nothing.
    • '-block' option: specifies the block size (default is 64 KB).
    • '-nolog' option: specifies that this transfer will not be logged (only on requester side).

    • '-start' "time start" as yyyyMMddHHmmss (override previous -delay options)

    • '-delay' "+delay in ms" as delay in ms from current time(override previous -start options)

    • '-delay' "delay in ms" as time in ms (override previous -start options)

  • MultipleSubmitTransfer

    • This function is the same than SubmitTransfer except that multiple hosts and multiples files could be specifed using a comma (',') as separator to -file and -to arguments.
    • -client : If the Rule is a RECV method, passing the -client additionnal option allows the function to contact the remote servers in case wildcards are used (?*§~).
  • DirectTransfer

This is the second method to transfer a file. This method is direct, so as a synchronous operation. This time the client will do the real work (transfering the file). The options are exactly the same than with SubmitTransfer. However two cases exist:

  1. Heavy client: The client XML configuration file includes the database access. All transfer operations will be logged (eventually deleted at the end from the client side if the '-nolog' option is set). This option is useful for "production" clients in a data center.
  2. Light client: The client XML configuration file does not included the database access. All transfer operations will not be logged at all (at client side). This option is useful for "light client" like personal computers where transfering files to or from a data center is a necessity.

However, both clients can only be the initiator of the transfer (receive or send), since no service is running once the client is over.

  • MultipleDirectTransfer

    • This function is the same than DirectTransfer except that multiple hosts and multiples files could be specifed using a comma (',') as separator to -file and -to arguments.

  • SpooledDirectoryTransfer

    This method (runs a service, even using Windows or Unix services) allows to spool a directory to submit transfers or directly transferred files to remote partners. It is intend as an helper. Note however that in many cases, it is preferable to directly request a file transfer from the application that has produced the file, and not relying on a directory scan (this method could be wrong with special situations like slow writes to disk).

    • The method consider a file as acceptable once

      • the file is discovered at iteration 1,

      • modification time and size not changed at iteration 2,

      • and finally hash unchanged at iteration 3.

    • The statusFile keeps the current status on disk (in case of crash, or restart of the process), in order that a file is not send twice. However, to ensure this is not the case, we encourage that the rule used a POST-SEND task that will move or eraze this file, such that it will be removed from the track list.

    • The SpooledDirectoryTransfer will also check at startup if another daemon is already running with the very same status file (which is not allowed). If so, the new daemon is immediately stopped.

Note : for the SpooledDirectoryTransfer service to be able to inform the Waarp server of its status, you must allow explicitely this client to use "Business" actions. This can be done by allowing this using the Business XML file and the ConfigImport -business file or ServerInitDatabase -loadBusiness file, or by adding directly this in the "Business Configuration:" field of the System page of the Web Administrator Waarp page with a content as in the Business XML file:

<business><businessid>hosta</businessid></business>

To launch the SpooledDirectoryTransfer, 2 options exist:

A) Through service (for Windows) by setting the Spooled program as a service (have a look in the example/Windows/servicespooled/ directory - using the Apache Commons Daemon)

B) Through command line (Windows or Linux - the last might be setup as a service too) by starting the command R66SPOOLEDSEND with the necessary arguments

 

The arguments might be setup in 2 ways:

  1. Using a standard client XML configuration file as first argument, using database access is recommended (No Database support might be ok if using -direct mode) plus extra arguments as command line (or for Windows service putting those arguments in the spooled.conf file - see example)
  2. Using an extended client XML configuration file as unique argument (or throufg the spooled.conf as unique variable setup), which enable multiple SpooledDirectoryTransfer monitor running in one unique JVM, but with the lack of differentiation in term of stopping the spooled dameons (all stopping at the same time, all starting at the same time). The XML file is described in one XSD and one JAXE format as OpenR66-SpooledDirectory file in xample directory.
    • The client XML configuration file as first argument, the one including database access
    • '-to remoteHost' : single or multiple with a comma separated list
    • '-directory namedir' : the directory to spool ; namedir could be a comma separated list too, in order to scan several directories in one and unique SpooledDirectoryTransfer service
    • '-statusfile file' : the file (fullpath) to use as permanent status - if process is killed or aborts -. Note this file must be unique per SpooledDirectoryTransfer instance.
    • '-stopfile file' : the file (fullpath) to use when its existence will stop the daemon
    • '-rule rule' : the rule to use for the transfers
    • Options:

      • '-info' option: specifies the optional information that is send at the same time with the transfer request (extra information that could be needed by the remote host).

      • '-md5' option: specifies that each block transfer will be checked with a MD5 key. If the rule used is already in MD5 mode, this option will change nothing.

      • '-block' option: specifies the block size (default is 64 KB).

      • '-nolog' option: specifies that this transfer will not be logged (only on requester side).

      • '-regex regex': regular expression to filter file names from directory source

      • '-elapse elapse': elapse time bewteen 2 checks of the directory in milliseconds (default 1000 ms)

      • '-submit' (default) OR '-direct': to submit only or to directly transfer only

      • '-recursive' allows the service to scan also subdirectories

      • '-name' allows to specify explicitely a name (used in conjonction with the following -waarp option as the name of the remote SpooledDirectory service), instead of the name of the client and of the directories (default). The name should be globally unique (per Waarp server listed in -waarp option).

      • '-waarp remoteHosts' : single or multiple (comma separated list) of Waarp servers that will be informed on the current status.
        • Note that those informations are only saved in memory on Waarp Servers side. It allows to quickly see the status under the Web Https Administrator interface, from Transfer element (Spooled.html).
        • The page shows a status of how much the status is fresh (green to orange to red), which could implies that the SpooledDirectory is stopped or broken.
        • If the '-parallel' option is specified, the information will be sent to the Waarp servers at fix interval (potentially set by -elapseWaarp option). On the contrary (-sequential), the information will be sent only at the end of a full check and send operation (-elapse time plus time to check and send).
        • '-parallel' or '-sequential' to allow parallism between send actions and information to Waarp server 
        • '-elapseWaarp elapse': elapse time between 2 sends to Waarp servers (default 5000ms). If less than 1000 ms, will be run sequentially after each check and not in parallel (could lead to great elapse time if a lot of files or big files must be sent in one step).
      • '-limitParallel limit' to specify the number of concurrent actions (send) in -direct mode only: It is recommended to set it to a low number, as 1. 
      • '-minimalSize limit' to specify the minimal size of each file that will be transferred (default: no limit)

      • '-notlogWarn' | '-logWarn' to deactivate or activate (default) the logging in Warn mode of Send/Remove information of the spool

  • SendThroughClient

This method is not a full implemented method. It is a way to route a file transfer from one protocol in Java to OpenR66. For instance, if a protocol like HTTP upload or FTP upload allows to get a file upload by packet, you can then route this file transfer through Waarp R66 to a final Waarp R66 Server using the protocol but without writing an intermediary file (directly write the bytes from the upload to the Waarp R66 SendThroughClient interface). A simple example is shown in TestSendThroughClient.

So this method needs some minor developments to be implemented.

  • RecvThroughClient

This method is not a full implemented method. It is a way to route a file transfer to one protocol in Java to OpenR66. For instance, if a protocol like HTTP or FTP download allows to get a file downloaded by packet, you can then route directly the file transfer from Waarp R66 from a remote Waarp R66 Server using the protocol directly but without writing an intermediary file (directly write the bytes from the download from the Waarp R66 RecvThroughHandler interface). A simple example is shown in TestRecvThroughClient.

So this method needs some minor developments to be implemented.

  • ProgressBarTransfer

This method is not a full implemented method. It is a way to implement a file transfer within a Graphical User Interface or to get information on progression during the transfer. Such an example is presented in the R66GUI.

To be used, this method needs some minor developments to be implemented, according to business needs.
  • RequestTransfer

This utility is used to get information for a specific runner or to have an action on this runner.

The arguments are the following:

  • The first argument is the client XML configuration file including the database access.
  • '-id' option: this is the Runner Id.
  • '-to' or '-from' option (exclusive): this specifies the way of the request transfer. '-to' specifies that the original request was initiated by the current running host to the remote specified host. '-from' specifies that the original request was initiated by the remote specifed host.
  • Optional arguments (exclusive): without any of those arguments, the request only returns the current information of the runner.
    • '-cancel' option: the runner will be canceled. Any file currently in writing will be deleted.
    • '-stop' option: the runner will be stopped (but not canceled).
    • '-restart' option: the runner will be restart (if stopped). Optionnally the following options can be applied in addition:

      • '-start' "time start" as yyyyMMddHHmmss (override previous -delay options)

      • '-delay' "+delay in ms" as delay in ms from current time(override previous -start options)

      • '-delay' "delay in ms" as time in ms (override previous -start options)

  • RequestInformation

This utility is used to get information for a file, a directory, with or without wildcard characters ('*' and '?").

The arguments are the following:

  • The first argument is the client XML configuration file including the database access.

  • '-to' option: specifies the requested host.

  • '-rule' option: the rule

  • '-file' the optional file for which to get info (may contain wildcard characters)

Optional arguments (exclusive): without any of those arguments, the request only returns the existence of the file or directory.

  • '-exist' to test the existence.

  • '-detail' to get the detail on file.

  • '-list' to get the list of files.

  • '-mlsx' to get the list and details of files

  • Message

This utility is used to send a short message to another server (like a ping in Waarp R66 protocol)

The arguments are the following:

  • The first argument is the client XML configuration file
  • '-to' option: specifies the requested host.
  • '-msg' option: the msg to send