gLite Data Transfer Agents Module Install Guide

In order to use the gLite File Transfer Agent, you need the following external dependencies installed on the system:

To install that libraries, you can use the following RPMs from the GLite repository:

except from the oracle instant client package (oracle-instantclient-basic-10.1.0.3-1.i386.rpm or greater) that should be downloaded from the Oracle web site.

The locations where those modules are installed are:

In order to install the GLite GLite Data Transfer Agent, you need to download from the GLite repository the RPMs of the appropiate version of the following modules

You can then install those RPMs issuing the command

rpm -ivh <rpm_list> 

here <rpm_list> is the list of RPMS of the modules reported above.

The default location where those modules are installed is /opt/glite.

Create the following directories under the glite location directory (assuming /opt/glite):

mkdir /opt/glite/etc
mkdir /var/log/glite
mkdir /var/glite/lock
mkdir /var/glite/tmp

The VO agents are intended to be installed one for each VO, you need to create a pair of the configuration files for each VO. The same also apply for a Channel Agent: you need an agent for each Channel you want to mange. At this purpose, you can run the GLite Data ConfigGenerator tool, passing the proper configuration template provided by the org.glite.data.transfer-agents module and installed in /opt/glite/share/config/glite-data-transfer-agents. In order to configure a GLite Data Transfer VO Agent, for example, you need to run:

export GLITE_LOCATION=/opt/glite
export LD_LIBRARY_PATH=$GLITE_LOCATION/externals/lib:\
   $GLITE_LOCATION/lib:$LD_LIBRARY_PATH

/opt/glite/bin/glite_data_config_generator \
 -f /opt/glite/share/config/glite-data-transfer-agents/glite-transfer-vo-agent-<type>-<dao>.config.xml \
 -o /opt/glite/etc/glite-transfer-vo-agent-<VO_NAME>.properties.xml \
 -l /opt/glite/etc/glite-transfer-vo-agent-<VO_NAME>.log-properties

where <dao> is the name of the DAO plugin to use (can be "mysql" or "oracle"), <type> is the chosen deployment model ("fts" or "fps") and <VO_NAME> is the name of the Virtual Organization the agent will serve. For more details concerning the deployment scenarios (File Transfer Service or File Placement Service), please have a look to the README file.

The output destination must be either $GLITE_LOCATION_USER/etc, $GLITE_LOCATION/etc or $GLITE_LOCATION_VAR/etc.

All you need to doi is to enter the value of the configuration parameters that you want to put into the configuration. If you wnat to simplify the configuration, don't switch to the advanced mode and fill just the mandatory parameters, leaving the default value for all the others.

In order to create the cofniguration for a Channel agent, you need to run:

/opt/glite/bin/glite_data_config_generator \
 -f /opt/glite/share/config/glite-data-transfer-agents/glite-transfer-channel-agent-<dao>.config.xml \
 -o /opt/glite/etc/glite-transfer-channel-agent-<CHANNEL_NAME>.properties.xml \
 -l /opt/glite/etc/glite-transfer-channel-agent-<CHANNEL_NAME>.log-properties

where <dao> is the name of the DAO plugin to use (can be "mysql" or "oracle"), and <CHANNEL_NAME> is the name of the Channel the agent is responsible for.

In order to help the user in generating the configuration files, two additional templates have been provided with just the properties required by the supported transfer service

/opt/glite/bin/glite_data_config_generator \
 -f /opt/glite/share/config/glite-data-transfer-agents/glite-transfer-channel-agent-<ts>-<dao>.config.xml \
 -o /opt/glite/etc/glite-transfer-channel-agent-<CHANNEL_NAME>.properties.xml \
 -l /opt/glite/etc/glite-transfer-channel-agent-<CHANNEL_NAME>.log-properties

where <ts> is the name of the Urlcopy Transfer Type configuration (can be "urlcopy" or "srmcopy").

You can also run the glite_data_config_generator tool without user interaction. To do that, you need to use the -p option and provide property file (key-value pairs) with the values you want to assign to the configuration parameters.

/opt/glite/bin/glite_data_config_generator \
 -f /opt/glite/share/config/glite-data-transfer-agents/glite-transfer-vo-agent-<type>-<dao>.config.xml \
 -o /opt/glite/etc/glite-transfer-vo-agent-<VO_NAME>.properties.xml \
 -l /opt/glite/etc/glite-transfer-vo-agent-<VO_NAME>.log-properties
 -p <glite-transfer-vo-agent-<type>-<VO_NAME>.properties>

where <glite-transfer-vo-agent-<type>-<VO_NAME>.properties> is the property file you have to pass.

For a Channel Agent, you need to run:

/opt/glite/bin/glite_data_config_generator \
 -f /opt/glite/share/config/glite-data-transfer-agents/glite-transfer-channel-agent-<dao>.config.xml \
 -o /opt/glite/etc/glite-transfer-channel-agent-<CHANNEL_NAME>.properties.xml \
 -l /opt/glite/etc/glite-transfer-channel-agent-<CHANNEL_NAME>.log-properties
 -p <glite-transfer-channel-agent-<CHANNEL_NAME>.properties>

The structure of that property file requires that the key should be composed by the name of the component, as reported on the config template file, followed by the property name (separeted by a "."), and then "=" and the value to be assigned.

<component_1_name>.<property_a_name> = <property_a_value>
<component_1_name>.<property_b_name> = <property_b_value>
...
<component_n_name>.<property_x_name> = <property_x_value>
log.FileName         = <log_file_destination>
log.Priority         = <log_priority>

("log" is a reserverd key to generate the logging configuration file)

If this option is used, rember that the property file should set all the mandatory properties, otherwise an error is returned.

From the org.glite.data.config-service version 1.1.10 it would be possible to provides multiple properties files as input parameters (specifying multiple -p options), as well as specifying directly the value of some parameters on the command line (using the -O option). These two methods can also be combined. You can then think to configure an agent in a similar way:

/opt/glite/bin/glite_data_config_generator \
 -f /opt/glite/share/config/glite-data-transfer-agents/glite-transfer-channel-agent-<dao>.config.xml \
 -o /opt/glite/etc/glite-transfer-channel-agent-<CHANNEL_NAME>.properties.xml \
 -l /opt/glite/etc/glite-transfer-channel-agent-<CHANNEL_NAME>.log-properties
 -p db_connection.properties -p timeout.properties 
 -O transfer-channel-agent.Name=<CHANNEL_NAME> 
 -O transfer-channel-agent.Contact=<your_email_address> 

The properties files can then be reused for installing different agent instances.

Since the FTA configuration files (/opt/glite/etc/glite-transfer-vo-agent-<VO_NAME>.properties.xml and /opt/glite/etc/glite-transfer-channel-agent-<CHANNEL_NAME>.properties.xml) may contain some sensible information (like the user and the password to connect to the DB server), it's recommended that you change the privileges of these files so they can be read just by the account that is used to run the agent.

Before running the daemon, you need to set up the environment variables GLITE_LOCATION_USER, GLITE_LOCATION, GLITE_LOCATION_VAR and LD_LIBRARY_PATH. You can do that either running the command illustrated above, creating a script or setting them in your shell profile. Remember to include GLITE_LOCATION/lib into the LD_LIBRARY_PATH. If Oracle is used as Data Source, the Oracle client library location should also be listed in LD_LIBRARY_PATH.

In addition, GLITE_LOCATION_USER, GLITE_LOCATION and GLITE_LOCATION_VAR could be set in the ~/.glite.conf or /etc/glite.conf and described by the GLite Developer's Guide.

Once all the previous steps are completed, you can run the daemon by using the following command (always with the service account credentials):

/opt/glite/bin/glite_data_config_service glite-transfer-vo-agent-<vo_name> start

or

/opt/glite/bin/glite_data_config_service glite-transfer-channel-agent-<channel_name> start

where <vo_name> is the name of the VO which the agent belongs to and <channel_name> is the name of the channel which the agent is responsible for.

Since the FTA will try to contect MyProxy in order to get the client delegated credentials for performing the transfer, contacting the ServiceDiscovery client and the Catalog, the use of a service/host certificate is compulsory.

The File Transfer Agent make use of the ServiceDiscovery in order to find the appropriate endpoint for the Catalog, the Storage Element and the MyPRoxy Server (if not configured). Moreover, the information returned by ServiceDiscovery is used to detect the site involved on a transfer, in order to properly perform the channel allocation. This then requires that the ServiceDiscovery environement is properly set in order to include the information concerning all the Catalog and SRM Servers that could be involved in the transfer and replication, i.e. in all the sites reachables through one of the available channels.

Since it's foreseen that different backends could be used as information system (file, RGMA, DBII), please refer to the ServiceDiscovery documentation to setup such environment.

For what concern the properties that the ServiceDiscovery should provide, the FTA requires that the information system contains the entries related to at least one Fireman catalog (service type is org.glite.FiremanCatalog) (only in case of FPS) and to any possible SRMs (service type SRM) that could be involved in a transfer. The SRM service should specify an additional property (<VO_NAME>:SEMountPoint) that specify the parent path where the files would be stored. The <VO_NAME> prefix is used in order to allow to use a different mount point for each VOs. In case this parameter is the same for all the VO the property can be named simply SEMountPoint.

All of these types are configurables as advanced parameters of the servicediscovery client module.

An example of a configuration for the file based ServiceDiscovery is:

<?xml version="1.0" encoding="UTF-8"?>
<services>
  <service name="CERN-Fireman">
    <parameters>
      <endpoint>https://lxb2028.cern.ch:8443/EGEE/glite-data-catalog-service-fr/services/FiremanCatalog</endpoint>
      <type>org.glite.FiremanCatalog</type>
      <version>2.0.0</version>
      <site>cern.ch</site>
    </parameters>
  </service>
  <service name="INFN-SRM">
    <parameters>
      <endpoint>httpg://griftp.infn.it:8443/srm/managerv1</endpoint>
      <type>SRM</type>
      <version>1.1.0</version>
      <site>infn.it</site>
      <param name="dteam:SEMountPoint">/castor/infn.it/grid/dteam/</param>
      <param name="alice:SEMountPoint">/castor/infn.it/grid/alice/</param>
      <param name="SEMountPoint">/castor/infn.it/tmp/</param>
    </parameters>
  </service>
  <service name="CERN-SRM">
    <parameters>
      <endpoint>httpg://griftp05.cern.ch:8443/srm/managerv1</endpoint>
      <type>SRM</type>
      <version>1.1.0</version>
      <site>cern.ch</site>
      <param name="SEMountPoint">/castor/cern.ch/user/g/gproduct/EGEETEST/SE/</param>
    </parameters>
  </service>
</services>