dCache installations, used as a frontend to tertiary storage system, need, at some point, to exchange data this such a system in order to store new, precious files and to retrieve files from the HSM if not yet, or no longer, available on one of the dCache pools. Unfortunately there is no well defined interface for such HSM operations. So the dCache overcomes this problem by calling configurable (dCache external) shell scripts or binaries whenever an HSM store or retrieve operation becomes necessary. The local HSM administrator is responsible for providing this procedure and to make it available and known to the dCache. This small writeup defines the way dCache will call such an external method.Defining the HSM interface.
Each individual pool, which is expected to exchange data with an HSM, has to define a dCache external method to flush/fetch datasets into/from one or more connected HSM's. The command decribed below has either to be given in the command line interface of the corresponding pool while the pool is active (don't forget so 'save') or may be added to the pool setup file commands prior to starting the pool.Syntax : hsm set <hsmName> -command=<fullPathToExternalCommand> Example : hsm set osm -command=/usr/d-cache/jobs/osm-hsmcp.sh or hsm set enstore -command=/usr/d-cache-deployment/jobs/real-enstore.shThe external method, which might be a shell script or a binary, is called by the dCache with a set of positional arguments (see below). In addition, options may be specified which are appended to the regular argument list on calling the external method.Syntax : hsm set <hsmName> -<key>=<value> Example : hsm set osm -command=/usr/d-cache/jobs/osm-hsmcp.sh hsm set osm -pnfs=/pnfs/desy.de -somethingElse=trueThis will result in excuting the following command line whenever a file has to be exchanged with an HSM./usr/d-cache/jobs/osm-hsmcp.sh put|get <pnfsId> <LocalFilename> \ -si=<See Below> \ -pnfs=/pnfs/desy.de \ -somethingElse=true
Calling sequence
The external script or binary is launed with 3 positional arguments and at least one option (-si=<storageInfo>). Additonial options may follow if defined so with the hsm set pool command. Arguments and options are separated by at least on blank character.Syntax : <binary> put|get <pnfsid> <localFileName> \ -si=<storageInfo> [more options]The put|get argument determines the data transfer direction seen from the HSM. put means, that data has to be stored into the HSM while get means it has be fetched out of the HSM.
The <storageInfo> option is a collection of key value pairs, separated by semicola. All these values are derived from the pnfs database. The possible keys slightly differ, depending on which HSM is addressed. The order of the key value pairs is not determined and may vary between calls. The -si= string shouldn't contain blank TAB or newline characters.
Mandatory StorageInfo keys
Key Meaning size Size of the file in bytes new false if file already in the dCache stored true if file already stored in the HSM sClass HSM depended. Used by the PoolManager for pool attraction cClass Parent Director tag (cacheClass). Used by the PoolManager for pool attraction. May be '-' hsm Storage Manager name (enstore/osm). Can be overwritten by parent directory tag (hsmType). Optional StorageInfo keys but used by all HSM's
Key Meaning flag-l Size of the file (if size exceeds 2G) flag-s * if file is defined sticky flag-c CRC value (currently 1:<hexAdler32> Enstore specific
Key Meaning group Storage Group (e.g. cdf,cms ...) family File family (e.g. sgi2test,h6nxl8, ...) bfid Bitfile Id (GET only) (e.g. B0MS105746894100000) volume Tape Volume (GET only) (e.g. IA6912) location Location on tape (GET only) (e.g. : 0000_000000000_0000117) OSM specific
Key Meaning store OSM store (e.g. zeus,h1, ...) group OSM Storage Group (e.g. h1raw99, ...) bfid Bitfile Id (GET only) (e.g. 000451243.2542452542.25424524) There might be more key values pairs which are used by the dCache internally and which should not affect the behaviour of the hsm copy script.
Return codes
Return Code Meaning Pool Behaviour Into HSM From HSM 30 <= rc < 40 User defined Deactivates request Reports Problem to PoolManager 41 No Space Left on device Pool Retries Disables Pool
Reports Problem to PoolManager42 Disk Read I/O Error 43 Disk Write I/O Error All other Pool Retries Reports Problem to PoolManager
Special Cases and exceptions
Reading vers. Writing HSM files
When fetching a file from an HSM, the command line contains sufficient information about the location of the dataset within the HSM to get the file. No additional interaction with pnfs is needed. So pnfs doesn't need to be mounted on read-only pools.This is different for storing files into an HSM. As a return from the actual HSM put operation, some data has to be stored in pnfs. Currently this has to be done directly by the corresponding external HSM script. So, other then for read pools, write pools still need to have pnfs mounted.
A future approache will be to transfer the necessay HSM information from the HSM copy script into the dCache using STDOUT. The dCache subsequently performs the necessary pnfs store operation through the PnfsManager.
Precious files are removed from Pnfs
In case a precious file is removed from pnfs BEFORE the hsmcopy-Script (osmcp.sh or real-encp.sh) is called, the copy on disk is removed and the hsmcopy-Script is not called.If the file is removed while the hsmcopy-Script is active, the script will encounter an error when writing HSM data into the various pnfs layers. In this case it's recommended to return an error code in the 30-39 range to have the request deactivated. So manual intervention is needed to get the situation cleaned up but no attempt is made by the dCache to get the corresponding dataset stored into the HSM again.