CERN Accelerating science

This website is no longer maintained. Its content may be obsolete. Please visit for current CERN information.

CERN home pageCERN home pageDocuments by ReferenceDocuments by ReferenceCNLsCNLsYear 2001Year 2001Help, Info about this page


Editorial Information
If you need help
Announcements Special 35th Anniversary Physics Computing Desktop Computing Internet Services and Network Scientific Applications and Software Engineering Desktop Publishing The Learning Zone User Documentation
Previous: European Union Gives 9.8 million Euros for GRID Development (January News)
Next:Desktop Computing
 (See printing version)

Move from HPSS to Castor : Will the hsm Command be Transparent?

Bernard Antoine , IT/PDP

For various reasons, CERN has decided to no longer use HPSS for storing files on tapes, and to switch to the home-grown product Castor (Castor fiber L.). Experiment's data already stored in HPSS is already actively being moved to Castor by the experiments themselves.

The other type of data stored in HPSS so far is the so-called "User Tapes" data, offered as a way to archive/restore large files (like n-tuples) on tertiary storage. This is achieved through a command called "hsm".

Time has come ... to convert this tool to use Castor rather than HPSS, and to help users move their data.

The obvious idea would be to change the hsm command to be fully transparent within the conversion period:

  • "hsm put" command would start putting files into Castor instead of HPSS.
  • "hsm get" command would try and locate files first in Castor, then in HPSS, so it is the Castor copy, if it exists, which will be retrieved.

Things are already less easy with the other subcommands:

  • "hsm ls" or "hsm query" should look both in Castor & HPSS, and give some sort of composite output, where the only HPSS files shown are those which have no counterpart in Castor.
  • "hsm delete" should look in both places, and delete all the existing copies of a file.

And the logic becomes over-convoluted for the "hsm rename" command. If file A exists in the HPSS tree, and file B in Castor, should "hsm rename A B" fail, even if B would have been created in the HPSS tree?

Although the design of this transparent hsm command has been looked at, it is felt that the coding and debugging are not worth the required work.

For this reason, we are going to implement a simpler, translucent, rather than transparent approach:

  • A copy tool is being written (it had to be anyway), and used either by the users themselves, or by the Computing Center on behalf of the users.
  • While the tool is running against a specific user's files, the hsm command is blocked for this user, with a message asking for some patience.
  • Once the copy is completed, and the files re-invigorated by a dose of Castor oil, the hsm command only deals with Castor files.
  • This change of state will be controlled by a hidden .hsmrc file in the user's home directory, which file should of course not be messed with.

We think that this simpler way is less likely to create problems, and will make users more aware of the change of the underlying Hierarchical Storage System.

For matters related to this article please contact the author.

Vol. XXXVI, issue no 2

Last Updated on Fri Aug 03 12:02:34 CEST 2001.
Copyright © CERN 2001 -- European Organization for Nuclear Research