Rclone¶
This guide describes how to use the Rclone WebDAV client for storing and retrieving files from Swestore. Rclone is versatile and supports many protocols through a simple command line interface (CLI).
Requirements¶
NAISS resources should already have rclone installed. To install rclone on your own computer, please follow your systems instructions at the official rclone documentation pages.
To access Swestore using rclone you need to be a member of a Swestore storage project, see Apply for Swestore.
Depending on your preference and security needs, you can use rclone with either certificate or username/password authentication.
Certificate authentication¶
This method requires you to obtain a security certificate and install it. While more cumbersome to set up, it provides a higher level of security.
We recommend this method when logged in to NAISS resources.
To use this authentication method you need to upload your certificate as described on the Prepare page. See also ARC - Requirements for more details on the requirements.
Username/Password¶
This method will require you to enter your username/password in the rclone configuration file. While this is convenient, it is also a security risk since a malicious user gaining access to the rclone configuration file or your login on the affected resource will also gain access to your Swestore dCache storage. This might be an acceptable risk on a single-user system such as a laptop or desktop, but avoid using this method on multi-user systems.
To use this authentication method you need to know your Username and password.
Quickstart¶
Swestore access URL¶
The WebDAV access URL for Swestore should be specified as:
https://webdav.swestore.se
However, do note that this isn't optimal for transferring huge volumes since transfers are tunneled through the dCache endpoint before reaching the source/destination storage pool. See Enabled Access Protocols for more options that might be relevant.
Basic commands¶
rclone config
- configure rclone. See Rclone
configuration for details.
rclone ls
- for listing files. Works similarly to ls
. See Rclone
ls.
rclone copyto
- for copying directories or separate files. Works
similarly to cp
with wildcards. See Rclone
copyto.
rclone copy
- for copying contents of directories. Works similarly
to cp -R path/*
. See Rclone
copy.
rclone mkdir
- for creating directories. Works similarly to mkdir
-p
. See Rclone mkdir.
rclone deletefile
- for removing specific files. Works similarly to
rm
. See Rclone
deletefile.
More powerful removal functions are available, but be careful with these;
rclone delete
- for removing all files under path - i.e. you loose
your data. Works similarly to find path -type f | xargs rm
. See
Rclone delete.
rclone rmdir
- removing path if empty. Works similarly to rm
-d
. See Rclone rmdir.
rclone rmdirs
- removing all empty directories under path. Works
similarly to find path -type d | awk '{ print length, $0 }' | sort
-nsr | cut -d" " -f2- | xargs rm -d
(example modified from
stackoverflow). See Rclone
rmdirs.
rclone purge
- be very careful using this since it will remove
all data under path, including path - i.e. you loose your
data. Works similarly to rm -rf
. See Rclone
purge.
Use man
and --help
to get more info on rclone and its commands.
Examples: man rclone
, rclone --help
or rclone copy --help
.
Paths¶
The rclone commands supports multiple storage protocols. Given that
you have configured https://webdav.swestore.se
as the swestore
remote, we recommend using WebDAV with paths on the form
swestore:/snic/YOUR_PROJECT_DIR/...
.
Configuration¶
You have to configure rclone with your access protocol, URL and possibly user login credentials for it to work with Swestore dCache.
Issue the following configuration command to interactively configure rclone:
rclone config
Then choose between the two configuration alternatives below, depending on you authentication preference.
For configuring Swestore dCache access using WebDAV and Certificate authentication:
n
for New remoteswestore
for namewebdav
for Storage (or the corresponding number in the list)https://webdav.swestore.se
for urlother
for vendor (or the corresponding number in the list)- Just press
<Enter>
for user (ie. leave blank) n
for No leave this optional password blank- Just press
<Enter>
for bearer_token n
for Edit advanced config?y
if you think the resulting config is correct, otherwisee
to edit again.q
to Quit config
Alternatively, for configuring Swestore dCache access using WebDAV and Username/Password authentication:
n
for New remoteswestore
for namewebdav
for Storage (or the corresponding number in the list)https://webdav.swestore.se
for urlother
for vendor (or the corresponding number in the list)yourusername
for user, should be on the formats_user
y
for Yes type in my own password- then enter your swestore password twice
- Just press
<Enter>
for bearer_token n
for Edit advanced config?y
if you think the resulting config is correct, otherwisee
to edit again.q
to Quit config
To see your configuration afterwards, run
cat ~/.config/rclone/rclone.conf
You can also list your configured remotes by issuing
rclone listremotes
In the following sections, we are assuming your swestore remote is
named swestore
.
Additional steps for Certificate authentication¶
Unfortunately rclone doesn't (yet) support adding the needed client certificate paths to the configuration, requiring the following invocation for all rclone commands:
rclone --client-cert=/tmp/x509up_u${UID} --client-key=/tmp/x509up_u${UID}
Since this is obviously very cumbersome, we strongly suggest adding an
alias for it. In the example below, an alias is added for
rclone-cert
to do the above invocation. The example assumes that the
bash shell is used:
echo 'alias rclone-cert="rclone --client-cert=/tmp/x509up_u${UID} --client-key=/tmp/x509up_u${UID}"' >> ~/.bashrc
The alias will be activated in a new shell or login session.
For all invocations using certificate authentication, use
rclone-cert
instead of rclone
to pass the needed arguments to
rclone.
Logging in¶
Certificate authentication¶
Your certificate needs to be unlocked, similar to a login process, in
order to enable access. This is done using the arcproxy
command.
See ARC - Unlock your certificate for details.
Remember to use rclone-cert
instead of rclone
as mentioned in
Additional steps for Certificate authentication
Username/Password authentication¶
When using this method, an explicite login isn't needed since the username/password is entered into the rclone configuration file. It's convenient, but less secure.
Copying files¶
Copying files to and from resources is accomplished using the rclone copy and copyto command.
Copying single files¶
Copying single files is accomplished in the same way as using the normal cp command as shown in the following example:
rclone copyto archive.tar.gz swestore:/snic/YOUR_PROJECT_DIR/archive.tar.gz
You can also use copyto in order to rename the file in the process of copying it to the destination, by specifying a different filename on the remote.
Recursive copying¶
Recursive copying of a directory is accomplished using the copy command. The command will only copy files that have changed on the source compared to the destination, which is determined by checksums and timestamps. Observe that the source directory is not copied over, only its contents. Also, empty directories are omitted.
Example:
rclone copy /path/to/src swestore:/snic/YOUR_PROJECT_DIR/DESTINATION_DIRECTORY
The option --no-traverse can be used to not list files on the destination (good for huge directories). --max-age can be used to select the most recently modified files for transfer, and -P gives you status on progress.
Example, copying the last days modifications, with progress:
rclone copy --max-age 24h --no-traverse -P /path/to/src swestore:/snic/YOUR_PROJECT_DIR/DESTINATION_DIRECTORY
NOTE: The above example will copy all files in the directory src
into the destination directory DESTINATION_DIRECTORY
. If you want
the directory src
to be part of the destination path you have to
explicitly supply it as shown in the example below:
rclone copy /path/to/src swestore:/snic/YOUR_PROJECT_DIR/DESTINATION_DIRECTORY/src
Listing¶
Rclone supports listing all files under a path.
Listing files and directories on a resources is done using the ls commands below. The simple ls command only lists objects and their sizes in the specified directory and all subdirectories due to recursion being enabled by default.
rclone ls swestore:/snic/YOUR_PROJECT_DIR
Further functionality can be be achieved from using any of the the following ls* commands;
lsl
long listing with additional infolsd
list only directorieslsf
list objects and directories, in a fashion good for scriptinglsjson
gives advanced output in JSON format
NOTE: The ls
and lsl
recurses to show all subdirectories by
default - use --max-depth 1
to disable it. For other commands you
need to specify the -R
flag to enable recursion.
Example 1:
rclone lsl --max-depth 1 swestore:/snic/YOUR_PROJECT_DIR
Example 2:
rclone lsd swestore:/snic/YOUR_PROJECT_DIR
Creating directories¶
Directories are generally created on demand. If you copy a file with the destination /snic/YOUR_PROJECT_DIR/newdir/dummyfile the newdir directory will be created if missing. But you can explicitly create directories using the mkdir command.
rclone mkdir swestore:/snic/YOUR_PROJECT_DIR/newdir
Removing files or directories¶
Beware that the command delete will recursively delete all file objects under specified path!
To remove the file dummyfile
under /snic/YOUR_PROJECT_DIR/newdir
,
rclone deletefile swestore:/snic/YOUR_PROJECT_DIR/newdir/dummyfile
To remove a directory, they have to be empty, and you use the command
rclone rmdir swestore:/snic/YOUR_PROJECT_DIR/newdir/
To remove all empty directories under a path, use
rclone rmdirs swestore:/snic/YOUR_PROJECT_DIR/newdir/
To recursively remove all files under
/snic/YOUR_PROJECT_DIR/newdir
, leaving the empty directory structure
in place, be careful;
rclone delete swestore:/snic/YOUR_PROJECT_DIR/newdir/
Do not use the command purge, as that will delete everything under the path specified.
FAQ¶
Q: I used rclone purge
or rclone delete
and have now deleted all
my files; can I get them back somehow?
A: No. Swestore does not currently support recovery of data for
which the user has explicably requested to be deleted from the system,
be it intentionally or by mistake. Therefore caution is advised
while using powerful tools such as rclone
.
Q: When trying to use rclone with certificate authentication, I get
an error message ending with 401 login failed
A: Check the following: 1) arcproxy -I
shows a valid
credential. 2) You're using rclone-cert
to get the proper rclone
invocation.