{"id":630,"date":"2021-02-26T22:36:58","date_gmt":"2021-02-26T22:36:58","guid":{"rendered":"http:\/\/mastercom.es\/?p=630"},"modified":"2023-08-21T19:39:19","modified_gmt":"2023-08-21T19:39:19","slug":"630-2","status":"publish","type":"post","link":"http:\/\/mastercom.es\/en\/630-2\/","title":{"rendered":"Configuring an rsync Task"},"content":{"rendered":"

[et_pb_section fb_built=\u00bb1″ _builder_version=\u00bb4.16″ _module_preset=\u00bbdefault\u00bb global_colors_info=\u00bb{}\u00bb theme_builder_area=\u00bbpost_content\u00bb][et_pb_row _builder_version=\u00bb4.16″ _module_preset=\u00bbdefault\u00bb global_colors_info=\u00bb{}\u00bb theme_builder_area=\u00bbpost_content\u00bb][et_pb_column type=\u00bb4_4″ _builder_version=\u00bb4.16″ _module_preset=\u00bbdefault\u00bb global_colors_info=\u00bb{}\u00bb theme_builder_area=\u00bbpost_content\u00bb][et_pb_text _builder_version=\u00bb4.16″ _module_preset=\u00bbdefault\u00bb custom_padding=\u00bb22px|||||\u00bb global_colors_info=\u00bb{}\u00bb theme_builder_area=\u00bbpost_content\u00bb]<\/p>\n

Configuring an rsync Task<\/h1>\n
How to configure automated data transfers using rysnc.<\/div>\n

<\/i>\u00a0 9 minute read<\/p>\n

Data often needs to be copied to another system for backup or when migrating to a new system. A fast and secure way of doing this is by using rsync.<\/p>\n

Rsync provides the ability to either\u00a0<\/span>push<\/em>\u00a0<\/span>or\u00a0<\/span>pull<\/em>\u00a0<\/span>data. When using rsync to\u00a0<\/span>push<\/em>, data is copied from a\u00a0<\/span>host<\/em>\u00a0<\/span>system to a\u00a0<\/span>remote<\/em>\u00a0<\/span>system. When using rsync to\u00a0<\/span>pull<\/em>, data is pulled from a\u00a0<\/span>remote<\/em>\u00a0<\/span>system and put on the\u00a0<\/span>host<\/em>\u00a0<\/span>system. To do this, a dataset with data available for transfer must exist on either the\u00a0<\/span>host<\/em>\u00a0<\/span>or the\u00a0<\/span>remote<\/em>\u00a0<\/span>system. See\u00a0<\/span>ZFS Datasets<\/a>\u00a0<\/span>for more details about creating a dataset.<\/p>\n

Rsync Service<\/path><\/path><\/svg><\/a><\/h2>\n

The rsync task does not work if the related system service is not turned on. To turn the rsync service on, go to\u00a0<\/span>Services<\/strong>\u00a0<\/span>and click the slider for\u00a0<\/span>rsync<\/em>. To activate the service whenever TrueNAS boots, set\u00a0<\/span>Start Automatically<\/em>.<\/p>\n

The rsync service settings can be configured by clicking\u00a0<\/span><\/i>. The default TCP port that rsync listens on can be changed, and additional auxiliary parameters from\u00a0<\/span>rsyncd.conf<\/a>\u00a0<\/span>can be added.<\/p>\n

Unless a specific setting is needed, it is recommended to use the default settings for the rsync service. Don\u2019t forget to click\u00a0<\/span>SAVE<\/em>\u00a0<\/span>after changing any settings.<\/p>\n

Create Rsync Task<\/path><\/path><\/svg><\/a><\/h2>\n

To create an rsync task, go to\u00a0<\/span>Tasks > Rsync Tasks<\/strong>\u00a0<\/span>and click\u00a0<\/span>ADD<\/em>. There are two primary rsync modes:\u00a0<\/span>Module<\/em>\u00a0<\/span>and\u00a0<\/span>SSH<\/em>. The requirements for each mode are different. Refer to the appropriate section below for your desired rsync mode.<\/p>\n

Rsync Mode: Module<\/path><\/path><\/svg><\/a><\/h3>\n

First, turn on the rsync service on the\u00a0<\/span>remote<\/em>\u00a0<\/span>system. Before creating an rsync task on the\u00a0<\/span>host<\/em>\u00a0<\/span>system, a module on the\u00a0<\/span>remote<\/em>\u00a0<\/span>system must be created. Create a module by going to\u00a0<\/span>Services<\/strong>\u00a0<\/span>and clicking\u00a0<\/span><\/i>\u00a0 for the rsync service. Then go to\u00a0<\/span>Rsync Module > ADD<\/strong>.<\/p>\n

Enter a descriptive name for the rsync module and use the file browser to define the path to the dataset to be transferred from the\u00a0<\/span>host<\/em>\u00a0<\/span>system.<\/p>\n

Select the permissions for the module by selecting an option from the\u00a0<\/span>Access Mode<\/em>\u00a0<\/span>dropdown.<\/p>\n

Set the\u00a0<\/span>Max Connections<\/em>. Entering\u00a0<\/span>0<\/em>\u00a0<\/span>sets the amount of connections to unlimited.<\/p>\n

When using a specific user or group during data transfers to and from the module, select them from the related dropdowns.<\/p>\n

You can also define a list of hosts that are allowed or denied access to the rsync module.<\/p>\n

\n

If a\u00a0<\/span>Hosts Allow<\/em>\u00a0<\/span>list is specified,\u00a0<\/span>only<\/strong>\u00a0<\/span>the IPs and hostnames on the list will be able to connect to the module.<\/p>\n<\/div>\n

Other options include specifying a comment for the rsync module and any additional parameters from\u00a0<\/span>rsyncd.conf<\/a>.<\/p>\n

When a module has been created on the\u00a0<\/span>remote<\/em>\u00a0<\/span>system, log in to the\u00a0<\/span>host<\/em>\u00a0<\/span>system interface, go to\u00a0<\/span>Tasks > Rsync Tasks<\/strong>, and click\u00a0<\/span>ADD<\/em>.<\/p>\n

Select the source dataset to be used for the rsync task. This dataset will be transferred to the remote module when\u00a0<\/span>pushing<\/em>. Alternately, this dataset will be used as the destination when\u00a0<\/span>pulling<\/em>\u00a0<\/span>data from the module.<\/p>\n

Select a user to run the rsync task. The user selected must have write permissions to the specified directory chosen for the module on the\u00a0<\/span>remote<\/em>\u00a0<\/span>system.<\/p>\n

Choose a direction for the rsync task. When\u00a0<\/span>push<\/em>\u00a0<\/span>is selected, data from the\u00a0<\/span>host<\/em>\u00a0<\/span>dataset copies to the\u00a0<\/span>remote<\/em>\u00a0<\/span>module. When\u00a0<\/span>pull<\/em>\u00a0<\/span>is selected, data from the\u00a0<\/span>remote<\/em>\u00a0<\/span>module is copied into the\u00a0<\/span>host<\/em>\u00a0<\/span>dataset.<\/p>\n

Select a schedule for the rsync task to run on. If a custom schedule is desired, select\u00a0<\/span>Custom<\/em>\u00a0<\/span>and fill out the custom scheduler to meet your needs. The custom scheduler can accept standard\u00a0<\/span>cron input strings<\/a>\u00a0<\/span>for the\u00a0<\/span>Minutes<\/em>,\u00a0<\/span>Hours<\/em>, and\u00a0<\/span>Days<\/em>.<\/p>\n

By default, the rsync task will include all subdirectories of the chosen\u00a0<\/span>host<\/em>\u00a0<\/span>dataset with the transer. To disable this, unset the\u00a0<\/span>Recursive<\/em>\u00a0<\/span>option.<\/p>\n

Next, enter the\u00a0<\/span>remote<\/em>\u00a0<\/span>host IP address or hostname. Use the format\u00a0<\/span>username@remote_host<\/em>\u00a0<\/span>if the username differs on the\u00a0<\/span>remote<\/em>\u00a0<\/span>host. Select\u00a0<\/span>Module<\/em>\u00a0<\/span>in the\u00a0<\/span>Rsync Mode<\/em>\u00a0<\/span>dropdown. Enter the\u00a0<\/span>Remote Module Name<\/em>\u00a0<\/span>exactly.<\/p>\n

Setting the\u00a0<\/span>Times<\/em>\u00a0<\/span>option preserves the modification times of files. Set\u00a0<\/span>Compress<\/em>\u00a0<\/span>to reduce the size of data to transmit. This is recommended for slow connections.<\/p>\n

Archives<\/em>\u00a0<\/span>can be set to run recursively, preserving symlinks, permissions, modification times, group, and special files. When run as root, the owner, device files, and special files are also preserved. This is equivalent to passing the flags\u00a0<\/span>-rlptgoD<\/code>\u00a0<\/span>to rsync.<\/p>\n

When\u00a0<\/span>Delete<\/em>\u00a0<\/span>is set, the rsync task will delete files in the destination directory that do not exist in the source directory. For example, if the task is set to\u00a0<\/span>pull<\/em>, this option will delete any files on the\u00a0<\/span>host<\/em>\u00a0<\/span>dataset that do not existing on the\u00a0<\/span>remote<\/em>\u00a0<\/span>module. Alternatively, if the task is set to\u00a0<\/span>push<\/em>, this option will delete any files on the\u00a0<\/span>remote<\/em>\u00a0<\/span>module that are not in the\u00a0<\/span>host<\/em>\u00a0<\/span>dataset. This is a\u00a0<\/span>destructive<\/strong>\u00a0<\/span>option. Use with caution as data can be deleted permanently.<\/p>\n

Set the\u00a0<\/span>Quiet<\/em>\u00a0<\/span>option to suppress informational messages from the\u00a0<\/span>remote<\/em>\u00a0<\/span>system. Set the option\u00a0<\/span>Preserve Permissions<\/em>\u00a0<\/span>to preserve original file permissions. This is useful when the user is set to root.<\/p>\n

Set\u00a0<\/span>Preserve Extended Attributes<\/em>\u00a0<\/span>to keep the advanced file system features and file metadata through the transfer (see\u00a0<\/span>Extended attributes<\/a>). Extended attributes must be supported by both systems.<\/p>\n

Set\u00a0<\/span>Delay Updates<\/em>\u00a0<\/span>to save the temporary files from updated files to a holding directory until the end of the transfer when all transferred files are renamed into place.<\/p>\n

Additional\u00a0<\/span>rsync<\/a>\u00a0<\/span>options can be included in\u00a0<\/span>Auxiliary Parameters<\/em>. Separate entries by pressing\u00a0<\/span>Enter<\/kbd>. The\u00a0<\/span>* character must be escaped with a backslash (\\<\/em>.txt) or used inside single quotes (‘*.txt\u2019).<\/p>\n

Unsetting\u00a0<\/span>Enabled<\/em>\u00a0<\/span>disables the task schedule. You can still save the rsync task and run it manually.<\/p>\n

Rsync Mode: SSH<\/path><\/path><\/svg><\/a><\/h3>\n

Using the\u00a0<\/span>SSH<\/em>\u00a0<\/span>rsync mode requires the SSH service to be enabled on the\u00a0<\/span>remote<\/em>\u00a0<\/span>system. To enable SSH, go to\u00a0<\/span>Services<\/strong>\u00a0<\/span>and click the slider for SSH. SSH settings can be configured by clicking\u00a0<\/span><\/i>. Take note of the TCP port set on the\u00a0<\/span>remote<\/em>\u00a0<\/span>system. The default port for the SSH service is port\u00a0<\/span>22<\/em>. It is required when creating the rsync task.<\/p>\n

Now, an ssh key pair must be created on the\u00a0<\/span>host<\/em>\u00a0<\/span>system. Open up the\u00a0<\/span>Shell<\/strong>\u00a0<\/span>on the\u00a0<\/span>host<\/em>\u00a0<\/span>system. To create the ssh key pair, the\u00a0<\/span>ssh-keygen<\/code>\u00a0<\/span>command is used. Typically, the rsync task is ran as the root user. If running the command as a different user, run\u00a0<\/span>su -\u00a0<\/span>username<\/i><\/code>\u00a0<\/span>where\u00a0<\/span>username<\/em>\u00a0<\/span>is the name of the user that will run the rsync task. When the appropriate user is selected, run the command\u00a0<\/span>ssh-keygen -t rsa<\/code>\u00a0<\/span>and follow the prompts. When prompted for a password, press\u00a0<\/span>Enter<\/kbd>. Setting a password will break the scheduled rsync task since it is automatic. Below is an example of running the command.<\/p>\n

\n
truenas# ssh-keygen -t rsa\nGenerating public\/private rsa key pair.\nEnter file in which to save the key (<\/span>\/root\/.ssh\/id_rsa)<\/span>:\nCreated directory '\/root\/.ssh'<\/span>.\nEnter passphrase (<\/span>empty for<\/span> no passphrase)<\/span>:\nEnter same passphrase again:\nYour identification has been saved in \/root\/.ssh\/id_rsa.\nYour public key has been saved in \/root\/.ssh\/id_rsa.pub.\nThe key fingerprint is:\nSHA256:NZMgbuPvTHeEqi3SA\/U5wW8un6AWrx8ZsRQdbJJHmR4 tester@truenas.local\nThe key'<\/span>s randomart image is:\n+---[<\/span>RSA 2048]<\/span>----+\n|<\/span>      . o<\/span>=<\/span>o+     |<\/span>\n|<\/span>     . .ooE.     |<\/span>\n|<\/span>      +.o==<\/span>.     |<\/span>\n|<\/span>     o.oo+.+     |<\/span>\n|<\/span>     ...S+. .    |<\/span>\n|<\/span>    . ..++o.     |<\/span>\n|<\/span>     o oB+. .    |<\/span>\n|<\/span>    . =<\/span>Bo+.o     |<\/span>\n|<\/span>     o<\/span>+==<\/span>oo      |<\/span>\n+----[<\/span>SHA256]<\/span>-----+\n<\/code><\/pre>\n<\/div>\n
The default location for the public key is\u00a0<\/span>~\/.ssh\/id_rsa.pub<\/code>. Run the command\u00a0<\/span>cat ~\/.ssh\/id_rsa.pub<\/code>\u00a0<\/span>to view the key. Copy it to the corresponding user account on the\u00a0<\/span>remote<\/em>\u00a0<\/span>system in\u00a0<\/span>Accounts > Users<\/strong>. Click\u00a0<\/span>EDIT<\/em>\u00a0<\/span>and paste the key in the\u00a0<\/span>SSH Public Key<\/em>\u00a0<\/span>field.<\/p>\n
Next, copy the host key of the\u00a0<\/span>remote<\/em>\u00a0<\/span>to the user\u2019s\u00a0<\/span>.ssh\/known_hosts<\/code>\u00a0<\/span>directory on the\u00a0<\/span>host<\/em>\u00a0<\/span>system. This is done using the\u00a0<\/span>ssh-keyscan<\/code>\u00a0<\/span>command. While on the\u00a0<\/span>host<\/em>\u00a0<\/span>system, open up the\u00a0<\/span>Shell<\/strong>\u00a0<\/span>and run\u00a0<\/span>ssh-keyscan -t rsa\u00a0<\/span>remoteIPaddress<\/i>\u00a0<\/span>\u00bb\u00a0<\/span>userknown_hostsDir<\/i><\/code>\u00a0<\/span>where\u00a0<\/span>remoteIPaddress<\/em>\u00a0<\/span>is the IP address of the\u00a0<\/span>remote<\/em>\u00a0<\/span>system and\u00a0<\/span>userknown_hostsDir<\/em>\u00a0<\/span>is the known_hosts directory on the\u00a0<\/span>host<\/em>\u00a0<\/span>system. Here is an example of the command:\u00a0<\/span>ssh-keyscan -t rsa 192.168.2.6 >> \/root\/.ssh\/known_hosts<\/code>.<\/p>\n
Next, enter the\u00a0<\/span>remote<\/em>\u00a0<\/span>host IP address or hostname. Use the format\u00a0<\/span>username@remote_host<\/em>\u00a0<\/span>if the username differs on the\u00a0<\/span>remote<\/em>host. Select\u00a0<\/span>SSH<\/em>\u00a0<\/span>in the\u00a0<\/span>Rsync Mode<\/em>\u00a0<\/span>dropdown. Enter the\u00a0<\/span>TCP port<\/em>\u00a0<\/span>set on the\u00a0<\/span>remote<\/em>\u00a0<\/span>system. The default is\u00a0<\/span>22<\/em>. Enter the remote path of the dataset for rsync to\u00a0<\/span>push<\/em>\u00a0<\/span>or\u00a0<\/span>pull<\/em>. Set\u00a0<\/span>Validate Remote Path<\/em>\u00a0<\/span>to automatically create the defined\u00a0<\/span>Remote Path<\/em>\u00a0<\/span>if it does not exist on the\u00a0<\/span>remote<\/em>\u00a0<\/span>system.<\/p>\n
Setting the\u00a0<\/span>Times<\/em>\u00a0<\/span>option preserves modification times of the files. Set\u00a0<\/span>Compress<\/em>\u00a0<\/span>to reduce the size of data to transmit. This is recommended for slow connections.\u00a0<\/span>Archives<\/em>\u00a0<\/span>can be set to run recursively, preserving symlinks, permissions, modification times, group, and special files. When run as root, the owner, device files, and special files are also preserved. This is equivalent to passing the flags\u00a0<\/span>-rlptgoD<\/code>\u00a0<\/span>to rsync. When\u00a0<\/span>Delete<\/em>\u00a0<\/span>is set, the rsync task will delete files in the destination directory that do not exist in the source directory. For example, if the task is set to\u00a0<\/span>pull<\/em>, this option will delete any files on the\u00a0<\/span>host<\/em>\u00a0<\/span>dataset that do not existing on the\u00a0<\/span>remote<\/em>\u00a0<\/span>module. Alternatively, if the task is set to\u00a0<\/span>push<\/em>, this option will delete any files on the\u00a0<\/span>remote<\/em>\u00a0<\/span>module that are not in the\u00a0<\/span>host<\/em>\u00a0<\/span>dataset. This is a\u00a0<\/span>destructive<\/strong>\u00a0<\/span>option. Use with caution as data can be deleted permanently. Set the\u00a0<\/span>Quiet<\/em>\u00a0<\/span>option to suppress informational messages from the\u00a0<\/span>remote<\/em>\u00a0<\/span>system. Set the option\u00a0<\/span>Preserve Permissions<\/em>\u00a0<\/span>to preserve original file permissions. This is useful when the user is set to root. Set\u00a0<\/span>Preserve Extended Attributes<\/em>\u00a0<\/span>to keep the advanced file system features and file metadata through the transfer (see\u00a0<\/span>Extended attributes<\/a>). Extended attributes must be supported by both systems. Set\u00a0<\/span>Delay Updates<\/em>\u00a0<\/span>to save the temporary files from updated files to a holding directory until the end of the transfer when all transferred files are renamed into place.<\/p>\n
Additional\u00a0<\/span>rsync(1)<\/a>\u00a0<\/span>options can be included in\u00a0<\/span>Auxiliary Parameters<\/em>. Separate entries by pressing\u00a0<\/span>Enter<\/kbd>. The\u00a0<\/span>* character must be escaped with a backslash (\\<\/em>.txt) or used inside single quotes (‘*.txt\u2019).<\/p>\n
Unsetting\u00a0<\/span>Enabled<\/em>\u00a0<\/span>disables the task schedule. You can still save the rsync task and run it manually.<\/p>\n
[\/et_pb_text][\/et_pb_column][\/et_pb_row][\/et_pb_section]<\/p>\n","protected":false},"excerpt":{"rendered":"
Configuring an rsync Task How to configure automated data transfers using rysnc. \u00a0 9 minute read Data often needs to be copied to another system for backup or when migrating to a new system. A fast and secure way of doing this is by using rsync. Rsync provides the ability to either\u00a0push\u00a0or\u00a0pull\u00a0data. When using rsync […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"on","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":""},"categories":[3],"tags":[],"class_list":["post-630","post","type-post","status-publish","format-standard","hentry","category-linux"],"_links":{"self":[{"href":"http:\/\/mastercom.es\/en\/wp-json\/wp\/v2\/posts\/630","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/mastercom.es\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/mastercom.es\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/mastercom.es\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/mastercom.es\/en\/wp-json\/wp\/v2\/comments?post=630"}],"version-history":[{"count":0,"href":"http:\/\/mastercom.es\/en\/wp-json\/wp\/v2\/posts\/630\/revisions"}],"wp:attachment":[{"href":"http:\/\/mastercom.es\/en\/wp-json\/wp\/v2\/media?parent=630"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/mastercom.es\/en\/wp-json\/wp\/v2\/categories?post=630"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/mastercom.es\/en\/wp-json\/wp\/v2\/tags?post=630"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
Finally, the rsync task is ready to be created. Go to\u00a0<\/span>Tasks > Rsync Tasks<\/strong>\u00a0<\/span>and click\u00a0<\/span>ADD<\/em>. Select the source dataset to be used for the rsync task. This dataset will be copied to the remote path chosen when\u00a0<\/span>pushing<\/em>. Alternatively, this dataset will be used as the destination when\u00a0<\/span>pulling<\/em>\u00a0<\/span>data from the remote path. Select the user to run the rsync task. Make sure this is the same user the ssh key pair was created for. In the example given, the user is root. Choose a direction for the rsync task. When\u00a0<\/span>push<\/em>\u00a0<\/span>is selected, data from the\u00a0<\/span>host<\/em>\u00a0<\/span>dataset copies to the\u00a0<\/span>remote<\/em>\u00a0<\/span>dataset. When\u00a0<\/span>pull<\/em>\u00a0<\/span>is selected, data from the\u00a0<\/span>remote<\/em>\u00a0<\/span>dataset is copied into the\u00a0<\/span>host<\/em>\u00a0<\/span>dataset. An optional description can be specified. Select a schedule for the rsync task. If a custom schedule is desired, select\u00a0<\/span>Custom<\/em>\u00a0<\/span>and fill out the custom scheduler to meet your needs. The custom scheduler can accept standard\u00a0<\/span>cron input strings<\/a>\u00a0<\/span>for the\u00a0<\/span>Minutes<\/em>,\u00a0<\/span>Hours<\/em>, and\u00a0<\/span>Days<\/em>. By default, the\u00a0<\/span>Recursive<\/em>\u00a0<\/span>option is set. If unset, the rsync task will not copy any subdirectories in the\u00a0<\/span>host<\/em>\u00a0<\/span>dataset.<\/p>\n