slow copy job from disk to LTO-6 (speed issue)

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

slow copy job from disk to LTO-6 (speed issue)

Caribe Schreiber
Hi all,

I'm looking for some advice on how to diagnose a speed issue with a disk-to-tape copy job from a relatively fast RAID array to an LTO-6 drive over SAS.

My local copy jobs seem to be limited to ~23MB/s which is well below the 54-160MB/s speed specified for the adaptive streaming on this drive which leads me to believe that the drive is shoe-shining.

I've checked the write speed to the tape, and the read speed from disk and they seem to be fine as shown below with btape, bonnie++, and dd output.  What else should I be looking at?  Is this typical of a transfer between devices within a single SD instance?

I understand that compression and encryption in the FD is not multithreaded, so a local backup could be CPU limited on those two processes. The initial backup of the below job ran at about 7.3MB/s, but was encrypted and got fairly decent compression so I'm perfectly willing to pay the speed penalty on the backup side of the equation.  My understanding is that a copy job should not be affected by encryption or compression as it's just doing a straight copy of the existing data from one SD device to another (basically a spool-type process), but I'm still seeing very slow transfer rates on copy jobs.

Here's the output for a moderate size copy job going from local disk (FileChgr1-Dev1) to the SAS attached LTO-6 tape (Drive-1) showing the Drive -1 speed as 23.22MB/s:

backup-sd JobId 64 Sending spooled attrs to the Director. Despooling 2,615 bytes ...

Elapsed time=02:03:24, Transfer rate=7.346 M Bytes/second backup-dir JobId 64 No Files found to prune. End auto prune. Bacula backup-dir 7.4.4 (202Sep16): Build OS: x86_64-pc-linux-gnu gentoo JobId: 64 Job: BackupImage.2017-03-24_21.45.00_45 Backup Level: Full Client: "primary-fd" 7.4.4 (202Sep16) x86_64-pc-linux-gnu,gentoo, FileSet: "BackupFileSet" 2017-03-24 21:45:00 Pool: "file.full" (From Job resource) Catalog: "MyCatalog" (From Client resource) Storage: "file" (From Pool resource) Scheduled time: 24-Mar-2017 21:45:00 Start time: 25-Mar-2017 00:12:38 End time: 25-Mar-2017 02:16:03 Elapsed time: 2 hours 3 mins 25 secs Priority: 13 FD Files Written: 7 SD Files Written: 7 FD Bytes Written: 54,391,511,056 (54.39 GB) SD Bytes Written: 54,391,515,157 (54.39 GB) Rate: 7345.2 KB/s Software Compression: 57.8% 2.4:1 Snapshot/VSS: no Encryption: yes Accurate: no Volume name(s): full-Vol-0024|full-Vol-0029 Volume Session Id: 49 Volume Session Time: 1490104159 Last Volume Bytes: 22,911,377,270 (22.91 GB) Non-fatal FD errors: 0 SD Errors: 0 FD termination status: OK SD termination status: OK Termination: Backup OK Begin pruning Jobs older than 6 months . No Jobs found to prune. Begin pruning Files. backup-dir Using Device "Drive-1" to write. backup-sd Elapsed time=00:39:02, Transfer rate=23.22 M Bytes/second backup-sd Alert: smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.4.39-gentoo] (local build) Alert: Copyright (C) 2002-15, Bruce Allen, Christian Franke, Alert: TapeAlert: OK Alert: === START OF READ SMART DATA SECTION === Alert: Alert: Error Counter logging not supported backup-sd Sending spooled attrs to the Director. Despooling 2,615 bytes ... Alert: Last n error events log page Alert: Alert:
I'm able to get good write speeds to the LTO-6 drive with btape as shown here: Wrote block=100000, file,blk=1,99999 VolBytes=6,451,135,488 rate=157.3 MB/s Wrote block=105000, file,blk=1,104999 VolBytes=6,773,695,488 rate=157.5 MB/s Wrote block=110000, file,blk=1,109999 VolBytes=7,096,255,488 rate=157.6 MB/s Wrote block=115000, file,blk=1,114999 VolBytes=7,418,815,488 rate=161.2 MB/s Wrote block=120000, file,blk=1,119999 VolBytes=7,741,375,488 rate=161.2 MB/s Wrote block=125000, file,blk=1,124999 VolBytes=8,063,935,488 rate=161.2 MB/s Wrote block=130000, file,blk=1,129999 VolBytes=8,386,495,488 rate=161.2 MB/s I'm also able to get good read speeds off the drive array as shown by bonnie++ here: backup ~ # bonnie++ -u root Using uid:0, gid:0. Writing a byte at a time...done Writing intelligently...done Rewriting...done Reading a byte at a time...done Reading intelligently...done start 'em...done...done...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version  1.97       ------Sequential Output------ --Sequential Input- --Random- Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP backup      126G  1211  97 601130  47 240240  17  5506  94 759982  22 698.1  10 Latency              9305us     710ms     646ms   15409us     276ms     519ms Version  1.97       ------Sequential Create------ --------Random Create-------- backup           -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP                  16 16535  15 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ Latency               385us     360us     714us     383us     370us     345us 1.97,1.97,backup,1,1490480328,126G,,1211,97,601130,47,240240,17,5506,94,759982,22,698.1,10,16,,,,,16535,15,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,9305us,710ms,646ms,15409us,276ms,519ms,385us,360us,714us,383us,370us,345us And it's pretty fast using dd to read a bacula file from the job as well: backup bacula # dd if=full-Vol-0029 of=/dev/null bs=4M 12799+1 records in 12799+1 records out 53687083817 bytes (54 GB, 50 GiB) copied, 111.637 s, 481 MB/s Here's my tape drive and local file storage config in the bacula-sd.conf: #   # A file changer for disk based backups # Autochanger {   Name = FileChgr1   Device = FileChgr1-Dev1   Changer Command = ""   Changer Device = /dev/null } Device {   Name = FileChgr1-Dev1   Media Type = file   Archive Device = /home/backups/bacula   LabelMedia = yes;                   # lets Bacula label unlabeled media   Random Access = Yes;   AutomaticMount = yes;               # when device opened, read it   RemovableMedia = no;   AlwaysOpen = no;   Maximum Concurrent Jobs = 5 } # # An HP 1/8 G2 Library device with one LTO-6 drive # Autochanger {   Name = AutochangerHP   Device = Drive-1   Changer Command = "/usr/libexec/bacula/mtx-changer %c %o %S %a %d"   Changer Device = /dev/sg3 } Device {   Name = Drive-1                      #   Media Type = LTO-6   Archive Device = /dev/nst0   AutomaticMount = yes;               # when device opened, read it   AlwaysOpen = yes;   RemovableMedia = yes;   RandomAccess = no;   AutoChanger = yes   Maximum File Size = 10GB   Alert Command = "sh -c 'smartctl -H -l error %c'" } I'm a little stumped about why the raw hardware read/write speeds are so different from the bacula copy job speeds. I'm hoping the collective wisdom of the list can give me some ideas on other places to look for speed issues or config changes so that I don't prematurely wear out this drive head with shoe-shining (of course...faster backups are always welcome too!). Thanks in advance! ~Caribe
Check out the vibrant tech community on one of the world's most
engaging tech sites,!
Bacula-users mailing list
[hidden email]