Configuring autochanger with SAS LTO-5 drives

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Configuring autochanger with SAS LTO-5 drives

Ivan Adzhubey
Hi,

I am trying to configure a Qualstar TLS-8466 autochanger, which has two IBM
LTO-5 drives installed. The changer itself is connected to a SCSI HBA
(Adaptec) but the drives are connected separately to another SAS HBA (LSI).
The server runs Ubuntu 16.04.2 LTS and bacula version 7.0.5 was installed from
Ubuntu repository.

Both tape drives and the autochanger devices are recognized correctly and
would pass various basic tests (e.g. mtx status, mt status, tapeinfo, etc).
Here is the system's SCSI devices list:

# lsscsi -g
[2:0:0:0]    disk    LSI      9750-16i4e DISK  5.12  /dev/sda   /dev/sg0
[2:0:1:0]    disk    LSI      9750-16i4e DISK  5.12  /dev/sdb   /dev/sg1
[7:0:0:0]    mediumx QUALSTAR TLS-8466         227d  /dev/sch0  /dev/sg4
[8:0:0:0]    tape    IBM      ULTRIUM-HH5      BBNF  /dev/st0   /dev/sg2
[8:0:1:0]    tape    IBM      ULTRIUM-HH5      BBNF  /dev/st1   /dev/sg3

But I am still confused about few bits of configuration not very well
explained in the documentation. I have used LTO-4 Device section from the
distribution bacula-sd.conf file as a template. Here is my questions:

a) I have added ChangerDevice directive to the Device resources for the
drives, pointing to their corresponding raw SCSI devices (/dev/sg2, /dev/
sga3), while Autochager has ChangerDevice configured as /dev/sg4. Is this
configuration correct? I am assuming these tape raw devices only used for the
purpose of running AlertCommand on the drives?

b) What is the effect of MaximumFileSize option and what would be its optimal
value for my IBM LTO-5 SAS drives? I have used 8GB value found in one of the
list posts, while the documentation suggests 2GB for LTO-4. But even set at
8GB this would create lots of EOF marks on a 1.5TB tape, do we really need so
many?

c) Should I try increasing the tape block size? Set it to fixed? I know this
topic has been rather controversial, so any recent experience from similar
drives/system would be appreciated. I plan to use spooling to a dedicated RAID
volume, so hopefully hard drives should not be a bottleneck.

Listed below is the devices configuration from my bacula-sd.conf:

Autochanger {
  Name = TLS
  Device = Drive-1, Drive-2
  Changer Command = "/etc/bacula/scripts/mtx-changer %c %o %S %a %d"
  Changer Device = /dev/sg4
}

Device {
  Name = Drive-1                      # IBM ULTRIUM-HH5 on /dev/sg2
  Drive Index = 0
  Media Type = LTO-5
  Archive Device = /dev/nst0
  AutomaticMount = yes;               # when device opened, read it
  AlwaysOpen = yes;
  RemovableMedia = yes;
  RandomAccess = no;
  Maximum File Size = 8GB
  AutoChanger = yes
  Changer Device = /dev/sg2
  #
  # Enable the Alert command only if you have the mtx package loaded
  # Note, apparently on some systems, tapeinfo resets the SCSI controller
  #  thus if you turn this on, make sure it does not reset your SCSI
  #  controller.  I have never had any problems, and smartctl does
  #  not seem to cause such problems.
  #
#  Alert Command = "sh -c 'tapeinfo -f %c |grep TapeAlert|cat'"
#  If you have smartctl, enable this, it has more info than tapeinfo
  Alert Command = "sh -c 'smartctl -H -l error %c'"
}

Device {
  Name = Drive-2                      # IBM ULTRIUM-HH5 on /dev/sg3
  Drive Index = 1
  Media Type = LTO-5
  Archive Device = /dev/nst1
  AutomaticMount = yes;               # when device opened, read it
  AlwaysOpen = yes;
  RemovableMedia = yes;
  RandomAccess = no;
  Maximum File Size = 8GB
  AutoChanger = yes
  Changer Device = /dev/sg3
  # Enable the Alert command only if you have the mtx package loaded
#  Alert Command = "sh -c 'tapeinfo -f %c |grep TapeAlert|cat'"
#  If you have smartctl, enable this, it has more info than tapeinfo
  Alert Command = "sh -c 'smartctl -H -l error %c'"
}

Thanks,
Ivan

--
Ivan Adzhubey, Ph.D.
Instructor
Division of Genetics, Dept of Medicine
Brigham & Women's Hospital
Harvard Medical School
New Research Building, Room 0464C
77 Avenue Louis Pasteur
Boston, MA 02115
tel.: (617) 525-4728
fax:  (617) 525-4705
web: http://genetics.bwh.harvard.edu/wiki/sunyaevlab/



The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Configuring autochanger with SAS LTO-5 drives

Cejka Rudolf
Ivan Adzhubey wrote (2017/06/01):
> b) What is the effect of MaximumFileSize option and what would be its optimal
> value for my IBM LTO-5 SAS drives? I have used 8GB value found in one of the
> list posts, while the documentation suggests 2GB for LTO-4. But even set at
> 8GB this would create lots of EOF marks on a 1.5TB tape, do we really need so
> many?

Hi, I do use 16 GB. Every EOF mark means around 3 seconds delay. So if you have
over 200 files on the tape using 8 GB, it is around 10 minutes extra per tape.

> c) Should I try increasing the tape block size? Set it to fixed? I know this

I think that there is no reason to use fixed tape block size. On the other side,
increasing tape block size would help, there are many discussions about that.

> topic has been rather controversial, so any recent experience from similar
> drives/system would be appreciated. I plan to use spooling to a dedicated RAID
> volume, so hopefully hard drives should not be a bottleneck.

Hard drives are potential bottleneck, they can not handle several write
streams and at the same time one read stream going 150-300 MB/s. Rather
use SSD drives.

Regards.

--
Rudolf Cejka <cejkar at fit.vutbr.cz> http://www.fit.vutbr.cz/~cejkar
Brno University of Technology, Faculty of Information Technology
Bozetechova 2, 612 66  Brno, Czech Republic

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Configuring autochanger with SAS LTO-5 drives

Ivan Adzhubey
Hi Rudolf,

Thanks for the prompt reply. Please scroll down for inline comments.

On Friday, June 02, 2017 09:52:58 AM Cejka Rudolf wrote:

> Ivan Adzhubey wrote (2017/06/01):
> > b) What is the effect of MaximumFileSize option and what would be its
> > optimal value for my IBM LTO-5 SAS drives? I have used 8GB value found in
> > one of the list posts, while the documentation suggests 2GB for LTO-4.
> > But even set at 8GB this would create lots of EOF marks on a 1.5TB tape,
> > do we really need so many?
>
> Hi, I do use 16 GB. Every EOF mark means around 3 seconds delay. So if you
> have over 200 files on the tape using 8 GB, it is around 10 minutes extra
> per tape.

Searching throughout the list, I am seeing widely varying numbers quoted for
working LTO tape configurations. I guess what it means is that the parameter is
not critical. But thanks for the estimate, looks like 10GB value makes sense.

> > c) Should I try increasing the tape block size? Set it to fixed? I know
> > this
> I think that there is no reason to use fixed tape block size. On the other
> side, increasing tape block size would help, there are many discussions
> about that.

Documentation states (rather vaguely, I admit) that you only would want to set
MaximumBlockSize to use fixed block sizes:

"Maximum block size = size-in-bytes On most modern tape drives, you will not
need to specify this directive. If you do so, it will most likely be to use
fixed block sizes (see Minimum block size above)."

http://www.bacula.org/7.0.x-manuals/en/main/Storage_Daemon_Configuratio.html#SECTION001730000000000000000

I am not sure I understand what the documentation is trying to tell us but, as
others have mentioned already, whatever it was it is probably no longer true
for more recent drive models/HBAs and server hardware configurations.

One related question. Default maximum block size is actually specified as
126*512 = 64512. This is not equal neither 64KB (base 1000) nor 64K (base
1024). Do you know why it is short two 512 blocks? Should I reserve the same
amount in my maximum block size setting as well?

> > topic has been rather controversial, so any recent experience from similar
> > drives/system would be appreciated. I plan to use spooling to a dedicated
> > RAID volume, so hopefully hard drives should not be a bottleneck.
>
> Hard drives are potential bottleneck, they can not handle several write
> streams and at the same time one read stream going 150-300 MB/s. Rather
> use SSD drives.

I have seen the SSD argument brought up several times but was never able to
see any supporting benchmarks. Maybe I misunderstand something important but
how RAID volume can be a limiting factor when our 3Ware/LSI SAS RAID
controller easily provides 350 MB/sec sustained read rates (750 MB/sec peak)?
I do not plan to run concurrent spooling/despooling jobs on the same
partition, so any drop in performance due to concurrency is unlikely as well,
even though my benchmarks suggest sustained 150 MB/sec speed for concurrent
read/write on our RAID6 array.

I would love to see any numbers for recent SSD models since we do not have any
installed here and I do not want to shell out large amounts of money on
something I am not sure will give us any significant benefits.

Thanks,
Ivan


--
Ivan Adzhubey, Ph.D.
Instructor
Division of Genetics, Dept of Medicine
Brigham & Women's Hospital
Harvard Medical School
New Research Building, Room 0464C
77 Avenue Louis Pasteur
Boston, MA 02115
tel.: (617) 525-4728
fax:  (617) 525-4705
web: http://genetics.bwh.harvard.edu/wiki/sunyaevlab/



The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Configuring autochanger with SAS LTO-5 drives

Ivan Adzhubey
In reply to this post by Ivan Adzhubey
On Thursday, June 1, 2017 11:27:49 PM EDT Ivan Adzhubey wrote:

> [7:0:0:0]    mediumx QUALSTAR TLS-8466         227d  /dev/sch0  /dev/sg4
> [8:0:0:0]    tape    IBM      ULTRIUM-HH5      BBNF  /dev/st0   /dev/sg2
> [8:0:1:0]    tape    IBM      ULTRIUM-HH5      BBNF  /dev/st1   /dev/sg3
>
> But I am still confused about few bits of configuration not very well
> explained in the documentation. I have used LTO-4 Device section from the
> distribution bacula-sd.conf file as a template. Here is my questions:
>
> a) I have added ChangerDevice directive to the Device resources for the
> drives, pointing to their corresponding raw SCSI devices (/dev/sg2, /dev/
> sga3), while Autochager has ChangerDevice configured as /dev/sg4. Is this
> configuration correct? I am assuming these tape raw devices only used for
> the purpose of running AlertCommand on the drives?

Obviously, the above assumption was wrong because btape crashes if I specify
anything other then the autochanger's raw SCSI device as the ChangerDevice
under Device resource for the tape drives.

Does this mean Device {...} will not localize variables specified inside them?
What about other resources? I could not find anything in the documentation
regarding the variables scope inside bacula-X.conf files.

Of course, I might simply hardcode /dev/sgX names into TapeAlert strings.

Thanks,
Ivan



The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Configuring autochanger with SAS LTO-5 drives

Kern Sibbald
In reply to this post by Ivan Adzhubey
Hello,

Thanks for pointing this out.  I have made a few updates to the 7.9
manual (on www.bacula.org).

In fact, it is the Minimum Block Size that is used to force fixed block
size.  That said, for any modern tape drive (LTO) variable block sizes
are the best to use so you should leave Minimum Block Size at 0.

The current maximum for Maximum Block Size is 4,000,000.  If you want to
use something bigger than this, you must edit the source code yourself
and set a larger value.  If you do so, you will possibly encounter two
problems:

1. Writing blocks bigger than 1GB can potentially lead to more drive
data errors and thus possible data loss.

2. Your restore times for individual files will be longer since the
index granularity for file restoration is the Maximum File Size.

Some people recommend very large block sizes -- 2-4MB.  You are welcome
to do it, but the performance gain is not enormous, and I personally
prefer to be conservative and to know that my data is safe.

Best regards,

Kern


On 06/02/2017 07:31 PM, Ivan Adzhubey wrote:

> Hi Rudolf,
>
> Thanks for the prompt reply. Please scroll down for inline comments.
>
> On Friday, June 02, 2017 09:52:58 AM Cejka Rudolf wrote:
>> Ivan Adzhubey wrote (2017/06/01):
>>> b) What is the effect of MaximumFileSize option and what would be its
>>> optimal value for my IBM LTO-5 SAS drives? I have used 8GB value found in
>>> one of the list posts, while the documentation suggests 2GB for LTO-4.
>>> But even set at 8GB this would create lots of EOF marks on a 1.5TB tape,
>>> do we really need so many?
>> Hi, I do use 16 GB. Every EOF mark means around 3 seconds delay. So if you
>> have over 200 files on the tape using 8 GB, it is around 10 minutes extra
>> per tape.
> Searching throughout the list, I am seeing widely varying numbers quoted for
> working LTO tape configurations. I guess what it means is that the parameter is
> not critical. But thanks for the estimate, looks like 10GB value makes sense.
>
>>> c) Should I try increasing the tape block size? Set it to fixed? I know
>>> this
>> I think that there is no reason to use fixed tape block size. On the other
>> side, increasing tape block size would help, there are many discussions
>> about that.
> Documentation states (rather vaguely, I admit) that you only would want to set
> MaximumBlockSize to use fixed block sizes:
>
> "Maximum block size = size-in-bytes On most modern tape drives, you will not
> need to specify this directive. If you do so, it will most likely be to use
> fixed block sizes (see Minimum block size above)."
>
> http://www.bacula.org/7.0.x-manuals/en/main/Storage_Daemon_Configuratio.html#SECTION001730000000000000000
>
> I am not sure I understand what the documentation is trying to tell us but, as
> others have mentioned already, whatever it was it is probably no longer true
> for more recent drive models/HBAs and server hardware configurations.
>
> One related question. Default maximum block size is actually specified as
> 126*512 = 64512. This is not equal neither 64KB (base 1000) nor 64K (base
> 1024). Do you know why it is short two 512 blocks? Should I reserve the same
> amount in my maximum block size setting as well?
>
>>> topic has been rather controversial, so any recent experience from similar
>>> drives/system would be appreciated. I plan to use spooling to a dedicated
>>> RAID volume, so hopefully hard drives should not be a bottleneck.
>> Hard drives are potential bottleneck, they can not handle several write
>> streams and at the same time one read stream going 150-300 MB/s. Rather
>> use SSD drives.
> I have seen the SSD argument brought up several times but was never able to
> see any supporting benchmarks. Maybe I misunderstand something important but
> how RAID volume can be a limiting factor when our 3Ware/LSI SAS RAID
> controller easily provides 350 MB/sec sustained read rates (750 MB/sec peak)?
> I do not plan to run concurrent spooling/despooling jobs on the same
> partition, so any drop in performance due to concurrency is unlikely as well,
> even though my benchmarks suggest sustained 150 MB/sec speed for concurrent
> read/write on our RAID6 array.
>
> I would love to see any numbers for recent SSD models since we do not have any
> installed here and I do not want to shell out large amounts of money on
> something I am not sure will give us any significant benefits.
>
> Thanks,
> Ivan
>
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Configuring autochanger with SAS LTO-5 drives

Kern Sibbald
In reply to this post by Ivan Adzhubey
Hello,

In Bacula the "variables" that are set in the bacula-sd.conf file are
normally referred to as Directives.  In general their scope is within
the enclosing brackets that define the particular resource (e.g. Device
{ ... } or Autochanger { ...}

Normally for a Device resource that is part of an autochanger, you do
not need to specify the Changer Device directive or the Changer Command
directive, because they already be defined in the Autochanger resource.  
Within the Autochanger resource, the Changer Device should point to the
scsi control channel for the changer -- in your case below this would be
/dev/sg4.

The Control Device directive within a Device resource is used to specify
the scsi control device for the Drive. This control device is used for
TapeAlerts.  I strongly recommend setting up tape alerts under Bacula
7.9 and later as it will let you know when there are drive problems.  In
prior Bacula version the implementation was too simplistic and so there
was no need to specify a Control Device. Even in 7.9 and later you are
not required to specify a Control Device within the Device resource.

Yes, the Control Device is primarily used for Tape Alerts, but it can
also be used in the Enterprise version for drive locking and autochanger
sharing.

Final note:  Both Changer Device and Changer Command are normally
specified only in an Autochanger resource.  However, the original
autochanger implementation was for a single drive autochanger, and in
that case, Bacula did not have an Autochanger resource as the Device
itself can be a single drive Autochanger by specifying the Changer
Device and Changer Command within the Device resource.  I hope that is
clear, and I see the documentation is not totally clear.  I have
attempted to correct this in the latest manual that is on www.bacula.org.

Best regards,
Kern

On 06/03/2017 05:02 AM, Ivan Adzhubey wrote:

> On Thursday, June 1, 2017 11:27:49 PM EDT Ivan Adzhubey wrote:
>> [7:0:0:0]    mediumx QUALSTAR TLS-8466         227d  /dev/sch0  /dev/sg4
>> [8:0:0:0]    tape    IBM      ULTRIUM-HH5      BBNF  /dev/st0   /dev/sg2
>> [8:0:1:0]    tape    IBM      ULTRIUM-HH5      BBNF  /dev/st1   /dev/sg3
>>
>> But I am still confused about few bits of configuration not very well
>> explained in the documentation. I have used LTO-4 Device section from the
>> distribution bacula-sd.conf file as a template. Here is my questions:
>>
>> a) I have added ChangerDevice directive to the Device resources for the
>> drives, pointing to their corresponding raw SCSI devices (/dev/sg2, /dev/
>> sga3), while Autochager has ChangerDevice configured as /dev/sg4. Is this
>> configuration correct? I am assuming these tape raw devices only used for
>> the purpose of running AlertCommand on the drives?
> Obviously, the above assumption was wrong because btape crashes if I specify
> anything other then the autochanger's raw SCSI device as the ChangerDevice
> under Device resource for the tape drives.
>
> Does this mean Device {...} will not localize variables specified inside them?
> What about other resources? I could not find anything in the documentation
> regarding the variables scope inside bacula-X.conf files.
>
> Of course, I might simply hardcode /dev/sgX names into TapeAlert strings.
>
> Thanks,
> Ivan
>
>
>
> The information in this e-mail is intended only for the person to whom it is
> addressed. If you believe this e-mail was sent to you in error and the e-mail
> contains patient information, please contact the Partners Compliance HelpLine at
> http://www.partners.org/complianceline . If the e-mail was sent to you in error
> but does not contain patient information, please contact the sender and properly
> dispose of the e-mail.
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Bacula-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Configuring autochanger with SAS LTO-5 drives

Ivan Adzhubey
Hi Kern,

Thank you, your reply was most helpful. I am making progress, slowly but
steadily. Hopefully, will have our new backup system up and running early next
week.

Would you recommend building & installing version 7.9 instead of 7.4.4, which
is the one available through Ubuntu repos? We do not need any of the more
advanced features of the recent versions but bug fixes would be welcome.

--Ivan

On Saturday, June 3, 2017 6:02:25 AM EDT Kern Sibbald wrote:

> Hello,
>
> In Bacula the "variables" that are set in the bacula-sd.conf file are
> normally referred to as Directives.  In general their scope is within
> the enclosing brackets that define the particular resource (e.g. Device
> { ... } or Autochanger { ...}
>
> Normally for a Device resource that is part of an autochanger, you do
> not need to specify the Changer Device directive or the Changer Command
> directive, because they already be defined in the Autochanger resource.
> Within the Autochanger resource, the Changer Device should point to the
> scsi control channel for the changer -- in your case below this would be
> /dev/sg4.
>
> The Control Device directive within a Device resource is used to specify
> the scsi control device for the Drive. This control device is used for
> TapeAlerts.  I strongly recommend setting up tape alerts under Bacula
> 7.9 and later as it will let you know when there are drive problems.  In
> prior Bacula version the implementation was too simplistic and so there
> was no need to specify a Control Device. Even in 7.9 and later you are
> not required to specify a Control Device within the Device resource.
>
> Yes, the Control Device is primarily used for Tape Alerts, but it can
> also be used in the Enterprise version for drive locking and autochanger
> sharing.
>
> Final note:  Both Changer Device and Changer Command are normally
> specified only in an Autochanger resource.  However, the original
> autochanger implementation was for a single drive autochanger, and in
> that case, Bacula did not have an Autochanger resource as the Device
> itself can be a single drive Autochanger by specifying the Changer
> Device and Changer Command within the Device resource.  I hope that is
> clear, and I see the documentation is not totally clear.  I have
> attempted to correct this in the latest manual that is on www.bacula.org.
>
> Best regards,
> Kern
>
> On 06/03/2017 05:02 AM, Ivan Adzhubey wrote:
> > On Thursday, June 1, 2017 11:27:49 PM EDT Ivan Adzhubey wrote:
> >> [7:0:0:0]    mediumx QUALSTAR TLS-8466         227d  /dev/sch0  /dev/sg4
> >> [8:0:0:0]    tape    IBM      ULTRIUM-HH5      BBNF  /dev/st0   /dev/sg2
> >> [8:0:1:0]    tape    IBM      ULTRIUM-HH5      BBNF  /dev/st1   /dev/sg3
> >>
> >> But I am still confused about few bits of configuration not very well
> >> explained in the documentation. I have used LTO-4 Device section from the
> >> distribution bacula-sd.conf file as a template. Here is my questions:
> >>
> >> a) I have added ChangerDevice directive to the Device resources for the
> >> drives, pointing to their corresponding raw SCSI devices (/dev/sg2, /dev/
> >> sga3), while Autochager has ChangerDevice configured as /dev/sg4. Is this
> >> configuration correct? I am assuming these tape raw devices only used for
> >> the purpose of running AlertCommand on the drives?
> >
> > Obviously, the above assumption was wrong because btape crashes if I
> > specify anything other then the autochanger's raw SCSI device as the
> > ChangerDevice under Device resource for the tape drives.
> >
> > Does this mean Device {...} will not localize variables specified inside
> > them? What about other resources? I could not find anything in the
> > documentation regarding the variables scope inside bacula-X.conf files.
> >
> > Of course, I might simply hardcode /dev/sgX names into TapeAlert strings.
> >
> > Thanks,
> > Ivan


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Configuring autochanger with SAS LTO-5 drives

Kern Sibbald
Hello Ivan,

See my comments below ...


On 06/04/2017 12:45 AM, Ivan Adzhubey wrote:
> Hi Kern,
>
> Thank you, your reply was most helpful. I am making progress, slowly but
> steadily. Hopefully, will have our new backup system up and running early next
> week.
Good luck with Bacula :-)
>
> Would you recommend building & installing version 7.9 instead of 7.4.4, which
> is the one available through Ubuntu repos? We do not need any of the more
> advanced features of the recent versions but bug fixes would be welcome.
Version 7.9 is a beta version for the moment.  It is what I am running,
and it has a huge number of changes and corrections over 7.4.4. I am
pretty conservative so especially for someone new to Bacula, I would
recommend 7.4.7 for the moment and when version 9.0.0 is released (I
expect before the end of June).

By that time, I hope to have Bacula project binaries for it --
graciously built for me by Bacula Systems, and that might be the time to
consider upgrading to the new version.

Even though I am a Ubuntu LTS user, concerning Bacula, I build from
source, so I am not sure what is available through their repos or ppa.

Best regards,
Kern

>
> --Ivan
>
> On Saturday, June 3, 2017 6:02:25 AM EDT Kern Sibbald wrote:
>> Hello,
>>
>> In Bacula the "variables" that are set in the bacula-sd.conf file are
>> normally referred to as Directives.  In general their scope is within
>> the enclosing brackets that define the particular resource (e.g. Device
>> { ... } or Autochanger { ...}
>>
>> Normally for a Device resource that is part of an autochanger, you do
>> not need to specify the Changer Device directive or the Changer Command
>> directive, because they already be defined in the Autochanger resource.
>> Within the Autochanger resource, the Changer Device should point to the
>> scsi control channel for the changer -- in your case below this would be
>> /dev/sg4.
>>
>> The Control Device directive within a Device resource is used to specify
>> the scsi control device for the Drive. This control device is used for
>> TapeAlerts.  I strongly recommend setting up tape alerts under Bacula
>> 7.9 and later as it will let you know when there are drive problems.  In
>> prior Bacula version the implementation was too simplistic and so there
>> was no need to specify a Control Device. Even in 7.9 and later you are
>> not required to specify a Control Device within the Device resource.
>>
>> Yes, the Control Device is primarily used for Tape Alerts, but it can
>> also be used in the Enterprise version for drive locking and autochanger
>> sharing.
>>
>> Final note:  Both Changer Device and Changer Command are normally
>> specified only in an Autochanger resource.  However, the original
>> autochanger implementation was for a single drive autochanger, and in
>> that case, Bacula did not have an Autochanger resource as the Device
>> itself can be a single drive Autochanger by specifying the Changer
>> Device and Changer Command within the Device resource.  I hope that is
>> clear, and I see the documentation is not totally clear.  I have
>> attempted to correct this in the latest manual that is on www.bacula.org.
>>
>> Best regards,
>> Kern
>>
>> On 06/03/2017 05:02 AM, Ivan Adzhubey wrote:
>>> On Thursday, June 1, 2017 11:27:49 PM EDT Ivan Adzhubey wrote:
>>>> [7:0:0:0]    mediumx QUALSTAR TLS-8466         227d  /dev/sch0  /dev/sg4
>>>> [8:0:0:0]    tape    IBM      ULTRIUM-HH5      BBNF  /dev/st0   /dev/sg2
>>>> [8:0:1:0]    tape    IBM      ULTRIUM-HH5      BBNF  /dev/st1   /dev/sg3
>>>>
>>>> But I am still confused about few bits of configuration not very well
>>>> explained in the documentation. I have used LTO-4 Device section from the
>>>> distribution bacula-sd.conf file as a template. Here is my questions:
>>>>
>>>> a) I have added ChangerDevice directive to the Device resources for the
>>>> drives, pointing to their corresponding raw SCSI devices (/dev/sg2, /dev/
>>>> sga3), while Autochager has ChangerDevice configured as /dev/sg4. Is this
>>>> configuration correct? I am assuming these tape raw devices only used for
>>>> the purpose of running AlertCommand on the drives?
>>> Obviously, the above assumption was wrong because btape crashes if I
>>> specify anything other then the autochanger's raw SCSI device as the
>>> ChangerDevice under Device resource for the tape drives.
>>>
>>> Does this mean Device {...} will not localize variables specified inside
>>> them? What about other resources? I could not find anything in the
>>> documentation regarding the variables scope inside bacula-X.conf files.
>>>
>>> Of course, I might simply hardcode /dev/sgX names into TapeAlert strings.
>>>
>>> Thanks,
>>> Ivan
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Bacula-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Configuring autochanger with SAS LTO-5 drives

Cejka Rudolf
In reply to this post by Cejka Rudolf
Cejka Rudolf wrote (2017/06/02):
> Ivan Adzhubey wrote (2017/06/01):
> > b) What is the effect of MaximumFileSize option and what would be its optimal
> > value for my IBM LTO-5 SAS drives? I have used 8GB value found in one of the
> > list posts, while the documentation suggests 2GB for LTO-4. But even set at
> > 8GB this would create lots of EOF marks on a 1.5TB tape, do we really need so
> > many?
>
> Hi, I do use 16 GB. Every EOF mark means around 3 seconds delay. So if you have
> over 200 files on the tape using 8 GB, it is around 10 minutes extra per tape.

Hi, small fix. It seems that it is even around 5-6 seconds delay resulting in
extra 20 minutes per tape. And worse, I'm afraid that the tape drive do not
stop nor slowdown tape movement, which would mean over 10 % of the tape lost
because of filemarks. Hope that I'm wrong.

--
Rudolf Cejka <cejkar at fit.vutbr.cz> http://www.fit.vutbr.cz/~cejkar
Brno University of Technology, Faculty of Information Technology
Bozetechova 2, 612 66  Brno, Czech Republic

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Configuring autochanger with SAS LTO-5 drives

Alan Brown
On 05/06/17 11:57, Cejka Rudolf wrote:
>
>> Hi, I do use 16 GB. Every EOF mark means around 3 seconds delay. So if you have
>> over 200 files on the tape using 8 GB, it is around 10 minutes extra per tape.
> Hi, small fix. It seems that it is even around 5-6 seconds delay resulting in
> extra 20 minutes per tape. And worse, I'm afraid that the tape drive do not
> stop nor slowdown tape movement, which would mean over 10 % of the tape lost
> because of filemarks. Hope that I'm wrong.
>
Most drives stop and restart. You won't hear it unless you're listening
for it and in a quiet room (I use a stethoscope)


If your OS and kernel supports Tapestat, then you can see it quite
clearly in throughput.


IMHO: The best thing to do with LTO is use the largest block size bacula
will accept and a file size of 16GB or larger - and make sure your (ssd)
spool is large enough to avoid filling it up.

Don't forget: If you alter the block size, you MUST close off all
existing tapes and initialise/recycle new ones. Bacula doesn't mind
different block sizes on different tapes but encountering a change
partway through a reel "may give undesirable results" when reading.



--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Configuring autochanger with SAS LTO-5 drives

Alan Brown
On 05/06/17 15:48, Alan Brown wrote:

> IMHO: The best thing to do with LTO is use the largest block size
> bacula will accept and a file size of 16GB or larger - and make sure
> your (ssd) spool is large enough to avoid filling it up.
>
> Don't forget: If you alter the block size, you MUST close off all
> existing tapes and initialise/recycle new ones. Bacula doesn't mind
> different block sizes on different tapes but encountering a change
> partway through a reel "may give undesirable results" when reading.
>
>
>

Given comments about the block size, I think this should be noted:


Following up on Kern's comment about block sizes (Using BEE 8.8.3, but
should be close to the community versions) and that bacula-sd will
accept larger block sizes than the 2M in documentation.


Yes, it will accept the larger block sizes, BUT: If you set the block
size larger than 2M, bacula-sd silently truncate it to 64kB


Be warned. (Kern, perhaps you should consider this a bug, given that
'bacula-sd -t' doesn't throw an error message)


Experimentation here showed that the throughput advantage tapered off
after 512kB, but 2MB blocks means the drives are being given 72-250
writes/second instead of 1300-2500 @ 64kB, which translates to lower
overall loading on the system (interrupt loads).


There's some confusion between tape buffers, block sizes and actual
drive buffers.


LTO5 drives contain at least 1GB of ram which is used to buffer what's
coming from the computer and feed the internal compression/encryption
engines, then buffer the output of those sections to the actual tape itself.


Block sizes are about what's written to tape.


Tape buffering in this context is the data sitting between the "written
to tape" phase and "confirmed OK" phase (which happens as the written
data is readback inflight from the adjacent read head. If a LTO drive
says something has been written, it _HAS_ been written and verified.)


One of the reasons raised by Baculasystems for not allowing larger block
sizes is the time burned when verification fails and the block has to be
re-recorded, but the reality is that would still only be 1/10-1/20 of
one second at 140MB/s (and it's all handled internally to the drive anyway).


One gotcha is that different LTO drives actually have different maximum
block sizes. IBM FH tend to be 8MB whilst HP FH are 16MB and this may
cause problems trying to read tapes recorded on one drive, on another
one (The HH units all seem to be 8MB).


I believe a better option than manually setting (or defaulting to 64kB)
is to interrogate the drive using sg_tools or tapeinfo and set the
maximum block size based on what it reports. (ie, autotuning):

# tapeinfo -f  /etc/bacula/DEVICES/drive-1-sg
Product Type: Tape Drive
Vendor ID: 'IBM     '
Product ID: 'ULTRIUM-TD6     '
Revision: 'G350'
Attached Changer API: No
SerialNumber: 'F3A2930094'
MinBlock: 1
MaxBlock: 8388608
SCSI ID: 1
SCSI LUN: 0
Ready: no

# sg_read_block_limits /etc/bacula/DEVICES/MSSLYF-1-sg
Read Block Limits results:
         Minimum block size: 1 byte(s)
         Maximum block size: 8388608 byte(s), 8192 KB, 8 MB



--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Configuring autochanger with SAS LTO-5 drives

Kern Sibbald
Hello,

Alain is certainly, for me, the tape expert.  However, some of the
things Alan says about Bacula might need clarification, and after having
reviewed the code, some of the things I recently wrote about Bacula
block sizes are incorrect.

Here are the things I have just reviewed in the code:
- When reading blocks if a size exceeds 20M, an error message is printed.
- If you set the block size to more than 4M, Bacula will detect a
problem and print and error telling you the block size is too big and
that it is being reset to the default.  This is detected when the SD is
starting, so Bacula attempts to the message and it is also sent to the
syslog.  Note, in 7.9.x and greater it should also be put in the Job
output of the first Job that runs.
- There are various other errors that Bacula can detect with the block
size (eg spool size not at least 8 times the block size). In all cases,
it attempts to print an error.  In most cases, these messages should at
least end up in the syslog. As mentioned above these daemon messages
will also be put in the Job output of the first job that runs.

By the way, the Enterprise 8.8.x code and the community 7.9.x code are
as Alain says pretty much the same.  Enterprises 8.6.x code and the
community 7.4.x code are pretty much the same and somewhat different
from the 8.8.x code which has been heavily refactored and many bug fixes
applied (many found while testing the refactoring).

Before release the 9.0.0 official release, I will increase the maximum
block size to 20M.

Best regards,
Kern

On 08/06/2017 17:11, Alan Brown wrote:

> On 05/06/17 15:48, Alan Brown wrote:
>> IMHO: The best thing to do with LTO is use the largest block size
>> bacula will accept and a file size of 16GB or larger - and make sure
>> your (ssd) spool is large enough to avoid filling it up.
>>
>> Don't forget: If you alter the block size, you MUST close off all
>> existing tapes and initialise/recycle new ones. Bacula doesn't mind
>> different block sizes on different tapes but encountering a change
>> partway through a reel "may give undesirable results" when reading.
>>
>>
>>
>
> Given comments about the block size, I think this should be noted:
>
>
> Following up on Kern's comment about block sizes (Using BEE 8.8.3, but
> should be close to the community versions) and that bacula-sd will
> accept larger block sizes than the 2M in documentation.
>
>
> Yes, it will accept the larger block sizes, BUT: If you set the block
> size larger than 2M, bacula-sd silently truncate it to 64kB
>
>
> Be warned. (Kern, perhaps you should consider this a bug, given that
> 'bacula-sd -t' doesn't throw an error message)
>
>
> Experimentation here showed that the throughput advantage tapered off
> after 512kB, but 2MB blocks means the drives are being given 72-250
> writes/second instead of 1300-2500 @ 64kB, which translates to lower
> overall loading on the system (interrupt loads).
>
>
> There's some confusion between tape buffers, block sizes and actual
> drive buffers.
>
>
> LTO5 drives contain at least 1GB of ram which is used to buffer what's
> coming from the computer and feed the internal compression/encryption
> engines, then buffer the output of those sections to the actual tape
> itself.
>
>
> Block sizes are about what's written to tape.
>
>
> Tape buffering in this context is the data sitting between the
> "written to tape" phase and "confirmed OK" phase (which happens as the
> written data is readback inflight from the adjacent read head. If a
> LTO drive says something has been written, it _HAS_ been written and
> verified.)
>
>
> One of the reasons raised by Baculasystems for not allowing larger
> block sizes is the time burned when verification fails and the block
> has to be re-recorded, but the reality is that would still only be
> 1/10-1/20 of one second at 140MB/s (and it's all handled internally to
> the drive anyway).
>
>
> One gotcha is that different LTO drives actually have different
> maximum block sizes. IBM FH tend to be 8MB whilst HP FH are 16MB and
> this may cause problems trying to read tapes recorded on one drive, on
> another one (The HH units all seem to be 8MB).
>
>
> I believe a better option than manually setting (or defaulting to
> 64kB) is to interrogate the drive using sg_tools or tapeinfo and set
> the maximum block size based on what it reports. (ie, autotuning):
>
> # tapeinfo -f  /etc/bacula/DEVICES/drive-1-sg
> Product Type: Tape Drive
> Vendor ID: 'IBM     '
> Product ID: 'ULTRIUM-TD6     '
> Revision: 'G350'
> Attached Changer API: No
> SerialNumber: 'F3A2930094'
> MinBlock: 1
> MaxBlock: 8388608
> SCSI ID: 1
> SCSI LUN: 0
> Ready: no
>
> # sg_read_block_limits /etc/bacula/DEVICES/MSSLYF-1-sg
> Read Block Limits results:
>         Minimum block size: 1 byte(s)
>         Maximum block size: 8388608 byte(s), 8192 KB, 8 MB
>
>
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Configuring autochanger with SAS LTO-5 drives

Cejka Rudolf
In reply to this post by Alan Brown
Alan Brown wrote (2017/06/08):
> LTO5 drives contain at least 1GB of ram which is used to buffer what's
> coming from the computer and feed the internal compression/encryption
> engines, then buffer the output of those sections to the actual tape itself.

Hello, please, which ones? I just found LTO5 drives with 256 MB from HP
and 512 MB from IBM, but no one with 1 GB.

with 1 GB, I have found LTO6 from IBM or LTO7 from both HP and IBM.

Regards.

--
Rudolf Cejka <cejkar at fit.vutbr.cz> http://www.fit.vutbr.cz/~cejkar
Brno University of Technology, Faculty of Information Technology
Bozetechova 2, 612 66  Brno, Czech Republic

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Configuring autochanger with SAS LTO-5 drives

Cejka Rudolf
In reply to this post by Cejka Rudolf
Cejka Rudolf wrote (2017/06/05):

> Cejka Rudolf wrote (2017/06/02):
> > Ivan Adzhubey wrote (2017/06/01):
> > > b) What is the effect of MaximumFileSize option and what would be its optimal
> > > value for my IBM LTO-5 SAS drives? I have used 8GB value found in one of the
> > > list posts, while the documentation suggests 2GB for LTO-4. But even set at
> > > 8GB this would create lots of EOF marks on a 1.5TB tape, do we really need so
> > > many?
> >
> > Hi, I do use 16 GB. Every EOF mark means around 3 seconds delay. So if you have
> > over 200 files on the tape using 8 GB, it is around 10 minutes extra per tape.
>
> Hi, small fix. It seems that it is even around 5-6 seconds delay resulting in
> extra 20 minutes per tape. And worse, I'm afraid that the tape drive do not
> stop nor slowdown tape movement, which would mean over 10 % of the tape lost
> because of filemarks. Hope that I'm wrong.

Hi, I finally have done and finished real testing with Maximum File Size option.
Tested on LTO-6 drive with LTO-5 tape, so the results are relevant for LTO-5,
used btape utility and fill s (simplified) command.

Fortunately, I has negligible impact on tape capacity, but unfortunately, I has
really dramatical impact on overall write speed and you have to count not just 5-6
seconds, but better 8-10 seconds per file. I also added time to read one file at
140 MB/s, which is important for seeking during restores.

  File   Simplified   Simplified           Bytes     Files   Rough time to
  size    fill time    fill rate         written   written   read one file

  1 GB      6:44:18    61.8 MB/s   1498688192512      1499       00:00:07
  2 GB      4:51:30    85.7 MB/s   1498737278976       750       00:00:14
  4 GB      3:54:57   106.3 MB/s   1498762641408       375       00:00:29
  8 GB      3:24:22   122.2 MB/s   1499224932352       188       00:00:57
 16 GB      3:11:09   130.7 MB/s   1499456405504        94       00:01:54
 32 GB      3:04:35   135.4 MB/s   1499690303488        47       00:03:49
 64 GB      3:01:17   137.8 MB/s   1499574960128        24       00:07:37
128 GB      2:59:37   139.1 MB/s   1499603664896        12       00:15:14

And it seems that the results are perfectly reproducible. Three successive
tests with 16 GB file size have had exactly the same fill time 3:11:09!
Tests were done by mistake, when I wondered how it is possible that the
time stays exactly the same :o)

Regards.

--
Rudolf Cejka <cejkar at fit.vutbr.cz> http://www.fit.vutbr.cz/~cejkar
Brno University of Technology, Faculty of Information Technology
Bozetechova 2, 612 66  Brno, Czech Republic

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Configuring autochanger with SAS LTO-5 drives

Ivan Adzhubey
Hi,

Thank you, Rudolf. This is very useful information! Thank you for taking effort
to benchmark this.

Best,
Ivan

On Friday, June 23, 2017 03:53:04 PM Cejka Rudolf wrote:

> Cejka Rudolf wrote (2017/06/05):
> > Cejka Rudolf wrote (2017/06/02):
> > > Ivan Adzhubey wrote (2017/06/01):
> > > > b) What is the effect of MaximumFileSize option and what would be its
> > > > optimal value for my IBM LTO-5 SAS drives? I have used 8GB value
> > > > found in one of the list posts, while the documentation suggests 2GB
> > > > for LTO-4. But even set at 8GB this would create lots of EOF marks on
> > > > a 1.5TB tape, do we really need so many?
> > >
> > > Hi, I do use 16 GB. Every EOF mark means around 3 seconds delay. So if
> > > you have over 200 files on the tape using 8 GB, it is around 10 minutes
> > > extra per tape.>
> > Hi, small fix. It seems that it is even around 5-6 seconds delay resulting
> > in extra 20 minutes per tape. And worse, I'm afraid that the tape drive
> > do not stop nor slowdown tape movement, which would mean over 10 % of the
> > tape lost because of filemarks. Hope that I'm wrong.
>
> Hi, I finally have done and finished real testing with Maximum File Size
> option. Tested on LTO-6 drive with LTO-5 tape, so the results are relevant
> for LTO-5, used btape utility and fill s (simplified) command.
>
> Fortunately, I has negligible impact on tape capacity, but unfortunately, I
> has really dramatical impact on overall write speed and you have to count
> not just 5-6 seconds, but better 8-10 seconds per file. I also added time
> to read one file at 140 MB/s, which is important for seeking during
> restores.
>
>   File   Simplified   Simplified           Bytes     Files   Rough time to
>   size    fill time    fill rate         written   written   read one file
>
>   1 GB      6:44:18    61.8 MB/s   1498688192512      1499       00:00:07
>   2 GB      4:51:30    85.7 MB/s   1498737278976       750       00:00:14
>   4 GB      3:54:57   106.3 MB/s   1498762641408       375       00:00:29
>   8 GB      3:24:22   122.2 MB/s   1499224932352       188       00:00:57
>  16 GB      3:11:09   130.7 MB/s   1499456405504        94       00:01:54
>  32 GB      3:04:35   135.4 MB/s   1499690303488        47       00:03:49
>  64 GB      3:01:17   137.8 MB/s   1499574960128        24       00:07:37
> 128 GB      2:59:37   139.1 MB/s   1499603664896        12       00:15:14
>
> And it seems that the results are perfectly reproducible. Three successive
> tests with 16 GB file size have had exactly the same fill time 3:11:09!
> Tests were done by mistake, when I wondered how it is possible that the
> time stays exactly the same :o)
>
> Regards.



The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users