Bacula Install Ubuntu 16.04

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Bacula Install Ubuntu 16.04

Brian Blater
Anyone have a nice easy to follow How to for installing and configuring Bacula on Ubuntu 16.04? I've tried several I've found online and they all fall short or don't deal with 16.04.

Thanks,
Brian

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Bacula Install Ubuntu 16.04

compdoc

I would install ubuntu server 16.04x and select lamp if you like mysql, as well as openssh, and any other options that are listed on the initial setup screen. Dont use the option to choose more packages, which opens additional screens. Doing that never goes well for me. you can add that stuff later.

then once you've booted and updated and rebooted, there are packages you can install with: sudo apt install bacula

And I think there are variations like:  sudo apt install bacula-fd

Search for which are available.

After that, you need to modify the files in /etc/bacula/ to fit.

Because i have more control of how/where bacula is installed on the server, I compile the files from sourceforge.

I have notes for that, but its a lot of work.



On 05/29/2017 12:18 PM, Brian Blater wrote:
Anyone have a nice easy to follow How to for installing and configuring Bacula on Ubuntu 16.04? I've tried several I've found online and they all fall short or don't deal with 16.04.

Thanks,
Brian


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Bacula Install Ubuntu 16.04

compdoc
On 05/31/2017 09:42 AM, Brian Blater wrote:
> This is an area where I need help. Lots of configuration files in this
> directory and coming from a corporate world using netbackup or
> symantec I'm a little confused.

You dont have to mess with the files in /etc/bacula/scripts/, except if
you need to run any of those. Only the .conf files in /etc/bacula/.

There are lots of tutorials online. Many of which are from the bacula
people, and others from individuals.

I see bacula as a kind of programming language for backups, but you dont
have to start from scratch. Just take examples and modify them to suit.

Like programming, it takes lots of reading and experimenting with the
options to understand how, or which work best for you.

You're right - not much like symantec-type backup programs. But
extremely reliable and customizable, and very good support. Free or paid.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Bacula Install Ubuntu 16.04

Wanderlei Huttel
In reply to this post by Brian Blater
Hello Brian


Best regards

Wanderlei Hüttel

2017-05-29 15:18 GMT-03:00 Brian Blater <[hidden email]>:
Anyone have a nice easy to follow How to for installing and configuring Bacula on Ubuntu 16.04? I've tried several I've found online and they all fall short or don't deal with 16.04.

Thanks,
Brian

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Bacula LTO-5

Steven Hammond-2
I've been reading through some of the articles about settings for my HP
LTO-5 drive.  I have a question concerning FIXED vs VARIABLE block size.

1. Is it safe (Bacula 7.XX) to set the block size to something other
than 64K?
2. Does increasing the block size increase the throughput?  (2 x 3.6Ghz
PROCS - 24 cores, 32GB RAM, 6x600GB 15K RPM SAS, external SAS  / HP LTO-5)
3. Are there any down sides to increasing the block size?  (e.g.,
existing tapes created)

SD configuration looks like:

Device {
   Name = LTO-5
   Media Type = LTO-5
   Archive Device = /dev/nst0
   AutomaticMount = Yes
   AlwaysOpen = Yes
   RemovableMedia = Yes
   RandomAccess = No
   AutoChanger = No
   Maximum Spool Size = 500G
   Maximum Job Spool Size = 100G
   Maximum Concurrent Jobs = 5
   Spool Directory = /var/spool/bacula
}

Despooling speeds avg 60+ MB/s.  I also have a quad nic bonded
(bond-mode 6 with MTU 9000).  I did run the btape test and speed. It's
results were much faster.

block size = 64K
btape -c /etc/bacula/bacula-sd.conf /dev/nst0
speed file_size=5 nb_file=8
Zero data / Raw Avg: 280 MB/s
Random data / Raw Avg: 111 MB/s
Zero / bacula Avg: 168 MB/s
Random / bacula Avg: 108 MB/s

O/S: Ubuntu 17.04 (so we could get a later version of bacula without
compiling from source)

I'm not overly concerned about the speed (everything backs up over night
fine) but I was hoping to get over 100 MB/s seeing that the LTO-5
theoretically can go up to 144 MB/s.
Any tips would be greatly appreciated.

Steven Hammond

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Bacula LTO-5

Cejka Rudolf
Steven Hammond wrote (2017/05/31):
> 1. Is it safe (Bacula 7.XX) to set the block size to something other
> than 64K?

You have to try it, but it should be safe in Linux. Write data with
bigger blocks and read them back with source comparison. btape should
do the job too.

> 2. Does increasing the block size increase the throughput?  (2 x 3.6Ghz
> PROCS - 24 cores, 32GB RAM, 6x600GB 15K RPM SAS, external SAS  / HP LTO-5)

If the disks are not a bottleneck, then in Linux it should increase the
throughput very much. Yes, you have very good disks, but experiences say,
that you really need SSD disks to take full advantage of LTO-5.

> 3. Are there any down sides to increasing the block size?  (e.g.,
> existing tapes created)

Lower portability. If you want to read the tape back, you need a system,
which does support atleast the same block size, which depends on used
operating system, host adapter and its driver. It also depends on tape
drive, but just do not use block bigger than 8 MB and you are safe.
However, it is recommended do not use blocks bigger than 2 MB and please
do not exceed it if you do not know what you are doing.

> Despooling speeds avg 60+ MB/s.  I also have a quad nic bonded

It is too low for LTO-5. It is at the lower limit before tape
shoe shining, problably with some tape stops already here.
The real limit is 47 MB/s for HP and 40 MB/s for IBM, but no
guarantee it is correct.

> O/S: Ubuntu 17.04 (so we could get a later version of bacula without
> compiling from source)
>
> I'm not overly concerned about the speed (everything backs up over night

You are concerned, because you want to protect your tape drive from
a malfunction.

> fine) but I was hoping to get over 100 MB/s seeing that the LTO-5
> theoretically can go up to 144 MB/s.

Yes, my typical speeds with LTO-5 are 140-280 MB/s on FreeBSD with
64 KB blocks, but FreeBSD is not so sensitive to block sizes as is Linux.

> Any tips would be greatly appreciated.

Yes, really try to do something with increasing tape write speed and do
not go under 100 MB/s even in low peaks.

--
Rudolf Cejka <cejkar at fit.vutbr.cz> http://www.fit.vutbr.cz/~cejkar
Brno University of Technology, Faculty of Information Technology
Bozetechova 2, 612 66  Brno, Czech Republic

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Bacula LTO-5

Kern Sibbald
In reply to this post by Steven Hammond-2
On 31/05/2017 21:43, Steven Hammond wrote:
> I've been reading through some of the articles about settings for my
> HP LTO-5 drive.  I have a question concerning FIXED vs VARIABLE block
> size.
>
> 1. Is it safe (Bacula 7.XX) to set the block size to something other
> than 64K?
Yes.  However, once you have written a tape with a given block size, you
cannot changed the block size unless you re-initialized the tape.

> 2. Does increasing the block size increase the throughput?  (2 x
> 3.6Ghz PROCS - 24 cores, 32GB RAM, 6x600GB 15K RPM SAS, external SAS  
> / HP LTO-5)

Yes.  If you are running lots of jobs, it can significantly increase
performance.

> 3. Are there any down sides to increasing the block size?  (e.g.,
> existing tapes created)
Yes.  The optimal size, IMO, is about 512K.  Making the block size much
bigger can increase the chances of a tape error.  That said there are
Bacula users that use 1MB or even large without problems.

On my system I use 256K because it performs very well, and I am not
interested to squeeze every tiny bit of performance from the system.

>
> SD configuration looks like:
>
> Device {
>   Name = LTO-5
>   Media Type = LTO-5
>   Archive Device = /dev/nst0
>   AutomaticMount = Yes
>   AlwaysOpen = Yes
>   RemovableMedia = Yes
>   RandomAccess = No
>   AutoChanger = No
>   Maximum Spool Size = 500G
>   Maximum Job Spool Size = 100G
>   Maximum Concurrent Jobs = 5
>   Spool Directory = /var/spool/bacula
> }
>
> Despooling speeds avg 60+ MB/s.  I also have a quad nic bonded
> (bond-mode 6 with MTU 9000).  I did run the btape test and speed. It's
> results were much faster.
>
> block size = 64K
> btape -c /etc/bacula/bacula-sd.conf /dev/nst0
> speed file_size=5 nb_file=8
> Zero data / Raw Avg: 280 MB/s
> Random data / Raw Avg: 111 MB/s
> Zero / bacula Avg: 168 MB/s
> Random / bacula Avg: 108 MB/s
>
> O/S: Ubuntu 17.04 (so we could get a later version of bacula without
> compiling from source)
>
> I'm not overly concerned about the speed (everything backs up over
> night fine) but I was hoping to get over 100 MB/s seeing that the
> LTO-5 theoretically can go up to 144 MB/s.
> Any tips would be greatly appreciated.

You will need multiple concurrent jobs to see high speed writes. You
also need very fast spooling disks.

Best regards,
Kern

>
> Steven Hammond
>
> ------------------------------------------------------------------------------
>
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Bacula-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Bacula LTO-5

Cejka Rudolf
In reply to this post by Cejka Rudolf
Richard Fox wrote (2017/06/01):
> Otherwise, this advice is a little contradictory to the documentation which states "On most modern tape drives, you will not need to specify this directive.".

Given that Linux with LTO-X tape drive is probably a majority system here (not counting
configurations without tape drives), the statement is slightly misleading. I'm convinced
that it is really not needed because of tape drive, server nor HBA, but it seems that it
is really needed because of Linux. However it is not a real problem, because Linux allows
to increase the block size "naturally", with the exception that you have limiting HBA.

> More importantly however, the documentation (for Bacula 7.2 anyways) says: "The maximum size-in-bytes possible is 2,000,000." which contradicts the assertion that these can be specified as 2MB which is not the same thing. Is the documentation inaccurate on this subject?

I wrote 2 MB as a general recommendation over various manufacturers and software
developers, with non-written suggested value 256 KB or 512 KB as max., so please
take my 2 MB limit just loosely :o)

--
Rudolf Cejka <cejkar at fit.vutbr.cz> http://www.fit.vutbr.cz/~cejkar
Brno University of Technology, Faculty of Information Technology
Bozetechova 2, 612 66  Brno, Czech Republic

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Bacula LTO-5

Cejka Rudolf
In reply to this post by Cejka Rudolf
Cejka Rudolf wrote (2017/06/01):
> It is too low for LTO-5. It is at the lower limit before tape
> shoe shining, problably with some tape stops already here.
> The real limit is 47 MB/s for HP and 40 MB/s for IBM, but no
> guarantee it is correct.

Small clarification about this: Speeds 47 MB/s or 40 MB/s are
physical writing speeds after compression, so uncompressed datas
from server to tape drive have to be faster. And because of
variable compression ratio over time, typically between 1:1 and
2:1, then you are safe just with much bigger writing speeds.
From the other side, if you have writing speed under 100 MB/s,
it is sometimes recommended to switch off compression, so that
even speeds between 50-100 MB/s are safe.

--
Rudolf Cejka <cejkar at fit.vutbr.cz> http://www.fit.vutbr.cz/~cejkar
Brno University of Technology, Faculty of Information Technology
Bozetechova 2, 612 66  Brno, Czech Republic

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Bacula LTO-5

Steven Hammond-2
In reply to this post by Kern Sibbald
Kern,

Ok, thanks.  I am running concurrent jobs (10) and spooling the
data/attributes.  I will try 512K.  I assume I will need to set both the
MINIMUM BLOCK SIZE and MAXIMUM BLOCK SIZE.  Or is just setting the
MAXIMUM BLOCK SIZE sufficient?  Also, what would you recommend for a
MAXIMUM FILE SIZE?

Oh, just fyi, here is a graph I found on the Internet.
https://www.bareos.org/en/Whitepapers/articles/Speed_Tuning_of_Tape_Drives.html

Steven Hammond


On 6/1/2017 7:04 AM, Kern Sibbald wrote:

> On 31/05/2017 21:43, Steven Hammond wrote:
>> I've been reading through some of the articles about settings for my
>> HP LTO-5 drive.  I have a question concerning FIXED vs VARIABLE block
>> size.
>>
>> 1. Is it safe (Bacula 7.XX) to set the block size to something other
>> than 64K?
> Yes.  However, once you have written a tape with a given block size,
> you cannot changed the block size unless you re-initialized the tape.
>
>> 2. Does increasing the block size increase the throughput?  (2 x
>> 3.6Ghz PROCS - 24 cores, 32GB RAM, 6x600GB 15K RPM SAS, external SAS  
>> / HP LTO-5)
>
> Yes.  If you are running lots of jobs, it can significantly increase
> performance.
>
>> 3. Are there any down sides to increasing the block size?  (e.g.,
>> existing tapes created)
> Yes.  The optimal size, IMO, is about 512K.  Making the block size
> much bigger can increase the chances of a tape error.  That said there
> are Bacula users that use 1MB or even large without problems.
>
> On my system I use 256K because it performs very well, and I am not
> interested to squeeze every tiny bit of performance from the system.
>
>>
>> SD configuration looks like:
>>
>> Device {
>>   Name = LTO-5
>>   Media Type = LTO-5
>>   Archive Device = /dev/nst0
>>   AutomaticMount = Yes
>>   AlwaysOpen = Yes
>>   RemovableMedia = Yes
>>   RandomAccess = No
>>   AutoChanger = No
>>   Maximum Spool Size = 500G
>>   Maximum Job Spool Size = 100G
>>   Maximum Concurrent Jobs = 5
>>   Spool Directory = /var/spool/bacula
>> }
>>
>> Despooling speeds avg 60+ MB/s.  I also have a quad nic bonded
>> (bond-mode 6 with MTU 9000).  I did run the btape test and speed.
>> It's results were much faster.
>>
>> block size = 64K
>> btape -c /etc/bacula/bacula-sd.conf /dev/nst0
>> speed file_size=5 nb_file=8
>> Zero data / Raw Avg: 280 MB/s
>> Random data / Raw Avg: 111 MB/s
>> Zero / bacula Avg: 168 MB/s
>> Random / bacula Avg: 108 MB/s
>>
>> O/S: Ubuntu 17.04 (so we could get a later version of bacula without
>> compiling from source)
>>
>> I'm not overly concerned about the speed (everything backs up over
>> night fine) but I was hoping to get over 100 MB/s seeing that the
>> LTO-5 theoretically can go up to 144 MB/s.
>> Any tips would be greatly appreciated.
>
> You will need multiple concurrent jobs to see high speed writes. You
> also need very fast spooling disks.
>
> Best regards,
> Kern
>>
>> Steven Hammond
>>
>> ------------------------------------------------------------------------------
>>
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Bacula-users mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>>
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Bacula LTO-5

Steven Hammond-2
In reply to this post by Kern Sibbald
Oh, should I do hardware compression or software compression? Reason I
ask, I tried just hardware but didn't seem to get much out of the tape
(1.7TB).  However, with client side compression, my file server was
compressed nearly 80+%.  I assume the LTO-5 is set to hardware
compression by default (it appears to be enabled when I run tapeinfo).  
I don't mind the extra time needed for client side compression, but I
don't want to slow down the tape drive trying to compress data that is
already compressed.

Steven Hammond


On 6/1/2017 7:04 AM, Kern Sibbald wrote:

> On 31/05/2017 21:43, Steven Hammond wrote:
>> I've been reading through some of the articles about settings for my
>> HP LTO-5 drive.  I have a question concerning FIXED vs VARIABLE block
>> size.
>>
>> 1. Is it safe (Bacula 7.XX) to set the block size to something other
>> than 64K?
> Yes.  However, once you have written a tape with a given block size,
> you cannot changed the block size unless you re-initialized the tape.
>
>> 2. Does increasing the block size increase the throughput?  (2 x
>> 3.6Ghz PROCS - 24 cores, 32GB RAM, 6x600GB 15K RPM SAS, external SAS  
>> / HP LTO-5)
>
> Yes.  If you are running lots of jobs, it can significantly increase
> performance.
>
>> 3. Are there any down sides to increasing the block size?  (e.g.,
>> existing tapes created)
> Yes.  The optimal size, IMO, is about 512K.  Making the block size
> much bigger can increase the chances of a tape error.  That said there
> are Bacula users that use 1MB or even large without problems.
>
> On my system I use 256K because it performs very well, and I am not
> interested to squeeze every tiny bit of performance from the system.
>
>>
>> SD configuration looks like:
>>
>> Device {
>>   Name = LTO-5
>>   Media Type = LTO-5
>>   Archive Device = /dev/nst0
>>   AutomaticMount = Yes
>>   AlwaysOpen = Yes
>>   RemovableMedia = Yes
>>   RandomAccess = No
>>   AutoChanger = No
>>   Maximum Spool Size = 500G
>>   Maximum Job Spool Size = 100G
>>   Maximum Concurrent Jobs = 5
>>   Spool Directory = /var/spool/bacula
>> }
>>
>> Despooling speeds avg 60+ MB/s.  I also have a quad nic bonded
>> (bond-mode 6 with MTU 9000).  I did run the btape test and speed.
>> It's results were much faster.
>>
>> block size = 64K
>> btape -c /etc/bacula/bacula-sd.conf /dev/nst0
>> speed file_size=5 nb_file=8
>> Zero data / Raw Avg: 280 MB/s
>> Random data / Raw Avg: 111 MB/s
>> Zero / bacula Avg: 168 MB/s
>> Random / bacula Avg: 108 MB/s
>>
>> O/S: Ubuntu 17.04 (so we could get a later version of bacula without
>> compiling from source)
>>
>> I'm not overly concerned about the speed (everything backs up over
>> night fine) but I was hoping to get over 100 MB/s seeing that the
>> LTO-5 theoretically can go up to 144 MB/s.
>> Any tips would be greatly appreciated.
>
> You will need multiple concurrent jobs to see high speed writes. You
> also need very fast spooling disks.
>
> Best regards,
> Kern
>>
>> Steven Hammond
>>
>> ------------------------------------------------------------------------------
>>
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Bacula-users mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>>
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Bacula LTO-5

Alan Brown
On 01/06/17 15:02, Steven Hammond wrote:
> Oh, should I do hardware compression or software compression? Reason I
> ask, I tried just hardware but didn't seem to get much out of the tape
> (1.7TB).  However, with client side compression, my file server was
> compressed nearly 80+%.  I assume the LTO-5 is set to hardware
> compression by default (it appears to be enabled when I run
> tapeinfo).  I don't mind the extra time needed for client side
> compression, but I don't want to slow down the tape drive trying to
> compress data that is already compressed.

The hardware compression on LTO is a lot faster than the raw speed of
the tape - and if what comes out of the compressor is enlarged (which
happens when compressing already-compressed data) it stands aside.

There are no circumstances where what's fed as input into the tape drive
should slow the drive down below the raw speed of the tape (140MB/s).

On the other hand precompressing on the client side uses substantial
resources and may slow your backups down considerably.


If you're "only" seeing despool speeds of 60MB/s then there's a
bottleneck somewhere in the system which needs investigating and fixing.
Despool speed should be the same as the raw speed of the tape (for
uncompressible data) or faster.

It is generally counterprpoductive to use client-side or client-server
compression over networks running at 1Gb/s or faster.

>
> Steven Hammond
>
>
> On 6/1/2017 7:04 AM, Kern Sibbald wrote:
>> On 31/05/2017 21:43, Steven Hammond wrote:
>>> I've been reading through some of the articles about settings for my
>>> HP LTO-5 drive.  I have a question concerning FIXED vs VARIABLE
>>> block size.
>>>
>>> 1. Is it safe (Bacula 7.XX) to set the block size to something other
>>> than 64K?
>> Yes.  However, once you have written a tape with a given block size,
>> you cannot changed the block size unless you re-initialized the tape.
>>
>>> 2. Does increasing the block size increase the throughput?  (2 x
>>> 3.6Ghz PROCS - 24 cores, 32GB RAM, 6x600GB 15K RPM SAS, external
>>> SAS  / HP LTO-5)
>>
>> Yes.  If you are running lots of jobs, it can significantly increase
>> performance.
>>
>>> 3. Are there any down sides to increasing the block size?  (e.g.,
>>> existing tapes created)
>> Yes.  The optimal size, IMO, is about 512K.  Making the block size
>> much bigger can increase the chances of a tape error.  That said
>> there are Bacula users that use 1MB or even large without problems.
>>
>> On my system I use 256K because it performs very well, and I am not
>> interested to squeeze every tiny bit of performance from the system.
>>
>>>
>>> SD configuration looks like:
>>>
>>> Device {
>>>   Name = LTO-5
>>>   Media Type = LTO-5
>>>   Archive Device = /dev/nst0
>>>   AutomaticMount = Yes
>>>   AlwaysOpen = Yes
>>>   RemovableMedia = Yes
>>>   RandomAccess = No
>>>   AutoChanger = No
>>>   Maximum Spool Size = 500G
>>>   Maximum Job Spool Size = 100G
>>>   Maximum Concurrent Jobs = 5
>>>   Spool Directory = /var/spool/bacula
>>> }
>>>
>>> Despooling speeds avg 60+ MB/s.  I also have a quad nic bonded
>>> (bond-mode 6 with MTU 9000).  I did run the btape test and speed.
>>> It's results were much faster.
>>>
>>> block size = 64K
>>> btape -c /etc/bacula/bacula-sd.conf /dev/nst0
>>> speed file_size=5 nb_file=8
>>> Zero data / Raw Avg: 280 MB/s
>>> Random data / Raw Avg: 111 MB/s
>>> Zero / bacula Avg: 168 MB/s
>>> Random / bacula Avg: 108 MB/s
>>>
>>> O/S: Ubuntu 17.04 (so we could get a later version of bacula without
>>> compiling from source)
>>>
>>> I'm not overly concerned about the speed (everything backs up over
>>> night fine) but I was hoping to get over 100 MB/s seeing that the
>>> LTO-5 theoretically can go up to 144 MB/s.
>>> Any tips would be greatly appreciated.
>>
>> You will need multiple concurrent jobs to see high speed writes. You
>> also need very fast spooling disks.
>>
>> Best regards,
>> Kern
>>>
>>> Steven Hammond
>>>
>>> ------------------------------------------------------------------------------
>>>
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>> _______________________________________________
>>> Bacula-users mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>>>
>>
>
>
> ------------------------------------------------------------------------------
>
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Bacula-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>


--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Bacula LTO-5

Kern Sibbald
In reply to this post by Cejka Rudolf
I don't seem to have the original post of Richard Fox, so could you
please specify what "this directive" is in the sentence:

Otherwise, this advice is a little contradictory to the documentation which states "On most modern tape drives, you will not need to specify this directive.

Best regards,
Kern


On 06/01/2017 02:51 PM, Cejka Rudolf wrote:

> Richard Fox wrote (2017/06/01):
>> Otherwise, this advice is a little contradictory to the documentation which states "On most modern tape drives, you will not need to specify this directive.".
> Given that Linux with LTO-X tape drive is probably a majority system here (not counting
> configurations without tape drives), the statement is slightly misleading. I'm convinced
> that it is really not needed because of tape drive, server nor HBA, but it seems that it
> is really needed because of Linux. However it is not a real problem, because Linux allows
> to increase the block size "naturally", with the exception that you have limiting HBA.
>
>> More importantly however, the documentation (for Bacula 7.2 anyways) says: "The maximum size-in-bytes possible is 2,000,000." which contradicts the assertion that these can be specified as 2MB which is not the same thing. Is the documentation inaccurate on this subject?
> I wrote 2 MB as a general recommendation over various manufacturers and software
> developers, with non-written suggested value 256 KB or 512 KB as max., so please
> take my 2 MB limit just loosely :o)
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Bacula LTO-5

Kern Sibbald
In reply to this post by Steven Hammond-2
Hello Steven,

I recommend setting the Minimum Block Size to 0 (the default).

For LTO-5 I recommend using Maximum File Size = 5 GB

Best regards,

Kern




On 06/01/2017 03:58 PM, Steven Hammond wrote:

> Kern,
>
> Ok, thanks.  I am running concurrent jobs (10) and spooling the
> data/attributes.  I will try 512K.  I assume I will need to set both
> the MINIMUM BLOCK SIZE and MAXIMUM BLOCK SIZE.  Or is just setting the
> MAXIMUM BLOCK SIZE sufficient?  Also, what would you recommend for a
> MAXIMUM FILE SIZE?
>
> Oh, just fyi, here is a graph I found on the Internet.
> https://www.bareos.org/en/Whitepapers/articles/Speed_Tuning_of_Tape_Drives.html
>
> Steven Hammond
>
>
> On 6/1/2017 7:04 AM, Kern Sibbald wrote:
>> On 31/05/2017 21:43, Steven Hammond wrote:
>>> I've been reading through some of the articles about settings for my
>>> HP LTO-5 drive.  I have a question concerning FIXED vs VARIABLE
>>> block size.
>>>
>>> 1. Is it safe (Bacula 7.XX) to set the block size to something other
>>> than 64K?
>> Yes.  However, once you have written a tape with a given block size,
>> you cannot changed the block size unless you re-initialized the tape.
>>
>>> 2. Does increasing the block size increase the throughput?  (2 x
>>> 3.6Ghz PROCS - 24 cores, 32GB RAM, 6x600GB 15K RPM SAS, external
>>> SAS  / HP LTO-5)
>>
>> Yes.  If you are running lots of jobs, it can significantly increase
>> performance.
>>
>>> 3. Are there any down sides to increasing the block size?  (e.g.,
>>> existing tapes created)
>> Yes.  The optimal size, IMO, is about 512K.  Making the block size
>> much bigger can increase the chances of a tape error.  That said
>> there are Bacula users that use 1MB or even large without problems.
>>
>> On my system I use 256K because it performs very well, and I am not
>> interested to squeeze every tiny bit of performance from the system.
>>
>>>
>>> SD configuration looks like:
>>>
>>> Device {
>>>   Name = LTO-5
>>>   Media Type = LTO-5
>>>   Archive Device = /dev/nst0
>>>   AutomaticMount = Yes
>>>   AlwaysOpen = Yes
>>>   RemovableMedia = Yes
>>>   RandomAccess = No
>>>   AutoChanger = No
>>>   Maximum Spool Size = 500G
>>>   Maximum Job Spool Size = 100G
>>>   Maximum Concurrent Jobs = 5
>>>   Spool Directory = /var/spool/bacula
>>> }
>>>
>>> Despooling speeds avg 60+ MB/s.  I also have a quad nic bonded
>>> (bond-mode 6 with MTU 9000).  I did run the btape test and speed.
>>> It's results were much faster.
>>>
>>> block size = 64K
>>> btape -c /etc/bacula/bacula-sd.conf /dev/nst0
>>> speed file_size=5 nb_file=8
>>> Zero data / Raw Avg: 280 MB/s
>>> Random data / Raw Avg: 111 MB/s
>>> Zero / bacula Avg: 168 MB/s
>>> Random / bacula Avg: 108 MB/s
>>>
>>> O/S: Ubuntu 17.04 (so we could get a later version of bacula without
>>> compiling from source)
>>>
>>> I'm not overly concerned about the speed (everything backs up over
>>> night fine) but I was hoping to get over 100 MB/s seeing that the
>>> LTO-5 theoretically can go up to 144 MB/s.
>>> Any tips would be greatly appreciated.
>>
>> You will need multiple concurrent jobs to see high speed writes. You
>> also need very fast spooling disks.
>>
>> Best regards,
>> Kern
>>>
>>> Steven Hammond
>>>
>>> ------------------------------------------------------------------------------
>>>
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>> _______________________________________________
>>> Bacula-users mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>>>
>>
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Bacula LTO-5

Kern Sibbald
In reply to this post by Steven Hammond-2
For most people tape compression is the way to go.  In your particular
case you might want to use Client compression, but if you do, it would
be better to disable compression on the tape drive.  Doing so will
probably improve your throughput.


Best regards,

Kern


On 06/01/2017 04:02 PM, Steven Hammond wrote:

> Oh, should I do hardware compression or software compression? Reason I
> ask, I tried just hardware but didn't seem to get much out of the tape
> (1.7TB).  However, with client side compression, my file server was
> compressed nearly 80+%.  I assume the LTO-5 is set to hardware
> compression by default (it appears to be enabled when I run
> tapeinfo).  I don't mind the extra time needed for client side
> compression, but I don't want to slow down the tape drive trying to
> compress data that is already compressed.
>
> Steven Hammond
>
>
> On 6/1/2017 7:04 AM, Kern Sibbald wrote:
>> On 31/05/2017 21:43, Steven Hammond wrote:
>>> I've been reading through some of the articles about settings for my
>>> HP LTO-5 drive.  I have a question concerning FIXED vs VARIABLE
>>> block size.
>>>
>>> 1. Is it safe (Bacula 7.XX) to set the block size to something other
>>> than 64K?
>> Yes.  However, once you have written a tape with a given block size,
>> you cannot changed the block size unless you re-initialized the tape.
>>
>>> 2. Does increasing the block size increase the throughput?  (2 x
>>> 3.6Ghz PROCS - 24 cores, 32GB RAM, 6x600GB 15K RPM SAS, external
>>> SAS  / HP LTO-5)
>>
>> Yes.  If you are running lots of jobs, it can significantly increase
>> performance.
>>
>>> 3. Are there any down sides to increasing the block size?  (e.g.,
>>> existing tapes created)
>> Yes.  The optimal size, IMO, is about 512K.  Making the block size
>> much bigger can increase the chances of a tape error.  That said
>> there are Bacula users that use 1MB or even large without problems.
>>
>> On my system I use 256K because it performs very well, and I am not
>> interested to squeeze every tiny bit of performance from the system.
>>
>>>
>>> SD configuration looks like:
>>>
>>> Device {
>>>   Name = LTO-5
>>>   Media Type = LTO-5
>>>   Archive Device = /dev/nst0
>>>   AutomaticMount = Yes
>>>   AlwaysOpen = Yes
>>>   RemovableMedia = Yes
>>>   RandomAccess = No
>>>   AutoChanger = No
>>>   Maximum Spool Size = 500G
>>>   Maximum Job Spool Size = 100G
>>>   Maximum Concurrent Jobs = 5
>>>   Spool Directory = /var/spool/bacula
>>> }
>>>
>>> Despooling speeds avg 60+ MB/s.  I also have a quad nic bonded
>>> (bond-mode 6 with MTU 9000).  I did run the btape test and speed.
>>> It's results were much faster.
>>>
>>> block size = 64K
>>> btape -c /etc/bacula/bacula-sd.conf /dev/nst0
>>> speed file_size=5 nb_file=8
>>> Zero data / Raw Avg: 280 MB/s
>>> Random data / Raw Avg: 111 MB/s
>>> Zero / bacula Avg: 168 MB/s
>>> Random / bacula Avg: 108 MB/s
>>>
>>> O/S: Ubuntu 17.04 (so we could get a later version of bacula without
>>> compiling from source)
>>>
>>> I'm not overly concerned about the speed (everything backs up over
>>> night fine) but I was hoping to get over 100 MB/s seeing that the
>>> LTO-5 theoretically can go up to 144 MB/s.
>>> Any tips would be greatly appreciated.
>>
>> You will need multiple concurrent jobs to see high speed writes. You
>> also need very fast spooling disks.
>>
>> Best regards,
>> Kern
>>>
>>> Steven Hammond
>>>
>>> ------------------------------------------------------------------------------
>>>
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>> _______________________________________________
>>> Bacula-users mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>>>
>>
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Bacula LTO-5

rfox
In reply to this post by Kern Sibbald
Hi,

On Fri, Jun 02, 2017 at 01:24:46PM +0200, Kern Sibbald wrote:
> I don't seem to have the original post of Richard Fox, so could you please
> specify what "this directive" is in the sentence:
>
> Otherwise, this advice is a little contradictory to the documentation which states "On most modern tape drives, you will not need to specify this directive.

My apologies, I had sent the message from the wrong address and
cancelled moderation on it. I was hoping nobody would notice.

My original message asked if this discussion was in regard to the
"Maximum block size" (and presumably "Minimum block size") from the
device resource.

"Are you both referring to "Maximum block size" (and presumably "Minimum block size") from the
Device resource?
If not please ignore the rest of this message. Otherwise, this advice is a little contradictory
to the documentation which states "On most modern tape drives, you will not need to specify this
directive.".
More importantly however, the documentation (for Bacula 7.2 anyways) says: "The maximum
size-in-bytes possible is 2,000,000." which contradicts the assertion that these can be specified
as 2MB which is not the same thing. Is the documentation inaccurate on
this subject?"

> On 06/01/2017 02:51 PM, Cejka Rudolf wrote:
> > Richard Fox wrote (2017/06/01):
> > > Otherwise, this advice is a little contradictory to the documentation which states "On most modern tape drives, you will not need to specify this directive.".
> > Given that Linux with LTO-X tape drive is probably a majority system here (not counting
> > configurations without tape drives), the statement is slightly misleading. I'm convinced
> > that it is really not needed because of tape drive, server nor HBA, but it seems that it
> > is really needed because of Linux. However it is not a real problem, because Linux allows
> > to increase the block size "naturally", with the exception that you have limiting HBA.
> >
> > > More importantly however, the documentation (for Bacula 7.2 anyways) says: "The maximum size-in-bytes possible is 2,000,000." which contradicts the assertion that these can be specified as 2MB which is not the same thing. Is the documentation inaccurate on this subject?
> > I wrote 2 MB as a general recommendation over various manufacturers and software
> > developers, with non-written suggested value 256 KB or 512 KB as max., so please
> > take my 2 MB limit just loosely :o)

Thanks,
Rich.

--


 Rich Fox
 Systems Administrator
 JBPC - Marine Biological Laboratory
 [hidden email] - [hidden email]
 508-289-7669



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Bacula LTO-5

Kern Sibbald
Hello,

See below ...



On 06/02/2017 01:44 PM, Richard Fox wrote:

> Hi,
>
> On Fri, Jun 02, 2017 at 01:24:46PM +0200, Kern Sibbald wrote:
>> I don't seem to have the original post of Richard Fox, so could you please
>> specify what "this directive" is in the sentence:
>>
>> Otherwise, this advice is a little contradictory to the documentation which states "On most modern tape drives, you will not need to specify this directive.
> My apologies, I had sent the message from the wrong address and
> cancelled moderation on it. I was hoping nobody would notice.
>
> My original message asked if this discussion was in regard to the
> "Maximum block size" (and presumably "Minimum block size") from the
> device resource.
>
> "Are you both referring to "Maximum block size" (and presumably "Minimum block size") from the
> Device resource?
As for me: Yes.
> If not please ignore the rest of this message. Otherwise, this advice is a little contradictory
> to the documentation which states "On most modern tape drives, you will not need to specify this
> directive.".
Yes, this is completely correct.  If you really want to squeeze every
ounce of performance in writing drives, you might want to increase the
Maximum Block Size.

> More importantly however, the documentation (for Bacula 7.2 anyways) says: "The maximum
> size-in-bytes possible is 2,000,000." which contradicts the assertion that these can be specified
> as 2MB which is not the same thing. Is the documentation inaccurate on
> this subject?"
Unless I am missing something, in Bacula 1000000  and 1 MB are the same
thing.  In Bacula 2 M is 1048576 bytes.  This may be a bit confusing,
but it is historic.

Best regards,
Kern

>
>> On 06/01/2017 02:51 PM, Cejka Rudolf wrote:
>>> Richard Fox wrote (2017/06/01):
>>>> Otherwise, this advice is a little contradictory to the documentation which states "On most modern tape drives, you will not need to specify this directive.".
>>> Given that Linux with LTO-X tape drive is probably a majority system here (not counting
>>> configurations without tape drives), the statement is slightly misleading. I'm convinced
>>> that it is really not needed because of tape drive, server nor HBA, but it seems that it
>>> is really needed because of Linux. However it is not a real problem, because Linux allows
>>> to increase the block size "naturally", with the exception that you have limiting HBA.
>>>
>>>> More importantly however, the documentation (for Bacula 7.2 anyways) says: "The maximum size-in-bytes possible is 2,000,000." which contradicts the assertion that these can be specified as 2MB which is not the same thing. Is the documentation inaccurate on this subject?
>>> I wrote 2 MB as a general recommendation over various manufacturers and software
>>> developers, with non-written suggested value 256 KB or 512 KB as max., so please
>>> take my 2 MB limit just loosely :o)
> Thanks,
> Rich.
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Bacula LTO-5

Steven Hammond-2
Thanks for all the input.  I've made some changes and was able to get a
bit more throughput.  I'll will continue to *tweak* the settings and
test.  Thanks again Kern for a great product.

Steven Hammond


On 6/2/2017 8:28 AM, Kern Sibbald wrote:

> Hello,
>
> See below ...
>
>
>
> On 06/02/2017 01:44 PM, Richard Fox wrote:
>> Hi,
>>
>> On Fri, Jun 02, 2017 at 01:24:46PM +0200, Kern Sibbald wrote:
>>> I don't seem to have the original post of Richard Fox, so could you
>>> please
>>> specify what "this directive" is in the sentence:
>>>
>>> Otherwise, this advice is a little contradictory to the
>>> documentation which states "On most modern tape drives, you will not
>>> need to specify this directive.
>> My apologies, I had sent the message from the wrong address and
>> cancelled moderation on it. I was hoping nobody would notice.
>>
>> My original message asked if this discussion was in regard to the
>> "Maximum block size" (and presumably "Minimum block size") from the
>> device resource.
>>
>> "Are you both referring to "Maximum block size" (and presumably
>> "Minimum block size") from the
>> Device resource?
> As for me: Yes.
>> If not please ignore the rest of this message. Otherwise, this advice
>> is a little contradictory
>> to the documentation which states "On most modern tape drives, you
>> will not need to specify this
>> directive.".
> Yes, this is completely correct.  If you really want to squeeze every
> ounce of performance in writing drives, you might want to increase the
> Maximum Block Size.
>
>> More importantly however, the documentation (for Bacula 7.2 anyways)
>> says: "The maximum
>> size-in-bytes possible is 2,000,000." which contradicts the assertion
>> that these can be specified
>> as 2MB which is not the same thing. Is the documentation inaccurate on
>> this subject?"
> Unless I am missing something, in Bacula 1000000  and 1 MB are the
> same thing.  In Bacula 2 M is 1048576 bytes.  This may be a bit
> confusing, but it is historic.
>
> Best regards,
> Kern
>
>>
>>> On 06/01/2017 02:51 PM, Cejka Rudolf wrote:
>>>> Richard Fox wrote (2017/06/01):
>>>>> Otherwise, this advice is a little contradictory to the
>>>>> documentation which states "On most modern tape drives, you will
>>>>> not need to specify this directive.".
>>>> Given that Linux with LTO-X tape drive is probably a majority
>>>> system here (not counting
>>>> configurations without tape drives), the statement is slightly
>>>> misleading. I'm convinced
>>>> that it is really not needed because of tape drive, server nor HBA,
>>>> but it seems that it
>>>> is really needed because of Linux. However it is not a real
>>>> problem, because Linux allows
>>>> to increase the block size "naturally", with the exception that you
>>>> have limiting HBA.
>>>>
>>>>> More importantly however, the documentation (for Bacula 7.2
>>>>> anyways) says: "The maximum size-in-bytes possible is 2,000,000."
>>>>> which contradicts the assertion that these can be specified as 2MB
>>>>> which is not the same thing. Is the documentation inaccurate on
>>>>> this subject?
>>>> I wrote 2 MB as a general recommendation over various manufacturers
>>>> and software
>>>> developers, with non-written suggested value 256 KB or 512 KB as
>>>> max., so please
>>>> take my 2 MB limit just loosely :o)
>> Thanks,
>> Rich.
>>
>
>
> ------------------------------------------------------------------------------
>
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Bacula-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/bacula-users


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users