Mini-Status report

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Mini-Status report

Kern Sibbald
Hello,

This is a mini-status report just to let you know that someone on one of
these lists said that the new white papers advise using:

  Maximum Concurrent Jobs = 1

In the bacula-sd.conf Device resource.  I have reviewed the three white
papers listed on the bacula.org site and I find no such mention.  Almost
all the examples use MaximumConcurrentJobs=5 and several set it to 10.

If someone can point me to the place (if it exists) where 1 in
mentioned, I will be happy to fix it.

In addition, another person pointed out that manual_prune.pl does not
seem to be available. That was an oversight on my part, and that file is
now included as a download item under White Papers.  The essential
elements of what manual_prune.pl does has been integrated directly into
the Bacula source code, and in the next few days I will update the
documentation to refer to the new way of doing "manual" pruning (i.e.
pruning once a day rather than at the end of every Job).

For those of you who are not aware of the pruning issues, you need to
know that when one has many jobs (say 1000 per day) and many Volumes,
the normal Job pruning that Bacula does at the end of each Job can
become a performance problem (not likely with less than 50 jobs per
day).  To resolve it, it is best to turn off automatic pruning but then
to schedule an Admin job once a day that will do the equivalent.  The
program (now integrated into the code) manual_prune.pl was written by
Bacula Systems for the Enterprise customers, but is published for your
use with the community version.

Finally, I have made a few very minor changes to the Best Practices for
Disk Based Backup document.

Best regards,
Kern

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Bacula-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-devel
Reply | Threaded
Open this post in threaded view
|

Re: Mini-Status report

Kern Sibbald
Hello,

It was a little faster than I had thought, and I have now updated the
Best Practices for Disk Based Backup white paper to include the
information on the new "manual" pruning option.

Best regards,
Kern

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Bacula-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Bacula-users] Mini-Status report

Ana Emília M. Arruda
In reply to this post by Kern Sibbald
Hello Kern,

Is there any problem to set Maximum Concurrent Jobs = 1 in Devices definition?

Setting this directive to a higher value and, in the case of using tape libraries, combined with data spooling for reducing data interleave, should make our backups goes faster. But if someone really don´t need that more than one job writes to one tape and want to have no data interleave, is this configuration a problem?

Best regards,
Ana

On Tue, Nov 25, 2014 at 10:42 AM, Kern Sibbald <[hidden email]> wrote:
Hello,

This is a mini-status report just to let you know that someone on one of
these lists said that the new white papers advise using:

  Maximum Concurrent Jobs = 1

In the bacula-sd.conf Device resource.  I have reviewed the three white
papers listed on the bacula.org site and I find no such mention.  Almost
all the examples use MaximumConcurrentJobs=5 and several set it to 10.

If someone can point me to the place (if it exists) where 1 in
mentioned, I will be happy to fix it.

In addition, another person pointed out that manual_prune.pl does not
seem to be available. That was an oversight on my part, and that file is
now included as a download item under White Papers.  The essential
elements of what manual_prune.pl does has been integrated directly into
the Bacula source code, and in the next few days I will update the
documentation to refer to the new way of doing "manual" pruning (i.e.
pruning once a day rather than at the end of every Job).

For those of you who are not aware of the pruning issues, you need to
know that when one has many jobs (say 1000 per day) and many Volumes,
the normal Job pruning that Bacula does at the end of each Job can
become a performance problem (not likely with less than 50 jobs per
day).  To resolve it, it is best to turn off automatic pruning but then
to schedule an Admin job once a day that will do the equivalent.  The
program (now integrated into the code) manual_prune.pl was written by
Bacula Systems for the Enterprise customers, but is published for your
use with the community version.

Finally, I have made a few very minor changes to the Best Practices for
Disk Based Backup document.

Best regards,
Kern

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Bacula-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Bacula-users] Mini-Status report

Kern Sibbald
Hello Ana,

Nice to hear from you :-)

On 11/25/2014 05:43 PM, Ana Emília M. Arruda wrote:
Hello Kern,

Is there any problem to set Maximum Concurrent Jobs = 1 in Devices definition?

No, but if you have multiple disk devices that point to the same directory, you will get a certain amount of disk fragmentation, whereas if you allow Maximum Concurrent Jobs to be a bit higher (say 5), you will get interleaving of jobs but not disk fragmentation (it probably works out to be roughly the same performance wise).

So, it is only a question of efficiency.

For tapes, see below ...


Setting this directive to a higher value and, in the case of using tape libraries, combined with data spooling for reducing data interleave, should make our backups goes faster. But if someone really don´t need that more than one job writes to one tape and want to have no data interleave, is this configuration a problem?

For tapes, Bacula will definitely run faster if you do data spooling, and as you note it will reduce interleaving too.  For best performance try to have the fastest disks possible, perhaps with some form of RAID (for speed) if you are writing to an LTO-5 or LTO-6 drive. 

If someone does not need more than one job writing to a tape and wants no data interleaving, then setting Maximum Concurrent Jobs = 1 is the best solution.

Best regards,
Kern


On Tue, Nov 25, 2014 at 10:42 AM, Kern Sibbald <[hidden email]> wrote:
Hello,

This is a mini-status report just to let you know that someone on one of
these lists said that the new white papers advise using:

  Maximum Concurrent Jobs = 1

In the bacula-sd.conf Device resource.  I have reviewed the three white
papers listed on the bacula.org site and I find no such mention.  Almost
all the examples use MaximumConcurrentJobs=5 and several set it to 10.

If someone can point me to the place (if it exists) where 1 in
mentioned, I will be happy to fix it.

In addition, another person pointed out that manual_prune.pl does not
seem to be available. That was an oversight on my part, and that file is
now included as a download item under White Papers.  The essential
elements of what manual_prune.pl does has been integrated directly into
the Bacula source code, and in the next few days I will update the
documentation to refer to the new way of doing "manual" pruning (i.e.
pruning once a day rather than at the end of every Job).

For those of you who are not aware of the pruning issues, you need to
know that when one has many jobs (say 1000 per day) and many Volumes,
the normal Job pruning that Bacula does at the end of each Job can
become a performance problem (not likely with less than 50 jobs per
day).  To resolve it, it is best to turn off automatic pruning but then
to schedule an Admin job once a day that will do the equivalent.  The
program (now integrated into the code) manual_prune.pl was written by
Bacula Systems for the Enterprise customers, but is published for your
use with the community version.

Finally, I have made a few very minor changes to the Best Practices for
Disk Based Backup document.

Best regards,
Kern

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users



------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Bacula-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Bacula-users] Mini-Status report

Ana Emília M. Arruda
Hi Kern,

Now it´s very clear for me :-)
Thank you a lot!

Best regards,
Ana

On Tue, Nov 25, 2014 at 4:21 PM, Kern Sibbald <[hidden email]> wrote:
Hello Ana,

Nice to hear from you :-)

On 11/25/2014 05:43 PM, Ana Emília M. Arruda wrote:
Hello Kern,

Is there any problem to set Maximum Concurrent Jobs = 1 in Devices definition?

No, but if you have multiple disk devices that point to the same directory, you will get a certain amount of disk fragmentation, whereas if you allow Maximum Concurrent Jobs to be a bit higher (say 5), you will get interleaving of jobs but not disk fragmentation (it probably works out to be roughly the same performance wise).

So, it is only a question of efficiency.

For tapes, see below ...


Setting this directive to a higher value and, in the case of using tape libraries, combined with data spooling for reducing data interleave, should make our backups goes faster. But if someone really don´t need that more than one job writes to one tape and want to have no data interleave, is this configuration a problem?

For tapes, Bacula will definitely run faster if you do data spooling, and as you note it will reduce interleaving too.  For best performance try to have the fastest disks possible, perhaps with some form of RAID (for speed) if you are writing to an LTO-5 or LTO-6 drive. 

If someone does not need more than one job writing to a tape and wants no data interleaving, then setting Maximum Concurrent Jobs = 1 is the best solution.

Best regards,
Kern



On Tue, Nov 25, 2014 at 10:42 AM, Kern Sibbald <[hidden email]> wrote:
Hello,

This is a mini-status report just to let you know that someone on one of
these lists said that the new white papers advise using:

  Maximum Concurrent Jobs = 1

In the bacula-sd.conf Device resource.  I have reviewed the three white
papers listed on the bacula.org site and I find no such mention.  Almost
all the examples use MaximumConcurrentJobs=5 and several set it to 10.

If someone can point me to the place (if it exists) where 1 in
mentioned, I will be happy to fix it.

In addition, another person pointed out that manual_prune.pl does not
seem to be available. That was an oversight on my part, and that file is
now included as a download item under White Papers.  The essential
elements of what manual_prune.pl does has been integrated directly into
the Bacula source code, and in the next few days I will update the
documentation to refer to the new way of doing "manual" pruning (i.e.
pruning once a day rather than at the end of every Job).

For those of you who are not aware of the pruning issues, you need to
know that when one has many jobs (say 1000 per day) and many Volumes,
the normal Job pruning that Bacula does at the end of each Job can
become a performance problem (not likely with less than 50 jobs per
day).  To resolve it, it is best to turn off automatic pruning but then
to schedule an Admin job once a day that will do the equivalent.  The
program (now integrated into the code) manual_prune.pl was written by
Bacula Systems for the Enterprise customers, but is published for your
use with the community version.

Finally, I have made a few very minor changes to the Best Practices for
Disk Based Backup document.

Best regards,
Kern

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users




------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Bacula-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-devel