bacula-sd file storage question

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

bacula-sd file storage question

Timo Neuvonen
Pasted below is a piece of the  default bacula-sd.conf
My installation is from epel-bacula repo for CentOS 7, but exactly the same
conf example can be found from the latest Bacula source tarball.

Could someone explain why a file storage device is definend as an "dummy"
autochanger, and the two file storage devices to which the autochanger
refers? I can't get the point what this autochanger does - changer command
is an empty string etc, and both the devices use the same /tmp path.

I understand that this could be used as a skeleton, to provide more advanced
autochanger functionality. But since this is the most simplest file storage
device example in the conf file, I'm still wondering if this really is the
simplest way of implementing the file storage?

What would I lose if I simply had only one of the two device resources, no
autochanger resource at all, and the jobs would refer directly to the device
resource? I think it should work too...


Regards,

Timo


#
# Define a Virtual autochanger
#
Autochanger {
  Name = FileChgr1
  Device = FileChgr1-Dev1, FileChgr1-Dev2
  Changer Command = ""
  Changer Device = /dev/null
}

Device {
  Name = FileChgr1-Dev1
  Media Type = File1
  Archive Device = /tmp
  LabelMedia = yes;                   # lets Bacula label unlabeled media
  Random Access = Yes;
  AutomaticMount = yes;               # when device opened, read it
  RemovableMedia = no;
  AlwaysOpen = no;
  Maximum Concurrent Jobs = 5
}

Device {
  Name = FileChgr1-Dev2
  Media Type = File1
  Archive Device = /tmp
  LabelMedia = yes;                   # lets Bacula label unlabeled media
  Random Access = Yes;
  AutomaticMount = yes;               # when device opened, read it
  RemovableMedia = no;
  AlwaysOpen = no;
  Maximum Concurrent Jobs = 5
}



------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: bacula-sd file storage question

Kern Sibbald
Hello,

The Autochanger definition for the SD you show below is a Virtual Autochanger.  It is anything but a dummy, though that is arguable.

If you want more information about it, there are two whitepapers on the bacula.org web site that talk about this feature of Bacula.  See: www.bacula.org -->Documentation-->Whitepapers.  They are:
  • Disk Backup Design PDF
  • Best Practices for Disk Backup PDF

Best regards,

Kern


On 01/09/2017 08:55 PM, Timo Neuvonen wrote:
Pasted below is a piece of the  default bacula-sd.conf
My installation is from epel-bacula repo for CentOS 7, but exactly the same
conf example can be found from the latest Bacula source tarball.

Could someone explain why a file storage device is definend as an "dummy"
autochanger, and the two file storage devices to which the autochanger
refers? I can't get the point what this autochanger does - changer command
is an empty string etc, and both the devices use the same /tmp path.

I understand that this could be used as a skeleton, to provide more advanced
autochanger functionality. But since this is the most simplest file storage
device example in the conf file, I'm still wondering if this really is the
simplest way of implementing the file storage?

What would I lose if I simply had only one of the two device resources, no
autochanger resource at all, and the jobs would refer directly to the device
resource? I think it should work too...


Regards,

Timo


#
# Define a Virtual autochanger
#
Autochanger {
  Name = FileChgr1
  Device = FileChgr1-Dev1, FileChgr1-Dev2
  Changer Command = ""
  Changer Device = /dev/null
}

Device {
  Name = FileChgr1-Dev1
  Media Type = File1
  Archive Device = /tmp
  LabelMedia = yes;                   # lets Bacula label unlabeled media
  Random Access = Yes;
  AutomaticMount = yes;               # when device opened, read it
  RemovableMedia = no;
  AlwaysOpen = no;
  Maximum Concurrent Jobs = 5
}

Device {
  Name = FileChgr1-Dev2
  Media Type = File1
  Archive Device = /tmp
  LabelMedia = yes;                   # lets Bacula label unlabeled media
  Random Access = Yes;
  AutomaticMount = yes;               # when device opened, read it
  RemovableMedia = no;
  AlwaysOpen = no;
  Maximum Concurrent Jobs = 5
} 



------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users



------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: bacula-sd file storage question

Dimitri Maziuk
In reply to this post by Timo Neuvonen
On 01/09/2017 01:55 PM, Timo Neuvonen wrote:

> I'm still wondering if this really is the
> simplest way of implementing the file storage?
>
> What would I lose if I simply had only one of the two device resources, no
> autochanger resource at all, and the jobs would refer directly to the device
> resource? I think it should work too...

What are you trying to achieve? I agree that "virtual autochanger" is a
mind boggle with no obvious practical use, but why are you looking at it
in the first place? One file storage device is exactly how it works,
moreover it's the only way it works natively.

--
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users

signature.asc (197 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: bacula-sd file storage question

Timo Neuvonen
> > I'm still wondering if this really is the
> > simplest way of implementing the file storage?
> >
> > What would I lose if I simply had only one of the two device resources,
> > no autochanger resource at all, and the jobs would refer directly to the
> > device resource? I think it should work too...

> What are you trying to achieve? I agree that "virtual autochanger" is a
> mind boggle with no obvious practical use, but why are you looking at it
> in the first place? One file storage device is exactly how it works,
> moreover it's the only way it works natively.


I'm not trying to achieve anything special, I'm rather trying to avoid doing
anything unnecessarily complicated.
I just want to set up (and  to fully understand my set-up) a simple but
working file storage device for a small business / home office environments,
with no more than just a few concurrent jobs.
By now, I've used Bacula for a decade with tape drives, but only now I'm
about to switch over to disk-based backup.


What confused me is that the example conf files supplied with current Bacula
releases don't give an example of the very basic disk-based setup, but just
show how to setup a virtual disk autochanger.
The whitepapers that Kern referred gave me an impression that the virtual
autochanger setup is primarily for high-end applications where may be
thousands of concurrent jobs, and where it is essential to have a more
precise control on how concurrent jobs simultaneously use several volumes.
I hope I got this right?

So I guess that within my small sandbox I'll be fine with just one file
storage device :-)


Kern and Dimitri, thank you for the answers!

Regards,
Timo



------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: bacula-sd file storage question

Dimitri Maziuk
On 2017-01-10 03:49, Timo Neuvonen wrote:

> What confused me is that the example conf files supplied with current Bacula
> releases don't give an example of the very basic disk-based setup, but just
> show how to setup a virtual disk autochanger.
> The whitepapers that Kern referred gave me an impression that the virtual
> autochanger setup is primarily for high-end applications where may be
> thousands of concurrent jobs, and where it is essential to have a more
> precise control on how concurrent jobs simultaneously use several volumes.
> I hope I got this right?

I haven't looked at example files in quite some time so I wasn't sure.

Personally I have hard time figuring out what practical problem would be
solved by having 2 "storages" writing to the same directory -- maybe if
you want to run backups and restores at the same time all the time... I
use an SSD spool and it's despooling pretty much as fast as the spinning
rust will take it. I expect adding a concurrent writer would only add
contention and slow it all down. YMMV and all that of course.

Dima


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: bacula-sd file storage question

Josh Fisher
In reply to this post by Timo Neuvonen

On 1/10/2017 4:49 AM, Timo Neuvonen wrote:

>
> I'm not trying to achieve anything special, I'm rather trying to avoid doing
> anything unnecessarily complicated.
> I just want to set up (and  to fully understand my set-up) a simple but
> working file storage device for a small business / home office environments,
> with no more than just a few concurrent jobs.
> By now, I've used Bacula for a decade with tape drives, but only now I'm
> about to switch over to disk-based backup.
>
>
> What confused me is that the example conf files supplied with current Bacula
> releases don't give an example of the very basic disk-based setup, but just
> show how to setup a virtual disk autochanger.
> The whitepapers that Kern referred gave me an impression that the virtual
> autochanger setup is primarily for high-end applications where may be
> thousands of concurrent jobs, and where it is essential to have a more
> precise control on how concurrent jobs simultaneously use several volumes.
> I hope I got this right?
>
> So I guess that within my small sandbox I'll be fine with just one file
> storage device :-)

More precisely, that is 'one storage device at a time'. For example,
with RDX or USB drives, or hot-swap SAS/SATA, you may have several
devices that are used for backup volume storage, but only one at a time
can be used with s particular SD-defined Device resource. Of course, for
fixed disk, it truly is a single storage device. I would not recommend
using fixed disk, unless it is used as an intermediate backup storage in
a disk-disk-tape type setup. I guess RAIDed fixed disk might be OK, but
it is important that there be at least two copies of backups on separate
media to protect from media failure.

The simplest approach is just a single Device resource in
bacula-sd.conf, such as:

Device {
   Name = FileStorage
   Media Type = File
   Archive Device = /mnt/backup
   LabelMedia = yes;
   Random Access = Yes;
   AutomaticMount = yes;
   RemovableMedia = no;
   AlwaysOpen = no;
}

Let's say you want to use USB drives. You would then mount, whether
manually or through udev rules, one USB drive at a time at /mnt/backup.
This will allow an unlimited number of volume files by using an
unlimited number of USB drives, one at a time. However, since they can
only be mounted one at a time, only a subset of those volumes can be
made available at any one time. If a job needs volumes from more than
one USB drive, then manually intervention will be needed.

Only one SD Device resource means that only one volume may be used at a
time. It is still possible to run concurrent jobs, however those
concurrent jobs will interlace their data onto the same volume file.
That can make restores slower, but other than that is not too much of an
issue. It is also possible to turn on data spooling, which will spool
concurrent jobs to a temporary spool directory on fixed disk, then
despool the jobs one at a time to the USB drive, eliminating the
interlacing. It depends on the media being used. Generally, USB drives
are far too slow to support concurrent jobs, and it is just as fast to
run one job at a time in the first place.

For faster storage devices like RDX, hot-swap SATA, or if using Virtual
Full jobs, or if the ability to restore while other backups are
occurring is needed, then it is necessary to utilize more than one
volume file concurrently. That is where the virtual autochanger comes
in. The autochanger allows a using multiple Device resources as a single
logical resource known as a Autochanger resource. With the virtual
autochanger, there is still only one mountpoint, so one physical device.
Each Device resource associated with the Autochanger resource defines
the ArchiveDevice to the same directory path, and so reads/writes volume
files in the same (shared) mountpoint directory. But since there are
multiple Device resources, and each can have a volume file open, the
virtual autochanger is a single logical resource that can have more than
one volume file open at a time.

To use tape terminology, a virtual autochanger is similar to a tape
autoloader with a library consisting of a single tape caddie containing
available tapes (the volume files in the ArchiveDevice directory) and
multiple tape drives (the Device resources associated with the
Autochanger resource). Each drive can hold one tape, and a particular
tape can only be loaded into one drive at a time. Each physical disk is
a tape caddie containing volumes, and only one caddie at a time can be
loaded into the library. The simpler single Device resource (no
autochanger) approach is like a tape autoloader with a single-caddie
library and having only a single tape drive. Any tape in the caddie can
be used, but only one at a time.

Then there is vchanger, which uses a different method of implementing a
virtual autochanger. It also uses an Autochanger resource with any
number of associated Device resources, but each Device resource has a
different ArchiveDevice that points to a dynamically generated symlink.
This allows the use of more than one mountpoint, so multiple physical
disks at a time. Again with tape terminology, it simulates a tape
autoloader with multiple drives, like the built-in virtual autochanger,
but with a multi-caddie tape library instead of a single caddie tape
library.

And last, but not least, a future version of Bacula will support a Cloud
storage resource, which I think should allow read/write to an unlimited
number of volumes stored in cloud storage, such as MyCloud.


>
>
> Kern and Dimitri, thank you for the answers!
>
> Regards,
> Timo
>
>
>
> ------------------------------------------------------------------------------
> Developer Access Program for Intel Xeon Phi Processors
> Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today. http://sdm.link/xeonphi
> _______________________________________________
> Bacula-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/bacula-users


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: bacula-sd file storage question

Alan Brown
In reply to this post by Dimitri Maziuk
On 09/01/17 23:01, Dimitri Maziuk wrote:
>
> What are you trying to achieve? I agree that "virtual autochanger" is a
> mind boggle with no obvious practical use, but why are you looking at it
> in the first place?

The primary advantage of the virtual autochanger is you can more easily
use removable hard drives or other temporarily-connected storage.

For fixed storage, I'd say it's not worth bothering - but I'll give this
warning.

Backups are for data you care enough about to keep in at least 2
separate physical (not virtual) locations. Anything attached to the
backup system is not separate.

In the event of a system compromise where the intruder deletes files: If
the backups are in any way, shape or form able to be deleted without
manual physical intervention (ie, on mounted drives, or even on
connected but not mounted drives(which can be mounted for formatted)),
then you should expect that they _WILL_ be deleted.

I know of several organisations who have found this out the hard way.
One had set sets of disks online used in GFS rotation (all three got
wiped by a particularly nasty script kiddy who took down an ISP
webserver holding about 2500 websites.) Others have had backups wiped
via accidental "rm -rf /" events.

Not all of them managed to stay in business. ALL of them suffered major
reputational damage and incurred costs trying to recover.

The same thing applies if all your backup tapes are in the changer (only
the tapes being currently written should be accessible, the rest should
be in a fire safe), but at least wiping them takes longer.





------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users