Add external drive rotation to existing backup regime?

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Add external drive rotation to existing backup regime?

David Andrzejewski
I'm *relatively* new to bacula. I have a working bacula setup where I have a bunch of machines backing up to a ZFS array on a server. This is working well and without error, I've tested restores; everything seems hunky-dory. Basically, I do a full backup of each client every couple of weeks, daily incrementals in between, and I don't prune backups until they're 60 days old.

The next step is to add a rotation of external USB hard drives to this setup, so that I can store backups offsite as well. My thought is that I can start with two drives, one on-site and one off-site, and periodically switch them out. Maybe add more drives to the rotation later on.

I've looked at vchanger, and I *think* that's what I need here. The problem is, I'm not entirely clear on how it will work with my current setup. I want to be sure that I can do complete restores even if the server and on-site external drive is totally destroyed. My worry is that if I misconfigure it, I will end up with some of my full backups on one of the drives but not both, incrementals on the other, etc.

I want to be sure I configure this to have the full backups and incremental backups on the ZFS pool as it is now, as well as on ALL external hard drives, so only one of the drives is needed to do a full restore of everything, at least up until the point where it was unmounted from the system.

How can I be sure of all of this?

Thanks!

- Dave

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Add external drive rotation to existing backup regime?

Dimitri Maziuk
On 05/09/2017 05:21 PM, David Andrzejewski wrote:

> I've looked at vchanger, and I *think* that's what I need here.

I don't know if you can combine vchanger and your zfs pool backups. I
would think copy jobs may be a better fit for what you want.

Although I'd be tempted to just cp the volume files to the usb drive in
a RunAfterJob script.

--
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users

signature.asc (197 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Add external drive rotation to existing backup regime?

Andrea Venturoli
In reply to this post by David Andrzejewski
On 05/10/17 00:21, David Andrzejewski wrote:

> I've looked at vchanger, and I *think* that's what I need here.

That's what I'm using.



> I want to be sure that I can do complete restores even if the
> server and on-site external drive is totally destroyed.

Well, of course you'll test this before disaster strikes...



> I want to be sure I configure this to have the full backups and
> incremental backups on the ZFS pool as it is now, as well as on ALL
> external hard drives, so only one of the drives is needed to do a full
> restore of everything, at least up until the point where it was
> unmounted from the system.
>
> How can I be sure of all of this?

I'm using copy jobs, altough I had to write a custom query to manually
select the latest Full, latest Differential and all subsequent Incrementals.

Consolidating them into a single volume would be nicer, but I found no
way to put that up with the fact that I still want bacula to use the
original volumes if they are avaliable.



  bye
        av.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Add external drive rotation to existing backup regime?

Martin Simmons
In reply to this post by David Andrzejewski
>>>>> On Tue, 09 May 2017 22:21:59 +0000, David Andrzejewski said:
>
> I'm *relatively* new to bacula. I have a working bacula setup where I have a
> bunch of machines backing up to a ZFS array on a server. This is working
> well and without error, I've tested restores; everything seems
> hunky-dory. Basically, I do a full backup of each client every couple of
> weeks, daily incrementals in between, and I don't prune backups until
> they're 60 days old.
>
> The next step is to add a rotation of external USB hard drives to this
> setup, so that I can store backups offsite as well. My thought is that I can
> start with two drives, one on-site and one off-site, and periodically switch
> them out. Maybe add more drives to the rotation later on.
>
> I've looked at vchanger, and I *think* that's what I need here. The problem
> is, I'm not entirely clear on how it will work with my current setup. I want
> to be sure that I can do complete restores even if the server and on-site
> external drive is totally destroyed. My worry is that if I misconfigure it,
> I will end up with some of my full backups on one of the drives but not
> both, incrementals on the other, etc.
>
> I want to be sure I configure this to have the full backups and incremental
> backups on the ZFS pool as it is now, as well as on ALL external hard
> drives, so only one of the drives is needed to do a full restore of
> everything, at least up until the point where it was unmounted from the
> system.
>
> How can I be sure of all of this?

If each USB hard disk is large enough to contain all of your backups, then the
simplest approach is to rsync the volumes from the server to the USB disks.

In the (hopefully unlikely) case where you need to restore from the USB disk,
either rsync them back to the server or change the bacula-sd config to point
to the USB disk.

__Martin

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Add external drive rotation to existing backup regime?

Josip Deanovic
On Wednesday 2017-05-10 11:17:41 Martin Simmons wrote:
> If each USB hard disk is large enough to contain all of your backups,
> then the simplest approach is to rsync the volumes from the server to
> the USB disks.
>
> In the (hopefully unlikely) case where you need to restore from the USB
> disk, either rsync them back to the server or change the bacula-sd
> config to point to the USB disk.

I usually don't like the idea of rsyncing bacula volumes and although
an example where such approach would be acceptable probably exist I don't
think this is the case here because of the requirements that were set in
the initial post.

Rsyncing of the volumes would take a lot of time unless one is willing
to use "--append-verify" rsync option.
One would also have to make sure that at the time of rsyncing volumes
those volumes are not being modified.

The problem with this approach is that original poster is looking for
a way to keep one or multiple removable media (e.g. large usb disk)
offline and offsite for some period of time (e.g. month) and then
periodically replace the media.

--
Josip Deanovic

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Add external drive rotation to existing backup regime?

Josh Fisher
In reply to this post by David Andrzejewski

On 5/9/2017 7:11 PM, Dimitri Maziuk wrote:
On 05/09/2017 05:21 PM, David Andrzejewski wrote:

I've looked at vchanger, and I *think* that's what I need here.
I don't know if you can combine vchanger and your zfs pool backups. I
would think copy jobs may be a better fit for what you want.

Vchanger needs to create and delete symlinks to volume files on a mounted filesystem. It doesn't care what filesystem, as long as it can create symlinks to the volume files using the symlink() system call. The symlink itself is stored in the vchanger work directory, not on the filesystem containing volume files.

But I agree that copy jobs are a good fit. I would use copy jobs to copy jobs stored in volumes on the ZFS pool into volumes on USB disk using vchanger. This gives two different volumes, both physically and in the Bacula catalog. Since there are two unique volumes, the one going to USB disk can be placed in another pool named 'offsite' by using the NextPool directive for the copy job. This makes it easier to identify where the volumes are located.


Although I'd be tempted to just cp the volume files to the usb drive in
a RunAfterJob script.



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Add external drive rotation to existing backup regime?

Dimitri Maziuk
In reply to this post by Josip Deanovic
On 2017-05-10 05:44, Josip Deanovic wrote:
>
> I usually don't like the idea of rsyncing bacula volumes and although
> an example where such approach would be acceptable probably exist I don't
> think this is the case here because of the requirements that were set in
> the initial post.

I've a 'canoncal' vchanger setup with individual hard drives as
magazines. Post-job script rsyncs the current magazine to another drive.
The idea is that spinning rust fails and it'll more likely fail while
being written to. So if my current 'magazine' keels over, I'll only lose
the latest backup, there's a copy everything else on it.

The couple of problems I ran into is 'not current' disks failing and
bacula's volume selection algorithm: it randomly writes to a 'not
current' magazine sometimes and defeats my post-script. But overall it's
working OK.

It sounds to me that the ability to stream the same job to 2 storages at
once would solve my problem, and OP's. Which is something I asked about
in May 2013 and Kern replied "no but we'll implement copy jobs" -- those
don't work for me but may work for the OP if I understand his
requirement correctly.

Dima


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users