Delete volume file after retention time

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Delete volume file after retention time

Daniel Heitepriem
Hi guys,

I need some clarification regarding the terms "retention time" and "prune":

"Retention time": The time a file record and/or job record is stored in
the database. For example: "File retention = 30 days" means that a file
record is stored in the database for 30 days. If you do a restore in
this time you are able to select the file directly. After day 30 you can
just restore the whole job without a selection by files

"Prune": With the "AutoPrune" directive a volume is marked as "recycled"
and can be reused when the volume retention period is over. But the
volume isn't automatically rewritten unless "Recycle = yes" is present,
right?


My second question is whether it is possible for bacula to delete a
volume (also the created volume file on a hard disk) after a specific
amount of time (e.g. 4 weeks)? So I create a backup at the beginning of
the month and at the end of the month bacula is automatically deleting
all records from the database (File/Job/Volume retention would apply
here?) and also deletes the created volume file from the disk.

Thank you very much for enlightening me,

Daniel Heitepriem


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Delete volume file after retention time

Daniel Heitepriem
On 18.01.17 16:16, Wanderlei Huttel wrote:
Hello Daniel

There are 2 parameters (File Retention and Job Retention). They are responsible to keep files and jobs in the catalog for the period you have set.
Hello Wanderlei,

alright, got it.
Prune (Autoprune = Yes) is used to clean (Files and Jobs) from catalog using the parameters (File Retention and Job Retention)

There is another parameter (Volume Retention) that is used to keep the data safe in the Volume (physical volume)
So "Volume Retention" kind of protects the volume for a specific time to not get used again, right? If this period is over the volume gets purged so every data on it is wiped out and it can be rewritten with new data thus be seen as a "clean" volume like it had been freshly created.
Bacula never delete physical volume. If you want this, is necessary to create an admin job to run a script and do this job. 

Best Regards

Wanderlei Hüttel
 Ok an admin job which runs once every x weeks shouldn't be a big problem.

Thank you very much and best regards,
Daniel Heitepriem

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Delete volume file after retention time

Phil Stracchino-2
In reply to this post by Daniel Heitepriem
On 01/18/17 09:27, Daniel Heitepriem wrote:
> My second question is whether it is possible for bacula to delete a
> volume (also the created volume file on a hard disk) after a specific
> amount of time (e.g. 4 weeks)? So I create a backup at the beginning of
> the month and at the end of the month bacula is automatically deleting
> all records from the database (File/Job/Volume retention would apply
> here?) and also deletes the created volume file from the disk.


Bacula itself will not do this.  However, you can do it using a properly
constructed admin job.

Consider the following Pool definition:


Pool {
  Name = Incr-Disk
  Storage = babylon4-file
  Pool Type = Backup
  Recycle = no
  Recycle Oldest Volume = no
  Recycle Current Volume = no
  AutoPrune = yes
  Volume Retention = 1 month
  File Retention = 1 month
  Maximum Volume Jobs = 0
  Volume Use Duration = 23h
  Label Format =
"INCR-$Year${Month:p/2/0/r}${Day:p/2/0/r}-${Hour:p/2/0/r}:${Minute:p/2/0/r}"
  RecyclePool = Scratch
}


This pool auto-creates new disk volumes as needed, allows each volume to
be used for a single day's incremental backups, auto-prunes all of the
jobs on the volume when it is one month old, and then recycles it into
the Scratch pool.

Once in the Scratch pool, it is picked up by this admin job:


Job {
  Name = "Clean Expired Volumes"
  Type = Admin
  Enabled = Yes
  Pool = Scratch
  FileSet = Dummy
  Storage = babylon4-file
  Client = babylon4
  Level = Full
  RunBeforeJob = "/etc/bacula/clean_volumes -v"
  Rerun Failed Levels = yes
  Messages = Daemon
  Priority = 20
  Allow Duplicate Jobs = no
  Cancel Queued Duplicates = yes
  Schedule = "Volume Cleanup"
}


The admin job calls this Perl script, which calls bconsole to delete the
volumes from Bacula's catalog, and then physically deletes the disk files:


#!/usr/bin/perl
use strict;
use Getopt::Long;
use IPC::Open2;
use IO::Handle;


my $bconsole = '/usr/sbin/bconsole';
my (%opts, @purged, $pid);

GetOptions(\%opts,
           'verbose|v',
           'test');

my ($IN, $OUT) = (IO::Handle->new(), IO::Handle->new());

$pid = open2($OUT, $IN, $bconsole) || die "Unable to open bconsole";

if (scalar (@purged = check_volumes()))
{
    printf("Bacula reports the following purged volumes:\n\t%s\n",
           join("\n\t", @purged)) if ($opts{verbose});
    my $deleted = delete_volumes(@purged);
    print "$deleted volumes deleted.\n" if ($opts{verbose});
}
elsif ($opts{verbose})
{
    print "No purged volumes found to delete.\n";
}

print $IN "exit\n";
waitpid($pid, 0);

exit (0);


sub check_volumes
{
    my $dividers = 0;
    my (@purged, @row);

    print $IN "list volumes pool=Scratch\n";
    for (;;)
    {
        my $resp = <$OUT>;
        last if ($resp =~ /No results to list./);
        $dividers++ if ($resp =~ /^[\+\-]+$/);
        last if ($dividers == 3);
        @row = split(/\s+/, $resp);
        push (@purged, $row[3]) if ($row[5] eq 'Purged');
    }

    return (@purged);
}


sub delete_volumes
{
    my $volume_dir = '/spool/bacula/';
    my $count = 0;

    foreach my $vol (@_)
    {
        my $l;
        my $file = $volume_dir.$vol;

        print "Deleting volume $vol from catalog ... " if ($opts{verbose});
        print $IN "delete volume=$vol yes\n";
        $l = <$OUT>;
        $l = <$OUT>;
        print "Done.\nDeleting volume $file from disk ... " if
($opts{verbose});
        if (-f $file)
        {
            $count++;
            unlink ($file);
        }
        print "Done.\n" if ($opts{verbose});
    }

    return ($count);
}


--
  Phil Stracchino
  Babylon Communications
  [hidden email]
  [hidden email]
  Landline: 603.293.8485

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Delete volume file after retention time

Ana Emília M. Arruda
In reply to this post by Daniel Heitepriem
Hello Daniel and Wanderlei,

There is a misunderstanding regarding the 'Volume Retention'  period.

On Thu, Jan 19, 2017 at 10:54 AM, Daniel Heitepriem <[hidden email]> wrote:
On 18.01.17 16:16, Wanderlei Huttel wrote:
Prune (Autoprune = Yes) is used to clean (Files and Jobs) from catalog using the parameters (File Retention and Job Retention)

There is another parameter (Volume Retention) that is used to keep the data safe in the Volume (physical volume)
So "Volume Retention" kind of protects the volume for a specific time to not get used again, right? If this period is over the volume gets purged so every data on it is wiped out and it can be rewritten with new data thus be seen as a "clean" volume like it had been freshly created.

​If Recycle=yes and Autoprune=yes are configured in a Pool resource, a volume can be reused before its Volume Retention period is reached. This will happen if bacula needs a volume and it can't create (label) a new one and it doesn't find any available (Append or Purged status) in the Pool.

If this condition is reached, then bacula will look if the oldest volume (Used or Full) has no jobs/files entries in catalog (this means all the jobs and files backed up in this volume were previously pruned from catalog). If this happens, this volume can potentially be reused even before its retention time is expired.

For example:

1) a Client has configured: AutoPrune=yes, Job Retention = 90 days, File Retention = 30 days
2) the job for this client uses a Pool configured: Recycle=yes, AutoPrune=yes, Volume Retention = 180 days, no automatic label configured
3) let's suppose I run a full backup for this client every day and each job is backed up into one volume

Now suppose I have only 120  volumes in this pool:

VOL-0001 to VOL-0120

For the job run after 120 days, bacula will not be able to create a new one (automatic label is not configured) and I will have all 120 volumes marked as "Used" (so no available volume to be used).

In this case, bacula will start the recycling algorithm (because of Recycle=yes).

Since I have AutoPrune=yes for my client, all jobs and files older than 90 days had been pruned (after each job run for this client, the autoprune of jobs and files is done). So I will have the 30 oldest volumes with no more jobs and files entries in Catalog.

In this case, the recycling algorithm will select the oldest volume in the Pool (last time written) and check if it has jobs and files entries in catalog. Since the jobs and files associated to this volume were previously pruned (after 90 days, because of client having AutoPrune configured), bacula will reuse this volume despite of the fact its retention period was not reached (180 days).

Hope this helps.

Best regards,
Ana


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users