Quantcast

Files are added to catalog at the end of backup

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Files are added to catalog at the end of backup

Daniel Heitepriem
Hi guys,

I'm using Bacula 7.4.4 and got a script which runs after every backup
and expects a list of file names which were backed up. Now I noticed
that bacula is writing the files to the Catalog at the end of the backup
process, thus the script receives an empty list and doesn't run
properly. Below I got an excerpt of this script:

export HOME; HOME="/opt/bacula"
export LOG; LOG="$HOME/var/cleanup.$1.log"

echo "Cleaning job $1">>$LOG

$HOME/bin/bconsole <<EOF
@output /dev/null
messages
@output $HOME/var/cleanup.$1.set
llist files jobid="$1"
EOF


for file in $( egrep "^\|" $HOME/var/cleanup.$1.set | sed -e "s/\|//g" )
do
   if [ $file != "Filename" ]
   then
     AGE=$(perl -e  "print (-M \"$file\");"|cut -d'.' -f1) ## Get age of
file

     if test -d $file ; then ## If line is  a directory
       echo "Directory $file ... skipping...">>$LOG
     elif echo $file|egrep '.gz$' 2>&1 >/dev/null ; then ## Test if file
has ".gz" suffix
       if [ "$AGE" -gt "$KEEP" ] ; then
         echo "$file is older than $KEEP days -> removing">>$LOG
         pfexec rm $file  ## Removing files older than 7 days
       else
         echo "$file is younger than $KEEP days -> keeping">>$LOG

In Bacula 5 this issue isn't present so the file list is available
almost instantly. Did something change between the versions regarding
this behavior?

Thanks and best regards,
Daniel Heitepriem

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Files are added to catalog at the end of backup

Kern Sibbald
Hello,

When a Bacula backup job terminates, all the File table entries are
already in the catalog.  So about the only thing that makes any sense is
that you are running the script before the backup completes.  Perhaps by
simply adding a
"wait" just before your llist, would solve the problem.  However, be
aware "wait" waits until no jobs are running which may delay the output
if you are running multiple jobs.  There is a "wait JobId=nnn" but I am
not 100% sure it really works correctly (I have a vague recollection of
some problems).

Alternatively, it seems to me that use the appropriate RunScript +
options to do the work, you can guarantee that it runs at the end of the
job.

Best regards,
Kern

On 02/02/2017 09:57 AM, Daniel Heitepriem wrote:

> Hi guys,
>
> I'm using Bacula 7.4.4 and got a script which runs after every backup
> and expects a list of file names which were backed up. Now I noticed
> that bacula is writing the files to the Catalog at the end of the backup
> process, thus the script receives an empty list and doesn't run
> properly. Below I got an excerpt of this script:
>
> export HOME; HOME="/opt/bacula"
> export LOG; LOG="$HOME/var/cleanup.$1.log"
>
> echo "Cleaning job $1">>$LOG
>
> $HOME/bin/bconsole <<EOF
> @output /dev/null
> messages
> @output $HOME/var/cleanup.$1.set
> llist files jobid="$1"
> EOF
>
>
> for file in $( egrep "^\|" $HOME/var/cleanup.$1.set | sed -e "s/\|//g" )
> do
>     if [ $file != "Filename" ]
>     then
>       AGE=$(perl -e  "print (-M \"$file\");"|cut -d'.' -f1) ## Get age of
> file
>
>       if test -d $file ; then ## If line is  a directory
>         echo "Directory $file ... skipping...">>$LOG
>       elif echo $file|egrep '.gz$' 2>&1 >/dev/null ; then ## Test if file
> has ".gz" suffix
>         if [ "$AGE" -gt "$KEEP" ] ; then
>           echo "$file is older than $KEEP days -> removing">>$LOG
>           pfexec rm $file  ## Removing files older than 7 days
>         else
>           echo "$file is younger than $KEEP days -> keeping">>$LOG
>
> In Bacula 5 this issue isn't present so the file list is available
> almost instantly. Did something change between the versions regarding
> this behavior?
>
> Thanks and best regards,
> Daniel Heitepriem
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Bacula-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Files are added to catalog at the end of backup

Daniel Heitepriem

Hello Kern,

thank you for your answer. Here is my job definition:

Job {
  Name = "monitor-host.logs"
  JobDefs = "DefaultJob"
  Storage = EAST
  Schedule = "Early"
  Write Bootstrap = "/opt/bacula/var/bacula/working/monitor-host.bsr"
  Client = monitor-host
  FileSet = "monitor-host.logs"
  RunScript {
    RunsWhen = After
    FailJobOnError = No
    Command = "/opt/bacula/libexec/cleanup.sh %i"
    RunsOnSuccess = Yes
    RunsOnClient = Yes
  }
}

FileSet {
  Name = "monitor-host.logs"
  Include {
    Options {
      signature = MD5
      compression=GZIP
    }
    File = /logs/nagios/
  }
  Exclude {
    File = /logs/nagios/nagios.log
  }
}

As you can see the jox uses the "RunScript"-directive. Here is the cleanup.$jobid$.set file

llist files jobid="79"
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
           JobId: 79
             Job: monitor-host.logs.2017-02-01_23.59.00_14
            Name: monitor-host.logs
     PurgedFiles: 0
            Type: B
           Level: I
        ClientId: 31
      ClientName: monitor-host
       JobStatus: R
       SchedTime: 2017-02-01 23:59:00
       StartTime: 2017-02-01 23:59:11
         EndTime: NULL
     RealEndTime: NULL
        JobTDate: 1,485,989,951
    VolSessionId: 0
  VolSessionTime: 0
        JobFiles: 0
        JobBytes: 0
       ReadBytes: 0
       JobErrors: 0
 JobMissingFiles: 0
          PoolId: 1
        PooLname: Incremental
      PriorJobId: 0
       FileSetId: 65
         FileSet: monitor-host.logs
         HasBase: 0
        HasCache: 0
         Comment:

You have messages.

and the corresponding log file

-bash-4.1$ cat cleanup.79.log
Cleaning job 79
-bash-4.1$

When I run the file daemon with debug options this is the output when the script is running:

HOST-fd: job.c:314-79 Quit command loop. Canceled=0
HOST-fd: runscript.c:99-79 runscript: running all RUNSCRIPT object (ClientAfterJob) JobStatus=T
HOST-fd: runscript.c:120-79 runscript: try to run *None*:/opt/bacula/libexec/cleanup.sh %i
HOST-fd: runscript.c:158-79 runscript: Run it because SCRIPT_After (/opt/bacula/libexec/cleanup.sh %i,1,0,T)
HOST-fd: runscript.c:219-79 runscript: running a RUNSCRIPT object type=79
HOST-fd: util.c:831-79 edit_job_codes: /opt/bacula/libexec/cleanup.sh %i
HOST-fd: runscript.c:226-79 runscript: running '/opt/bacula/libexec/cleanup.sh 79'...
HOST-fd: runscript.c:254-79 runscript OK

As you can see the output of "llist files jobid=79" is empty and therefore the script does not run as intended. I hope that I made this a little bit clearer.

Best regards,

Daniel


Am 02.02.17 um 14:39 schrieb Kern Sibbald:
Hello,

When a Bacula backup job terminates, all the File table entries are already in the catalog.  So about the only thing that makes any sense is that you are running the script before the backup completes.  Perhaps by simply adding a
"wait" just before your llist, would solve the problem.  However, be aware "wait" waits until no jobs are running which may delay the output if you are running multiple jobs.  There is a "wait JobId=nnn" but I am not 100% sure it really works correctly (I have a vague recollection of some problems).

Alternatively, it seems to me that use the appropriate RunScript + options to do the work, you can guarantee that it runs at the end of the job.

Best regards,
Kern

On 02/02/2017 09:57 AM, Daniel Heitepriem wrote:
Hi guys,

I'm using Bacula 7.4.4 and got a script which runs after every backup
and expects a list of file names which were backed up. Now I noticed
that bacula is writing the files to the Catalog at the end of the backup
process, thus the script receives an empty list and doesn't run
properly. Below I got an excerpt of this script:

export HOME; HOME="/opt/bacula"
export LOG; LOG="$HOME/var/cleanup.$1.log"

echo "Cleaning job $1">>$LOG

$HOME/bin/bconsole <<EOF
@output /dev/null
messages
@output $HOME/var/cleanup.$1.set
llist files jobid="$1"
EOF


for file in $( egrep "^\|" $HOME/var/cleanup.$1.set | sed -e "s/\|//g" )
do
    if [ $file != "Filename" ]
    then
      AGE=$(perl -e  "print (-M \"$file\");"|cut -d'.' -f1) ## Get age of
file

      if test -d $file ; then ## If line is  a directory
        echo "Directory $file ... skipping...">>$LOG
      elif echo $file|egrep '.gz$' 2>&1 >/dev/null ; then ## Test if file
has ".gz" suffix
        if [ "$AGE" -gt "$KEEP" ] ; then
          echo "$file is older than $KEEP days -> removing">>$LOG
          pfexec rm $file  ## Removing files older than 7 days
        else
          echo "$file is younger than $KEEP days -> keeping">>$LOG

In Bacula 5 this issue isn't present so the file list is available
almost instantly. Did something change between the versions regarding
this behavior?

Thanks and best regards,
Daniel Heitepriem

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users



--
Mit freundlichen Gruessen / Best regards
Daniel Heitepriem

pribas GmbH

Valterweg 24-25
65817 Eppstein-Bremthal
Germany

Phone  +49 (0) 6198 57146400
Fax       +49 (0) 6198 57146433
eMail   [hidden email]

Corporate Headquarters: Huenfelden-Dauborn Managing Director: Arnulf Pribas
Registration: Amtsgericht Limburg a. d. Lahn 7HRB874 Tax ID: DE113840457
________________________________________________________________________
This e-mail is confidential. Information in this e-mail is intended for the exclusive use of the individual or entity named above and may constitute information that is privileged or confidential or otherwise protected from disclosure. The information in this e-mail may be read, published, copied and/or forwarded only by the individual or entity named above. Dissemination, distribution, forwarding or copying of this e-mail by anyone other than the intended recipient is prohibited. If you have received this e-mail in error, please notify us immediately by telephone or e-mail and completely delete or destroy any and all disseminated, distributed, forwarded electronic or other copies of the original message and any attachments.



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Files are added to catalog at the end of backup

Kern Sibbald
I see you are running the job on the Client.  On the client, the Job is finished so the script is run, but in the SD and Dir, the Job is still running.  Though it is a bit odd and perhaps confusing, Bacula is functioning as it was designed.

Best regards,
Kern

On 02/02/2017 02:59 PM, Daniel Heitepriem wrote:

Hello Kern,

thank you for your answer. Here is my job definition:

Job {
  Name = "monitor-host.logs"
  JobDefs = "DefaultJob"
  Storage = EAST
  Schedule = "Early"
  Write Bootstrap = "/opt/bacula/var/bacula/working/monitor-host.bsr"
  Client = monitor-host
  FileSet = "monitor-host.logs"
  RunScript {
    RunsWhen = After
    FailJobOnError = No
    Command = "/opt/bacula/libexec/cleanup.sh %i"
    RunsOnSuccess = Yes
    RunsOnClient = Yes
  }
}

FileSet {
  Name = "monitor-host.logs"
  Include {
    Options {
      signature = MD5
      compression=GZIP
    }
    File = /logs/nagios/
  }
  Exclude {
    File = /logs/nagios/nagios.log
  }
}

As you can see the jox uses the "RunScript"-directive. Here is the cleanup.$jobid$.set file

llist files jobid="79"
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
           JobId: 79
             Job: monitor-host.logs.2017-02-01_23.59.00_14
            Name: monitor-host.logs
     PurgedFiles: 0
            Type: B
           Level: I
        ClientId: 31
      ClientName: monitor-host
       JobStatus: R
       SchedTime: 2017-02-01 23:59:00
       StartTime: 2017-02-01 23:59:11
         EndTime: NULL
     RealEndTime: NULL
        JobTDate: 1,485,989,951
    VolSessionId: 0
  VolSessionTime: 0
        JobFiles: 0
        JobBytes: 0
       ReadBytes: 0
       JobErrors: 0
 JobMissingFiles: 0
          PoolId: 1
        PooLname: Incremental
      PriorJobId: 0
       FileSetId: 65
         FileSet: monitor-host.logs
         HasBase: 0
        HasCache: 0
         Comment:

You have messages.

and the corresponding log file

-bash-4.1$ cat cleanup.79.log
Cleaning job 79
-bash-4.1$

When I run the file daemon with debug options this is the output when the script is running:

HOST-fd: job.c:314-79 Quit command loop. Canceled=0
HOST-fd: runscript.c:99-79 runscript: running all RUNSCRIPT object (ClientAfterJob) JobStatus=T
HOST-fd: runscript.c:120-79 runscript: try to run *None*:/opt/bacula/libexec/cleanup.sh %i
HOST-fd: runscript.c:158-79 runscript: Run it because SCRIPT_After (/opt/bacula/libexec/cleanup.sh %i,1,0,T)
HOST-fd: runscript.c:219-79 runscript: running a RUNSCRIPT object type=79
HOST-fd: util.c:831-79 edit_job_codes: /opt/bacula/libexec/cleanup.sh %i
HOST-fd: runscript.c:226-79 runscript: running '/opt/bacula/libexec/cleanup.sh 79'...
HOST-fd: runscript.c:254-79 runscript OK

As you can see the output of "llist files jobid=79" is empty and therefore the script does not run as intended. I hope that I made this a little bit clearer.

Best regards,

Daniel


Am 02.02.17 um 14:39 schrieb Kern Sibbald:
Hello,

When a Bacula backup job terminates, all the File table entries are already in the catalog.  So about the only thing that makes any sense is that you are running the script before the backup completes.  Perhaps by simply adding a
"wait" just before your llist, would solve the problem.  However, be aware "wait" waits until no jobs are running which may delay the output if you are running multiple jobs.  There is a "wait JobId=nnn" but I am not 100% sure it really works correctly (I have a vague recollection of some problems).

Alternatively, it seems to me that use the appropriate RunScript + options to do the work, you can guarantee that it runs at the end of the job.

Best regards,
Kern

On 02/02/2017 09:57 AM, Daniel Heitepriem wrote:
Hi guys,

I'm using Bacula 7.4.4 and got a script which runs after every backup
and expects a list of file names which were backed up. Now I noticed
that bacula is writing the files to the Catalog at the end of the backup
process, thus the script receives an empty list and doesn't run
properly. Below I got an excerpt of this script:

export HOME; HOME="/opt/bacula"
export LOG; LOG="$HOME/var/cleanup.$1.log"

echo "Cleaning job $1">>$LOG

$HOME/bin/bconsole <<EOF
@output /dev/null
messages
@output $HOME/var/cleanup.$1.set
llist files jobid="$1"
EOF


for file in $( egrep "^\|" $HOME/var/cleanup.$1.set | sed -e "s/\|//g" )
do
    if [ $file != "Filename" ]
    then
      AGE=$(perl -e  "print (-M \"$file\");"|cut -d'.' -f1) ## Get age of
file

      if test -d $file ; then ## If line is  a directory
        echo "Directory $file ... skipping...">>$LOG
      elif echo $file|egrep '.gz$' 2>&1 >/dev/null ; then ## Test if file
has ".gz" suffix
        if [ "$AGE" -gt "$KEEP" ] ; then
          echo "$file is older than $KEEP days -> removing">>$LOG
          pfexec rm $file  ## Removing files older than 7 days
        else
          echo "$file is younger than $KEEP days -> keeping">>$LOG

In Bacula 5 this issue isn't present so the file list is available
almost instantly. Did something change between the versions regarding
this behavior?

Thanks and best regards,
Daniel Heitepriem

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users



--
Mit freundlichen Gruessen / Best regards
Daniel Heitepriem

pribas GmbH

Valterweg 24-25
65817 Eppstein-Bremthal
Germany

Phone  +49 (0) 6198 57146400
Fax       +49 (0) 6198 57146433
eMail   [hidden email]

Corporate Headquarters: Huenfelden-Dauborn Managing Director: Arnulf Pribas
Registration: Amtsgericht Limburg a. d. Lahn 7HRB874 Tax ID: DE113840457
________________________________________________________________________
This e-mail is confidential. Information in this e-mail is intended for the exclusive use of the individual or entity named above and may constitute information that is privileged or confidential or otherwise protected from disclosure. The information in this e-mail may be read, published, copied and/or forwarded only by the individual or entity named above. Dissemination, distribution, forwarding or copying of this e-mail by anyone other than the intended recipient is prohibited. If you have received this e-mail in error, please notify us immediately by telephone or e-mail and completely delete or destroy any and all disseminated, distributed, forwarded electronic or other copies of the original message and any attachments.




------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Loading...