bacula ignoring mtimeonly

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

bacula ignoring mtimeonly

scar-2
I am using bacula 5.x and I recently replaced the hardware for a client
(new machine).  I used "rsync -a" to copy over the contents of one of
the filesystems.  Now bacula wants to back up everything again even
though nothing is changed.  The filesystem is the same and the path to
the files is the same.  I thought this might have to do with ctime
changing, so I tried using "mtimeonly=yes" option for the FileSet
(Ignore fileset changes=yes is already set), yet when I manually run an
Incremental backup, it doesn't get upgraded to a Full yet it is still
grabbing all the files that haven't changed.  Here is an example file
that bacula is grabbing which it shouldn't:

[scar@oldserver ~]$ ls -l <file>
-rw-rw-r-- 1 10022 101 422186903 Aug 26  2016 <file>
[scar@oldserver ~]$ ls -lc <file>
-rw-rw-r-- 1 10022 101 422186903 Aug 26  2016 <file>

[scar@newserver ~]$ ls -l <file>
-rw-rw-r-- 1 10022 101 422186903 Aug 26  2016 <file>
[scar@newserver ~]$ ls -lc <file>
-rw-rw-r-- 1 10022 101 422186903 Dec  7 14:20 <file>


So we can see the ctime changed, yet with mtimeonly=yes, bacula is still
grabbing the file.  Is there some other option I am over looking?

Thanks


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: bacula ignoring mtimeonly

Josip Deanovic
On Tuesday 2017-03-28 19:44:17 scar wrote:
> I am using bacula 5.x and I recently replaced the hardware for a client
> (new machine).  I used "rsync -a" to copy over the contents of one of
> the filesystems.  Now bacula wants to back up everything again even
> though nothing is changed.  The filesystem is the same and the path to
> the files is the same.  I thought this might have to do with ctime

The file system is not the same, you have replaced the hardware,
created a new file system (probably with the same parameters) but
it is still a new file system.

You have used "rsync -a" which has preserved the modification time
but rsync deals with the data on the file level (not block level)
and thus all the copied files will be created with the different
inodes (rsync can't do anything about it because it's up to file system
to assign an inode number).

What I am saying is that your files probably have different ctime
on the file systems in question.

Please use the "ls -lc" or stat(1) command to check it.

> changing, so I tried using "mtimeonly=yes" option for the FileSet
> (Ignore fileset changes=yes is already set), yet when I manually run an
> Incremental backup, it doesn't get upgraded to a Full yet it is still
> grabbing all the files that haven't changed.  Here is an example file
> that bacula is grabbing which it shouldn't:
>
> [scar@oldserver ~]$ ls -l <file>
> -rw-rw-r-- 1 10022 101 422186903 Aug 26  2016 <file>
> [scar@oldserver ~]$ ls -lc <file>
> -rw-rw-r-- 1 10022 101 422186903 Aug 26  2016 <file>
>
> [scar@newserver ~]$ ls -l <file>
> -rw-rw-r-- 1 10022 101 422186903 Aug 26  2016 <file>
> [scar@newserver ~]$ ls -lc <file>
> -rw-rw-r-- 1 10022 101 422186903 Dec  7 14:20 <file>
>
>
> So we can see the ctime changed, yet with mtimeonly=yes, bacula is still
> grabbing the file.  Is there some other option I am over looking?

I suppose that you have added "mtimeonly=yes" only recently when
you found out that you will have to replace the storage but not
before the last full backup.

I believe that this could be a problem.
It could be that bacula saved files with ctime during the last full
backup and now sees all the files as changed even with the "mtimeonly"
set to "yes".

If this is the case it would be a good idea to change that behavior
(unless there is a good reason not to).


--
Josip Deanovic

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: bacula ignoring mtimeonly

scar-2
Josip Deanovic wrote on 03/28/2017 09:10 PM:
> I believe that this could be a problem.
> It could be that bacula saved files with ctime during the last full
> backup and now sees all the files as changed even with the "mtimeonly"
> set to "yes".
>
> If this is the case it would be a good idea to change that behavior
> (unless there is a good reason not to).


Thanks, so are you saying this is potentially a bug with bacula?  Do you
know how i could continue with incremental backups in the mean time?
This particular fileset is 4 levels deep yet it contains over 21 million
files and about 4 terabytes of gzip-compressed data.  Soon I will set up
a new storage daemon with compressing filesystem so won't have to worry
about gzip on the client side... but this particular full backup takes
about 10 days to complete... would like to avoid it if possible.



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: bacula ignoring mtimeonly

Josip Deanovic
On Wednesday 2017-03-29 14:07:43 scar wrote:

> Josip Deanovic wrote on 03/28/2017 09:10 PM:
> > I believe that this could be a problem.
> > It could be that bacula saved files with ctime during the last full
> > backup and now sees all the files as changed even with the "mtimeonly"
> > set to "yes".
> >
> > If this is the case it would be a good idea to change that behavior
> > (unless there is a good reason not to).
>
> Thanks, so are you saying this is potentially a bug with bacula?  Do you
> know how i could continue with incremental backups in the mean time?
> This particular fileset is 4 levels deep yet it contains over 21
> million files and about 4 terabytes of gzip-compressed data.  Soon I
> will set up a new storage daemon with compressing filesystem so won't
> have to worry about gzip on the client side... but this particular full
> backup takes about 10 days to complete... would like to avoid it if
> possible.

Let's wait for some of the developers to give their opinion.
I might be completely wrong about this.

What I said is a speculation.
However, the File table in the bacula database contains a LStat
column with information about files.
Several values that represent different file attributes are space
delimited and base64 encoded (this is bacula specific base64 and
not a standard base64 o it's a bit tricky to decode it but there is
a source (src/findlib/attribs.c) so it's not impossible).

Anyway, when decoded those values would give these file attributes:
st_dev
st_ino
st_mode
st_nlink
st_uid
st_gid
st_rdev
st_size
st_blksize
st_blocks
st_atime
st_mtime
st_ctime
LinkFI
st_flags
data

Note that I am using bacula 5.0.0 here but it might be that
this still applies in the current bacula release.

Anyway, until the developers join this thread, if I were you
I would setup a small job and perform a full backup with and without
"mtimeonly" option and then I would try to determine if there is any
difference between the st_ctime and st_ino between those two full
backup jobs.

That could give you and idea if there is anything you can do
in the database to help yourself.

Now let's wait for the developers to tell me that I am completely
wrong and that LStat has nothing to do with this. :-)

--
Josip Deanovic

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Loading...