Problem - Volume data error

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Problem - Volume data error

Wanderlei Huttel
Hello

A friend in Brazil has a serious problem. One of his servers has died and he is trying to restore a Full Job in bacula but is receiving an error, because one of those volumes has some error.

He has 12 volumes, but at Volume-Diario-0009 has an error and bacula terminates with error instead of continue and try restore data from another volumes.

Volume-Diario-0006
Volume-Diario-0007
Volume-Diario-0008
Volume-Diario-0009
Volume-Diario-0010
Volume-Diario-0011
Volume-Diario-0012
Volume-Diario-0013
Volume-Diario-0014
Volume-Diario-0015
Volume-Diario-0016
Volume-Diario-0017

2017-03-09 14:25:59   srv-backup04-dir JobId 2389: Start Restore Job RestoreFiles.2017-03-09_14.25.57_14
2017-03-09 14:25:59   srv-backup04-dir JobId 2389: Using Device "DiscoLocal3" to read.
2017-03-09 14:26:00   srv-backup04-SD JobId 2389: Ready to read from volume "Volume-Diario-0006" on file device "DiscoLocal3" (/home/bacula/volume03).
2017-03-09 14:26:00   srv-backup04-SD JobId 2389: Forward spacing Volume "Volume-Diario-0006" to file:block 1:620956522.
2017-03-09 14:29:28   srv-backup04-SD JobId 2389: End of Volume at file 5 on device "DiscoLocal3" (/home/bacula/volume03), Volume "Volume-Diario-0006"
2017-03-09 14:29:28   srv-backup04-SD JobId 2389: Ready to read from volume "Volume-Diario-0007" on file device "DiscoLocal3" (/home/bacula/volume03).
2017-03-09 14:29:28   srv-backup04-SD JobId 2389: Forward spacing Volume "Volume-Diario-0007" to file:block 0:64731.
2017-03-09 14:33:46   srv-backup04-SD JobId 2389: End of Volume at file 5 on device "DiscoLocal3" (/home/bacula/volume03), Volume "Volume-Diario-0007"
2017-03-09 14:33:46   srv-backup04-SD JobId 2389: Ready to read from volume "Volume-Diario-0008" on file device "DiscoLocal3" (/home/bacula/volume03).
2017-03-09 14:33:46   srv-backup04-SD JobId 2389: Forward spacing Volume "Volume-Diario-0008" to file:block 0:219.
2017-03-09 14:38:04   srv-backup04-SD JobId 2389: End of Volume at file 5 on device "DiscoLocal3" (/home/bacula/volume03), Volume "Volume-Diario-0008"
2017-03-09 14:38:04   srv-backup04-SD JobId 2389: Ready to read from volume "Volume-Diario-0009" on file device "DiscoLocal3" (/home/bacula/volume03).
2017-03-09 14:38:04   srv-backup04-SD JobId 2389: Forward spacing Volume "Volume-Diario-0009" to file:block 0:64731.
2017-03-09 14:40:04   srv-backup04-SD JobId 2389: Error: block_util.c:356 Volume data error at 2:3123444791!
Block checksum mismatch in block=712541 len=64512: calc=a43dc97 blk=cfbe9c6a
2017-03-09 14:40:04   srv-backup04-fd JobId 2389: Error: attribs.c:559 File size of restored file /home/bacula/volume02/wowza/opt/www/camara/public_videos/Ano2013/Abril/3.flv not correct. Original 356550489, restored 136249344.
2017-03-09 14:40:04   srv-backup04-SD JobId 2389: Elapsed time=00:14:04, Transfer rate=42.42 M Bytes/second
2017-03-09 14:40:04   srv-backup04-SD JobId 2389: Fatal error: fd_cmds.c:201 Command error with FD, hanging up.
2017-03-09 14:40:04   srv-backup04-dir JobId 2389: Error: Bacula srv-backup04-dir 7.4.4 (202Sep16):
  Build OS:               x86_64-redhat-linux-gnu redhat Five)
  JobId:                  2389
  Job:                    RestoreFiles.2017-03-09_14.25.57_14
  Restore Client:         srv-backup04-fd
  Start time:             24-fev-2017 14:25:59
  End time:               24-fev-2017 14:40:04
  Files Expected:         2,344
  Files Restored:         1,845
  Bytes Restored:         37,094,010,161
  Rate:                   43898.2 KB/s
  FD Errors:              1
  FD termination status:  Error
  SD termination status:  Error
  Termination:            *** Restore Error ***
2017-03-09 14:40:04   srv-backup04-dir JobId 2389: Begin pruning Jobs older than 1 year .
2017-03-09 14:40:04   srv-backup04-dir JobId 2389: No Jobs found to prune.
2017-03-09 14:40:04   srv-backup04-dir JobId 2389: Begin pruning Files.
2017-03-09 14:40:04   srv-backup04-dir JobId 2389: No Files found to prune.
2017-03-09 14:40:04   srv-backup04-dir JobId 2389: End auto prune.


I suggested him to try to restore in bconsole and get the bootstrap file and edit it and remove this volume from bootstrap file and use bextract to restore

Any tips how to restore this data? Or why this volume has an error?


Best regards

Wanderlei Hüttel

------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Problem - Volume data error

Kern Sibbald
Hello,

You can try adding the -p option to the command line of the Storage daemon.  The SD will then try to continue after errors, but the error is serious enough that it is unlikely to work, the SD will at some point probably seg fault when it read bad data.

Another approach would be to attempt to restore an older backup, and if that is successful restore one by one each job that ran after the backup that worked.   Then you can examine in detail the restore that failed, and see if you can find older backups of the files.  Also, if you use the -p option on the Job that fails, you may be able to get something.

A more radical approach would be to force the Director to save the bsr, then you modify it and remove either just the references to bad blocks or all records for Volume-Diario-0009 and try the restore.  Obviously this will not get back everything, but maybe if you are lucky, it will get back what you need.  You can then use the catalog to find out what files are on the Volume and see if they were previously saved either elsewhere on the same volume or preferably on a different Volume.  This will be a good deal of work, but if you dig into it and understand pretty much what you are doing, you will be surprised what you can recover.

Best regards,
Kern

On 03/09/2017 08:44 PM, Wanderlei Huttel wrote:
Hello

A friend in Brazil has a serious problem. One of his servers has died and he is trying to restore a Full Job in bacula but is receiving an error, because one of those volumes has some error.

He has 12 volumes, but at Volume-Diario-0009 has an error and bacula terminates with error instead of continue and try restore data from another volumes.

Volume-Diario-0006
Volume-Diario-0007
Volume-Diario-0008
Volume-Diario-0009
Volume-Diario-0010
Volume-Diario-0011
Volume-Diario-0012
Volume-Diario-0013
Volume-Diario-0014
Volume-Diario-0015
Volume-Diario-0016
Volume-Diario-0017

2017-03-09 14:25:59   srv-backup04-dir JobId 2389: Start Restore Job RestoreFiles.2017-03-09_14.25.57_14
2017-03-09 14:25:59   srv-backup04-dir JobId 2389: Using Device "DiscoLocal3" to read.
2017-03-09 14:26:00   srv-backup04-SD JobId 2389: Ready to read from volume "Volume-Diario-0006" on file device "DiscoLocal3" (/home/bacula/volume03).
2017-03-09 14:26:00   srv-backup04-SD JobId 2389: Forward spacing Volume "Volume-Diario-0006" to file:block 1:620956522.
2017-03-09 14:29:28   srv-backup04-SD JobId 2389: End of Volume at file 5 on device "DiscoLocal3" (/home/bacula/volume03), Volume "Volume-Diario-0006"
2017-03-09 14:29:28   srv-backup04-SD JobId 2389: Ready to read from volume "Volume-Diario-0007" on file device "DiscoLocal3" (/home/bacula/volume03).
2017-03-09 14:29:28   srv-backup04-SD JobId 2389: Forward spacing Volume "Volume-Diario-0007" to file:block 0:64731.
2017-03-09 14:33:46   srv-backup04-SD JobId 2389: End of Volume at file 5 on device "DiscoLocal3" (/home/bacula/volume03), Volume "Volume-Diario-0007"
2017-03-09 14:33:46   srv-backup04-SD JobId 2389: Ready to read from volume "Volume-Diario-0008" on file device "DiscoLocal3" (/home/bacula/volume03).
2017-03-09 14:33:46   srv-backup04-SD JobId 2389: Forward spacing Volume "Volume-Diario-0008" to file:block 0:219.
2017-03-09 14:38:04   srv-backup04-SD JobId 2389: End of Volume at file 5 on device "DiscoLocal3" (/home/bacula/volume03), Volume "Volume-Diario-0008"
2017-03-09 14:38:04   srv-backup04-SD JobId 2389: Ready to read from volume "Volume-Diario-0009" on file device "DiscoLocal3" (/home/bacula/volume03).
2017-03-09 14:38:04   srv-backup04-SD JobId 2389: Forward spacing Volume "Volume-Diario-0009" to file:block 0:64731.
2017-03-09 14:40:04   srv-backup04-SD JobId 2389: Error: block_util.c:356 Volume data error at 2:3123444791!
Block checksum mismatch in block=712541 len=64512: calc=a43dc97 blk=cfbe9c6a
2017-03-09 14:40:04   srv-backup04-fd JobId 2389: Error: attribs.c:559 File size of restored file /home/bacula/volume02/wowza/opt/www/camara/public_videos/Ano2013/Abril/3.flv not correct. Original 356550489, restored 136249344.
2017-03-09 14:40:04   srv-backup04-SD JobId 2389: Elapsed time=00:14:04, Transfer rate=42.42 M Bytes/second
2017-03-09 14:40:04   srv-backup04-SD JobId 2389: Fatal error: fd_cmds.c:201 Command error with FD, hanging up.
2017-03-09 14:40:04   srv-backup04-dir JobId 2389: Error: Bacula srv-backup04-dir 7.4.4 (202Sep16):
  Build OS:               x86_64-redhat-linux-gnu redhat Five)
  JobId:                  2389
  Job:                    RestoreFiles.2017-03-09_14.25.57_14
  Restore Client:         srv-backup04-fd
  Start time:             24-fev-2017 14:25:59
  End time:               24-fev-2017 14:40:04
  Files Expected:         2,344
  Files Restored:         1,845
  Bytes Restored:         37,094,010,161
  Rate:                   43898.2 KB/s
  FD Errors:              1
  FD termination status:  Error
  SD termination status:  Error
  Termination:            *** Restore Error ***
2017-03-09 14:40:04   srv-backup04-dir JobId 2389: Begin pruning Jobs older than 1 year .
2017-03-09 14:40:04   srv-backup04-dir JobId 2389: No Jobs found to prune.
2017-03-09 14:40:04   srv-backup04-dir JobId 2389: Begin pruning Files.
2017-03-09 14:40:04   srv-backup04-dir JobId 2389: No Files found to prune.
2017-03-09 14:40:04   srv-backup04-dir JobId 2389: End auto prune.


I suggested him to try to restore in bconsole and get the bootstrap file and edit it and remove this volume from bootstrap file and use bextract to restore

Any tips how to restore this data? Or why this volume has an error?


Best regards

Wanderlei Hüttel


------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford


_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users



------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Loading...