btape fill failure on HP LTO6/4 drives

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

btape fill failure on HP LTO6/4 drives

Roberts, Ben
Hi all,

While setting up a new tape library with HP LTO6 drives, btape test runs successfully, but btape fill fails with what looks like a problem handling the last block on the tape (perhaps an off-by-one error?). The outputs below are repeatable with both LTO6 and LTO4 media, but I've been able to do a multi-tape backup and restore OK; so I think the problem is isolated to btape rather than Bacula-sd and any common code between them. Will do some more testing, but obviously filling multiple LTO6 tapes takes a little while!

Has anyone seen this before?

This is on Solaris 5.11 (x86_64) running Bacula 5.2.13, locally compiled.

------------
btape output:
..snip..
Wrote block=19090000, file,blk=954,10936 VolBytes=2,499,540,811,776 rate=107.7 MB/s
19-Mar 15:23 btape JobId 0: End of Volume "TestVolume1" at 954:11250 on device "drive-0-tapestore1" (/dev/rmt/0mbn). Write of 131072 bytes got 0.
19-Mar 15:23 btape JobId 0: Error: Backspace record at EOT failed. ERR=I/O error
btape: btape.c:2714-0 Last block at: 954:11249 this_dev_block_num=11250
btape: btape.c:2749-0 End of tape 954:-1. Volume Bytes=2,499,581,968,384. Write rate = 107.7 MB/s
btape: btape.c:2320-0 Wrote 1000 blocks on second tape. Done.
Done writing 0 records ...
btape: btape.c:2389-0 Wrote state file last_block_num1=11249 last_block_num2=0
btape: btape.c:2404-0

15:23:52 Done filling tape at 954:-1. Now beginning re-read of tape ...
btape: btape.c:2485-0 Enter do_unfill
19-Mar 15:24 btape JobId 0: Ready to read from volume "TestVolume1" on device "drive-0-tapestore1" (/dev/rmt/0mbn).
Rewinding.
Reading the first 10000 records from 0:0.
10000 records read now at 1:2502
Reposition from 1:2502 to 954:11249
Reposition error. ERR=dev.c:1366 ioctl MTFSR 11249 error on "drive-0-tapestore1" (/dev/rmt/0mbn). ERR=I/O error.

btape: btape.c:2412-0 do_unfill failed.

------------
tapeinfo -f /dev/rmt/0mbn
Product Type: Tape Drive
Vendor ID: 'HP      '
Product ID: 'Ultrium 6-SCSI  '
Revision: '338B'
Attached Changer API: No
SerialNumber: 'xxxxxxxx'
MinBlock: 1
MaxBlock: 16777215
Ready: no

------------
Relevant Bacula-sd.conf snippet:

Autochanger {
  Name = tapestore1-autochanger
  Device = drive-0-tapestore1, drive-1-tapestore1
  Device = drive-2-tapestore1, drive-3-tapestore1
  Changer Device = /dev/scsi/changer/c1t5d1
  Changer Command = "/opt/bacula/etc/mtx-changer %c %o %S %a %d"
}

Device {
  Name = drive-0-tapestore1
  Archive Device = /dev/rmt/0mbn
  Device Type = Tape
  Media Type = LT06
  AutoChanger = yes
  Removable media = yes;
  Label Media = no
  Random access = no;
  Requires Mount = no;
  Maximum Changer Wait = 180
  Drive Index = 0
  Maximum Spool Size = 100G
}

------------
Full btape test output:
Tape block granularity is 1024 bytes.
btape: butil.c:290-0 Using device: "/dev/rmt/0mbn" for writing.
btape: btape.c:477-0 open device "drive-0-tapestore1" (/dev/rmt/0mbn): OK
*test

=== Write, rewind, and re-read test ===

I'm going to write 10000 records and an EOF
then write 10000 records and an EOF, then rewind,
and re-read the data to verify that it is correct.

This is an *essential* feature ...

btape: btape.c:1157-0 Wrote 10000 blocks of 130972 bytes.
btape: btape.c:609-0 Wrote 1 EOF to "drive-0-tapestore1" (/dev/rmt/0mbn)
btape: btape.c:1173-0 Wrote 10000 blocks of 130972 bytes.
btape: btape.c:609-0 Wrote 1 EOF to "drive-0-tapestore1" (/dev/rmt/0mbn)
btape: btape.c:1215-0 Rewind OK.
10000 blocks re-read correctly.
Got EOF on tape.
10000 blocks re-read correctly.
=== Test Succeeded. End Write, rewind, and re-read test ===

btape: btape.c:1283-0 Block position test
btape: btape.c:1295-0 Rewind OK.
Reposition to file:block 0:4
Block 5 re-read correctly.
Reposition to file:block 0:200
Block 201 re-read correctly.
Reposition to file:block 0:9999
Block 10000 re-read correctly.
Reposition to file:block 1:0
Block 10001 re-read correctly.
Reposition to file:block 1:600
Block 10601 re-read correctly.
Reposition to file:block 1:9999
Block 20000 re-read correctly.
=== Test Succeeded. End Write, rewind, and re-read test ===



=== Append files test ===

This test is essential to Bacula.

I'm going to write one record  in file 0,
                   two records in file 1,
             and three records in file 2

btape: btape.c:579-0 Rewound "drive-0-tapestore1" (/dev/rmt/0mbn)
btape: btape.c:1914-0 Wrote one record of 130972 bytes.
btape: btape.c:1916-0 Wrote block to device.
btape: btape.c:609-0 Wrote 1 EOF to "drive-0-tapestore1" (/dev/rmt/0mbn)
btape: btape.c:1914-0 Wrote one record of 130972 bytes.
btape: btape.c:1916-0 Wrote block to device.
btape: btape.c:1914-0 Wrote one record of 130972 bytes.
btape: btape.c:1916-0 Wrote block to device.
btape: btape.c:609-0 Wrote 1 EOF to "drive-0-tapestore1" (/dev/rmt/0mbn)
btape: btape.c:1914-0 Wrote one record of 130972 bytes.
btape: btape.c:1916-0 Wrote block to device.
btape: btape.c:1914-0 Wrote one record of 130972 bytes.
btape: btape.c:1916-0 Wrote block to device.
btape: btape.c:1914-0 Wrote one record of 130972 bytes.
btape: btape.c:1916-0 Wrote block to device.
btape: btape.c:609-0 Wrote 1 EOF to "drive-0-tapestore1" (/dev/rmt/0mbn)
btape: btape.c:477-0 open device "drive-0-tapestore1" (/dev/rmt/0mbn): OK
btape: btape.c:579-0 Rewound "drive-0-tapestore1" (/dev/rmt/0mbn)
btape: btape.c:1427-0 Now moving to end of medium.
btape: btape.c:630-0 Moved to end of medium.
We should be in file 3. I am at file 3. This is correct!

Now the important part, I am going to attempt to append to the tape.

btape: btape.c:1914-0 Wrote one record of 130972 bytes.
btape: btape.c:1916-0 Wrote block to device.
btape: btape.c:609-0 Wrote 1 EOF to "drive-0-tapestore1" (/dev/rmt/0mbn)
btape: btape.c:579-0 Rewound "drive-0-tapestore1" (/dev/rmt/0mbn)
Done appending, there should be no I/O errors

Doing Bacula scan of blocks:
1 block of 131008 bytes in file 1
End of File mark.
2 blocks of 131008 bytes in file 2
End of File mark.
3 blocks of 131008 bytes in file 3
End of File mark.
1 block of 131008 bytes in file 4
End of File mark.
Total files=4, blocks=7, bytes = 917,056
End scanning the tape.
We should be in file 4. I am at file 4. This is correct!

The above Bacula scan should have output identical to what follows.
Please double check it ...
=== Sample correct output ===
1 block of 64448 bytes in file 1
End of File mark.
2 blocks of 64448 bytes in file 2
End of File mark.
3 blocks of 64448 bytes in file 3
End of File mark.
1 block of 64448 bytes in file 4
End of File mark.
Total files=4, blocks=7, bytes = 451,136
=== End sample correct output ===

If the above scan output is not identical to the
sample output, you MUST correct the problem
or Bacula will not be able to write multiple Jobs to
the tape.


=== Write, backup, and re-read test ===

I'm going to write three records and an EOF
then backup over the EOF and re-read the last record.
Bacula does this after writing the last block on the
tape to verify that the block was written correctly.

This is not an *essential* feature ...

btape: btape.c:579-0 Rewound "drive-0-tapestore1" (/dev/rmt/0mbn)
btape: btape.c:816-0 Wrote first record of 130972 bytes.
btape: btape.c:827-0 Wrote second record of 130972 bytes.
btape: btape.c:838-0 Wrote third record of 130972 bytes.
btape: btape.c:609-0 Wrote 1 EOF to "drive-0-tapestore1" (/dev/rmt/0mbn)
btape: btape.c:854-0 Backspaced over EOF OK.
btape: btape.c:859-0 Backspace record OK.
btape: btape.c:877-0
Block re-read correct. Test succeeded!
=== End Write, backup, and re-read test ===



=== Forward space files test ===

This test is essential to Bacula.

I'm going to write five files then test forward spacing

btape: btape.c:579-0 Rewound "drive-0-tapestore1" (/dev/rmt/0mbn)
btape: btape.c:1914-0 Wrote one record of 130972 bytes.
btape: btape.c:1916-0 Wrote block to device.
btape: btape.c:609-0 Wrote 1 EOF to "drive-0-tapestore1" (/dev/rmt/0mbn)
btape: btape.c:1914-0 Wrote one record of 130972 bytes.
btape: btape.c:1916-0 Wrote block to device.
btape: btape.c:1914-0 Wrote one record of 130972 bytes.
btape: btape.c:1916-0 Wrote block to device.
btape: btape.c:609-0 Wrote 1 EOF to "drive-0-tapestore1" (/dev/rmt/0mbn)
btape: btape.c:1914-0 Wrote one record of 130972 bytes.
btape: btape.c:1916-0 Wrote block to device.
btape: btape.c:1914-0 Wrote one record of 130972 bytes.
btape: btape.c:1916-0 Wrote block to device.
btape: btape.c:1914-0 Wrote one record of 130972 bytes.
btape: btape.c:1916-0 Wrote block to device.
btape: btape.c:609-0 Wrote 1 EOF to "drive-0-tapestore1" (/dev/rmt/0mbn)
btape: btape.c:1914-0 Wrote one record of 130972 bytes.
btape: btape.c:1916-0 Wrote block to device.
btape: btape.c:1914-0 Wrote one record of 130972 bytes.
btape: btape.c:1916-0 Wrote block to device.
btape: btape.c:609-0 Wrote 1 EOF to "drive-0-tapestore1" (/dev/rmt/0mbn)
btape: btape.c:1914-0 Wrote one record of 130972 bytes.
btape: btape.c:1916-0 Wrote block to device.
btape: btape.c:609-0 Wrote 1 EOF to "drive-0-tapestore1" (/dev/rmt/0mbn)
btape: btape.c:579-0 Rewound "drive-0-tapestore1" (/dev/rmt/0mbn)
btape: btape.c:1641-0 Now forward spacing 1 file.
We should be in file 1. I am at file 1. This is correct!
btape: btape.c:1653-0 Now forward spacing 2 files.
We should be in file 3. I am at file 3. This is correct!
btape: btape.c:579-0 Rewound "drive-0-tapestore1" (/dev/rmt/0mbn)
btape: btape.c:1666-0 Now forward spacing 4 files.
We should be in file 4. I am at file 4. This is correct!

btape: btape.c:1684-0 Now forward spacing 1 more file.
We should be in file 5. I am at file 5. This is correct!

=== End Forward space files test ===


Ah, I see you have an autochanger configured.
To test the autochanger you must have a blank tape
that I can write on in Slot 1.

Do you wish to continue with the Autochanger test? (y/n): n
*

Regards,
Ben Roberts




This email and any files transmitted with it contain confidential and proprietary information and is solely for the use of the intended recipient.  If you are not the intended recipient please return the email to the sender and delete it from your computer and you must not use, disclose, distribute, copy, print or rely on this email or its contents.  This communication is for informational purposes only.  It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction.   Any comments or statements made herein do not necessarily reflect those of GSA Capital. GSA Capital Partners LLP is authorised and regulated by the Financial Conduct Authority and is registered in England and Wales at Stratton House, 5 Stratton Street, London W1J 8LA, number OC309261. GSA Capital Services Limited is registered in England and Wales at the same address, number 5320529.


------------------------------------------------------------------------------
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: btape fill failure on HP LTO6/4 drives

Kern Sibbald
Hello,

btape uses all the same libraries as Bacula, and while the sequence of
calls to the low level libraries may be different from what Bacula does,
it does indicate a problem.  It appears that the OS tape driver does not
properly implement back space record after an EOT.  This is a defect of
the operating system driver, but it is not fatal for Bacula.

You will very likely see this defect show up when Bacula fills a tape
and writes the final EOT mark then tries to verify that the last block
was written correctly.  Due to the OS driver defect this will fail, but
it is only a check and your data may still be good.  The results from
btape re-reading what was written does not look encouraging, and it may
indicate that the last block is not properly written.  I personally
would be worried.

This can happen if you are not running the tape drive in the right mode
-- since I have not looked at the Solaris tape drive naming conventions
for about 8 years now, I forget.  I suggest that you verify that using

/dev/rmt/0mbn

is what is recommended in the manual.

If you can really do multi-volume restores correctly and you have
verified that every byte is correct then you are probably OK.

Best regards,
Kern

On 03/28/2014 01:01 PM, Roberts, Ben wrote:

> Hi all,
>
> While setting up a new tape library with HP LTO6 drives, btape test runs successfully, but btape fill fails with what looks like a problem handling the last block on the tape (perhaps an off-by-one error?). The outputs below are repeatable with both LTO6 and LTO4 media, but I've been able to do a multi-tape backup and restore OK; so I think the problem is isolated to btape rather than Bacula-sd and any common code between them. Will do some more testing, but obviously filling multiple LTO6 tapes takes a little while!
>
> Has anyone seen this before?
>
> This is on Solaris 5.11 (x86_64) running Bacula 5.2.13, locally compiled.
>
> ------------
> btape output:
> ..snip..
> Wrote block=19090000, file,blk=954,10936 VolBytes=2,499,540,811,776 rate=107.7 MB/s
> 19-Mar 15:23 btape JobId 0: End of Volume "TestVolume1" at 954:11250 on device "drive-0-tapestore1" (/dev/rmt/0mbn). Write of 131072 bytes got 0.
> 19-Mar 15:23 btape JobId 0: Error: Backspace record at EOT failed. ERR=I/O error
> btape: btape.c:2714-0 Last block at: 954:11249 this_dev_block_num=11250
> btape: btape.c:2749-0 End of tape 954:-1. Volume Bytes=2,499,581,968,384. Write rate = 107.7 MB/s
> btape: btape.c:2320-0 Wrote 1000 blocks on second tape. Done.
> Done writing 0 records ...
> btape: btape.c:2389-0 Wrote state file last_block_num1=11249 last_block_num2=0
> btape: btape.c:2404-0
>
> 15:23:52 Done filling tape at 954:-1. Now beginning re-read of tape ...
> btape: btape.c:2485-0 Enter do_unfill
> 19-Mar 15:24 btape JobId 0: Ready to read from volume "TestVolume1" on device "drive-0-tapestore1" (/dev/rmt/0mbn).
> Rewinding.
> Reading the first 10000 records from 0:0.
> 10000 records read now at 1:2502
> Reposition from 1:2502 to 954:11249
> Reposition error. ERR=dev.c:1366 ioctl MTFSR 11249 error on "drive-0-tapestore1" (/dev/rmt/0mbn). ERR=I/O error.
>
> btape: btape.c:2412-0 do_unfill failed.
>
> ------------
> tapeinfo -f /dev/rmt/0mbn
> Product Type: Tape Drive
> Vendor ID: 'HP      '
> Product ID: 'Ultrium 6-SCSI  '
> Revision: '338B'
> Attached Changer API: No
> SerialNumber: 'xxxxxxxx'
> MinBlock: 1
> MaxBlock: 16777215
> Ready: no
>
> ------------
> Relevant Bacula-sd.conf snippet:
>
> Autochanger {
>   Name = tapestore1-autochanger
>   Device = drive-0-tapestore1, drive-1-tapestore1
>   Device = drive-2-tapestore1, drive-3-tapestore1
>   Changer Device = /dev/scsi/changer/c1t5d1
>   Changer Command = "/opt/bacula/etc/mtx-changer %c %o %S %a %d"
> }
>
> Device {
>   Name = drive-0-tapestore1
>   Archive Device = /dev/rmt/0mbn
>   Device Type = Tape
>   Media Type = LT06
>   AutoChanger = yes
>   Removable media = yes;
>   Label Media = no
>   Random access = no;
>   Requires Mount = no;
>   Maximum Changer Wait = 180
>   Drive Index = 0
>   Maximum Spool Size = 100G
> }
>
> ------------
> Full btape test output:
> Tape block granularity is 1024 bytes.
> btape: butil.c:290-0 Using device: "/dev/rmt/0mbn" for writing.
> btape: btape.c:477-0 open device "drive-0-tapestore1" (/dev/rmt/0mbn): OK
> *test
>
> === Write, rewind, and re-read test ===
>
> I'm going to write 10000 records and an EOF
> then write 10000 records and an EOF, then rewind,
> and re-read the data to verify that it is correct.
>
> This is an *essential* feature ...
>
> btape: btape.c:1157-0 Wrote 10000 blocks of 130972 bytes.
> btape: btape.c:609-0 Wrote 1 EOF to "drive-0-tapestore1" (/dev/rmt/0mbn)
> btape: btape.c:1173-0 Wrote 10000 blocks of 130972 bytes.
> btape: btape.c:609-0 Wrote 1 EOF to "drive-0-tapestore1" (/dev/rmt/0mbn)
> btape: btape.c:1215-0 Rewind OK.
> 10000 blocks re-read correctly.
> Got EOF on tape.
> 10000 blocks re-read correctly.
> === Test Succeeded. End Write, rewind, and re-read test ===
>
> btape: btape.c:1283-0 Block position test
> btape: btape.c:1295-0 Rewind OK.
> Reposition to file:block 0:4
> Block 5 re-read correctly.
> Reposition to file:block 0:200
> Block 201 re-read correctly.
> Reposition to file:block 0:9999
> Block 10000 re-read correctly.
> Reposition to file:block 1:0
> Block 10001 re-read correctly.
> Reposition to file:block 1:600
> Block 10601 re-read correctly.
> Reposition to file:block 1:9999
> Block 20000 re-read correctly.
> === Test Succeeded. End Write, rewind, and re-read test ===
>
>
>
> === Append files test ===
>
> This test is essential to Bacula.
>
> I'm going to write one record  in file 0,
>                    two records in file 1,
>              and three records in file 2
>
> btape: btape.c:579-0 Rewound "drive-0-tapestore1" (/dev/rmt/0mbn)
> btape: btape.c:1914-0 Wrote one record of 130972 bytes.
> btape: btape.c:1916-0 Wrote block to device.
> btape: btape.c:609-0 Wrote 1 EOF to "drive-0-tapestore1" (/dev/rmt/0mbn)
> btape: btape.c:1914-0 Wrote one record of 130972 bytes.
> btape: btape.c:1916-0 Wrote block to device.
> btape: btape.c:1914-0 Wrote one record of 130972 bytes.
> btape: btape.c:1916-0 Wrote block to device.
> btape: btape.c:609-0 Wrote 1 EOF to "drive-0-tapestore1" (/dev/rmt/0mbn)
> btape: btape.c:1914-0 Wrote one record of 130972 bytes.
> btape: btape.c:1916-0 Wrote block to device.
> btape: btape.c:1914-0 Wrote one record of 130972 bytes.
> btape: btape.c:1916-0 Wrote block to device.
> btape: btape.c:1914-0 Wrote one record of 130972 bytes.
> btape: btape.c:1916-0 Wrote block to device.
> btape: btape.c:609-0 Wrote 1 EOF to "drive-0-tapestore1" (/dev/rmt/0mbn)
> btape: btape.c:477-0 open device "drive-0-tapestore1" (/dev/rmt/0mbn): OK
> btape: btape.c:579-0 Rewound "drive-0-tapestore1" (/dev/rmt/0mbn)
> btape: btape.c:1427-0 Now moving to end of medium.
> btape: btape.c:630-0 Moved to end of medium.
> We should be in file 3. I am at file 3. This is correct!
>
> Now the important part, I am going to attempt to append to the tape.
>
> btape: btape.c:1914-0 Wrote one record of 130972 bytes.
> btape: btape.c:1916-0 Wrote block to device.
> btape: btape.c:609-0 Wrote 1 EOF to "drive-0-tapestore1" (/dev/rmt/0mbn)
> btape: btape.c:579-0 Rewound "drive-0-tapestore1" (/dev/rmt/0mbn)
> Done appending, there should be no I/O errors
>
> Doing Bacula scan of blocks:
> 1 block of 131008 bytes in file 1
> End of File mark.
> 2 blocks of 131008 bytes in file 2
> End of File mark.
> 3 blocks of 131008 bytes in file 3
> End of File mark.
> 1 block of 131008 bytes in file 4
> End of File mark.
> Total files=4, blocks=7, bytes = 917,056
> End scanning the tape.
> We should be in file 4. I am at file 4. This is correct!
>
> The above Bacula scan should have output identical to what follows.
> Please double check it ...
> === Sample correct output ===
> 1 block of 64448 bytes in file 1
> End of File mark.
> 2 blocks of 64448 bytes in file 2
> End of File mark.
> 3 blocks of 64448 bytes in file 3
> End of File mark.
> 1 block of 64448 bytes in file 4
> End of File mark.
> Total files=4, blocks=7, bytes = 451,136
> === End sample correct output ===
>
> If the above scan output is not identical to the
> sample output, you MUST correct the problem
> or Bacula will not be able to write multiple Jobs to
> the tape.
>
>
> === Write, backup, and re-read test ===
>
> I'm going to write three records and an EOF
> then backup over the EOF and re-read the last record.
> Bacula does this after writing the last block on the
> tape to verify that the block was written correctly.
>
> This is not an *essential* feature ...
>
> btape: btape.c:579-0 Rewound "drive-0-tapestore1" (/dev/rmt/0mbn)
> btape: btape.c:816-0 Wrote first record of 130972 bytes.
> btape: btape.c:827-0 Wrote second record of 130972 bytes.
> btape: btape.c:838-0 Wrote third record of 130972 bytes.
> btape: btape.c:609-0 Wrote 1 EOF to "drive-0-tapestore1" (/dev/rmt/0mbn)
> btape: btape.c:854-0 Backspaced over EOF OK.
> btape: btape.c:859-0 Backspace record OK.
> btape: btape.c:877-0
> Block re-read correct. Test succeeded!
> === End Write, backup, and re-read test ===
>
>
>
> === Forward space files test ===
>
> This test is essential to Bacula.
>
> I'm going to write five files then test forward spacing
>
> btape: btape.c:579-0 Rewound "drive-0-tapestore1" (/dev/rmt/0mbn)
> btape: btape.c:1914-0 Wrote one record of 130972 bytes.
> btape: btape.c:1916-0 Wrote block to device.
> btape: btape.c:609-0 Wrote 1 EOF to "drive-0-tapestore1" (/dev/rmt/0mbn)
> btape: btape.c:1914-0 Wrote one record of 130972 bytes.
> btape: btape.c:1916-0 Wrote block to device.
> btape: btape.c:1914-0 Wrote one record of 130972 bytes.
> btape: btape.c:1916-0 Wrote block to device.
> btape: btape.c:609-0 Wrote 1 EOF to "drive-0-tapestore1" (/dev/rmt/0mbn)
> btape: btape.c:1914-0 Wrote one record of 130972 bytes.
> btape: btape.c:1916-0 Wrote block to device.
> btape: btape.c:1914-0 Wrote one record of 130972 bytes.
> btape: btape.c:1916-0 Wrote block to device.
> btape: btape.c:1914-0 Wrote one record of 130972 bytes.
> btape: btape.c:1916-0 Wrote block to device.
> btape: btape.c:609-0 Wrote 1 EOF to "drive-0-tapestore1" (/dev/rmt/0mbn)
> btape: btape.c:1914-0 Wrote one record of 130972 bytes.
> btape: btape.c:1916-0 Wrote block to device.
> btape: btape.c:1914-0 Wrote one record of 130972 bytes.
> btape: btape.c:1916-0 Wrote block to device.
> btape: btape.c:609-0 Wrote 1 EOF to "drive-0-tapestore1" (/dev/rmt/0mbn)
> btape: btape.c:1914-0 Wrote one record of 130972 bytes.
> btape: btape.c:1916-0 Wrote block to device.
> btape: btape.c:609-0 Wrote 1 EOF to "drive-0-tapestore1" (/dev/rmt/0mbn)
> btape: btape.c:579-0 Rewound "drive-0-tapestore1" (/dev/rmt/0mbn)
> btape: btape.c:1641-0 Now forward spacing 1 file.
> We should be in file 1. I am at file 1. This is correct!
> btape: btape.c:1653-0 Now forward spacing 2 files.
> We should be in file 3. I am at file 3. This is correct!
> btape: btape.c:579-0 Rewound "drive-0-tapestore1" (/dev/rmt/0mbn)
> btape: btape.c:1666-0 Now forward spacing 4 files.
> We should be in file 4. I am at file 4. This is correct!
>
> btape: btape.c:1684-0 Now forward spacing 1 more file.
> We should be in file 5. I am at file 5. This is correct!
>
> === End Forward space files test ===
>
>
> Ah, I see you have an autochanger configured.
> To test the autochanger you must have a blank tape
> that I can write on in Slot 1.
>
> Do you wish to continue with the Autochanger test? (y/n): n
> *
>
> Regards,
> Ben Roberts
>
>
>
>
> This email and any files transmitted with it contain confidential and proprietary information and is solely for the use of the intended recipient.  If you are not the intended recipient please return the email to the sender and delete it from your computer and you must not use, disclose, distribute, copy, print or rely on this email or its contents.  This communication is for informational purposes only.  It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction.   Any comments or statements made herein do not necessarily reflect those of GSA Capital. GSA Capital Partners LLP is authorised and regulated by the Financial Conduct Authority and is registered in England and Wales at Stratton House, 5 Stratton Street, London W1J 8LA, number OC309261. GSA Capital Services Limited is registered in England and Wales at the same address, number 5320529.
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Bacula-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>



------------------------------------------------------------------------------
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: btape fill failure on HP LTO6/4 drives

Roberts, Ben
Hi Kern,

> It appears that the OS tape driver does not properly
> implement back space record after an EOT.  This is a defect of the operating
> system driver, but it is not fatal for Bacula.
>
> You will very likely see this defect show up when Bacula fills a tape and
> writes the final EOT mark then tries to verify that the last block was written
> correctly.  Due to the OS driver defect this will fail, but it is only a check
> and your data may still be good.  The results from btape re-reading what was
> written does not look encouraging, and it may indicate that the last block is
> not properly written.  I personally would be worried.

Indeed I was seeing the same failure to backspace over EOT error in the job logs:
End of Volume "GSA784L6" at 3910:8005 on device "drive-1-tapestore1" (/dev/rmt/1mbn). Write of 64512 bytes got 0.
Error: Backspace record at EOT failed. ERR=I/O error
End of medium on Volume "GSA784L6" Bytes=3,909,704,343,552 Blocks=60,604,295 at 28-Mar-2014 21:28

> This can happen if you are not running the tape drive in the right mode
> -- since I have not looked at the Solaris tape drive naming conventions for
> about 8 years now, I forget.  I suggest that you verify that using
> /dev/rmt/0mbn
> is what is recommended in the manual.

Rechecked the manual for this. I think 0mbn would have been an acceptable mode.
m=medium density (I've since removed this to let the tape run at its highest density and am retesting now).
b=BSD-compatible (Needed by Bacula)
n=non-rewind (Wanted by Bacula)

> If you can really do multi-volume restores correctly and you have verified
> that every byte is correct then you are probably OK.

I've successfully restored 3 jobs that were written in the same way. All three were bpipe backups of zfs streams so I'm fairly confident the restore is byte-perfect, or the zfs recv would have bailed out. I've updated the config with "Backward Space Record = no" to disable this check.

Regards,
Ben Roberts

This email and any files transmitted with it contain confidential and proprietary information and is solely for the use of the intended recipient.  If you are not the intended recipient please return the email to the sender and delete it from your computer and you must not use, disclose, distribute, copy, print or rely on this email or its contents.  This communication is for informational purposes only.  It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction.   Any comments or statements made herein do not necessarily reflect those of GSA Capital. GSA Capital Partners LLP is authorised and regulated by the Financial Conduct Authority and is registered in England and Wales at Stratton House, 5 Stratton Street, London W1J 8LA, number OC309261. GSA Capital Services Limited is registered in England and Wales at the same address, number 5320529.


------------------------------------------------------------------------------
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: btape fill failure on HP LTO6/4 drives

Allan Black
On 01/04/14 10:26, Roberts, Ben wrote:
>> It appears that the OS tape driver does not properly
>> implement back space record after an EOT.  This is a defect of the operating
>> system driver, but it is not fatal for Bacula.

>> You will very likely see this defect show up when Bacula fills a tape and
>> writes the final EOT mark then tries to verify that the last block was written
>> correctly.  Due to the OS driver defect this will fail, but it is only a check
>> and your data may still be good.  The results from btape re-reading what was
>> written does not look encouraging, and it may indicate that the last block is
>> not properly written.  I personally would be worried.

> Indeed I was seeing the same failure to backspace over EOT error in the job logs:
> End of Volume "GSA784L6" at 3910:8005 on device "drive-1-tapestore1" (/dev/rmt/1mbn). Write of 64512 bytes got 0.
> Error: Backspace record at EOT failed. ERR=I/O error
> End of medium on Volume "GSA784L6" Bytes=3,909,704,343,552 Blocks=60,604,295 at 28-Mar-2014 21:28

I am getting exactly the same symptoms, also under Solaris 11, except this time
with an LTO2 drive. The drive worked perfectly under Solaris 10, though, and I
only started seeing this after upgrading to 11.1.

The btape fill/m test gave me this at the end of the first tape:

Wrote block=3160000, file,blk=204,13499 VolBytes=203,857,855,488 rate=20.62 MB/s
08-May 16:59 btape JobId 0: End of Volume "TestVolume1" at 204:15112 on device
"lto" (/dev/rmt/4cbn). Write of 64512 bytes got 0.
08-May 16:59 btape JobId 0: Error: Backspace record at EOT failed. ERR=I/O error
btape: btape.c:2702 Last block at: 204:15111 this_dev_block_num=15112
btape: btape.c:2737 End of tape 204:-1. Volume Bytes=203,961,913,344. Write rate
= 20.60 MB/s
08-May 16:59 btape JobId 0: End of medium on Volume "TestVolume1"
Bytes=203,961,913,344 Blocks=3,161,612 at 08-May-2014 16:59.

>> This can happen if you are not running the tape drive in the right mode
>> /dev/rmt/0mbn
>> is what is recommended in the manual.

I'm using /dev/rmt/4cbn, as under Solaris 10, which I believe is correct.

So I tried a few experiments. Using mt to test the first tape, I found this:

mt -f /dev/rmt/4 rew
mt -f /dev/rmt/4cbn fsf 204
mt -f /dev/rmt/4cbn fsr 15111
... gave me an I/O error

After some trial and error, I found I could fsf/fsr forward to 204/14360, but
If I tried to seek to 204/14361, I got an I/O error. The last 51 records appear
to be missing from the tape. Not good.

I have difficulty believing that the Solaris mtio and/or st modules fail to
handle EOT properly, or that Bacula doesn't either. However, this is a different
OS, different build, and different version of GCC (although it's the same Bacula
source). I guess it's worth looking everywhere for the culprit.

I have a DLT IV drive attached to the same system, so I tried btape on that, and
the fill/m test worked perfectly.

I also tried running my Solaris 10 Bacula build in a solaris10 branded zone, but
got the same results. (I wanted to eliminate my Solaris 11/GCC4 build of Bacula
- the Solaris 10 version was compiled with GCC3.)

All that's left now, that I can possibly think of, is that both the S10 and S11
versions of Bacula are 32-bit builds. I'll try building 64-bit binaries, and
install Sun Studio 12 if I have to. It may be a data structure
type/stride/alignment problem with ioctls - the block numbers on the LTO test go
much higher than on the DLT, and Sun's binary compatibility guarantee does not
include device interfaces.

OTOH, could this be a buffering problem? (Long shot, I know) :-) The SCSI
channel the LTO2 is on is clocking at 20 MB/s, but the other channel, where the
DLT is, is only at 5 MB/s. Could writes to the LTO be so far ahead that not
enough tape is left between logical and physical EOT for the data already
written by btape by the time the LEOT was reported?

>> If you can really do multi-volume restores correctly and you have verified
>> that every byte is correct then you are probably OK.
> I've successfully restored 3 jobs that were written in the same way. All
>three were bpipe backups of zfs streams so I'm fairly confident the restore
>is byte-perfect, or the zfs recv would have bailed out. I've updated the
>config with "Backward Space Record = no" to disable this check.

I'm not convinced that "Backward Space Record = no" is a good idea, given the
results I got with btape. I wouldn't trust the backup even if I managed to pull
back the data without any *reported* errors, and in any case I *like* having the
last block re-read. What I have done as a temporary workaround is to use Maximum
Volume Bytes. I know that with compression I get about 270 GB per tape, so I
have limited the volumes to 260 GB. That way I still get the last block test and
I have some time to figure out what the problem is (yesterday I was considering
a re-install of Solaris 10).

I have to admit that I'm completely stumped over this one - I'm not accustomed
to that and I *don't* like it :-)

Allan


------------------------------------------------------------------------------
Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
• 3 signs your SCM is hindering your productivity
• Requirements for releasing software faster
• Expert tips and advice for migrating your SCM now
http://p.sf.net/sfu/perforce
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: btape fill failure on HP LTO6/4 drives

Allan Black
On 09/05/14 14:09, Allan Black wrote:

> On 01/04/14 10:26, Roberts, Ben wrote:
>>> It appears that the OS tape driver does not properly
>>> implement back space record after an EOT.  This is a defect of the operating
>>> system driver, but it is not fatal for Bacula.
>> Indeed I was seeing the same failure to backspace over EOT error in the job logs:
>> End of Volume "GSA784L6" at 3910:8005 on device "drive-1-tapestore1" (/dev/rmt/1mbn). Write of 64512 bytes got 0.
>> Error: Backspace record at EOT failed. ERR=I/O error
>> End of medium on Volume "GSA784L6" Bytes=3,909,704,343,552 Blocks=60,604,295 at 28-Mar-2014 21:28
> I am getting exactly the same symptoms, also under Solaris 11, except this time
> with an LTO2 drive. The drive worked perfectly under Solaris 10, though, and I
> only started seeing this after upgrading to 11.1.
>
> The btape fill/m test gave me this at the end of the first tape:
>
> Wrote block=3160000, file,blk=204,13499 VolBytes=203,857,855,488 rate=20.62 MB/s
> 08-May 16:59 btape JobId 0: End of Volume "TestVolume1" at 204:15112 on device
> "lto" (/dev/rmt/4cbn). Write of 64512 bytes got 0.
> 08-May 16:59 btape JobId 0: Error: Backspace record at EOT failed. ERR=I/O error
> btape: btape.c:2702 Last block at: 204:15111 this_dev_block_num=15112
> btape: btape.c:2737 End of tape 204:-1. Volume Bytes=203,961,913,344. Write rate
> = 20.60 MB/s
> 08-May 16:59 btape JobId 0: End of medium on Volume "TestVolume1"
> Bytes=203,961,913,344 Blocks=3,161,612 at 08-May-2014 16:59.

This is quite an old thread, but I'm reviving it because I have something significant
to add - I'm pleased to report that BSR over EOT is now working after an upgrade to
Solaris 11.3.

> I have difficulty believing that the Solaris mtio and/or st modules fail to
> handle EOT properly,

I now believe it, though :-)

A while ago, when researching this, I came across a document on support.oracle.com.
(Doc ID 1919928.1 if you want to go looking for it!) It wasn't really the same problem,
but it did contain a statement that struck a chord with me:

"Solaris 11 changed the way initial tape position is determined. st now uses Long Form
Read Position to determine the tapes position."

No **** man :-)
In other words, Sun/Oracle rewrote a large part of the st driver, and broke it.

The btape fill test would write to the tape OK with "Backward Space Record = no", but
the unfill test would get an I/O error trying to read the end of tape 1. I believe the
data were all written correctly but btape couldn't position the tape properly for unfill
due to problems with the Solaris 11 st driver.

I suspect that Bacula under early Solaris 11 wouldn't be able to recover some data from
a backup, if that data happened to be near the end of a tape. Restoring an entire backup
would possibly be OK though.

[ufsdump/ufsrestore work fine with multiple tapes because ufsrestore doesn't seek]

I can't be sure exactly where the bug appeared and disappeared, but it's a good bet it's
been there since Solaris 11 was first released. Based on the Solaris versions I have
used, I can say this much:

Solaris 10 - Works as expected
Solaris 11.1.0.24.2 - BSR over EOT causes I/O error
Solaris 11.1.18.5.0 - BSR over EOT causes I/O error
Solaris 11.2.8.4.0 - BSR over EOT causes I/O error
Solaris 11.3.13.4.0 - Works as expected

So it was fixed somewhere between 11.2.8 and 11.3.13 - now I wish I had upgraded to
Solaris 11.3 months ago!

Allan

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: btape fill failure on HP LTO6/4 drives

Kern Sibbald
Very interesting.  It seems we do not run a full fill test on real tape
drives very often nor do we run it on alternative systems like
Solaris.    Too bad Solaris did not run the full Bacula regression tests :-)

Kern

On 01/28/2017 01:00 PM, Allan Black wrote:

> On 09/05/14 14:09, Allan Black wrote:
>> On 01/04/14 10:26, Roberts, Ben wrote:
>>>> It appears that the OS tape driver does not properly
>>>> implement back space record after an EOT.  This is a defect of the operating
>>>> system driver, but it is not fatal for Bacula.
>>> Indeed I was seeing the same failure to backspace over EOT error in the job logs:
>>> End of Volume "GSA784L6" at 3910:8005 on device "drive-1-tapestore1" (/dev/rmt/1mbn). Write of 64512 bytes got 0.
>>> Error: Backspace record at EOT failed. ERR=I/O error
>>> End of medium on Volume "GSA784L6" Bytes=3,909,704,343,552 Blocks=60,604,295 at 28-Mar-2014 21:28
>> I am getting exactly the same symptoms, also under Solaris 11, except this time
>> with an LTO2 drive. The drive worked perfectly under Solaris 10, though, and I
>> only started seeing this after upgrading to 11.1.
>>
>> The btape fill/m test gave me this at the end of the first tape:
>>
>> Wrote block=3160000, file,blk=204,13499 VolBytes=203,857,855,488 rate=20.62 MB/s
>> 08-May 16:59 btape JobId 0: End of Volume "TestVolume1" at 204:15112 on device
>> "lto" (/dev/rmt/4cbn). Write of 64512 bytes got 0.
>> 08-May 16:59 btape JobId 0: Error: Backspace record at EOT failed. ERR=I/O error
>> btape: btape.c:2702 Last block at: 204:15111 this_dev_block_num=15112
>> btape: btape.c:2737 End of tape 204:-1. Volume Bytes=203,961,913,344. Write rate
>> = 20.60 MB/s
>> 08-May 16:59 btape JobId 0: End of medium on Volume "TestVolume1"
>> Bytes=203,961,913,344 Blocks=3,161,612 at 08-May-2014 16:59.
> This is quite an old thread, but I'm reviving it because I have something significant
> to add - I'm pleased to report that BSR over EOT is now working after an upgrade to
> Solaris 11.3.
>
>> I have difficulty believing that the Solaris mtio and/or st modules fail to
>> handle EOT properly,
> I now believe it, though :-)
>
> A while ago, when researching this, I came across a document on support.oracle.com.
> (Doc ID 1919928.1 if you want to go looking for it!) It wasn't really the same problem,
> but it did contain a statement that struck a chord with me:
>
> "Solaris 11 changed the way initial tape position is determined. st now uses Long Form
> Read Position to determine the tapes position."
>
> No **** man :-)
> In other words, Sun/Oracle rewrote a large part of the st driver, and broke it.
>
> The btape fill test would write to the tape OK with "Backward Space Record = no", but
> the unfill test would get an I/O error trying to read the end of tape 1. I believe the
> data were all written correctly but btape couldn't position the tape properly for unfill
> due to problems with the Solaris 11 st driver.
>
> I suspect that Bacula under early Solaris 11 wouldn't be able to recover some data from
> a backup, if that data happened to be near the end of a tape. Restoring an entire backup
> would possibly be OK though.
>
> [ufsdump/ufsrestore work fine with multiple tapes because ufsrestore doesn't seek]
>
> I can't be sure exactly where the bug appeared and disappeared, but it's a good bet it's
> been there since Solaris 11 was first released. Based on the Solaris versions I have
> used, I can say this much:
>
> Solaris 10 - Works as expected
> Solaris 11.1.0.24.2 - BSR over EOT causes I/O error
> Solaris 11.1.18.5.0 - BSR over EOT causes I/O error
> Solaris 11.2.8.4.0 - BSR over EOT causes I/O error
> Solaris 11.3.13.4.0 - Works as expected
>
> So it was fixed somewhere between 11.2.8 and 11.3.13 - now I wish I had upgraded to
> Solaris 11.3 months ago!
>
> Allan
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Bacula-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: btape fill failure on HP LTO6/4 drives

Roberts, Ben

This is very interesting to hear, I was just about to upgrade my backup boxes to 11.3 this month. I’ve been ignoring the EOT error for a long time since it has no impact on my ability to take or restore backups, but I shall look forward to confirming your findings in the near future.

 

Regards,

Ben Roberts

 

From: Kern Sibbald [mailto:[hidden email]]
Sent: 28 January 2017 13:03
To: Allan Black <[hidden email]>; [hidden email]
Subject: Re: [Bacula-users] btape fill failure on HP LTO6/4 drives

 

Very interesting. It seems we do not run a full fill test on real tape
drives very often nor do we run it on alternative systems like
Solaris. Too bad Solaris did not run the full Bacula regression tests :-)

Kern

On 01/28/2017 01:00 PM, Allan Black wrote:
> On 09/05/14 14:09, Allan Black wrote:
>> On 01/04/14 10:26, Roberts, Ben wrote:
>>>> It appears that the OS tape driver does not properly
>>>> implement back space record after an EOT. This is a defect of the operating
>>>> system driver, but it is not fatal for Bacula.
>>> Indeed I was seeing the same failure to backspace over EOT error in the job logs:
>>> End of Volume "GSA784L6" at 3910:8005 on device "drive-1-tapestore1" (/dev/rmt/1mbn). Write of 64512 bytes got 0.
>>> Error: Backspace record at EOT failed. ERR=I/O error
>>> End of medium on Volume "GSA784L6" Bytes=3,909,704,343,552 Blocks=60,604,295 at 28-Mar-2014 21:28
>> I am getting exactly the same symptoms, also under Solaris 11, except this time
>> with an LTO2 drive. The drive worked perfectly under Solaris 10, though, and I
>> only started seeing this after upgrading to 11.1.
>>
>> The btape fill/m test gave me this at the end of the first tape:
>>
>> Wrote block=3160000, file,blk=204,13499 VolBytes=203,857,855,488 rate=20.62 MB/s
>> 08-May 16:59 btape JobId 0: End of Volume "TestVolume1" at 204:15112 on device
>> "lto" (/dev/rmt/4cbn). Write of 64512 bytes got 0.
>> 08-May 16:59 btape JobId 0: Error: Backspace record at EOT failed. ERR=I/O error
>> btape: btape.c:2702 Last block at: 204:15111 this_dev_block_num=15112
>> btape: btape.c:2737 End of tape 204:-1. Volume Bytes=203,961,913,344. Write rate
>> = 20.60 MB/s
>> 08-May 16:59 btape JobId 0: End of medium on Volume "TestVolume1"
>> Bytes=203,961,913,344 Blocks=3,161,612 at 08-May-2014 16:59.
> This is quite an old thread, but I'm reviving it because I have something significant
> to add - I'm pleased to report that BSR over EOT is now working after an upgrade to
> Solaris 11.3.
>
>> I have difficulty believing that the Solaris mtio and/or st modules fail to
>> handle EOT properly,
> I now believe it, though :-)
>
> A while ago, when researching this, I came across a document on support.oracle.com.
> (Doc ID 1919928.1 if you want to go looking for it!) It wasn't really the same problem,
> but it did contain a statement that struck a chord with me:
>
> "Solaris 11 changed the way initial tape position is determined. st now uses Long Form
> Read Position to determine the tapes position."
>
> No **** man :-)
> In other words, Sun/Oracle rewrote a large part of the st driver, and broke it.
>
> The btape fill test would write to the tape OK with "Backward Space Record = no", but
> the unfill test would get an I/O error trying to read the end of tape 1. I believe the
> data were all written correctly but btape couldn't position the tape properly for unfill
> due to problems with the Solaris 11 st driver.
>
> I suspect that Bacula under early Solaris 11 wouldn't be able to recover some data from
> a backup, if that data happened to be near the end of a tape. Restoring an entire backup
> would possibly be OK though.
>
> [ufsdump/ufsrestore work fine with multiple tapes because ufsrestore doesn't seek]
>
> I can't be sure exactly where the bug appeared and disappeared, but it's a good bet it's
> been there since Solaris 11 was first released. Based on the Solaris versions I have
> used, I can say this much:
>
> Solaris 10 - Works as expected
> Solaris 11.1.0.24.2 - BSR over EOT causes I/O error
> Solaris 11.1.18.5.0 - BSR over EOT causes I/O error
> Solaris 11.2.8.4.0 - BSR over EOT causes I/O error
> Solaris 11.3.13.4.0 - Works as expected
>
> So it was fixed somewhere between 11.2.8 and 11.3.13 - now I wish I had upgraded to
> Solaris 11.3 months ago!
>
> Allan
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Bacula-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users


This email and any files transmitted with it contain confidential and proprietary information and is solely for the use of the intended recipient. If you are not the intended recipient please return the email to the sender and delete it from your computer and you must not use, disclose, distribute, copy, print or rely on this email or its contents. This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. Any comments or statements made herein do not necessarily reflect those of GSA Capital. GSA Capital Partners LLP is authorised and regulated by the Financial Conduct Authority and is registered in England and Wales at Stratton House, 5 Stratton Street, London W1J 8LA, number OC309261. GSA Capital Services Limited is registered in England and Wales at the same address, number 5320529.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users