LZO compression on Windows Client

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

LZO compression on Windows Client

Norbert Gomes
Hi list !

GZIP compression is quite slow on our Windows clients (5.2.10), it
increases a lot the duration of theses backups. The speed of the network
transfer is at: 20 MB/s with GZIP and 100 MB/s without compression

I read that LZO would be faster, but when I enable it (compression="LZO"
in the FileSet Include Options), no compression is applied

I'm using the binary on
https://sourceforge.net/projects/bacula/files/Win32_64/5.2.10/

Is LZO enabled on these program ?


And also, have you got some ideas to speed up my backups using
compression (GZIP1 or GZIP6 gives the same level of performance) ?


Regards

Norbert


--
Norbert GOMES
Université d'Orléans



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: LZO compression on Windows Client

Kern Sibbald
On 02/17/2017 10:18 AM, Norbert Gomes wrote:

> Hi list !
>
> GZIP compression is quite slow on our Windows clients (5.2.10), it
> increases a lot the duration of theses backups. The speed of the network
> transfer is at: 20 MB/s with GZIP and 100 MB/s without compression
>
> I read that LZO would be faster, but when I enable it (compression="LZO"
> in the FileSet Include Options), no compression is applied
>
> I'm using the binary on
> https://sourceforge.net/projects/bacula/files/Win32_64/5.2.10/
>
> Is LZO enabled on these program ?

No.

However, if I am not mistaken, it is available in newer 7.4.x Windows
FDs.  Note the SD and Dir must be on the same or later version as all
FDs.  That is Bacula SD and Dir will work with older FDs but not (unless
you are really lucky) with newer FDs.

Regards,
Kern

>
>
> And also, have you got some ideas to speed up my backups using
> compression (GZIP1 or GZIP6 gives the same level of performance) ?
>
>
> Regards
>
> Norbert
>
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: LZO compression on Windows Client

Heitor Faria
Hello, Norbert,

>> And also, have you got some ideas to speed up my backups using
>> compression (GZIP1 or GZIP6 gives the same level of performance) ?

No, they don't.
Please find this performance analysis: http://bacula.us/bacula-compression-analysis/

>> Regards
>>
>> Norbert

Regards,
--
===========================================================================
Heitor Medrado de Faria - LPIC-III | ITIL-F | Bacula Systems Certified Administrator II
• Do you need Bacula training? http://bacula.us/video-classes/ 
+55 61 8268-4220 | http://bacula.us 
===========================================================================

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: LZO compression on Windows Client

compdoc
In reply to this post by Kern Sibbald
On 02/17/2017 10:18 AM, Norbert Gomes wrote:

> I read that LZO would be faster, but when I enable it (compression="LZO"
> in the FileSet Include Options), no compression is applied

I use LZO to backup my jpeg images, that are stored on SSDs. I used to
use gzip2 for speed, and in the hopes it wouldn't damage the images as
much as higher levels of compression might.

Now, I hope LZO doesnt damage them, and I consider restoring them from
backup is not ideal, but better than not having them.

Would it be better to create a job that uses no compression for .jpg files?


Currently:

FileSet {
     Enable VSS = yes
     Include {
     Options {
     ...
        }
     Options {
     compression = LZO
     signature = SHA1
     }
     File = "C:/"
     File = "D:/"
     File = "F:/"
     }
}


 From the logs:

   Backup Level:           Differential, since=2017-02-13 12:45:02
   Client:                 "" 7.4.4 (28Sep16) Microsoft Professional
(build 9200), 64-bit,Cross-compile,Win64
   Priority:               10
   FD Files Written:       132
   SD Files Written:       132
   FD Bytes Written:       17,883,070 (17.88 MB)
   SD Bytes Written:       17,902,783 (17.90 MB)
   Rate:                   95.1 KB/s
   Software Compression:   65.9% 2.9:1
   Snapshot/VSS:           yes
   Encryption:             no
   Accurate:               no





------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: LZO compression on Windows Client

Phil Stracchino-2
On 02/18/17 15:01, compdoc wrote:
> On 02/17/2017 10:18 AM, Norbert Gomes wrote:
>
>> I read that LZO would be faster, but when I enable it (compression="LZO"
>> in the FileSet Include Options), no compression is applied
>
> I use LZO to backup my jpeg images, that are stored on SSDs. I used to
> use gzip2 for speed, and in the hopes it wouldn't damage the images as
> much as higher levels of compression might.


This is a misconception.  No level of lossless compression will damage
the compressed data.  A *lossy* compression algorithm would, but Bacula
for obvious reasons does not use any lossy compression algorithms.  That
would defeat the entire point of making backups.


--
  Phil Stracchino
  Babylon Communications
  [hidden email]
  [hidden email]
  Landline: 603.293.8485

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: LZO compression on Windows Client

Mike Ruskai
In reply to this post by Norbert Gomes
On 2/17/2017 04:18, Norbert Gomes wrote:

> Hi list !
>
> GZIP compression is quite slow on our Windows clients (5.2.10), it
> increases a lot the duration of theses backups. The speed of the network
> transfer is at: 20 MB/s with GZIP and 100 MB/s without compression
>
> I read that LZO would be faster, but when I enable it (compression="LZO"
> in the FileSet Include Options), no compression is applied
>
> I'm using the binary on
> https://sourceforge.net/projects/bacula/files/Win32_64/5.2.10/
>
> Is LZO enabled on these program ?
>
>
> And also, have you got some ideas to speed up my backups using
> compression (GZIP1 or GZIP6 gives the same level of performance) ?
>

As far as I know, LZO support for Windows wasn't added until version
7.x, which is only available in the enterprise version (though you can
use that build for personal use now).

I can confirm that it works in 7.0.5, and is considerably faster than
gzip1, and all other compression methods available in Bacula.

A short test on this machine with a 1.6GB database table shows LZO
(lzop.exe with default options, which should be equivalent to what
Bacula does) running more than three times faster than gzip -1.
146MB/sec vs 46MB/sec, or 11.1 seconds vs 35.1 seconds.  That's at a
cost of around 10% worth of compression - LZO was 52.5% of the original,
while gzip -1 was 42.9% of the original.  And just for comparison, bzip2
-9 ran at 9.5MB/sec for 172.2 seconds, compressing to 38.6% of the
original size.

Something else very much worth doing is to simply disable compression
for file extensions that indicate already-compressed data.  Something
like this within the file set definition:

         Options
             {
             Signature = MD5
             Wild = *.jpg
             Wild = *.png
             Wild = *.gif
             Wild = *.zip
             Wild = *.rar
             Wild = *.7z
             Wild = *.r??
             Wild = *.mpg
             Wild = *.wmv
             Wild = *.avi
             Wild = *.mov
             Wild = *.mkv
             Wild = *.mp3
             Wild = *.mp4
             Wild = *.gz
             Wild = *.bz2
             }
         Options
             {
             Signature = MD5
             Compression = LZO
             }

That uses LZO for all files that don't match the wildcard specification
in the first option block.  Trying to use LZO on something already
compressed, especially something compressed more highly, is going to
accomplish either nothing or worse than nothing (i.e. output larger than
the input).


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: LZO compression on Windows Client

Kern Sibbald
In reply to this post by compdoc
On 02/18/2017 09:01 PM, compdoc wrote:

> On 02/17/2017 10:18 AM, Norbert Gomes wrote:
>
>> I read that LZO would be faster, but when I enable it (compression="LZO"
>> in the FileSet Include Options), no compression is applied
> I use LZO to backup my jpeg images, that are stored on SSDs. I used to
> use gzip2 for speed, and in the hopes it wouldn't damage the images as
> much as higher levels of compression might.
>
> Now, I hope LZO doesnt damage them, and I consider restoring them from
> backup is not ideal, but better than not having them.

All types and levels of compression in Bacula decompress to *exactly*
the same bytes.  There is no loss.

As you mention when going from certain files such as .wav to .mp3 they
lose a certain amount of information.  It all depends on the compression
technique used.


Regards,
Kern

>
> Would it be better to create a job that uses no compression for .jpg files?
>
>
> Currently:
>
> FileSet {
>       Enable VSS = yes
>       Include {
>       Options {
>       ...
>          }
>       Options {
>       compression = LZO
>       signature = SHA1
>       }
>       File = "C:/"
>       File = "D:/"
>       File = "F:/"
>       }
> }
>
>
>   From the logs:
>
>     Backup Level:           Differential, since=2017-02-13 12:45:02
>     Client:                 "" 7.4.4 (28Sep16) Microsoft Professional
> (build 9200), 64-bit,Cross-compile,Win64
>     Priority:               10
>     FD Files Written:       132
>     SD Files Written:       132
>     FD Bytes Written:       17,883,070 (17.88 MB)
>     SD Bytes Written:       17,902,783 (17.90 MB)
>     Rate:                   95.1 KB/s
>     Software Compression:   65.9% 2.9:1
>     Snapshot/VSS:           yes
>     Encryption:             no
>     Accurate:               no
>
>
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Bacula-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: LZO compression on Windows Client

Norbert Gomes
Hi

Thank you for all your suggestions.

We work currently with the distributions packages, and they are in 5.x
version (in CentOS 7). But I saw that Debian Stretch will provide 7.4
version, it may be a good opportunity to migrate :-)

Also, good idea to disable compression for some file extensions, I'll
give it a look !

Regards

Norbert

Norbert GOMES
Université d'Orléans


Le 19/02/2017 à 09:53, Kern Sibbald a écrit :

> On 02/18/2017 09:01 PM, compdoc wrote:
>> On 02/17/2017 10:18 AM, Norbert Gomes wrote:
>>
>>> I read that LZO would be faster, but when I enable it (compression="LZO"
>>> in the FileSet Include Options), no compression is applied
>> I use LZO to backup my jpeg images, that are stored on SSDs. I used to
>> use gzip2 for speed, and in the hopes it wouldn't damage the images as
>> much as higher levels of compression might.
>>
>> Now, I hope LZO doesnt damage them, and I consider restoring them from
>> backup is not ideal, but better than not having them.
> All types and levels of compression in Bacula decompress to *exactly*
> the same bytes.  There is no loss.
>
> As you mention when going from certain files such as .wav to .mp3 they
> lose a certain amount of information.  It all depends on the compression
> technique used.
>
>
> Regards,
> Kern
>
>> Would it be better to create a job that uses no compression for .jpg files?
>>
>>
>> Currently:
>>
>> FileSet {
>>       Enable VSS = yes
>>       Include {
>>       Options {
>>       ...
>>          }
>>       Options {
>>       compression = LZO
>>       signature = SHA1
>>       }
>>       File = "C:/"
>>       File = "D:/"
>>       File = "F:/"
>>       }
>> }
>>
>>
>>   From the logs:
>>
>>     Backup Level:           Differential, since=2017-02-13 12:45:02
>>     Client:                 "" 7.4.4 (28Sep16) Microsoft Professional
>> (build 9200), 64-bit,Cross-compile,Win64
>>     Priority:               10
>>     FD Files Written:       132
>>     SD Files Written:       132
>>     FD Bytes Written:       17,883,070 (17.88 MB)
>>     SD Bytes Written:       17,902,783 (17.90 MB)
>>     Rate:                   95.1 KB/s
>>     Software Compression:   65.9% 2.9:1
>>     Snapshot/VSS:           yes
>>     Encryption:             no
>>     Accurate:               no
>>
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Bacula-users mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Bacula-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users