Quantcast

Fwd: Re: Bandwidth in Bacula (slow rate)

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Fwd: Re: Bandwidth in Bacula (slow rate)

Norbert Gomes




-------- Message transféré --------
Sujet : Re: [Bacula-users] Bandwidth in Bacula (slow rate)
Date : Wed, 22 Mar 2017 14:57:11 +0100
De : Norbert Gomes [hidden email]
Pour : Josh Fisher [hidden email]


Hi

I've been confronted at the same situation, and that was due to the compression that was enabled on the FileSet. You can try to disable it and see the network transfer performance.

The workaround was to disable compression for some file extensions

Regards

Norbert


Le 21/03/2017 à 15:35, Josh Fisher a écrit :

On 3/21/2017 9:44 AM, Petar Kozić wrote:
Hi,

I will try to describe my problem with slow transfer rate.

I was setup Bacula server and now every day I will put one by one instances in backup.

Yesterday I was put one instances and backup was done fine. But for about 47,000 files, sum size 930 MB, bacula is transfer for 11 minutes. Transfer rate is about 1,230 KB/s.

That's very slow. I try to measure bandwidth with iperf. Both instances are on linux. That was Gigabite network between two nodes. 

Iperf says:
Interval            Transfer             Bandwidth
0.0-10.0 sec     1.08 GBytes    928 Mbits/sec 

I don’t have any Bandwidth statement in configuration file.
How can I run backup faster ?


What are the SpoolData and SpoolAttributes settings for the job? This is likely due to database issues. Make sure that SpoolAttributes=yes so that database updates happen in a batch at the end of the job, rather than during the job. If writing to tape, then make sure SpoolData=yes. If writing to disk, then SpoolData=no.





------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: Re: Bandwidth in Bacula (slow rate)

Alan Brown


FWIW: There is virtually no benefit in network compression for link speeds of 1Gb/s or faster.

It's a net benefit on WAN links or on 100Mb/s networks, but I found it had a tendency to slow things down (and use a lot of CPU!) on 1Gb/s networks vs letting the networking traffic run uncompressed. On 10Gb/s interconnects it is definitely slower than letting things run uncompressed.

Experimentation has shown that most modern CPUs seem to max out about 120-150MB/s throughput on singlethreaded simple compression, or about 35-40MB/s if gzip is used.

As in many things there are tradeoffs of CPU vs speed and if your network is "fast enough" then all compressing data in transit is really achieving is making your system run hotter with no real benefit.

As far as I'm aware bacula is not using using multithreaded compression libraries.

Kern, can you correct me if I'm wrong?

Alan


On 22/03/17 15:15, Norbert Gomes wrote:




-------- Message transféré --------
Sujet : Re: [Bacula-users] Bandwidth in Bacula (slow rate)
Date : Wed, 22 Mar 2017 14:57:11 +0100
De : Norbert Gomes [hidden email]
Pour : Josh Fisher [hidden email]


Hi

I've been confronted at the same situation, and that was due to the compression that was enabled on the FileSet. You can try to disable it and see the network transfer performance.

The workaround was to disable compression for some file extensions

Regards

Norbert


Le 21/03/2017 à 15:35, Josh Fisher a écrit :

On 3/21/2017 9:44 AM, Petar Kozić wrote:
Hi,

I will try to describe my problem with slow transfer rate.

I was setup Bacula server and now every day I will put one by one instances in backup.

Yesterday I was put one instances and backup was done fine. But for about 47,000 files, sum size 930 MB, bacula is transfer for 11 minutes. Transfer rate is about 1,230 KB/s.

That's very slow. I try to measure bandwidth with iperf. Both instances are on linux. That was Gigabite network between two nodes. 

Iperf says:
Interval            Transfer             Bandwidth
0.0-10.0 sec     1.08 GBytes    928 Mbits/sec 

I don’t have any Bandwidth statement in configuration file.
How can I run backup faster ?


What are the SpoolData and SpoolAttributes settings for the job? This is likely due to database issues. Make sure that SpoolAttributes=yes so that database updates happen in a batch at the end of the job, rather than during the job. If writing to tape, then make sure SpoolData=yes. If writing to disk, then SpoolData=no.





------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: Re: Bandwidth in Bacula (slow rate)

Kern Sibbald

Hello Alan,

I think you are speaking about what we call Comm Line compression in Bacula.  Note, currently (until community version 9.0.x) Comm line compression exists only in the Bacula Enterprise Edition, so it is not generally an issue on this list.  There is a data compression feature in Bacula, but the primary purpose of that is for data storage compression providing you are not using deduplication.  Obviously compressed data will speed up comm line transfers, but whether or not there is any total throughput speed up is a bit hard to estimate because it depends on the compression algorithm, the FD CPU speed, and the comm line speed.

I forget the exact numbers but if I remember right for lz4, which is what Bacula Enterprise uses for the comm line compression, the CPU will max out at about 400-500 MB/s.  The comm line compression is by default on if both components (mostly FD<->SD) support it, but can be turned off with a directive depending on your situation.

For the moment, Bacula is not using multithread compression, but this is planned in the future.

I don't think you are wrong in anything you wrote.

Best regards,

Kern


On 03/22/2017 06:31 PM, Alan Brown wrote:


FWIW: There is virtually no benefit in network compression for link speeds of 1Gb/s or faster.

It's a net benefit on WAN links or on 100Mb/s networks, but I found it had a tendency to slow things down (and use a lot of CPU!) on 1Gb/s networks vs letting the networking traffic run uncompressed. On 10Gb/s interconnects it is definitely slower than letting things run uncompressed.

Experimentation has shown that most modern CPUs seem to max out about 120-150MB/s throughput on singlethreaded simple compression, or about 35-40MB/s if gzip is used.

As in many things there are tradeoffs of CPU vs speed and if your network is "fast enough" then all compressing data in transit is really achieving is making your system run hotter with no real benefit.

As far as I'm aware bacula is not using using multithreaded compression libraries.

Kern, can you correct me if I'm wrong?

Alan


On 22/03/17 15:15, Norbert Gomes wrote:




-------- Message transféré --------
Sujet : Re: [Bacula-users] Bandwidth in Bacula (slow rate)
Date : Wed, 22 Mar 2017 14:57:11 +0100
De : Norbert Gomes [hidden email]
Pour : Josh Fisher [hidden email]


Hi

I've been confronted at the same situation, and that was due to the compression that was enabled on the FileSet. You can try to disable it and see the network transfer performance.

The workaround was to disable compression for some file extensions

Regards

Norbert


Le 21/03/2017 à 15:35, Josh Fisher a écrit :

On 3/21/2017 9:44 AM, Petar Kozić wrote:
Hi,

I will try to describe my problem with slow transfer rate.

I was setup Bacula server and now every day I will put one by one instances in backup.

Yesterday I was put one instances and backup was done fine. But for about 47,000 files, sum size 930 MB, bacula is transfer for 11 minutes. Transfer rate is about 1,230 KB/s.

That's very slow. I try to measure bandwidth with iperf. Both instances are on linux. That was Gigabite network between two nodes. 

Iperf says:
Interval            Transfer             Bandwidth
0.0-10.0 sec     1.08 GBytes    928 Mbits/sec 

I don’t have any Bandwidth statement in configuration file.
How can I run backup faster ?


What are the SpoolData and SpoolAttributes settings for the job? This is likely due to database issues. Make sure that SpoolAttributes=yes so that database updates happen in a batch at the end of the job, rather than during the job. If writing to tape, then make sure SpoolData=yes. If writing to disk, then SpoolData=no.





------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users




------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: Re: Bandwidth in Bacula (slow rate)

Petar Kozić
I try to reconfigure bacula director and configure with

—enable-batch-insert

My transfer rate now is increase for more of 50% and I’m happy for this.

Despooling also is much faster, but...

I have now another problem.

Verify DiskToCatalog is very slow and my Scheduled job is waiting too long.

Is there a way to increase speed of verify.


From one client:

Files=212,578 Bytes=33,157,419,846 AveBytes/sec=646,709 LastBytes/sec=646,709 Errors=0

From another client:

Files=8,337 Bytes=170,692,703 AveBytes/sec=14,933 LastBytes/sec=28,204 Errors=0




Petar Kozić
System Administrator

On March 23, 2017 at 10:13:20 AM, Kern Sibbald ([hidden email]) wrote:

Hello Alan,

I think you are speaking about what we call Comm Line compression in Bacula.  Note, currently (until community version 9.0.x) Comm line compression exists only in the Bacula Enterprise Edition, so it is not generally an issue on this list.  There is a data compression feature in Bacula, but the primary purpose of that is for data storage compression providing you are not using deduplication.  Obviously compressed data will speed up comm line transfers, but whether or not there is any total throughput speed up is a bit hard to estimate because it depends on the compression algorithm, the FD CPU speed, and the comm line speed.

I forget the exact numbers but if I remember right for lz4, which is what Bacula Enterprise uses for the comm line compression, the CPU will max out at about 400-500 MB/s.  The comm line compression is by default on if both components (mostly FD<->SD) support it, but can be turned off with a directive depending on your situation.

For the moment, Bacula is not using multithread compression, but this is planned in the future.

I don't think you are wrong in anything you wrote.

Best regards,

Kern


On 03/22/2017 06:31 PM, Alan Brown wrote:


FWIW: There is virtually no benefit in network compression for link speeds of 1Gb/s or faster.

It's a net benefit on WAN links or on 100Mb/s networks, but I found it had a tendency to slow things down (and use a lot of CPU!) on 1Gb/s networks vs letting the networking traffic run uncompressed. On 10Gb/s interconnects it is definitely slower than letting things run uncompressed.

Experimentation has shown that most modern CPUs seem to max out about 120-150MB/s throughput on singlethreaded simple compression, or about 35-40MB/s if gzip is used.

As in many things there are tradeoffs of CPU vs speed and if your network is "fast enough" then all compressing data in transit is really achieving is making your system run hotter with no real benefit.

As far as I'm aware bacula is not using using multithreaded compression libraries.

Kern, can you correct me if I'm wrong?

Alan


On 22/03/17 15:15, Norbert Gomes wrote:




-------- Message transféré --------
Sujet : Re: [Bacula-users] Bandwidth in Bacula (slow rate)
Date : Wed, 22 Mar 2017 14:57:11 +0100
De : Norbert Gomes [hidden email]
Pour : Josh Fisher [hidden email]


Hi

I've been confronted at the same situation, and that was due to the compression that was enabled on the FileSet. You can try to disable it and see the network transfer performance.

The workaround was to disable compression for some file extensions

Regards

Norbert


Le 21/03/2017 à 15:35, Josh Fisher a écrit :

On 3/21/2017 9:44 AM, Petar Kozić wrote:
Hi,

I will try to describe my problem with slow transfer rate.

I was setup Bacula server and now every day I will put one by one instances in backup.

Yesterday I was put one instances and backup was done fine. But for about 47,000 files, sum size 930 MB, bacula is transfer for 11 minutes. Transfer rate is about 1,230 KB/s.

That's very slow. I try to measure bandwidth with iperf. Both instances are on linux. That was Gigabite network between two nodes. 

Iperf says:
Interval            Transfer             Bandwidth
0.0-10.0 sec     1.08 GBytes    928 Mbits/sec 

I don’t have any Bandwidth statement in configuration file.
How can I run backup faster ?


What are the SpoolData and SpoolAttributes settings for the job? This is likely due to database issues. Make sure that SpoolAttributes=yes so that database updates happen in a batch at the end of the job, rather than during the job. If writing to tape, then make sure SpoolData=yes. If writing to disk, then SpoolData=no.





------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users




------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Loading...