Hartbeat during Client Run Before Job

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Hartbeat during Client Run Before Job

Jan Gazda

Hello all,

I have a problem with network timeout during backup.

Backup server with bacula is on different subnet than clients and there is a Sonicwall with 15 minutes timeout for TCP connection.

I set up Heartbeat interval to 60 seconds but it seems that heartbeat is not issued during script which backs up the database and when it takes longer the firewall just breaks the connection and backup fails.

My client versions:

bacula-client-5.0.3-1

bacula-client-5.2.13-19

 

 

Current workaround was to extend the timeout but it is not really a temporary solution.

 

Thank you

 

Jan



Sapienza Logo Jan Gazda
ECLIPSE & Office IT Support Technician
Sapienza Consulting
Kapteynstraat 1
2201 BB Noordwijk
Netherlands
Tel: +31 (0) 71 407 6518
Fax: +31 (0) 71 407 6536
Mob: +31 682607099


Follow us on:
LinkedinTwitterYouTube


The contents of this e-mail and any attachment are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email and/or any attachment in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not distribute, disclose, use, copy, print or rely on this e-mail.

Communicating by email is not 100% secure, error or virus free. It is your responsibility to ensure that viruses do not adversely affect your system. If you communicate with us by e-mail you agree to take appropriate measures to minimise these risks when e-mailing us.

We may monitor, record, store and use any email communication with you in order to check any instructions given to us, for training purposes, for crime prevention, to improve the quality of our customer service and to keep records of our business communications.

Sapienza Consulting Limited is a limited company registered in England and Wales with its registered office at 61 Rodney Street, Liverpool, Merseyside, L1 9ER and registered number 2986023

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hartbeat during Client Run Before Job

Kern Sibbald
Hello,

Where are you running the script?  On the Director or on the Client.  Once I have a better idea what you are doing, I can provide a definitive answer. It is possible that Bacula is not in control when a script is running and that it does not get control to issue the Heartbeat, but I would have to check because in most cases (in the SD and the FD) there are separate threads that keep the heartbeat running.  I recommend that you set the heartbeat interval at 300.  60 seconds is way too short in my view and may be a cause of your problem.  In addition, if it is the FD that is timing out, you must be very careful to setup the heartbeat on both the Director and the FD.

The internet standard is 2 hours, and Bacula always request that the network keep the network up, so in fact a 15 minute timeout is too short by internet standards and good quality switches should respect the KEEP_ALIVE request that Bacula issues -- unfortunately most switches do not.

Final comment: your Bacula version is *very* old, and though I only remember early (when I am not sure) changes to the heartbeat, it is possible that later versions do a much better job.

Best regards,
Kern

On 11/10/2016 08:43 AM, Jan Gazda wrote:

Hello all,

I have a problem with network timeout during backup.

Backup server with bacula is on different subnet than clients and there is a Sonicwall with 15 minutes timeout for TCP connection.

I set up Heartbeat interval to 60 seconds but it seems that heartbeat is not issued during script which backs up the database and when it takes longer the firewall just breaks the connection and backup fails.

My client versions:

bacula-client-5.0.3-1

bacula-client-5.2.13-19

 

 

Current workaround was to extend the timeout but it is not really a temporary solution.

 

Thank you

 

Jan



Sapienza Logo Jan Gazda
ECLIPSE & Office IT Support Technician
Sapienza Consulting
Kapteynstraat 1
2201 BB Noordwijk
Netherlands
Tel: +31 (0) 71 407 6518
Fax: +31 (0) 71 407 6536
Mob: +31 682607099


Follow us on:
LinkedinTwitterYouTube


The contents of this e-mail and any attachment are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email and/or any attachment in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not distribute, disclose, use, copy, print or rely on this e-mail.

Communicating by email is not 100% secure, error or virus free. It is your responsibility to ensure that viruses do not adversely affect your system. If you communicate with us by e-mail you agree to take appropriate measures to minimise these risks when e-mailing us.

We may monitor, record, store and use any email communication with you in order to check any instructions given to us, for training purposes, for crime prevention, to improve the quality of our customer service and to keep records of our business communications.

Sapienza Consulting Limited is a limited company registered in England and Wales with its registered office at 61 Rodney Street, Liverpool, Merseyside, L1 9ER and registered number 2986023


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi


_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users



------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hartbeat during Client Run Before Job

Josh Fisher

On 11/10/2016 3:43 AM, Kern Sibbald wrote:
Hello,

Where are you running the script?  On the Director or on the Client.  Once I have a better idea what you are doing, I can provide a definitive answer. It is possible that Bacula is not in control when a script is running and that it does not get control to issue the Heartbeat, but I would have to check because in most cases (in the SD and the FD) there are separate threads that keep the heartbeat running.  I recommend that you set the heartbeat interval at 300.  60 seconds is way too short in my view and may be a cause of your problem.  In addition, if it is the FD that is timing out, you must be very careful to setup the heartbeat on both the Director and the FD.

The internet standard is 2 hours, and Bacula always request that the network keep the network up, so in fact a 15 minute timeout is too short by internet standards and good quality switches should respect the KEEP_ALIVE request that Bacula issues -- unfortunately most switches do not.

My experience is that it is not the KEEP_ALIVE requests not being honored, but rather the handling (or mishandling) of the return from 802.3az Low Power Idle state. Disabling 802.3az support in the NIC driver on both ends seems to help. It definitely works if there is only one switch in between, since 802.3az requires backward compatibility. Even if only one of the NICs has 802.3az disabled, the other is supposed to work in legacy/standard mode. So the NIC-to-switch link can be placed in legacy/standard mode, but it is not clear (at least to me) how this affects any switch-to-switch links when there are multiple switches between the two NICs. In any case, I recommend returning the heartbeat and socket timeouts to their default values and then try disabling 802.3az support for the affected NICs.



Final comment: your Bacula version is *very* old, and though I only remember early (when I am not sure) changes to the heartbeat, it is possible that later versions do a much better job.

Best regards,
Kern

On 11/10/2016 08:43 AM, Jan Gazda wrote:

Hello all,

I have a problem with network timeout during backup.

Backup server with bacula is on different subnet than clients and there is a Sonicwall with 15 minutes timeout for TCP connection.

I set up Heartbeat interval to 60 seconds but it seems that heartbeat is not issued during script which backs up the database and when it takes longer the firewall just breaks the connection and backup fails.

My client versions:

bacula-client-5.0.3-1

bacula-client-5.2.13-19

 

 

Current workaround was to extend the timeout but it is not really a temporary solution.

 

Thank you

 

Jan



Sapienza Logo Jan Gazda
ECLIPSE & Office IT Support Technician
Sapienza Consulting
Kapteynstraat 1
2201 BB Noordwijk
Netherlands
Tel: +31 (0) 71 407 6518
Fax: +31 (0) 71 407 6536
Mob: +31 682607099


Follow us on:
LinkedinTwitterYouTube


The contents of this e-mail and any attachment are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email and/or any attachment in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not distribute, disclose, use, copy, print or rely on this e-mail.

Communicating by email is not 100% secure, error or virus free. It is your responsibility to ensure that viruses do not adversely affect your system. If you communicate with us by e-mail you agree to take appropriate measures to minimise these risks when e-mailing us.

We may monitor, record, store and use any email communication with you in order to check any instructions given to us, for training purposes, for crime prevention, to improve the quality of our customer service and to keep records of our business communications.

Sapienza Consulting Limited is a limited company registered in England and Wales with its registered office at 61 Rodney Street, Liverpool, Merseyside, L1 9ER and registered number 2986023


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi


_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users




------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi


_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hartbeat during Client Run Before Job

Jan Gazda
In reply to this post by Kern Sibbald

Hello Kern,

Thank you for your suggestion we are running some pre-backup scripts on DBs on our clients.

I will soon be able to set up clean test environment where I’d like to try latest version of bacula server and client and check if problem persist.

 

Thank you

 

Jan

 

From: Kern Sibbald [mailto:[hidden email]]
Sent: Thursday, November 10, 2016 09:43
To: Jan Gazda <[hidden email]>; [hidden email]
Subject: Re: [Bacula-users] Hartbeat during Client Run Before Job

 

Hello,

Where are you running the script?  On the Director or on the Client.  Once I have a better idea what you are doing, I can provide a definitive answer. It is possible that Bacula is not in control when a script is running and that it does not get control to issue the Heartbeat, but I would have to check because in most cases (in the SD and the FD) there are separate threads that keep the heartbeat running.  I recommend that you set the heartbeat interval at 300.  60 seconds is way too short in my view and may be a cause of your problem.  In addition, if it is the FD that is timing out, you must be very careful to setup the heartbeat on both the Director and the FD.

The internet standard is 2 hours, and Bacula always request that the network keep the network up, so in fact a 15 minute timeout is too short by internet standards and good quality switches should respect the KEEP_ALIVE request that Bacula issues -- unfortunately most switches do not.

Final comment: your Bacula version is *very* old, and though I only remember early (when I am not sure) changes to the heartbeat, it is possible that later versions do a much better job.

Best regards,
Kern

On 11/10/2016 08:43 AM, Jan Gazda wrote:

Hello all,

I have a problem with network timeout during backup.

Backup server with bacula is on different subnet than clients and there is a Sonicwall with 15 minutes timeout for TCP connection.

I set up Heartbeat interval to 60 seconds but it seems that heartbeat is not issued during script which backs up the database and when it takes longer the firewall just breaks the connection and backup fails.

My client versions:

bacula-client-5.0.3-1

bacula-client-5.2.13-19

 

 

Current workaround was to extend the timeout but it is not really a temporary solution.

 

Thank you

 

Jan

 

Sapienza Logo

Jan Gazda
ECLIPSE & Office IT Support Technician
Sapienza Consulting
Kapteynstraat 1
2201 BB Noordwijk
Netherlands
Tel: +31 (0) 71 407 6518
Fax: +31 (0) 71 407 6536
Mob: +31 682607099


Follow us on:
LinkedinTwitterYouTube



The contents of this e-mail and any attachment are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email and/or any attachment in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not distribute, disclose, use, copy, print or rely on this e-mail.

Communicating by email is not 100% secure, error or virus free. It is your responsibility to ensure that viruses do not adversely affect your system. If you communicate with us by e-mail you agree to take appropriate measures to minimise these risks when e-mailing us.

We may monitor, record, store and use any email communication with you in order to check any instructions given to us, for training purposes, for crime prevention, to improve the quality of our customer service and to keep records of our business communications.

Sapienza Consulting Limited is a limited company registered in England and Wales with its registered office at 61 Rodney Street, Liverpool, Merseyside, L1 9ER and registered number 2986023



------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi




_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users

 



Sapienza Logo Jan Gazda
ECLIPSE & Office IT Support Technician
Sapienza Consulting
Kapteynstraat 1
2201 BB Noordwijk
Netherlands
Tel: +31 (0) 71 407 6518
Fax: +31 (0) 71 407 6536
Mob: +31 682607099


Follow us on:
LinkedinTwitterYouTube


The contents of this e-mail and any attachment are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email and/or any attachment in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not distribute, disclose, use, copy, print or rely on this e-mail.

Communicating by email is not 100% secure, error or virus free. It is your responsibility to ensure that viruses do not adversely affect your system. If you communicate with us by e-mail you agree to take appropriate measures to minimise these risks when e-mailing us.

We may monitor, record, store and use any email communication with you in order to check any instructions given to us, for training purposes, for crime prevention, to improve the quality of our customer service and to keep records of our business communications.

Sapienza Consulting Limited is a limited company registered in England and Wales with its registered office at 61 Rodney Street, Liverpool, Merseyside, L1 9ER and registered number 2986023

------------------------------------------------------------------------------

_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Loading...