Volume retention starting time (documentation error?)

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Volume retention starting time (documentation error?)

Ivan Adzhubey
Hi,

In section "24.2 Pruning Directives", Bacula documentation claims that the
Volume Retention period begins only *after* the volume has already switched to
a status other than Append:

---quote---
A key point is that this time period [Volume Retention] is not even considered
as long at the Volume remains appendable. The Volume Retention period count
down begins only when the Append status has been changed to some othe status
(Full, Used, Purged, ...).
---quote ends---

I believe this is not true. In the rest of documentation, the three
independent retention periods (File, Job, and Volume) described as all
starting at the same point: specifically, at the end run time of a job.

Thanks,
Ivan

--
Ivan Adzhubey, Ph.D.
Instructor
Division of Genetics, Dept of Medicine
Brigham & Women's Hospital
Harvard Medical School
New Research Building, Room 0464C
77 Avenue Louis Pasteur
Boston, MA 02115
tel.: (617) 525-4728
fax:  (617) 525-4705
web: http://genetics.bwh.harvard.edu/wiki/sunyaevlab/



The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Volume retention starting time (documentation error?)

Kern Sibbald
Hello,

I haven't taken a look at the code, so what I say is subject to
confirmation.

I believe what is written is basically correct but with a nuance that is
not clearly explained.  What is correct is that the Volume cannot be
recycled until it is in a state other than Append, exactly as described
below.  Consequently, the data will remain on the Volume even if there
are no records in the Catalog that reference it, until the Volume has
been recycled. However, once its records are subject to pruning, they
will be pruned if the retention period measured from the last write time
has expired.

Each pruning (File, Job, Volume) is independent, and in addition the
most important one in terms of "data loss" is Volume Retention, so
drawing conclusions for Volume pruning based on pruning for File or Job
records is not likely to work.

I will look at the code to ensure that what I am saying is correct then
attempt to improve the documentation ...

Best regards,
Kern

PS: Note, the algorithm is designed to preserve the data as long as is
possible.  However, there are a few quirks of the current implementation:

1. Assuming you have a Job that writes on a Volume.  That Volume will
not automatically be pruned until the Job is run again.

2. If a Volume is only partially full so it remains in Append state.  As
long as no Jobs write to the Volume to fill it, it will not be
pruned/purged.

One of my projects over the next year or two is to implement a new
pruning/recycling algorithm that will be *much* simpler to understand
and easier to calculate  how long the data will be in the Catalog and on
the Volume.



On 06/18/2017 12:13 AM, Ivan Adzhubey wrote:

> Hi,
>
> In section "24.2 Pruning Directives", Bacula documentation claims that the
> Volume Retention period begins only *after* the volume has already switched to
> a status other than Append:
>
> ---quote---
> A key point is that this time period [Volume Retention] is not even considered
> as long at the Volume remains appendable. The Volume Retention period count
> down begins only when the Append status has been changed to some othe status
> (Full, Used, Purged, ...).
> ---quote ends---
>
> I believe this is not true. In the rest of documentation, the three
> independent retention periods (File, Job, and Volume) described as all
> starting at the same point: specifically, at the end run time of a job.
>
> Thanks,
> Ivan
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Volume retention starting time (documentation error?)

Ivan Adzhubey
Hi Kern,

I was referring only to the specific claim that the retention window for
volume is counted from the moment of Append->Full/Purged/Used transition of
the volume's status. The phrase in section 24.2 which says this is very
unequivocal and is also totally wrong. The rest of the pruning process is
described correctly, as far as I was able to comprehend it.

--Ivan

On Sunday, June 18, 2017 2:53:56 AM EDT Kern Sibbald wrote:

> Hello,
>
> I haven't taken a look at the code, so what I say is subject to
> confirmation.
>
> I believe what is written is basically correct but with a nuance that is
> not clearly explained.  What is correct is that the Volume cannot be
> recycled until it is in a state other than Append, exactly as described
> below.  Consequently, the data will remain on the Volume even if there
> are no records in the Catalog that reference it, until the Volume has
> been recycled. However, once its records are subject to pruning, they
> will be pruned if the retention period measured from the last write time
> has expired.
>
> Each pruning (File, Job, Volume) is independent, and in addition the
> most important one in terms of "data loss" is Volume Retention, so
> drawing conclusions for Volume pruning based on pruning for File or Job
> records is not likely to work.
>
> I will look at the code to ensure that what I am saying is correct then
> attempt to improve the documentation ...
>
> Best regards,
> Kern
>
> PS: Note, the algorithm is designed to preserve the data as long as is
> possible.  However, there are a few quirks of the current implementation:
>
> 1. Assuming you have a Job that writes on a Volume.  That Volume will
> not automatically be pruned until the Job is run again.
>
> 2. If a Volume is only partially full so it remains in Append state.  As
> long as no Jobs write to the Volume to fill it, it will not be
> pruned/purged.
>
> One of my projects over the next year or two is to implement a new
> pruning/recycling algorithm that will be *much* simpler to understand
> and easier to calculate  how long the data will be in the Catalog and on
> the Volume.
>
> On 06/18/2017 12:13 AM, Ivan Adzhubey wrote:
> > Hi,
> >
> > In section "24.2 Pruning Directives", Bacula documentation claims that the
> > Volume Retention period begins only *after* the volume has already
> > switched to a status other than Append:
> >
> > ---quote---
> > A key point is that this time period [Volume Retention] is not even
> > considered as long at the Volume remains appendable. The Volume Retention
> > period count down begins only when the Append status has been changed to
> > some othe status (Full, Used, Purged, ...).
> > ---quote ends---
> >
> > I believe this is not true. In the rest of documentation, the three
> > independent retention periods (File, Job, and Volume) described as all
> > starting at the same point: specifically, at the end run time of a job.
> >
> > Thanks,
> > Ivan



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Volume retention starting time (documentation error?)

Kern Sibbald
Hello Ivan,

You are free to view the text any way you want, but since I wrote it,
what I see is:

The problem is that the text does not say when the retention time
begins.  It only says when the retention period can begin to be applied
by Bacula.

Kern


On 06/18/2017 09:19 PM, Ivan Adzhubey wrote:

> Hi Kern,
>
> I was referring only to the specific claim that the retention window for
> volume is counted from the moment of Append->Full/Purged/Used transition of
> the volume's status. The phrase in section 24.2 which says this is very
> unequivocal and is also totally wrong. The rest of the pruning process is
> described correctly, as far as I was able to comprehend it.
>
> --Ivan
>
> On Sunday, June 18, 2017 2:53:56 AM EDT Kern Sibbald wrote:
>> Hello,
>>
>> I haven't taken a look at the code, so what I say is subject to
>> confirmation.
>>
>> I believe what is written is basically correct but with a nuance that is
>> not clearly explained.  What is correct is that the Volume cannot be
>> recycled until it is in a state other than Append, exactly as described
>> below.  Consequently, the data will remain on the Volume even if there
>> are no records in the Catalog that reference it, until the Volume has
>> been recycled. However, once its records are subject to pruning, they
>> will be pruned if the retention period measured from the last write time
>> has expired.
>>
>> Each pruning (File, Job, Volume) is independent, and in addition the
>> most important one in terms of "data loss" is Volume Retention, so
>> drawing conclusions for Volume pruning based on pruning for File or Job
>> records is not likely to work.
>>
>> I will look at the code to ensure that what I am saying is correct then
>> attempt to improve the documentation ...
>>
>> Best regards,
>> Kern
>>
>> PS: Note, the algorithm is designed to preserve the data as long as is
>> possible.  However, there are a few quirks of the current implementation:
>>
>> 1. Assuming you have a Job that writes on a Volume.  That Volume will
>> not automatically be pruned until the Job is run again.
>>
>> 2. If a Volume is only partially full so it remains in Append state.  As
>> long as no Jobs write to the Volume to fill it, it will not be
>> pruned/purged.
>>
>> One of my projects over the next year or two is to implement a new
>> pruning/recycling algorithm that will be *much* simpler to understand
>> and easier to calculate  how long the data will be in the Catalog and on
>> the Volume.
>>
>> On 06/18/2017 12:13 AM, Ivan Adzhubey wrote:
>>> Hi,
>>>
>>> In section "24.2 Pruning Directives", Bacula documentation claims that the
>>> Volume Retention period begins only *after* the volume has already
>>> switched to a status other than Append:
>>>
>>> ---quote---
>>> A key point is that this time period [Volume Retention] is not even
>>> considered as long at the Volume remains appendable. The Volume Retention
>>> period count down begins only when the Append status has been changed to
>>> some othe status (Full, Used, Purged, ...).
>>> ---quote ends---
>>>
>>> I believe this is not true. In the rest of documentation, the three
>>> independent retention periods (File, Job, and Volume) described as all
>>> starting at the same point: specifically, at the end run time of a job.
>>>
>>> Thanks,
>>> Ivan
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Reply | Threaded
Open this post in threaded view
|

Re: Volume retention starting time (documentation error?)

Ivan Adzhubey
Hi Kern,

Well, I am not a native English speaker so I might be wrong interpreting the
text as I do. This is one reason why I am seeking help posting on the list. I
think this configuration setting is important enough to make sure the
documentation is as clear as possible about it.

--Ivan

On Sunday, June 18, 2017 4:10:51 PM EDT Kern Sibbald wrote:

> Hello Ivan,
>
> You are free to view the text any way you want, but since I wrote it,
> what I see is:
>
> The problem is that the text does not say when the retention time
> begins.  It only says when the retention period can begin to be applied
> by Bacula.
>
> Kern
>
> On 06/18/2017 09:19 PM, Ivan Adzhubey wrote:
> > Hi Kern,
> >
> > I was referring only to the specific claim that the retention window for
> > volume is counted from the moment of Append->Full/Purged/Used transition
> > of
> > the volume's status. The phrase in section 24.2 which says this is very
> > unequivocal and is also totally wrong. The rest of the pruning process is
> > described correctly, as far as I was able to comprehend it.
> >
> > --Ivan
> >
> > On Sunday, June 18, 2017 2:53:56 AM EDT Kern Sibbald wrote:
> >> Hello,
> >>
> >> I haven't taken a look at the code, so what I say is subject to
> >> confirmation.
> >>
> >> I believe what is written is basically correct but with a nuance that is
> >> not clearly explained.  What is correct is that the Volume cannot be
> >> recycled until it is in a state other than Append, exactly as described
> >> below.  Consequently, the data will remain on the Volume even if there
> >> are no records in the Catalog that reference it, until the Volume has
> >> been recycled. However, once its records are subject to pruning, they
> >> will be pruned if the retention period measured from the last write time
> >> has expired.
> >>
> >> Each pruning (File, Job, Volume) is independent, and in addition the
> >> most important one in terms of "data loss" is Volume Retention, so
> >> drawing conclusions for Volume pruning based on pruning for File or Job
> >> records is not likely to work.
> >>
> >> I will look at the code to ensure that what I am saying is correct then
> >> attempt to improve the documentation ...
> >>
> >> Best regards,
> >> Kern
> >>
> >> PS: Note, the algorithm is designed to preserve the data as long as is
> >> possible.  However, there are a few quirks of the current implementation:
> >>
> >> 1. Assuming you have a Job that writes on a Volume.  That Volume will
> >> not automatically be pruned until the Job is run again.
> >>
> >> 2. If a Volume is only partially full so it remains in Append state.  As
> >> long as no Jobs write to the Volume to fill it, it will not be
> >> pruned/purged.
> >>
> >> One of my projects over the next year or two is to implement a new
> >> pruning/recycling algorithm that will be *much* simpler to understand
> >> and easier to calculate  how long the data will be in the Catalog and on
> >> the Volume.
> >>
> >> On 06/18/2017 12:13 AM, Ivan Adzhubey wrote:
> >>> Hi,
> >>>
> >>> In section "24.2 Pruning Directives", Bacula documentation claims that
> >>> the
> >>> Volume Retention period begins only *after* the volume has already
> >>> switched to a status other than Append:
> >>>
> >>> ---quote---
> >>> A key point is that this time period [Volume Retention] is not even
> >>> considered as long at the Volume remains appendable. The Volume
> >>> Retention
> >>> period count down begins only when the Append status has been changed to
> >>> some othe status (Full, Used, Purged, ...).
> >>> ---quote ends---
> >>>
> >>> I believe this is not true. In the rest of documentation, the three
> >>> independent retention periods (File, Job, and Volume) described as all
> >>> starting at the same point: specifically, at the end run time of a job.
> >>>
> >>> Thanks,
> >>> Ivan




The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/bacula-users