OpenLDAP: contextCSN check across mirrors

[r***@bt.com]

Hello,



I have configured three mirrors and am using a python script
(ldapSynchCheck.py) which check the sync status on retrieving the
contextCSN from both Provider and consumer



I tried deleting a tree from directory and checked what is the output of
the script and can still see "All of them are in sync". I believe this
is happening because contextCSN is not getting updated on the respective
directories and hence it will showing all in sync.



However sometimes I am getting below error and it seems one of the
consumer changed its contextCSN while provider was synching the data and
it also confirms that provider will not change the contextCSN until it
completes the sync to the consumer.



2010-05-06 15:15:51,459 - ldapSynchCheck.py - INFO - Provider is:
tardis03.nat.bt.com:389

2010-05-06 15:15:51,462 - ldapSynchCheck.py - DEBUG - Connecting to
ldap://tardis03.nat.bt.com:389

2010-05-06 15:15:51,464 - ldapSynchCheck.py - DEBUG - LDAP protocol
version 3

2010-05-06 15:15:51,465 - ldapSynchCheck.py - DEBUG - Binding with o=BT

2010-05-06 15:15:51,469 - ldapSynchCheck.py - INFO - Checking if
consumer tardis02.nat.bt.com:9011 is in SYNCH with provider

2010-05-06 15:15:51,470 - ldapSynchCheck.py - DEBUG - Connecting to
ldap://tardis02.nat.bt.com:9011

2010-05-06 15:15:51,471 - ldapSynchCheck.py - DEBUG - LDAP protocol
version 3

2010-05-06 15:15:51,472 - ldapSynchCheck.py - DEBUG - Binding with o=BT

2010-05-06 15:15:51,475 - ldapSynchCheck.py - DEBUG - Retrieving
Provider contextCSN

2010-05-06 15:15:51,482 - ldapSynchCheck.py - DEBUG - contextCSN =
20100506141551.044268Z#000000#003#000000

2010-05-06 15:15:51,483 - ldapSynchCheck.py - DEBUG - Retrieving
Consumer contextCSN

2010-05-06 15:15:51,513 - ldapSynchCheck.py - DEBUG - contextCSN =
20100506141551.044268Z#000000#003#000000

2010-05-06 15:15:51,515 - ldapSynchCheck.py - INFO - Provider and
consumer exactly in SYNCH

2010-05-06 15:15:51,516 - ldapSynchCheck.py - INFO - Checking if
consumer tardis01.nat.bt.com:9022 is in SYNCH with provider

2010-05-06 15:15:51,517 - ldapSynchCheck.py - DEBUG - Connecting to
ldap://tardis01.nat.bt.com:9022

2010-05-06 15:15:51,519 - ldapSynchCheck.py - DEBUG - LDAP protocol
version 3

2010-05-06 15:15:51,520 - ldapSynchCheck.py - DEBUG - Binding with o=BT

2010-05-06 15:15:51,523 - ldapSynchCheck.py - DEBUG - Retrieving
Provider contextCSN

2010-05-06 15:15:51,525 - ldapSynchCheck.py - DEBUG - contextCSN =
20100506141551.044268Z#000000#003#000000

2010-05-06 15:15:51,526 - ldapSynchCheck.py - DEBUG - Retrieving
Consumer contextCSN

2010-05-06 15:15:51,573 - ldapSynchCheck.py - DEBUG - contextCSN =
20100506141549.755004Z#000000#003#000000 <here it goes different>

Traceback (most recent call last):

File "./ldapSynchCheck.py", line 258, in <module>

main()

File "./ldapSynchCheck.py", line 253, in main

is_insynch(ldapprov, ldapcons, options.basedn, options.threshold,
logger)

File "./ldapSynchCheck.py", line 188, in is_insynch

delta = contextCSN_to_datetime(provcontextCSN) -
contextCSN_to_datetime(conscontextCSN)

File "./ldapSynchCheck.py", line 155, in contextCSN_to_datetime

return
datetime.datetime.fromtimestamp(time.mktime(time.strptime(gentime,"%Y%m%
d%H%M%S")))

File "/software/python/lib/python2.6/_strptime.py", line 454, in
_strptime_time

return _strptime(data_string, format)[0]

File "/software/python/lib/python2.6/_strptime.py", line 328, in
_strptime

data_string[found.end():])

ValueError: unconverted data remains: .044268



As I know putting a frequent checkpoint will generate a large amount of
checkpoint log file and will cause issues while database recovery during
crash/corrupt situations.



Can someone please suggest how make this more precise? Is there any
other way contextCSN keeps on changing frequently?



Please suggest.





Many Thanks in advance!!



Rahul Manchanda



GS Selfcare

Platform Build Team



Tel: [phone] Extn: 6178

SMTP: ***@bt.com



Office: Sharda Center ,6th Floor Annex

Off Karve Road, Erandwana ,Pune-04

[Quanah Gibson-Mount]

Post by r***@bt.com
However sometimes I am getting below error and it seems one of the
consumer changed its contextCSN while provider was synching the data and
it also confirms that provider will not change the contextCSN until it
completes the sync to the consumer.

What OpenLDAP release?

--Quanah


--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration

[r***@bt.com]

Hi,

It is 2.4.19.

Rahul Manchanda

GS Selfcare
Platform Build Team

Tel: [phone] Extn: 6178
SMTP: ***@bt.com

Office: Sharda Center ,6th Floor Annex
Off Karve Road, Erandwana ,Pune-04

-----Original Message-----
From: Quanah Gibson-Mount [mailto:***@zimbra.com]
Sent: Thursday, May 06, 2010 6:03 PM
To: Manchanda,RK,Rahul,DKE C; openldap-***@openldap.org
Subject: Re: OpenLDAP: contextCSN check across mirrors

Post by r***@bt.com
However sometimes I am getting below error and it seems one of the
consumer changed its contextCSN while provider was synching the data and
it also confirms that provider will not change the contextCSN until it
completes the sync to the consumer.

What OpenLDAP release?

--Quanah


--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration

[Quanah Gibson-Mount]

Post by r***@bt.com
Hi,
It is 2.4.19.

Please update your servers to 2.4.22 and see if it still occurs. Numerous
fixes have occurred since the 2.4.19 release. I've noted some specific
ones below that are likely of interest.

OpenLDAP 2.4.20 Release (2009/11/27)
Added slapd syncrepl contextCSN storing in subentry (ITS#6373)
Fixed slapd syncrepl deletes in MirrorMode (ITS#6368)
Fixed slapd syncrepl to use correct SID (ITS#6367)
Fixed slapo-syncprov checkpoint conversion (ITS#6370)
Fixed slapo-syncprov deadlock (ITS#6335)
Fixed slapo-syncprov memory leak (ITS#6376)
Fixed slapo-syncprov out of order changes (ITS#6346)
Fixed slapo-syncprov psearch with stale cookie (ITS#6397)

OpenLDAP 2.4.21 Release (2009/12/20)
Fixed slapd syncrepl freeing tasks from queue (ITS#6413)
Fixed slapd syncrepl parsing of tls defaults (ITS#6419)
Fixed slapd syncrepl uninitialized variables (ITS#6425)

OpenLDAP 2.4.22 Release (2010/04/24)
Fixed slapd-bdb contextCSN updates from updatedn (ITS#6469)

--Quanah

--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration

[r***@bt.com]

Thanks for the reply.

"Added slapd syncrepl contextCSN storing in subentry (ITS#6373)". In
this point what I can understand is that contextCSN will also get stored
in suffix entries as well as to sub entries as well. Please correct if
wrong.

I have some doubts if you can please help in making it more clear:

1) Can the contextCSN entry differ at the sub entry level if compared
with suffix(root) entry level?
2) If this is the case then a check will be required to each and every
entry if the sync is there at the dn level across mirrors and scripts
need to be modified so as to put a loop till all the entries has been
checked in that particular LDAP tree. Please correct if wrong.
3) how will the contextCSN entries will get changed. Does checkpoint
plays a role in changing the contextCSN value or not?
4) provider will not change the contextCSN until the whole data get
synched but will consumer will still be able to do so. Is this still the
scenario in the new version?

Many Thanks in advance!!

Rahul Manchanda

GS Selfcare
Platform Build Team

Tel: [phone] Extn: 6178
SMTP: ***@bt.com

Office: Sharda Center ,6th Floor Annex
Off Karve Road, Erandwana ,Pune-04


-----Original Message-----
From: Quanah Gibson-Mount [mailto:***@zimbra.com]
Sent: Friday, May 07, 2010 4:48 PM
To: Manchanda,RK,Rahul,DKE C; openldap-***@openldap.org
Subject: RE: OpenLDAP: contextCSN check across mirrors

Post by r***@bt.com
Hi,
It is 2.4.19.

Please update your servers to 2.4.22 and see if it still occurs.
Numerous
fixes have occurred since the 2.4.19 release. I've noted some specific
ones below that are likely of interest.

OpenLDAP 2.4.20 Release (2009/11/27)
Added slapd syncrepl contextCSN storing in subentry (ITS#6373)
Fixed slapd syncrepl deletes in MirrorMode (ITS#6368)
Fixed slapd syncrepl to use correct SID (ITS#6367)
Fixed slapo-syncprov checkpoint conversion (ITS#6370)
Fixed slapo-syncprov deadlock (ITS#6335)
Fixed slapo-syncprov memory leak (ITS#6376)
Fixed slapo-syncprov out of order changes (ITS#6346)
Fixed slapo-syncprov psearch with stale cookie (ITS#6397)

OpenLDAP 2.4.21 Release (2009/12/20)
Fixed slapd syncrepl freeing tasks from queue (ITS#6413)
Fixed slapd syncrepl parsing of tls defaults (ITS#6419)
Fixed slapd syncrepl uninitialized variables (ITS#6425)

OpenLDAP 2.4.22 Release (2010/04/24)
Fixed slapd-bdb contextCSN updates from updatedn (ITS#6469)

--Quanah

--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration

[Howard Chu]

Post by r***@bt.com
Thanks for the reply.
"Added slapd syncrepl contextCSN storing in subentry (ITS#6373)". In
this point what I can understand is that contextCSN will also get stored
in suffix entries as well as to sub entries as well. Please correct if
wrong.

That's wrong. Read the manpage and/or the ITS for the actual behavior.

Post by r***@bt.com
1) Can the contextCSN entry differ at the sub entry level if compared
with suffix(root) entry level?

It will only be in one place or the other, not both.

Post by r***@bt.com
2) If this is the case then a check will be required to each and every
entry if the sync is there at the dn level across mirrors and scripts
need to be modified so as to put a loop till all the entries has been
checked in that particular LDAP tree. Please correct if wrong.

No.

Post by r***@bt.com
3) how will the contextCSN entries will get changed. Does checkpoint
plays a role in changing the contextCSN value or not?

No.

Post by r***@bt.com
4) provider will not change the contextCSN until the whole data get
synched

False.

but will consumer will still be able to do so. Is this still the

Post by r***@bt.com
scenario in the new version?

I don't understand the question.

Post by r***@bt.com
Many Thanks in advance!!
Rahul Manchanda
GS Selfcare
Platform Build Team
Tel: [phone] Extn: 6178
Office: Sharda Center ,6th Floor Annex
Off Karve Road, Erandwana ,Pune-04
-----Original Message-----
Sent: Friday, May 07, 2010 4:48 PM
Subject: RE: OpenLDAP: contextCSN check across mirrors

Post by r***@bt.com
Hi,
It is 2.4.19.

Please update your servers to 2.4.22 and see if it still occurs.
Numerous
fixes have occurred since the 2.4.19 release. I've noted some specific
ones below that are likely of interest.
OpenLDAP 2.4.20 Release (2009/11/27)
Added slapd syncrepl contextCSN storing in subentry (ITS#6373)
Fixed slapd syncrepl deletes in MirrorMode (ITS#6368)
Fixed slapd syncrepl to use correct SID (ITS#6367)
Fixed slapo-syncprov checkpoint conversion (ITS#6370)
Fixed slapo-syncprov deadlock (ITS#6335)
Fixed slapo-syncprov memory leak (ITS#6376)
Fixed slapo-syncprov out of order changes (ITS#6346)
Fixed slapo-syncprov psearch with stale cookie (ITS#6397)
OpenLDAP 2.4.21 Release (2009/12/20)
Fixed slapd syncrepl freeing tasks from queue (ITS#6413)
Fixed slapd syncrepl parsing of tls defaults (ITS#6419)
Fixed slapd syncrepl uninitialized variables (ITS#6425)
OpenLDAP 2.4.22 Release (2010/04/24)
Fixed slapd-bdb contextCSN updates from updatedn (ITS#6469)
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration

[Michael Ströder]

Post by r***@bt.com
[..]
2010-05-06 15:15:51,573 - ldapSynchCheck.py - DEBUG - contextCSN =
*20100506141549.755004Z*#000000#003#000000 <*here it goes different*>
*Traceback (most recent call last): *
[..]
datetime.datetime.fromtimestamp(time.mktime(time.strptime(gentime,"%Y%m%d%H%M%S")))
*
* File "/software/python/lib/python2.6/_strptime.py", line 454, in
_strptime_time *
* return _strptime(data_string, format)[0] *
* File "/software/python/lib/python2.6/_strptime.py", line 328, in
_strptime *
* data_string[found.end():]) *
*ValueError: unconverted data remains: .044268 *

I think is rather a question about whether Python's standard lib function
time.strptime() can parse the input data with the format specifier "%Y%m%d%H%M%".

So I'd recommend to extract the input data in variable gendata and ask with
such a test case in news:comp.lang.python

Ciao, Michael.