slapcat generate extra "space" characters in LDIF output

[Frank Bonnet]

Hello

I'm in trouble with slapcat when generating a LDIF file
it puts some extra "space" characters into some dn longer than 80
characters.

is there a way to change the output format of slapcat command
to generate lines longer than 80 characters in the LDIF file ?

I need this because I need to duplicate our directory server
to another TLD and I need to substitute

dc=fr by dc=biz

but sometime I get this

------------------------------------------------dc=fr

sometime I get this

------------------------------------------------dc=
fr

sometime I get this

------------------------------------------------dc
=fr

sometime I get this

------------------------------------------------dc=
fr

Thank you

[Dieter Kluenter]

Post by Frank Bonnet
Hello
I'm in trouble with slapcat when generating a LDIF file
it puts some extra "space" characters into some dn longer than 80
characters.
is there a way to change the output format of slapcat command
to generate lines longer than 80 characters in the LDIF file ?
I need this because I need to duplicate our directory server
to another TLD and I need to substitute
dc=fr by dc=biz
but sometime I get this
------------------------------------------------dc=fr
sometime I get this
------------------------------------------------dc=
fr

This is intended behaviour, because the LDIF specifications (RFC2849)
require a max. line length of 76 characters, but allow line folding,
that is continuation of a line longer than 76 chars.

-Dieter

[Frank Bonnet]

Post by Dieter Kluenter

Post by Frank Bonnet
Hello
I'm in trouble with slapcat when generating a LDIF file
it puts some extra "space" characters into some dn longer than 80
characters.
is there a way to change the output format of slapcat command
to generate lines longer than 80 characters in the LDIF file ?
I need this because I need to duplicate our directory server
to another TLD and I need to substitute
dc=fr by dc=biz
but sometime I get this
------------------------------------------------dc=fr
sometime I get this
------------------------------------------------dc=
fr

This is intended behaviour, because the LDIF specifications (RFC2849)
require a max. line length of 76 characters, but allow line folding,
that is continuation of a line longer than 76 chars.
-Dieter

OK thank you dieter , but this is boring for my case ...

[Emmanuel Lecharny]

Post by Dieter Kluenter

Post by Frank Bonnet
Hello
I'm in trouble with slapcat when generating a LDIF file
it puts some extra "space" characters into some dn longer than 80
characters.
is there a way to change the output format of slapcat command
to generate lines longer than 80 characters in the LDIF file ?
I need this because I need to duplicate our directory server
to another TLD and I need to substitute
dc=fr by dc=biz
but sometime I get this
------------------------------------------------dc=fr
sometime I get this
------------------------------------------------dc=
fr

This is intended behaviour, because the LDIF specifications (RFC2849)
require a max. line length of 76 characters,

Nope :

...

10) When an attrval-spec, distinguishedName, or rdn is base64-
encoded, the encoding rules specified in [5] are used with the
following exceptions: a) ***The requirement that base64 output
streams must be represented as lines of no more than 76
characters is removed.***

[Mark J. Reed]

     10) When an attrval-spec, distinguishedName, or rdn is base64-
         encoded, the encoding rules specified in [5] are used with the
         following exceptions:  a) ***The requirement that base64 output
         streams must be represented as lines of no more than 76
         characters is removed.***

Point. Based on a quick reread of RFC 2849, LDIF doesn't mandate line
wrapping. But it does explicitly allow it, so tools that process LDIF
need to be able to deal with wrapped lines.

And it occurs to me that the unfold script can also be done a perl one-liner:

perl -pe 'BEGIN { undef $/; } s/\n //gms'

[m***@aero.polimi.it]

I'd note that while the current maximum width of lines is enforced to 76
chars, they happen to be 78 char long (because of an extra LDIF_KLUDGE set
to 1 and, I guess, of the leading blank).

In any case, in the spirit of being liberal when needed, I have nothing
against allowing OpenLDAP tools to visualize LDIF with arbitrary width, as
soon as the default behavior remains the original one, in order to avoid
breaking existing software.

p.

[m***@aero.polimi.it]

Post by m***@aero.polimi.it
I'd note that while the current maximum width of lines is enforced to 76
chars, they happen to be 78 char long (because of an extra LDIF_KLUDGE set
to 1 and, I guess, of the leading blank).
In any case, in the spirit of being liberal when needed, I have nothing
against allowing OpenLDAP tools to visualize LDIF with arbitrary width, as
soon as the default behavior remains the original one, in order to avoid
breaking existing software.

See <http://www.openldap.org/its?findid=6645>.

p.

[Mark J. Reed]

FWIW, I use this script to undo the line-folding of LDIF. Just pipe
the output of slapcat or ldapsearch or whatever through it and you'll
get something that's no longer RFC-compliant LDIF but is more amenable
to processing with text-based tools.

#!/usr/bin/perl
# Unfold LDIF so that each attribute is on a single line
my $acc;
while (<>) {
chomp;
if (s/^ //) {
$acc .= $_;
} elsif (/^\S/) {
print "$acc\n" if $acc;
$acc = $_;
} elsif (/^$/) {
print "$acc\n" if $acc;
print "$_\n";
$acc = undef;
}
}
print "$acc\n" if $acc;

Refolding is a one-liner, something like this:

perl -pe 's/^(.{76})(..*)$/$1\n $2/'

[Hallvard B Furuseth]

Post by Mark J. Reed
FWIW, I use this script to undo the line-folding of LDIF. Just pipe
the output of slapcat or ldapsearch or whatever through it and you'll
get something that's no longer RFC-compliant LDIF but is more amenable
to processing with text-based tools.
(..snip..)

This one-liner is sufficient to undo the line folding:

perl -p00e 's/\r?\n //g'

For people _reading_ an LDIF, remember you may need further decoding:
The LDIF may contain "attrtype:: base64-encoded value" or
"attrtype:< URL of value".

Post by Mark J. Reed
perl -pe 's/^(.{76})(..*)$/$1\n $2/'

The LDIF RFC doesn't require that, but it doens't hurt either.

However your snippet fails if you need to fold into more than 2 lines.
Also you should avoid folding in the middle of an UTF-8 sequence.

#!/usr/bin/perl -wp
# Fold LDIF lines > 76 chars (Note, not an LDIF requirement)

# Match {71 to 76} chars 1st time, then {70 to 75} since we prepended a space
# Accept either CRLF or LF as line endings. But always folds with just LF.
# Avoid splitting in the middle of an UTF-8 char, i.e. before [\200-\277].
s/\G ((?: \A[^\r\n] | (?!\A)) [^\r\n]{70,75}) (?=[^\r\n\200-\277]) /$1\n /gx;