LDAP exporter for HP OpenView ServiceDesk 4.5 & 5.x
version 2.00 [8th of October, 2008]
created by Radovan Skolnik, radovan@skolnik.info
Overview
This is a readme and a how-to file for LDAP export utility for HP OpenView ServiceDesk 4.5 & 5.x It produces XML file in the same format as the standard data exchange does (they call it CIM for some reason).
However the possibilities in standard data exchange (regarding LDAP) are quite limited. That's why I have created this little tool.
This tool is free. If it helps you achieve what you need any donations are highly welcome (click on the yellow button above).
Main features
- Highly configurable (through XML configuration file)
- Possibility to traverse LDAP hierarchies
- Possibility of evaluating expression on LDAP attributes
- Possibility to define many classes in one run
- Possibility to save parent/child relations
- Possibility to filter out (exclude) some records based on expressions not possible in LDAP queries
Requirements
I have created this software using Java Software Development Kit (JDK) version 1.5. However it should run flawlessly using 1.7, 1.6, 1.4, 1.3 and probably 1.2 series.
Java Runtime Environment (JRE) is sufficient - you do not need full Java SDK. It won't be able to run on Microsoft JVM. Please send me comments/issues regarding compatibility if you encounter any.
Download
You can download the whole archive with sample configuration file here
Usage
The usage is realy simple and same for all platforms:
java -jar SDLDAPExport.jar
In case you're traversing large directories you may also want to increase the memory available to Java runtime so that you do not run out of memory. This can be done like this (refer to Java docs about the switches):
java -Xms64m -Xmx1024m -jar SDLDAPExport.jar
I have added example.xml with some comments that shows most (or all) features available.
Description of configuration file
IMPORT_CONFIG
The whole configuration is enclosed in this tag that defines global parameters.
<IMPORT_CONFIG host="localhost" port="389" loginDN="" password="" outputFile="ldap.xml" logFile="ldap.log" timestampLDAPFormat="yyyyMMddHHmmss'Z'" timestampOutputFormat="dd/MM/yyyy HH:mm:ss" operationalAttributes="+" followReferrals="true" pageSize="0" useSSLTLS="false">
- host [required]
LDAP server host
- port [optional, 389 is used as default]
LDAP server port
- loginDN [optional, anonymous bind as default]
loginDN to use
- password [optional, anonymous bind as default]
password to use
- logFile [optional, stdout as default]
file to write logs
- outputFile [required]
target XML CIM file
- timestampLDAPFormat [optional, default yyyyMMddHHmmss'Z']
defines the format in which LDAP returns timestamps. For available options see Java docs for java.text.SimpleDateFormat. Some LDAP servers return the timestamp in like this "20050116213611.0Z" (as opposite to what most people and I have: "20050116213611Z"). In such case you should use "yyyyMMddHHmmss'.0Z'" for this parameter to make exporter recognize the timestamp.
- timestampOutputFormat [optional, default dd/MM/yyyy HH:mm:ss]
defines the format for outputting timestamp values. For available options see Java docs for java.text.SimpleDateFormat. Set this according to what you set as the default for 'Datetime format' for the account you use for importing.
- operationalAttributes [optional, see below for default list]
defines your own set of operational attributes to be fetched (for example to let you retrieve canonicalName attribute in M$ AD). It is stated in RFC 3673 that by specifying "+" you will get all operational attributes. So you may try this as well instead of specifying full list. If specifying list, comma (",") is used as separator.
- pageSize [optional, 0 (turned off) as default]
allows to retrieve results of search in pages of specified size. This is defined in RFC 2696. This option affects all searches!
- serverTimeLimit [optional, 30s as default]
allows to set server time limit for processing query.
- followReferrals [optional, default false]
allows to follow LDAP referrals (i.e. to transparently switch to another LDAP server where the rest of the data is stored).
- useSSLTLS [optional, default false]
allows to use TLS/SSL when connecting to LDAP server (some require this). LDAP server's certificate is not required nor checked!
- tag's value [optional, empty string as default]
Allows you to define 'global' variables, constants, functions - they will available to all <ATTRIBUTE> tags in all <ENTRY_CLASS> See example for timestamps processing.
ENTRY_CLASS
The <IMPORT_CONFIG> tag can contain one or more <ENTRY_CLASS> tags. This tag contains a definition of a CIM class.
For every LDAP entry retrieved one CIM class instance will be created.
<ENTRY_CLASS name="SD_ORGANIZATIONAL_UNIT" searchFilter="(objectclass=organizationalUnit)" searchScope="recursive" includeOperationalAttributes="true" searchBase="cn=intranetusers,dc=skolnik,dc=info" skipIf="" debug="true" maxEntries="0">
- name [required]
desired name of exported class
- searchFilter [optional, empty string as default]
filter to limit LDAP search - for example (objectClass=organizationalPerson). This is described in RFC 2254 Generally if you want to filter on 2 attributes via logical and operator the syntax would be like this: (&(objectclass=organizationalPerson)(ou=Hot Line Group)) Note the '&' that stands for single & (because of XML escaping)
- searchScope [opltional, default one]
scope of the search. One of the values {one, sub, base, recursive}
- includeOperationalAttributes [optional, default false]
specifies whether to include LDAP operational attributes - see below for list
- searchBase [required when inside IMPORT_CONFIG, otherwise optional]
LDAP search base (to start searching from)
- debug [optional, default false]
Dumps LDAP searches as well as contents of the namespace into log (for troubleshooting)
- maxEntries [optional, default 0]
Limits the number of returned entries for search operations - useful for testing & tweaking. Set to 0 to return all entries available.
- skipIf [optional, empty string as default]
Allows to provide additional condition. If evaluated to true, the record is skipped (not written to resulting XML). Condition is Java expression which should evaluate to boolean or Boolean that is constructed using namespace variables (read further for more explanation).
- tag's value [optional, empty string as default]
Allows you to define 'global' variables, constants, functions - they will available to all <ATTRIBUTE> tags in this <ENTRY_CLASS>
When exporter is run, it will connect to the server and find all LDAP entries that fulfill the searchScope/searchBase/filter criteria.
Every record will be written as a 'name' class into output file (list of attributes is defined in <ATTRIBUTES> - see below).
Here's a snippet from resulting XML showing tags resulting from <ENTRY_CLASS>:
<VALUE.OBJECT>
<INSTANCE CLASSNAME="SD_ORGANIZATIONAL_UNIT">
...
</INSTANCE>
</VALUE.OBJECT>
{one, sub, base} are standard scopes for LDAP searches (see some LDAP manual or Data Exchange guide from SD for explanation).
'recursive' is one of the main reasons for implementation of this tool.
It will do standard 'one' search on defined searchBase but for every entry found it will recursively do search with same parameters - only searchBase will be replaced with DN of found entry.
This allows recursive traversal of LDAP hierarchy (see <ENTRY_CLASS name="SD_ORGANIZATIONAL_UNIT"> in example.xml)
<ENTRY_CLASS> can contain another <ENTRY_CLASS> tags (children).
This will do a search from parent DN as searchBase using criteria in child <ENTRY_CLASS> (in example.xml this is used for finding Persons inside OUs).
For every entry all its attributes are fetched from server. If includeOperationalAttributes=true it also gets (otherwise invisible) operational attributes:
- createTimestamp
- creatorsName
- entryFlags
- federationBoundary
- localEntryID
- modifiersName
- modifyTimestamp
- structuralObjectClass
- subordinateCount
- subschemaSubentry
You can override the list of operation attributes in <IMPORT_CONFIG> by specifying operationalAttributes.
The attributes received make up a 'variable namespace' for the current record - i.e. as if you declared variables with the names and values of LDAP attributes.
Here's a sample from output log showing the situation:
CLASS=SD_ORGANIZATIONAL_UNIT, ID=0
modifiersName : java.lang.String = cn=intranetusers,dc=skolnik,dc=info
creatorsName : java.lang.String = cn=intranetusers,dc=skolnik,dc=info
ou : java.lang.String = General management
objectClass : java.lang.String = organizationalUnit
dn : java.lang.String = ou=General Management, cn=intranetusers,dc=skolnik,dc=info
subschemaSubentry : java.lang.String = cn=Subschema
modifyTimestamp : java.lang.String = 03/12/2004 09:05:51
createTimestamp : java.lang.String = 03/12/2004 09:05:51
All the received variables are declared as Java String types. You can use these variables in <ATTRIBUTE> tags to create (Java) expressions that will be output to resulting CIM XML.
The namespace also contains a special variable named $parent that holds a reference to a namespace of parent record - i.e.
- in case of recursive search the record one level up in the hierarchy (from point of view of current one)
- in case of nested <ENTRY_CLASS> tags the record of enclosing tag (from point of view of current one)
- in case of <ATTRIBUTE_CLASS> inside <ENTRY_CLASS> the record of enclosing <ENTRY_CLASS> (from the point of view of <ATTRIBUTE_CLASS>)
ATTRIBUTES and ATTRIBUTE
Attributes written into output XML are defined within <ATTRIBUTES> tag.
There can only be one <ATTRIBUTES> tag per <ENTRY_CLASS>. Inside <ATTRIBUTES> you put a set of <ATTRIBUTE>.
Each will be evaluated and the result of evaluation put into current class as its attribute.
<ATTRIBUTES>
This is only a placeholder that holds actual <ATTRIBUTE> definitions and has no attributes.
There are two forms of how to define <ATTRIBUTE>. First one uses value attribute for definition of the expression:
<ATTRIBUTE name="searchcode" value="(sn + givenName.substring(0,1)).toUpperCase()" includeInOutput="true"/>
The second one uses <![CDATA[ and ]]> pair (no need to do XML escaping there) to enable you to write multi-line code instead of value attribute.
<ATTRIBUTE name="accountExpires" includeInOutput="true">
<![CDATA[
import java.text.SimpleDateFormat; // yes we can freely import Java packages from runtime
String getExpiryDate (long accountExpiresL) { // we write a function that will be used for evaluation, we do not have to care about exception handlers
Calendar calendar = Calendar.getInstance();
calendar.clear();
calendar.set(1601, 0, 1, 0, 0);
accountExpiresL = accountExpiresL/10000 + calendar.getTime().getTime();
Date result = (new Date(accountExpiresL));
SimpleDateFormat sdf = new SimpleDateFormat("dd/MM/yyyy HH:mm:ss"); // same as timestampOutputFormat
return sdf.format(result);
}
// result of this expression will be used (instead of "1234567890" you would use the AD variable holding the value (accountExpires probably))
expiryDate = getExpiryDate((new Long("1234567890")).longValue());
]]>
</ATTRIBUTE>
- name [required]
desired name of exported attribute
- value [required]
This is an expression (Java syntax - see BeanShell documentation for details) whose evaluation will be provided as result. Alternatively you can provide the expression as the value of the tag itself using <![CDATA[ and ]]>. Please, see example file....
- includeInOutput [optional, true as default]
Let's you specify whether this should be included in output XML CIM - useful for intermediate variables used further in other formulas or in skipIf condition.
Note that evaluation (variable names) is case sensitive. If you get some strange values where you do not expect them it can be caused by case-sensitiveness.
If you find that a resulting value contains 'VOID' string, it means that 'value' in <ATTRIBUTE> tag contains reference to non-existent variable (most probably LDAP attribute not present in current record).
In such cases use can set debug=true flag for problem class and check out a log file for exact attribute names and values.
Also if you have attribute that sometimes is there and sometimes not, you would get 'VOID' values on occasions where it's missing from current LDAP record.
You can use Java ternary expression in this form: (variable==void ? "" : variable) as a workaround.
If the variable didn't exist it would be replaced by empty string (a pair of apostrophes - but as we're in XML we have to escape them as shown) otherwise its original value will be used.
The individual <ATTRIBUTE> tags are evaluated in the order they are written in configuration XML. This allows to define some attribute in one <ATTRIBUTE> and use it in more than one following tags.
Sample of CIM XML PROPERTY resulting from <ATTRIBUTE>.
<VALUE.OBJECT>
<INSTANCE CLASSNAME="SD_ORGANIZATIONAL_UNIT">
<PROPERTY NAME="ID" TYPE="string">
<VALUE>1</VALUE>
</PROPERTY>
<PROPERTY NAME="dn" TYPE="string">
<VALUE>ou=Others, cn=intranetusers,dc=skolnik,dc=info</VALUE>
</PROPERTY>
<PROPERTY NAME="ou" TYPE="string">
<VALUE>Others</VALUE>
</PROPERTY>
</INSTANCE>
</VALUE.OBJECT>
ATTRIBUTE_CLASS
<ENTRY_CLASS> can also contain <ATTRIBUTE_CLASS> tags.
The purpose of <ATTRIBUTE_CLASS> is to be able to create standalone class by going through array of values contained in attribute from its parent <ENTRY_CLASS>.
Some attributes behave like array - i.e. one attribute has an array of values.
For example objectclass contains many values, or attributes defining person's phone numbers or her/his belonging to some group are such examples.
The <ATTRIBUTE_CLASS> can only have one <ATTRIBUTES> tag inside that holds attributes to be retrieved.
<ATTRIBUTE_CLASS name="SD_PHONE" expandOn="mobile" skipIf="mobile.startsWith("0")" debug="true">
- name [required]
desired name of exported class
- expandOn [required]
name of the LDAP array attribute
- skipIf [optional, empty string as default]
Allows to provide additional condition. If evaluated to true, the record is skipped (not written to resulting XML). Condition is Java expression (should evaluate to boolean or Boolean) that can use both attributes that were received from LDAP server as well as calculated ones (they are added to namespace - read the the paragraph about <ATTRIBUTE> for more explanation).
- debug [optional, default false]
Dumps contents of namespace into log (for troubleshooting)
See also
I have created more tools for HP OpenView ServiceDesk 4.5 & 5.x that I would like you to have a look at. Here is a list:
- Single Sign On for ServiceDesk 4.5 - allows starting clients without providing username and password based on Windows NT Domain information or authenticating them against LDAP server. Also includes tools for enabling SSO for WEB-API applications as well as pre-configured Service Pages integrated with Windows NT Domain.
- Secure Attachments for ServiceDesk 4.5 and 5.x - allows you to store attachments on SSH server rather than FTP server.
- Enhanced Update All for ServiceDesk 4.5 - allows you to use functions and values of other fields in Update All in the same way you can use them in Database Rules.
Thanks
- I would like to thank Sven Storck of Heidelberg.com and Peter Gotthardt of HP for kindly sponsoring the development of paged searches by getting me some books of my interest from Amazon. Thank you very much.
- I want to thank Alexey Brodin, Andreas Schnur and Reinhard for testing, comments and enhancement proposals.
- Thanks to Maxim Baturlov (and some other people) for bugging me enough till I made TLS/SSL support working.
- Special greetings to Igor Barinov ;-)
History
- 0.90 [05/08/2004]
- 0.91 [05/08/2004]
small bugfix - debug flag for class was in effect for all classes (globally)
updated to latest CVS version of jldap
- 0.95 [27/09/2004]
updated documentation - searchFilter syntax RFC
added maxEntries option for <ENTRY_CLASS>
bugfix - now it works with Java version lower than 1.4 (had to switch to SAX 1.0 to be able to use JCLark parser - it previously used the one with JRE)
- 0.96 [06/12/2004]
added support for array values (expandOn in <ATTRIBUTE_CLASS>)
- 0.97 [11/01/2005]
updated documentation - avoiding VOIDs in <EXPRESSION> attributes
bugfix - strings with national characters (umlauts) were wrongly Base64 encoded so you got some mess instead (you can explicitly use com.novell.ldap.util.Base64.encode(value) to do this if you need it)
- 0.98 [15/02/2005]
added support for timestampLDAPFormat & timestampOutputFormat
bugfix - fixed bug with java.lang.NullPointerException when exporter could not parse the timestamp coming from LDAP
- 0.99 [13/04/2005]
added support for specifying your own set of operational attributes in <IMPORT_CONFIG>
- 1.00 [30/11/2006]
added support for Active Directory objectGUID formatting
great simplification (removal of DIRECT, SPECIAL, EXPRESSION tags and their replacement by ATTRIBUTE)
added support for following referrals via followReferrals attribute in <IMPORT_CONFIG>
ability to address variables in parent's namespace by using $parent.variable (can be used to address parent's attributes other than DN only as in previous versions)
- 1.10 [29/03/2007]
added paging support (as defined in RFC 2696) via pageSize attribute in <IMPORT_CONFIG>
- 1.20 [11/11/2007]
added Locator information from SAX parser - i.e. on errors and warnings it provides line and character number of a tag containing the problem
added possibility in <ATTRIBUTE> to provide the expression's value as the tag's value instead of value attribute - allows new lines (and other special characters without XML escaping using <![CDATA[ & ]]> pair)
added serverTimeLimit attribute for <ENTRY_CLASS>
- 1.30 [12/12/2007]
attempt to add support for TLS/SSL - this wasn't released publibly (it didn't work anyway)
- 1.40 [22/08/2008]
added working support for TLS/SSL (thank goes to Postgress JDBC driver for idea)
added support for skipIf attribute in <ENTRY_CLASS> and <ATTRIBUTE_CLASS> - allows further filtering of received results
added support for global functions/variables/constants in <IMPORT_CONFIG>
added support for includeInOutput attribute in <ATTRIBUTE>
removed timestampLDAPFormat and timestampOutputFormat as <IMPORT_CONFIG> attributes - they are now part of the global constants
rewrote this readme in pure HTML instead of crap M$ Word produced initially (I got sick of editing it), updated and enhanced some parts of it