XML::XQL - A perl module for querying XML tree structures with XQL

XPath functions and methods

Implementation Details
SEE ALSO
AUTHOR

NAME

XML::XQL - A perl module for querying XML tree structures with XQL

SYNOPSIS

 use XML::XQL;
 use XML::XQL::DOM;

 $parser = new XML::DOM::Parser;
 $doc = $parser->parsefile ("file.xml");

 # Return all elements with tagName='title' under the root element 'book'
 $query = new XML::XQL::Query (Expr => "book/title");
 @result = $query->solve ($doc);
 $query->dispose; # Avoid memory leaks - Remove circular references

 # Or (to save some typing)
 @result = XML::XQL::solve ("book/title", $doc);

 # Or (to save even more typing)
 @result = $doc->xql ("book/title");

DESCRIPTION

The XML::XQL module implements the XQL (XML Query Language) proposal submitted to the XSL Working Group in September 1998. The spec can be found at: http://www.w3.org/TandS/QL/QL98/pp/xql.html Most of the contents related to the XQL syntax can also be found in the the XML::XQL::Tutorial manpage that comes with this distribution. Note that XQL is not the same as XML-QL!

The current implementation only works with the the XML::DOM manpage module, but once the design is stable and the major bugs are flushed out, other extensions might follow, e.g. for XML::Grove.

XQL was designed to be extensible and this implementation tries to stick to that. Users can add their own functions, methods, comparison operators and data types. Plugging in a new XML tree structure (like XML::Grove) should be a piece of cake.

To use the XQL module, either

  use XML::XQL;

  use XML::XQL::Strict;

The Strict module only provides the core XQL functionality as found in the XQL spec. By default (i.e. by using XML::XQL) you get 'XQL+', which has some additional features.

See the section Additional Features in XQL+ for the differences.

This module is still in development. See the To-do list in XQL.pm for what still needs to be done. Any suggestions are welcome, the sooner these implementation issues are resolved, the faster we can all use this module.

If you find a bug, you would do me great favor by sending it to me in the form of a test case. See the file t/xql_template.t that comes with this distribution.

If you have written a cool comparison operator, function, method or XQL data type that you would like to share, send it to tjmather@tjmather.com and I will add it to this module.

XML::XQL global functions

solve (QUERY_STRING, INPUT_LIST...)

 @result = XML::XQL::solve ("doc//book", $doc);

This is provided as a shortcut for:

 $query = new XML::XQL::Query (Expr => "doc//book");
 @result = $query->solve ($doc);
 $query->dispose;

Note that with the XML::XQL::DOM manpage, you can also write (see the XML::DOM::Node manpage for details):

 @result = $doc->xql ("doc//book");

setDocParser (PARSER)

Sets the XML::DOM::Parser that is used by the new XQL+ document() method. By default it uses an XML::DOM::Parser that was created without any arguments, i.e.

  $PARSER = new XML::DOM::Parser;

defineFunction (NAME, FUNCREF, ARGCOUNT [, ALLOWED_OUTSIDE [, CONST, [QUERY_ARG]]])

Defines the XQL function (at the global level, i.e. for all newly created queries) with the specified NAME. The ARGCOUNT parameter can either be a single number or a reference to a list with numbers. A single number expands to [ARGCOUNT, ARGCOUNT]. The list contains pairs of numbers, indicating the number of arguments that the function allows. The value -1 means infinity. E.g. [2, 5, 7, 9, 12, -1] means that the function can have 2, 3, 4, 5, 7, 8, 9, 12 or more arguments. The number of arguments is checked when parsing the XQL query string.

The second parameter must be a reference to a Perl function or an anonymous sub. E.g. '\&my_func' or 'sub { ... code ... }'

If ALLOWED_OUTSIDE (default is 0) is set to 1, the function or method may also be used outside subqueries in node queries. (See NodeQuery parameter in Query constructor)

If CONST (default is 0) is set to 1, the function is considered to be ``constant''. See Constant Function Invocations for details.

If QUERY_ARG (default is 0) is not -1, the argument with that index is considered to be a 'query parameter'. If the query parameter is a subquery, that returns multiple values, the result list of the function invocation will contain one result value for each value of the subquery. E.g. 'length(book/author)' will return a list of Numbers, denoting the string lengths of all the author elements returned by 'book/author'.

Note that only methods (not functions) may appear after a Bang ``!'' operator. This is checked when parsing the XQL query string.

Additional Features in XQL+

Parent operator '..'

The '..' operator returns the parent of the current node, where '.' would return the current node. This is not part of any XQL standard, because you would normally use return operators, which are not implemented here.

Sequence operators ';' and ';;'

The sequence operators ';' (precedes) and ';;' (immediately precedes) are not in the XQL spec, but are described in 'The Design of XQL' by Jonathan Robie who is one of the designers of XQL. It can be found at http://www.texcel.no/whitepapers/xql-design.html See also the XQL Tutorial for a description of what they mean.

q// and qq// String Tokens

String tokens a la q// and qq// are allowed. q// evaluates like Perl's single quotes and qq// like Perl's double quotes. Note that the default XQL strings do not allow escaping etc., so it's not possible to define a string with both single and double quotes. If 'q' and 'qq' are not to your liking, you may redefine them to something else or undefine them altogether, by assigning undef to them. E.g:

 # at a global level - shared by all queries (that don't (re)define 'q')
 XML::XQL::defineTokenQ ('k');
 XML::XQL::defineTokenQQ (undef);

 # at a query level - only defined for this query
 $query = new XML::XQL::Query (Expr => "book/title", q => 'k', qq => undef);


From now on k// works like q// did and qq// doesn't work at all anymore.

Query strings can have embedded Comments

For example:

 $queryExpr = "book/title          # this comment is inside the query string
               [. = 'Moby Dick']"; # this comment is outside

Optional dollar delimiters and case-insensitive XQL keywords

The following XQL keywords are case-insensitive and the dollar sign delimiters may be omitted: $and$, $or$, $not$, $union$, $intersect$, $to$, $any$, $all$, $eq$, $ne$, $lt$, $gt$, $ge$, $le$, $ieq$, $ine$, $ilt$, $igt$, $ige$, $ile$.

E.g. $AND$, $And$, $aNd$, and, And, aNd are all valid replacements for $and$.

Note that XQL+ comparison operators ($match$, $no_match$, $isa$, $can$) still require dollar delimiters and are case-sensitive.

Comparison operator: $match$ or '=~'

E.g. ``book/title =~ '/(Moby|Dick)/']'' will return all book titles containing Moby or Dick. Note that the match expression needs to be quoted and should contain the // or m// delimiters for Perl.

When casting the values to be matched, both are converted to Text.

Comparison operator: $no_match$ or '!~'

E.g. ``book/title !~ '/(Moby|Dick)/']'' will return all book titles that don't contain Moby or Dick. Note that the match expression needs to be quoted and should contain the // or m// delimiters for Perl.

When casting the values to be matched, both are converted to Text.

Comparison operator: $isa$

E.g. '//. $isa$ ``XML::XQL::Date''' returns all elements for which the value() function returns an XML::XQL::Date object. (Note that the value() function can be overridden to return a specific object type for certain elements and attributes.) It uses expandType to expand XQL primitive type names.

Comparison operator: $can$

E.g. '//. $can$ ``swim''' returns all elements for which the value() function returns an object that implements the (Perl) swim() method. (Note that the value() function can be overridden to return a specific object type for certain elements and attributes.)

Function: once (QUERY)

E.g. 'once(id(``foo''))' will evaluate the QUERY expression only once per query. Certain query results (like the above example) will always return the same value within a query. Using once() will cache the QUERY result for the rest of the query.

Note that ``constant'' function invocations are always cached. See also Constant Function Invocations

Function: subst (QUERY, EXPR, EXPR [,MODIFIERS, [MODE]])

E.g. 'subst(book/title, ``[M|m]oby'', ``Dick'', ``g'')' will replace Moby or moby with Dick globally (``g'') in all book title elements. Underneath it uses Perl's substitute operator s///. Don't worry about which delimiters are used underneath. The function returns all the book/titles for which a substitution occurred. The default MODIFIERS string is ``'' (empty.) The function name may be abbreviated to ``s''.

For most Node types, it converts the value() to a string (with xql_toString) to match the string and xql_setValue to set the new value in case it matched. For XQL primitives (Boolean, Number, Text) and other data types (e.g. Date) it uses xql_toString to match the String and xql_setValue to set the result. Beware that performing a substitution on a primitive that was found in the original XQL query expression, changes the value of that constant.

If MODE is 0 (default), it treats Element nodes differently by matching and replacing text blocks occurring in the Element node. A text block is defined as the concatenation of the raw text of subsequent Text, CDATASection and EntityReference nodes. In this mode it skips embedded Element nodes. If a text block matches, it is replaced by a single Text node, regardless of the original node type(s).

If MODE is 1, it treats Element nodes like the other nodes, i.e. it converts the value() to a string etc. Note that the default implementation of value() calls text(), which normalizes whitespace and includes embedded Element descendants (recursively.) This is probably not what you want to use in most cases, but since I'm not a professional psychic... :-)

Function: map (QUERY, CODE)

E.g. 'map(book/title, ``s/[M|m]oby/Dick/g; $_'')' will replace Moby or moby with Dick globally (``g'') in all book title elements. Underneath it uses Perl's map operator. The function returns all the book/titles for which a change occurred.

??? add more specifics

Function: eval (EXPR [,TYPE])

Evaluates the Perl expression EXPR and returns an object of the specified TYPE. It uses expandType to expand XQL primitive type names. If the result of the eval was undef, the empty list [] is returned.

E.g. 'eval(``2 + 5'', ``Number'')' returns a Number object with the value 7, and 'eval(``%ENV{USER}'')' returns a Text object with the user name.

Consider using once() to cache the return value, when the invocation will return the same result for each invocation within a query.

??? add more specifics

Function: new (TYPE [, QUERY [, PAR] *])

Creates a new object of the specified object TYPE. The constructor may have any number of arguments. The first argument of the constructor (the 2nd argument of the new() function) is considered to be a 'query parameter'. See defineFunction for a definition of query parameter. It uses expandType to expand XQL primitive type names.

Function: document (QUERY) or doc (QUERY)

The document() function creates a new the XML::XML::Document manpage for each result of QUERY (QUERY may be a simple string expression, like ``/usr/enno/file.xml''. See t/xql_document.t or below for an example with a more complex QUERY.)

document() may be abbreviated to doc().

document() uses an XML::DOM::Parser underneath, which can be set with XML::XQL::setDocParser(). By default it uses a parser that was created without any arguments, i.e.

  $PARSER = new XML::DOM::Parser;

Let's try a more complex example, assuming $doc contains:

 <doc>
  <file name="file1.xml"/>
  <file name="file2.xml"/>
 </doc>

Then the following query will return two the XML::XML::Document manpages, one for file1.xml and one for file2.xml:

 @result = XML::XQL::solve ("document(doc/file/@name)", $doc);

The resulting documents can be used as input for following queries, e.g.

 @result = XML::XQL::solve ("document(doc/file/@name)/root/bla", $doc);

will return all /root/bla elements from the documents returned by document().

Method: DOM_nodeType ()

Returns the DOM node type. Note that these are mostly the same as nodeType(), except for CDATASection and EntityReference nodes. DOM_nodeType() returns 4 and 5 respectively, whereas nodeType() returns 3, because they are considered text nodes.

Function wrappers for Perl builtin functions

XQL function wrappers have been provided for most Perl builtin functions. When using a Perl builtin function like ``substr'' in an XQL+ querry, an XQL function wrapper will be generated on the fly. The arguments to these functions may be regular XQL+ subqueries (that return one or more values) for a query parameter (see generateFunction for a definition.) Most wrappers of Perl builtin functions have argument 0 for a query parameter, except for: chmod (parameter 1 is the query parameter), chown (2) and utime (2). The following functions have no query parameter, which means that all parameters should be a single value: atan2, rand, srand, sprintf, rename, unlink, system.

The function result is casted to the appropriate XQL primitive type (Number, Text or Boolean), or to an empty list if the result was undef.

XPath functions and methods

The following functions were found in the XPath specification:

Function: concat (STRING, STRING, STRING*)

The concat function returns the concatenation of its arguments.

Function: starts-with (STRING, STRING)

The starts-with function returns true if the first argument string starts with the second argument string, and otherwise returns false.

Function: contains (STRING, STRING)

The contains function returns true if the first argument string contains the second argument string, and otherwise returns false.

Function: substring-before (STRING, STRING)

The substring-before function returns the substring of the first argument string that precedes the first occurrence of the second argument string in the first argument string, or the empty string if the first argument string does not contain the second argument string. For example,

 substring-before("1999/04/01","/") returns 1999.

Function: substring-after (STRING, STRING)

The substring-after function returns the substring of the first argument string that follows the first occurrence of the second argument string in the first argument string, or the empty string if the first argument string does not contain the second argument string. For example,

 substring-after("1999/04/01","/") returns 04/01,

and

 substring-after("1999/04/01","19") returns 99/04/01.

Function: substring (STRING, NUMBER [, NUMBER] )

The substring function returns the substring of the first argument starting at the position specified in the second argument with length specified in the third argument. For example,

 substring("12345",2,3) returns "234".

If the third argument is not specified, it returns the substring starting at the position specified in the second argument and continuing to the end of the string. For example,

 substring("12345",2) returns "2345".

More precisely, each character in the string is considered to have a numeric position: the position of the first character is 1, the position of the second character is 2 and so on.

NOTE: This differs from the substr method , in which the method treats the position of the first character as 0.

The XPath spec says this about rounding, but that is not true in this implementation: The returned substring contains those characters for which the position of the character is greater than or equal to the rounded value of the second argument and, if the third argument is specified, less than the sum of the rounded value of the second argument and the rounded value of the third argument; the comparisons and addition used for the above follow the standard IEEE 754 rules; rounding is done as if by a call to the round function.

Method: string-length ( [ QUERY ] )

The string-length returns the number of characters in the string. If the argument is omitted, it defaults to the context node converted to a string, in other words the string-value of the context node.

Note that the generated XQL wrapper for the Perl built-in substr does not allow the argument to be omitted.

Method: normalize-space ( [ QUERY ] )

The normalize-space function returns the argument string with whitespace normalized by stripping leading and trailing whitespace and replacing sequences of whitespace characters by a single space. Whitespace characters are the same as those allowed by the S production in XML. If the argument is omitted, it defaults to the context node converted to a string, in other words the string-value of the context node.

Function: translate (STRING, STRING, STRING)

The translate function returns the first argument string with occurrences of characters in the second argument string replaced by the character at the corresponding position in the third argument string. For example,

 translate("bar","abc","ABC") returns the string BAr.

If there is a character in the second argument string with no character at a corresponding position in the third argument string (because the second argument string is longer than the third argument string), then occurrences of that character in the first argument string are removed. For example,

 translate("--aaa--","abc-","ABC") returns "AAA".

If a character occurs more than once in the second argument string, then the first occurrence determines the replacement character. If the third argument string is longer than the second argument string, then excess characters are ignored.

NOTE: The translate function is not a sufficient solution for case conversion in all languages. A future version may provide additional functions for case conversion.

This function was implemented using tr///d.

Function: sum ( QUERY )

The sum function returns the sum of the QUERY results, by converting the string values of each result to a number.

Function: floor (NUMBER)

The floor function returns the largest (closest to positive infinity) number that is not greater than the argument and that is an integer.

Function: ceiling (NUMBER)

The ceiling function returns the smallest (closest to negative infinity) number that is not less than the argument and that is an integer.

Function: round (NUMBER)

The round function returns the number that is closest to the argument and that is an integer. If there are two such numbers, then the one that is closest to positive infinity is returned.

Implementation Details

XQL Builtin Data Types

The XQL engine uses the following object classes internally. Only Number, Boolean and Text are considered primitive XQL types:

XML::XQL::Number
For integers and floating point numbers.
XML::XQL::Boolean
For booleans, e.g returned by true() and false().
XML::XQL::Text
For string values.
XML::XQL::Date
For date, time and date/time values. E.g. returned by the date() function.
XML::XQL::Node
Superclass of all XML node types. E.g. all subclasses of XML::DOM::Node subclass from this.
Perl list reference
Lists of values are passed by reference (i.e. using [] delimiters). The empty list [] has a double meaning. It also means 'undef' in certain situations, e.g. when a function invocation or comparison failed.

Type casting in comparisons

When two values are compared in an XML comparison (e.g. $eq$) the values are first casted to the same data type. Node values are first replaced by their value() (i.e. the XQL value() function is used, which returns a Text value by default, but may return any data type if the user so chooses.) The resulting values are then casted to the type of the object with the highest xql_primType() value. They are as follows: Node (0), Text (1), Number (2), Boolean (3), Date (4), other data types (4 by default, but this may be overriden by the user.)

E.g. if one value is a Text value and the other is a Number, the Text value is cast to a Number and the resulting low-level (Perl) comparison is (for $eq$):

 $number->xql_toString == $text->xql_toString

If both were Text values, it would have been

 $text1->xql_toString eq $text2->xql_toString

Note that the XQL spec is vague and even conflicting where it concerns type casting. This implementation resulted after talking to Joe Lapp, one of the spec writers.

Adding Data Types

If you want to add your own data type, make sure it derives from XML::XQL::PrimitiveType and implements the necessary methods.

I will add more stuff here to explain it all, but for now, look at the code for the primitive XQL types or the Date class (the XML::XQL::Date manpage in Date.pm.)

Document Order

The XQL spec states that query results always return their values in document order, which means the order in which they appeared in the original XML document. Values extracted from Nodes (e.g. with value(), text(), rawText(), nodeName(), etc.) always have a pointer to the reference node (i.e. the Node from which the value was extracted.) These pointers are acknowledged when (intermediate) result lists are sorted. Currently, the only place where a result list is sorted is in a $union$ expression, which is the only place where the result list can be unordered. (If you find that this is not true, let me know.)

Non-node values that have no associated reference node, always end up at the end of the result list in the order that they were added. The XQL spec states that the reference node for an XML Attribute is the Element to which it belongs, and that the order of values with the same reference node is undefined. This means that the order of an Element and its attributes would be undefined. But since the XML::DOM module keeps track of the order of the attributes, the XQL engine does the same, and therefore, the attributes of an Element are sorted and appear after their parent Element in a sorted result list.

Constant Function Invocations

If a function always returns the same value when given ``constant'' arguments, the function is considered to be ``constant''. A ``constant'' argument can be either an XQL primitive (Number, Boolean, Text) or a ``constant'' function invocation. E.g.

 date("12-03-1998")
 true()
 sin(0.3)
 length("abc")
 date(substr("12-03-1998 is the date", 0, 10))

are constant, but not:

 length(book[2])

Results of constant function invocations are cached and calculated only once for each query. See also the CONST parameter in defineFunction. It is not necessary to wrap constant function invocations in a once() call.

Constant XQL functions are: date, true, false and a lot of the XQL+ wrappers for Perl builtin functions. Function wrappers for certain builtins are not made constant on purpose to force the invocation to be evaluated every time, e.g. 'mkdir(``/user/enno/my_dir'', ``0644'')' (although constant in appearance) may return different results for multiple invocations. See %PerlFunc in Plus.pm for details.

Function: count ([QUERY])

The count() function has no parameters in the XQL spec. In this implementation it will return the number of QUERY results when passed a QUERY parameter.

Method: text ([RECURSE])

When expanding an Element node, the text() method adds the expanded text() value of sub-Elements. When RECURSE is set to 0 (default is 1), it will not include sub-elements. This is useful e.g. when using the $match$ operator in a recursive context (using the // operator), so it won't return parent Elements when one of the children matches.

Method: rawText ([RECURSE])

See text().

AUTHOR

Enno Derksen is the original author.

Please send bugs, comments and suggestions to T.J. Mather <tjmather@tjmather.com>

XML::XQL - A perl module for querying XML tree structures with XQL