PDL::NiceSlice - toward a nicer slicing syntax for PDL


NAME

PDL::NiceSlice - toward a nicer slicing syntax for PDL


SYNOPSYS

  use PDL::NiceSlice;
  $a(1:4) .= 2;             # concise syntax for ranges
  print $b((0),1:$end);     # use variables in the slice expression
  $a->xchg(0,1)->(($pos-1)) .= 0; # default method syntax
  $idx = long 1, 7, 3, 0;   # a piddle of indices
  $a(-3:2:2,$idx) += 3;     # mix explicit indexing and ranges
  $a->clump(1,2)->(0:30);   # 'default method' syntax
  $a(myfunc(0,$var),1:4)++; # when using functions in slice expressions
                            # use parentheses around args!
  # modifiers are specified in a ;-separated trailing block
  $a($a!=3;?)++;            # short for $a->where($a!=3)++
  $a(0:1114;_) .= 0;        # short for $a->flat->(0:1114)
  $b = $a(0:-1:3;|);        # short for $a(0:-1:3)->sever
  $n = sequence 3,1,4,1;
  $b = $n(;-);              # drop all dimensions of size 1 (AKA squeeze)
  $b = $n(0,0;-|);          # squeeze *and* sever
  $c = $a(0,3,0;-);         # more compact way of saying $a((0),(3),(0))
  # Use with perldl versions < v1.31 (or include these lines in .perldlrc)
  perldl> use PDL::NiceSlice;
  # next one is required, see below
  perldl> $PERLDL::PREPROCESS = \&PDL::NiceSlice::perldlpp;
  perldl> $a(4:5) .= xvals(2);


DESCRIPTION

PDL's slice syntax sucks. This module tries to rectify the situation to some degree. Using Perl's ability to use source filtering (which you can think of as a very powerful macro facility, see the perlfilter manpage) it introduces a more reasonable syntax for slicing PDL objects (AKA piddles). PDL::NiceSlice implements the source filter that provides this new feature.

NOTE: PDL::NiceSlice relies on several modules from CPAN that make source filtering and parsing easier. In particular it requires the Filter distribution and Text::Balanced. To make your life easier it is recommended that you use the CPAN module to resolve dependencies during installation automatically. Using the cpan installation shell installation should be as easy as this:

   $ perl -MCPAN -e shell
   cpan> install PDL::NiceSlice


Use in scripts and perldl shell

The new slicing syntax can be switched on and off in scripts and perl modules by using or unloading PDL::NiceSlice.

Note: this will not work in the perldl shell < v1.31. Because the perldl shell uses evals, and NiceSlice is a perl source filter, you have to set a special variable to use it within perldl. See below how to enable the new slicing syntax within older perldl.

But now back to scripts and modules. Everything after use PDL::NiceSlice will be translated and you can use the snew slicing syntax. Source filtering will continue until the end of the file is encountered. You can stop sourcefiltering before the end of the file by issuing a no PDL::NiceSlice statement.

Here is an example:

  use PDL::NiceSlice;
  # this code will be translated
  # and you can use the new slicing syntax
  no PDL::NiceSlice;
  # this code won't
  # and the new slicing syntax will raise errors!

See also the Filter::Simple manpage and example in this distribution for further examples.

NOTE: Unlike ``normal'' modules you need to include a use PDL::NiceSlice call in each and every file that contains code that uses the new slicing syntax. Imagine the following situation: a file test0.pl

   # start test0.pl
   use PDL;
   use PDL::NiceSlice;
   $a = sequence 10;
   print $a(0:4),"\n";
   require 'test1.pl';
   # end test0.pl

that requires a second file test1.pl

   # begin test1.pl
   $aa = sequence 11;
   print $aa(0:7),"\n";
   1;
   # end test1.pl

Following conventional perl wisdom everything should be alright since we used PDL and PDL::NiceSlice already from within test0.pl and by the time test1.pl is required things should be defined and imported, etc. A quick test run will, however, produce something like the following:

  perl test0.pl
 [0 1 2 3 4]
 syntax error at test1.pl line 3, near "0:"
 Compilation failed in require at test0.pl line 7.

This can be fixed by adding the line

  use PDL::NiceSlice;

before the code in test1.pl that uses the new slicing syntax (to play safe just include the line near the top of the file), e.g.

   # begin corrected test1.pl
   use PDL::NiceSlice;
   $aa = sequence 11;
   print $aa(0:7),"\n";
   1;
   # end test1.pl

Now things proceed more smoothly

  perl test0.pl
 [0 1 2 3 4]
 [0 1 2 3 4 5 6 7]

Note that we don't need to issue use PDL again. PDL::NiceSlice is a somewhat funny module in that respect. It is a consequence of the way source filtering works in Perl (see also the IMPLEMENTATION section below).

Usage with perldl

NOTE: This information only applies to versions of perldl earlier than 1.31 . From v1.31 onwards niceslicing is enabled by default, i.e. it should just work. See perldl for details.

For pre v1.31 perldls you need to add the following two lines to your .perldlrc file:

   use PDL::NiceSlice;
   $PERLDL::PREPROCESS = \&PDL::NiceSlice::perldlpp;

A more complete tool box of commands for experimentation is in the file local.perldlrc in the PDL::NiceSlice source directory. Just include the code in that file in your usual ~/.perldlrc and you can switch source filtering with PDL::NiceSlice on and off by typing trans and notrans, respectively. To see what and how your commands are translated switch reporting on:

  perldl> report 1;

Similarly, switch reporting off as needed

  perldl> report 0;

Note that these commands will only work if you included the contents of local.perldlrc in your perldl startup file. In perldl v1.31 and later these commands are available by default.

evals and PDL::NiceSlice

Due to PDL::NiceSlice being a source filter it won't work in the usual way within evals. The following will not do what you want:

  $a = sequence 10;
  eval << 'EOE';
  use PDL::NiceSlice;
  $b = $a(0:5);
  EOE
  print $b;

Instead say:

  use PDL::NiceSlice;
  $a = sequence 10;
  eval << 'EOE';
  $b = $a(0:5);
  EOE
  print $b;

Source filters must be executed at compile time to be effective. And PDL::NiceFilter is just a source filter (although it is not necessarily obvious for the casual user).


The new slicing syntax

Using PDL::NiceSlice slicing piddles becomes so much easier since, first of all, you don't need to make explicit method calls. No

  $pdl->slice(....);

calls, etc. Instead, PDL::NiceSlice introduces two ways in which to slice piddles without too much typing:

The format of the argument list is the same for both types of invocation and will be explained in more detail below.

Parentheses following a scalar variable name

An arglist in parentheses following directly after a scalar variable name that is not preceded by & will be resolved as a slicing command, e.g.

  $a(1:4) .= 2;         # only use this syntax on piddles
  $sum += $a(,(1));

However, if the variable name is immediately preceded by a &, for example

  &$a(4,5);

it will not be interpreted as a slicing expression. Rather, to avoid interfering with the current subref syntax, it will be treated as an invocation of the code reference $a with argumentlist (4,5).

The $a(ARGS) syntax collides in a minor way with the perl syntax. In particular, ``foreach $avar(LIST)'' appears like a PDL slicing call. NiceSlice avoids translating the ``for $avar(LIST)'' and ``foreach $avar(LIST)'' constructs for this reason. Since you can't use just any old lvalue expression in the 'foreach' 'for' constructs -- only a real perl scalar will do -- there's no functionality lost. If later versions of perl accept ``foreach <lvalue-expr> (LIST)'', then you can use the code ref syntax, below, to get what you want.

The default method syntax

The second syntax that will be recognized is what I called the default method syntax. It is the method arrow -> directly followed by an open parenthesis, e.g.

  $a->xchg(0,1)->(($pos)) .= 0;

Note that this conflicts with the use of normal code references, since you can write in plain Perl

  $sub = sub { print join ',', @_ };
  $sub->(1,'a');

NOTE: Once use PDL::NiceSlice is in effect (you can always switch it off with a line no PDL::NiceSlice; anywhere in the script) the source filter will incorrectly replace the above call to $sub with an invocation of the slicing method. This is one of the pitfalls of using a source filter that doesn't know anything about the runtime type of a variable (cf. the Implementation section).

This shouldn't be a major problem in practice; a simple workaround is to use the &-way of calling subrefs, e.g.:

  $sub = sub { print join ',', @_ };
  &$sub(1,'a');

When to use which syntax?

Why are there two different ways to invoke slicing? The first syntax $a(args) doesn't work with chained method calls. E.g.

  $a->xchg(0,1)(0);

won't work. It can only be used directly following a valid perl variable name. Instead, use the default method syntax in such cases:

  $a->xchg(0,1)->(0);

Similarly, if you have a list of piddles @pdls:

  $b = $pdls[5]->(0:-1);

The argument list

The argument list is a comma separated list. Each argument specifies how the corresponding dimension in the piddle is sliced. In contrast to usage of the slice method the arguments should not be quoted. Rather freely mix literals (1,3,etc), perl variables and function invocations, e.g.

  $a($pos-1:$end,myfunc(1,3)) .= 5;

There can even be other slicing commands in the arglist:

  $a(0:-1:$pdl($step)) *= 2;

NOTE: If you use function calls in the arglist make sure that you use parentheses around their argument lists. Otherwise the source filter will get confused since it splits the argument list on commas that are not protected by parentheses. Take the following example:

  sub myfunc { return 5*$_[0]+$_[1] }
  $a = sequence 10;
  $sl = $a(0:myfunc 1, 2);
  print $sl;
 PDL barfed: Error in slice:Too many dims in slice
 Caught at file /usr/local/bin/perldl, line 232, pkg main

The simple fix is

  $sl = $a(0:myfunc(1, 2));
  print $sl;
 [0 1 2 3 4 5 6 7]

Note that using prototypes in the definition of myfunc does not help. At this stage the source filter is simply not intelligent enough to make use of this information. So beware of this subtlety.

Another pitfall to be aware of: currently, you can't use the conditional operator in slice expressions (i.e., ?:, since the parser confuses them with ranges). For example, the following will cause an error:

  $a = sequence 10;
  $b = rand > 0.5 ? 0 : 1; # this one is ok
  print $a($b ? 1 : 2);    # error !
 syntax error at (eval 59) line 3, near "1,

For the moment, just try to stay clear of the conditional operator in slice expressions (or provide us with a patch to the parser to resolve this issue ;).

Modifiers

Following a suggestion originally put forward by Karl Glazebrook the latest versions of PDL::NiceSlice implement modifiers in slice expressions. Modifiers are convenient shorthands for common variations on PDL slicing. The general syntax is

    $pdl(<slice>;<modifier>)

Four modifiers are currently implemented:

Combining modifiers

Several modifiers can be used in the same expression, e.g.

  $c = $a(0;-|); # squeeze and sever

Other combinations are just as useful, e.g. ;_| to flatten and sever. The sequence in which modifiers are specified is not important.

A notable exception is the where modifier (?) which must not be combined with other flags (let me know if you see a good reason to relax this rule).

Repeating any modifier will raise an error:

  $c = $a(-1:1;|-|); # will cause error
 NiceSlice error: modifier | used twice or more

Modifiers are still a new and experimental feature of PDL::NiceSlice. I am not sure how many of you are actively using them. Please do so and experiment with the syntax. I think modifiers are very useful and make life a lot easier. Feedback is welcome as usual. The modifier syntax will likely be further tuned in the future but we will attempt to ensure backwards compatibility whenever possible.

Argument formats

In slice expressions you can use ranges and secondly, piddles as 1D index lists (although compare the description of the ?-modifier above for an exception).

As you might have expected ranges and index piddles can be freely mixed in slicing expressions:

  $a = random 5, 5;
  $b = $a(-1:2,pdl(3,0,1));

piddles as indices in ranges

You can use piddles to specify indices in ranges. No need to turn them into proper perl scalars with the new slicing syntax. However, make sure they contain not more than one element! Otherwise a runtime error will be triggered. First a couple of examples that illustrate proper usage:

  $a = sequence 5, 5;
  $rg = pdl(1,-1,3);
  print $a($rg(0):$rg(1):$rg(2),2);
 [
  [11 14]
 ]
  print $a($rg+1,:$rg(0));
 [
  [2 0 4]
  [7 5 9]
 ]

The next one raises an error

  print $a($rg+1,:$rg(0:1));
 multielement piddle where only one allowed at XXX/Core.pm line 1170.

The problem is caused by using the 2-element piddle $rg(0:1) as the stop index in the second argument :$rg(0:1) that is interpreted as a range by PDL::NiceSlice. You can use multielement piddles as index piddles as described above but not in ranges. And PDL::NiceSlice treats any expression with unprotected :'s as a range. Unprotected means as usual ``not occurring between matched parentheses''.


IMPLEMENTATION

PDL::NiceSlice exploits the ability of Perl to use source filtering (see also the perlfilter manpage). A source filter basically filters (or rewrites) your perl code before it is seen by the compiler. PDL::NiceSlice searches through your Perl source code and when it finds the new slicing syntax it rewrites the argument list appropriately and splices a call to the nslice method using the modified arg list into your perl code. You can see how this works in the perldl shell by switching on reporting (see above how to do that).

The nslice method is an extended version of mslice that knows how to deal with index piddles (and therefore combines slicing and dicing). Full documentation of nslice will be in the next PDL release.


BUGS

Conditional operator

The conditional operator can't be used in slice expressions (see above).

The DATA file handle

Note: To avoid clobbering the DATA filehandle PDL::NiceSlice switches itself off when encountering the __END__ or __DATA__ tokens. This should not be a problem for you unless you use SelfLoader to load PDL code including the new slicing from that section. It is even desirable when working with Inline::Pdlpp, see below.

Possible interaction with Inline::Pdlpp

There is currently an undesired interaction between PDL::NiceSlice and the new Inline::Pdlpp module (currently only in PDL CVS). Since PP code generally contains expressions of the type $var() (to access piddles, etc) PDL::NiceSlice recognizes those incorrectly as slice expressions and does its substitutions. This is not a problem if you use the DATA section for your Pdlpp code -- the recommended place for Inline code anyway. In that case PDL::NiceSlice will have switched itself off before encountering any Pdlpp code (see above):

    # use with Inline modules
  use PDL;
  use PDL::NiceSlice;
  use Inline Pdlpp;
  $a = sequence(10);
  print $a(0:5);
  __END__
  __Pdlpp__
  ... inline stuff

Otherwise switch PDL::NiceSlice explicitly off around the Inline::Pdlpp code:

  use PDL::NiceSlice;
  $a = sequence 10;
  $a(0:3)++;
  $a->inc;
  no PDL::NiceSlice; # switch off before Pdlpp code
  use Inline Pdlpp => "Pdlpp source code";

The cleaner solution is to always stick with the DATA way of including your Inline code as in the first example. That way you keep your nice Perl code at the top and all the ugly Pdlpp stuff etc at the bottom.

Bug reports

Feedback and bug reports are welcome. Please include an example that demonstrates the problem. Log bug reports in the PDL bug database at

  http://sourceforge.net/bugs/?group_id=612

or send them to the pdl-porters mailing list <pdl-porters@jach.hawaii.edu>.


COPYRIGHT

Copyright (c) 2001, 2002 Christian Soeller. All Rights Reserved. This module is free software. It may be used, redistributed and/or modified under the same terms as PDL itself (see http://pdl.perl.org).

 PDL::NiceSlice - toward a nicer slicing syntax for PDL