• NAME
• SYNOPSIS
• DESCRIPTION
• On Deploying Perl Applications
• PAR, the Perl Archive Toolkit
• Simple Packaging
• PAR Loaders
• Dependency Scanning
• Perl Packager: pp
• How it works
• Aggregating multiple programs
• Cross-platform Packages
• The Anatomy of a PAR file
• Special files
• Advantages over perlcc, PerlApp and Perl2exe
• MANIFEST: Best viewed with Mozilla
• META.yml: Metadata galore
• SIGNATURE: Signing and verifying packages
• Perl Servlets with Apache::PAR
• Hon Dah, A-par-che!
• On-demand library fetching
• Code Obfuscation
• Accessing packed files
• Packing GUI applications
• Precompiled CPAN distributions
• Platform-specific Tips
• Thank you!
• Bonus Slides: PAR Internals
• Overview of PAR.pm's Implementation
• Code References in @INC
• Source Filtering without Filter::* Modules
• Source Filtering without Filter::* Modules (cont.)
• Overriding DynaLoader::bootstrap
• Anatomy of a Self-Contained PAR executable
• Self-Bootstrapping Tricks
• Thank you (again)!
• SEE ALSO
• AUTHORS
• COPYRIGHT

NAME

PAR::Tutorial - Cross-Platform Packaging and Deployment with PAR

SYNOPSIS

This is a tutorial on PAR, first appeared at the 7th Perl Conference. The HTML version of this tutorial is available online as http://aut.dyndns.org/par-tutorial/.

DESCRIPTION

On Deploying Perl Applications

 % sshnuke.pl 10.2.2.2 -rootpw="Z1ON0101"
 Perl v5.6.1 required--this is only v5.6.0, stopped at sshnuke.pl line 1.
 BEGIN failed--compilation aborted at sshnuke.pl line 1.

• Q: ``Help! I can't run your program!''
  
• A1: Install Perl & perl -MCPAN -e'install(...)'
  
• How do we know which modules are needed?
  
• New versions of CPAN modules may break sshnuke.pl
  
• A2: Install Perl & tar zxf my_perllib.tgz
  
• Possibly overwriting existing modules; not cross-platform at all
  
• A3: Use the executable generated by perlcc sshnuke.pl
  
• Impossible to debug; perlcc usually does not work anyway
  

PAR, the Perl Archive Toolkit

• Do what JAR (Java Archive) does for Perl
  
• Aggregates modules, scripts and other files into a Zip file
  
• Easy to generate, update and extract
  
• Version consistency: solves forward-compatibility problems
  
• Developed by community: par@perl.org
  
• PAR files can be packed into self-contained scripts
  
• Automatically scans perl script for dependencies
  
• Bundles all necessary 3rd-party modules with it
  
• Requires only core Perl to run on the target machine
  
• PAR also comes with pp, the Perl Packager:
  

 % pp -o sshnuke.exe sshnuke.pl # stand-alone executable!

Simple Packaging

• PAR files are just Zip files with modules in it
  
• Any Zip tools can generate them:
  

 % zip foo.par Hello.pm World.pm        # pack two modules
 % zip -r bar.par lib/          # grab all modules in lib/

• To load modules from PAR files:
  

 use PAR;
 use lib "foo.par";             # the .par part is optional
 use Hello;

• This also works:
  

 use PAR "/home/mylibs/*.par";  # put all of them into @INC
 use Hello;

PAR Loaders

• Use par.pl to run files inside a PAR archive:
  

 % par.pl foo.par               # looks for 'main.pl' by default
 % par.pl foo.par test.pl       # runs script/test.pl in foo.par

• Same thing, with the stand-alone parl or parl.exe:
  

 % parl foo.par                 # no perl or PAR.pm needed!
 % parl foo.par test.pl         # ditto

• The PAR loader can prepend itself to a PAR file:
  
• -b bundles non-core modules needed by PAR.pm:
  

 % par.pl -b -O./foo.pl foo.par # self-contained script

• -B bundles core modules in addition to -b:
  

 % parl -B -O./foo.exe foo.par  # self-contained binary

Dependency Scanning

• Recursively scan dependencies with scandeps.pl:
  

 % scandeps.pl sshnuke.pl
 # Legend: [C]ore [X]ternal [S]ubmodule [?]NotOnCPAN
 'Crypt::SSLeay'       => '0', #  X   #
 'Net::HTTP'           => '0', #      #
 'Crypt::SSLeay::X509' => '0', # S    # Crypt::SSLeay
 'Net::HTTP::Methods'  => '0', # S    # Net::HTTP
 'Compress::Zlib'      => '0', #  X   # Net::HTTP::Methods

• Scan an one-liner, list all involved files:
  

 % scandeps.pl -V -e "use Dynaloader;"
 ...
 # auto/DynaLoader/dl_findfile.al [autoload]
 # auto/DynaLoader/extralibs.ld [autoload]
 # auto/File/Glob/Glob.bs [data]
 # auto/File/Glob/Glob.so [shared]
 ...

Perl Packager: pp

• Combines scanning, zipping and loader-embedding:
  

 % pp -o out.exe src.pl         # self-contained .exe
 % out.exe                      # runs anywhere on the same OS

• Bundle additional modules:
  

 % pp -o out.exe -M CGI src.pl  # pack CGI + its dependencies, too

• Pack one-liners:
  

 % pp -o out.exe -e 'print "Hi!"'   # turns one-liner into executable

• Generate PAR files instead of executables:
  

 % pp -p src.pl                 # makes 'source.par'
 % pp -B -p src.pl              # include core modules

How it works

• Command-line options are almost identical to perlcc's
  
• Also supports gcc-style long options:
  

 % pp --gui --verbose --output=out.exe src.pl

• Small initial overhead; no runtime overhead
  
• Dependencies are POD-stripped before packing
  
• Loads modules directly into memory on demand
  
• Shared libraries (DLLs) are extracted with File::Temp
  
• Works on Perl 5.6.0 or above
  
• Tested on Win32 (VC++ and MinGW), FreeBSD, NetBSD, Linux, MacOSX, Cygwin, AIX, Solaris, HP-UX, Tru64...
  

Aggregating multiple programs

• A common question:
  

 > I have used pp to make several standalone applications which work
 > great, the only problem is that for each executable that I make, I am
 > assuming the parl.exe is somehow bundled into the resulting exe.

• The obvious workaround:
  

 You can ship parl.exe by itself, along with .par files built
 by "pp -p", and run those PAR files by associating them to parl.exe.

• On platforms that have ln, there is a better solution:
  

 % pp --output=a.out a.pl b.pl  # two scripts in one!
 % ln a.out b.out               # symlink also works
 % ./a.out                      # runs a.pl
 % ./b.out                      # runs b.pl

Cross-platform Packages

• Of course, there is no cross-platform binary format
  
• Pure-perl PAR packages are cross-platform by default
  
• However, XS modules are specific to Perl version and platform
  
• Multiple versions of a XS module can co-exist in a PAR file
  
• Suppose we need out.par on both Win32 and Finix:
  

 C:\> pp --multiarch --output=out.par src.pl
 ...copy src.pl and out.par to a Finix machine...
 % pp --multiarch --output=out.par src.pl

• Now it works on both platforms:
  

 % parl out.par                 # runs src.pl
 % perl -MPAR=out.par -e '...'  # uses modules inside out.par

The Anatomy of a PAR file

• Modules can reside in several directories:
  

 /                      # casual packaging only
 /lib/                  # standard location
 /arch/                 # for creating from blib/ 
 /i386-freebsd/         # i.e. $Config{archname}
 /5.8.0/                # i.e. Perl version number
 /5.8.0/i386-freebsd/   # combination of the two above

• Scripts are stored in one of the two locations:
  

 /                      # casual packaging only
 /script/               # standard location

• Shared libraries may be architecture- or perl-version-specific:
  

 /shlib/(5.8.0/)?(i386-freebsd/)?

• PAR files may recursively contain other PAR files:
  

 /par/(5.8.0/)?(i386-freebsd/)?

Special files

• MANIFEST
  
• Index of all files inside PAR
  
• Can be parsed with ExtUtils::Manifest
  
• META.yml
  
• Dependency, license, runtime options
  
• Can be parsed with YAML
  
• SIGNATURE
  
• OpenPGP-signed digital signature
  
• Can be parsed and verified with Module::Signature
  

Advantages over perlcc, PerlApp and Perl2exe

• This is not meant to be a flame
  
• All three maintainers have contributed to PAR directly; I'm grateful
  
• perlcc
  
• ``The code generated in this way is not guaranteed to work... Use for production purposes is strongly discouraged.'' (from perldoc perlcc)
  
• Guaranteed to not work is more like it
  
• PerlApp / Perl2exe
  
• Expensive: Need to pay for each upgrade
  
• Non-portable: Only available for limited platforms
  
• Proprietary: Cannot extend its features or fix bugs
  
• Obfuscated: Vendor and black-hats can see your code, but you can't
  
• Inflexible: Does not work with existing Perl installations
  

MANIFEST: Best viewed with Mozilla

• The URL of MANIFEST inside /home/autrijus/foo.par:
  

 jar:file:///home/autrijus/foo.par!/MANIFEST

• Open it in a Gecko browser (e.g. Netscape 6+) with Javascript enabled:
  
• No needed to unzip anything; just click on files to view them
  

META.yml: Metadata galore

• Static, machine-readable distribution metadata
  
• Supported by Module::Build, ExtUtils::MakeMaker, Module::Install
  
• A typical pp-generated META.yml looks like this:
  

 build_requires: {}
 conflicts: {}
 dist_name: out.par
 distribution_type: par
 dynamic_config: 0
 generated_by: 'Perl Packager version 0.03'
 license: unknown
 par:
   clean: 0
   signature: ''
   verbatim: 0
   version: 0.68

• The par: settings controls its runtime behavior
  

SIGNATURE: Signing and verifying packages

• OpenPGP clear-signed manifest with SHA1 digests
  
• Supported by Module::Signature, CPANPLUS and Module::Build
  
• A typical SIGNATURE looks like this:
  

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1

 SHA1 8a014cd6d0f6775552a01d1e6354a69eb6826046 AUTHORS
 ...
 -----BEGIN PGP SIGNATURE-----
 ...
 -----END PGP SIGNATURE-----

• Use pp and cpansign to work with signatures:
  

 % pp -s -o foo.par bar.pl      # make and sign foo.par from bar.pl
 % cpansign -s foo.par  # sign this PAR file
 % cpansign -v foo.par  # verify this PAR file

Perl Servlets with Apache::PAR

• Framework for self-contained Web applications
  
• Similar to Java's ``Web Application Archive'' (WAR) files
  
• Works with mod_perl 1.x or 2.x
  
• A complete web application inside a .par file
  
• Apache configuration, static files, Perl modules...
  
• Supports Static, Registry and PerlRun handlers
  
• Can also load all PARs under a directory
  
• One additional special file: web.conf
  

 Alias /myapp/cgi-perl/ ##PARFILE##/
 <Location /myapp/cgi-perl>
     Options +ExecCGI
     SetHandler perl-script
     PerlHandler Apache::PAR::Registry
 </Location>

Hon Dah, A-par-che!

• First, make a hondah.par from an one-liner:
  

 # use the "web.conf" from the previous slide
 % pp -p -o hondah.par -e 'print "Hon Dah!\n"' \
      --add web.conf
 % chmod a+x hondah.par

• Add this to httpd.conf, then restart apache:
  

 <IfDefine MODPERL2>
 PerlModule Apache2
 </IfDefine>
 PerlAddVar PARInclude /home/autrijus/hondah.par
 PerlModule Apache::PAR

• Test it out:
  

 % GET http://localhost/myapp/cgi-perl/main.pl
 Hon Dah!

• Instant one-liner web application that works!
  

On-demand library fetching

• With LWP installed, your can use remote PAR files:
  

 use PAR;
 use lib 'http://aut.dyndns.org/par/DBI-latest.par';
 use DBI;    # always up to date!

• Modules are now cached under $ENV{PAR_GLOBAL_TEMP}
  
• Auto-updates with LWP::Simple::mirror
  
• Download only if modified
  
• Safe for offline use after the first time
  
• May use SIGNATURE to prevent DNS-spoofing
  
• Makes large-scale deployment a breeze
  
• Upgrades from a central location
  
• No installers needed
  

Code Obfuscation

• Also known as source-hiding techniques
  
• It is not encryption
  
• Offered by PerlApp, Perl2Exe, Stunnix...
  
• Usually easy to defeat
  
• Take optree dump from memory, feed to B::Deparse
  
• If you just want to stop a casual grep, ``deflate'' already works
  
• PAR now supports pluggable input filters with pp -f
  
• Bundled examples: Bleach, PodStrip and PatchContent
  
• True encryption using Crypt::*
  
• Or even _product activation_ over the internet
  
• Alternatively, just keep core logic in your server and use RPC
  

Accessing packed files

• To get the host archive from a packed program:
  

 my $zip = PAR::par_handle($0); # an Archive::Zip object
 my $content = $zip->contents('MANIFEST');

• Same thing, but with read_file():
  

 my $content = PAR::read_file('MANIFEST');

• Loaded PAR files are stored in %PAR::LibCache:
  

 use PAR '/home/mylibs/*.par';
 while (my ($filename, $zip) = each %PAR::LibCache) {
     print "[$filename - MANIFEST]\n";
     print $zip->contents('MANIFEST');
 }

Packing GUI applications

• GUI toolkits often need to link with shared libraries:
  

 # search for libncurses under library paths and pack it
 % pp -l ncurses curses_app.pl  # same for Tk, Wx, Gtk, Qt...

• Use pp --gui on Win32 to eliminate the console window:
  

 # pack 'src.pl' into a console-less 'out.exe' (Win32 only)
 % pp --gui -o out.exe src.pl

• ``Can't locate Foo/Widget/Bar.pm in @INC''?
  
• Some toolkits (notably Tk) autoloads modules without use or require
  
• Hence pp and Module::ScanDeps may fail to detect them
  
• Tk problems mostly fixed by now, but other toolkits may still break
  
• You can work around it with pp -M or an explicit require
  
• Or better, send a short test-case to par@perl.org so we can fix it
  

Precompiled CPAN distributions

• Installing XS extensions from CPAN was difficult
  
• Some platforms do not come with a compiler (Win32, MacOSX...)
  
• Some headers or libraries may be missing
  
• PAR.pm itself used to suffer from both problems
  
• ...but not anymore -- Module::Install to the rescue!
  

 # same old Makefile.PL, with a few changes
 use inc::Module::Install;      # was "use ExtUtils::MakeMaker;"
 WriteMakefile( ... );          # same as the original
 check_nmake();                 # make sure the user have nmake
 par_base('AUTRIJUS');          # your CPAN ID or a URL
 fetch_par() unless can_cc();   # use precompiled PAR only if necessary

• Users will not notice anything, except now it works
  
• Of course, you still need to type make par and upload the precompiled package
  
• PAR users can also install it directly with parl -i
  

Platform-specific Tips

• Win32 and other icon-savvy platforms
  
• Needs 3rd-party tools to add icons to pp-generated executables
  
• PE Header manipulation in Perl -- volunteers wanted!
  
• Linux and other libc-based platforms
  
• Try to avoid running pp on a bleeding-edge version of the OS
  
• Older versions with an earlier libc won't work with new ones
  
• Solaris and other zlib-lacking platforms (but not Win32)
  
• You need a static-linked Compress::Zlib before installing PAR
  
• In the future, PAR may depend on Compress::Zlib::Static instead
  
• Any platform with limited bandwidth or disk space
  
• Use UPX to minimize the executable size
  

Thank you!

• Additional resources
  
• Mailing list: par@perl.org
  
• Subscribe: Send a blank email to par-subscribe@perl.org
  
• List archive: http://nntp.x.perl.org/group/perl.par
  
• PAR::Intro: http://search.cpan.org/dist/PAR/lib/PAR/Intro.pod
  
• Apache::PAR: http://search.cpan.org/dist/Apache-PAR/
  
• Module::Install: http://search.cpan.org/dist/Module-Install/
  
• Any questions?
  

Bonus Slides: PAR Internals

Overview of PAR.pm's Implementation

• Here begins the scary part
  
• Grues, Dragons and Jabberwocks abound...
  
• You are going to learn weird things about Perl internals
  
• PAR invokes four areas of Perl arcana:
  
• @INC code references
  
• On-the-fly source filtering
  
• Overriding DynaLoader::bootstrap() to handle XS modules
  
• Making self-bootstrapping binary executables
  
• The first two only works on 5.6 or later
  
• DynaLoader and %INC are there since Perl 5 was born
  
• PAR currently needs 5.6, but a 5.005 port is possible
  

Code References in @INC

• On 1999-07-19, Ken Fox submitted a patch to P5P
  
• To _enable using remote modules_ by putting hooks in @INC
  
• It's accepted to come in Perl 5.6, but undocumented until 5.8
  
• Type perldoc -f require to read the nitty-gritty details
  
• Coderefs in @INC may return a fh, or undef to 'pass':
  

 push @INC, sub {
     my ($coderef, $filename) = @_;  # $coderef is \&my_sub
     open my $fh, "wget ftp://example.com/$filename |";
     return $fh;        # using remote modules, indeed!
 };

• Perl 5.8 let you open a file handle to a string, so we just use that:
  

        open my $fh, '<', \($zip->memberNamed($filename)->contents);
        return $fh;

• But Perl 5.6 does not have that, and I don't want to use temp files...
  

Source Filtering without Filter::* Modules

• ... Undocumented features to the rescue!
  
• It turns out that @INC hooks can return two values
  
• The first is still the file handle
  
• The second is a code reference for line-by-line source filtering!
  
• This is how Acme::use::strict::with::pride works:
  

 # Force all modules used to use strict and warnings
 open my $fh, "<", $filename or return;
 my @lines = ("use strict; use warnings;\n", "#line 1 \"$full\"\n");
 return ($fh, sub {
     return 0 unless @lines;    
     push @lines, $_; $_ = shift @lines; return length $_;
 });

Source Filtering without Filter::* Modules (cont.)

• But we don't really have a filehandle for anything
  
• Another undocumented feature saves the day!
  
• We can actually omit the first return value altogether:
  

 # Return all contents line-by-line from the file inside PAR
 my @lines = split(
     /(?<=\n)/,
     $zip->memberNamed($filename)->contents
 );
 return (sub {
     $_ = shift(@lines);
     return length $_;
 });

Overriding DynaLoader::bootstrap

• XS modules have dynamically loaded libraries
  
• They cannot be loaded as part of a zip file, so we extract them out
  
• Must intercept DynaLoader's library-finding process
  
• Module names are passed to bootstrap for XS loading
  
• During the process, it calls dl_findfile to locate the file
  
• So we install pre-hooks around both functions
  
• Our _bootstrap just checks if the library is in PARs
  
• If yes, extract it to a File::Temp temp file
  
• The file will be automatically cleaned up when the program ends
  
• It then pass the arguments to the original bootstrap
  
• Finally, our dl_findfile intercepts known filenames and return it
  

Anatomy of a Self-Contained PAR executable

• The par script ($0) itself
  
• May be in plain-text or native executable format
  
• Any number of embedded files
  
• Typically used to bootstrap PAR's various dependencies
  
• Each section begins with the magic string ``FILE''
  
• Length of filename in pack('N') format and the filename (auto/.../)
  
• File length in pack('N') and the file's content (not compressed)
  
• One PAR file
  
• Just a regular zip file with the magic string "PK\003\004"
  
• Ending section
  
• A pack('N') number of the total length of FILE and PAR sections
  
• Finally, there must be a 8-bytes magic string: "\012PAR.pm\012"
  

Self-Bootstrapping Tricks

• All we can expect is a working perl interpreter
  
• The self-contained script *must not* use any modules at all
  
• But to process PAR files, we need XS modules like Compress::Zlib
  
• Answer: bundle all modules + libraries used by PAR.pm
  
• That's what the FILE section in the previous slide is for
  
• Load modules to memory, and write object files to disk
  
• Then use a local @INC hook to load them on demand
  
• Minimizing the amount of temporary files
  
• First, try to load PerlIO::scalar and File::Temp
  
• Set up an END hook to unlink all temp files up to this point
  
• Load other bundled files, and look in the compressed PAR section
  
• This can be much easier with a pure-perl inflate(); patches welcome!
  

Thank you (again)!

• Any questions, please?
  

SEE ALSO

http://www.autrijus.org/par-tutorial/

http://www.autrijus.org/par-intro/ (English version)

http://www.autrijus.org/par-intro.zh/ (Chinese version)

PAR, pp, par.pl, parl

the ex::lib::zip manpage, the Acme::use::strict::with::pride manpage

the App::Packer manpage, the Apache::PAR manpage, CPANPLUS, the Module::Install manpage

AUTHORS

Audrey Tang <cpan@audreyt.org>

http://par.perl.org/ is the official PAR website. You can write to the mailing list at <par@perl.org>, or send an empty mail to <par-subscribe@perl.org> to participate in the discussion.

Please submit bug reports to <bug-par@rt.cpan.org>.

COPYRIGHT

Copyright 2003, 2004, 2005, 2006 by Audrey Tang <cpan@audreyt.org>.

This document is free documentation; you can redistribute it and/or modify it under the same terms as Perl itself.

See http://www.perl.com/perl/misc/Artistic.html