PDL::IO::FastRaw -- A simple, fast and convenient io format for PerlDL.


PDL::IO::FastRaw -- A simple, fast and convenient io format for PerlDL.


 use PDL;
 use PDL::IO::FastRaw;
 writefraw($pdl,"fname");         # write a raw file
 $pdl2 = readfraw("fname");       # read a raw file
 $pdl2 = PDL->readfraw("fname");
 $pdl3 = mapfraw("fname2",{ReadOnly => 1}); # mmap a file, don't read yet
 $pdl4 = maptextfraw("fname3",{...}); # map a text file into a 1-D pdl.


This is a very simple and fast io format for PerlDL. The disk data consists of two files, a header metadata file in ASCII and a binary file consisting simply of consecutive bytes, shorts or whatever.

It is hoped that this will not only make for a simple PerlDL module for saving and retrieving these files but also make it easy for other programs to use these files.

The format of the ASCII header is simply

        <dim0> <dim1> ...

The binary files are in general NOT interchangeable between different architectures since the binary file is simply dumped from the memory region of the piddle. This is what makes the approach efficient.

It is also possible to mmap the file which can give a large speedup in certain situations as well as save a lot of memory by using a disk file as virtual memory. When a file is mapped, parts of it are read only as they are accessed in the memory (or as the kernel decides: if you are reading the pages in order, it may well preread some for you).

Note that memory savings and copy-on-write are operating-system dependent - see Core.xs and your operating system documentation for exact semantics of whatever. Basically, if you write to a mmapped file without ReadOnly, the change will be reflected in the file immediately. ReadOnly doesn't really make it impossible to write to the piddle but maps the memory privately so the file will not be changed when you change the piddle. Be aware though that mmapping a 40Mb file without ReadOnly spends no virtual memory but with ReadOnly it does reserve 40Mb.



Read a raw format binary file

 $pdl2 = readfraw("fname");
 $pdl2 = PDL->readfraw("fname");


Write a raw format binary file



Memory map a raw format binary file (see the module docs also)

 $pdl3 = mapfraw("fname2",{ReadOnly => 1});

The mapfraw command supports the following options (not all combinations make sense):

Dims, Datatype
If creating a new file or if you want to specify your own header data for the file, you can give an array reference and a scalar, respectively.

Create the file. Also writes out a header for the file.

Set the file size. Automatically enabled with Creat. NOTE: This also clears the file to all zeroes.

Disallow writing to the file.


Memory map a text file (see the module docs also).

Note that this function maps the raw format so if you are using an operating system which does strange things to e.g. line delimiters upon reading a text file, you get the raw (binary) representation.

The file doesn't really need to be text but it is just mapped as one large binary chunk.

This function is just a convenience wrapper which firsts stats the file and sets the dimensions and datatype.

 $pdl4 = maptextfraw("fname", {options}

The options other than Dims, Datatype of mapfraw are supported.


Should be documented better. writefraw and readfraw should also have options (the author nowadays only uses mapfraw ;)


Copyright (C) Tuomas J. Lukka 1997. All rights reserved. There is no warranty. You are allowed to redistribute this software / documentation under certain conditions. For details, see the file COPYING in the PDL distribution. If this file is separated from the PDL distribution, the copyright notice should be included in the file.

 PDL::IO::FastRaw -- A simple, fast and convenient io format for PerlDL.