TopicDataHelperPlugin < System

You are here: Foswiki>System Web>TopicDataHelperPluginEditAttach
---+!! Topic Data helper plugin

Helper plugin for collecting, filtering and sorting data objects, to be used by other plugins.

This plugin is used by Foswiki:Extensions/AttachmentListPlugin and Foswiki:Extensions/FormFieldListPlugin to process this kind of parameters:

<verbatim>
%ATTACHMENTLIST{
   web="*"
   topic="*"
   excludetopic="WebHome, WebPreferences"
   extension="jpg,jpeg,gif,png"
   includefilepattern="(?i)^[A]"
   fromdate="2007/01/01"
   sort="$fileName"
   sortorder="descending"
}%
</verbatim>

In short, this plugin provides:
   * *Collecting*
      * Creation of a web-topic hash to pass one set of topics to process
      * Exclude topics that the current user does not have view permission for
      * Adding your custom data objects to this hash
   * *Filtering*
      * Filter your data objects by property (direct match or regular expression)
      * Filter by date range
   * *Listing* (for further processing)
      * Get a array of all data objects
      * Get a stringified array for object file storage or caching
   * *Sorting*
      * Sorting by property (primary key) and secondary key
      * Sort ascending or descending


%TOC%

---++ Background

With extending Foswiki:Extensions/FormListPlugin I found I had the same needs as with Foswiki:Extensions/AttachmentListPlugin. I needed almost the plugin syntax parameters! I decided to abstract out the collecting, filtering and sorting functions and provide them in a re-usable way.


---++ When to use this plugin

Any time you need to process a set of data - filtering, sorting - this plugin may make your life easier.

See for example how filters in !AttachmentListPlugin are created:

<verbatim>
# filter attachments by date range
if ( defined $inParams->{'fromdate'} || defined $inParams->{'todate'} ) {
   Foswiki::Plugins::TopicDataHelperPlugin::filterTopicDataByDateRange(
      \%topicData, $inParams->{'fromdate'},
      $inParams->{'todate'} );
}

# filter included/excluded filenames
if (   defined $inParams->{'file'}
   || defined $inParams->{'excludefile'} )
{
   Foswiki::Plugins::TopicDataHelperPlugin::filterTopicDataByProperty(
      \%topicData, 'name', 1, $inParams->{'file'},
      $inParams->{'excludefile'} );
}

# filter filenames by regular expression
if (   defined $inParams->{'includefilepattern'}
   || defined $inParams->{'excludefilepattern'} )
{
   Foswiki::Plugins::TopicDataHelperPlugin::filterTopicDataByRegexMatch(
      \%topicData, 'name',
      $inParams->{'includefilepattern'},
      $inParams->{'excludefilepattern'}
   );
}
</verbatim>

There is a relatively small burden for setting up your data for data collection - to enable it for filtering and sorting. After that it is more or less straightforward.

---++ How it works

Let us assume the processing order in your plugin would be:
   1 Finding the topics to find data in (filtering out unwanted topics)
   1 Collecting data from the topics
   1 Removing unwanted data
   1 Sorting the data
   1 Limiting the amount of data to show
   1 Formatting the data

<nop>TopicDataHelperPlugin does not prescribe any way to write your plugin. But let's follow this order and see how the plugin could help you.

---+++ Finding the topics to find data in

Almost all functions assume you have a hash object with web-topic-data relations. For <nop>AttachmentListPlugin this looks like:

<verbatim>
%topicData = (
   Web1 => {
      Topic1 => {
         picture.jpg => FileData object,
         me.PNG => FileData object,      
         ...
      },
   },
)
</verbatim>

The first step is to create a hash of web-topic relations, and that is what =createTopicData= does:

<blockquote style="margin-left:0;">
==createTopicData( $webs, $excludewebs, $topics, $excludetopics ) -> \%hash==
</blockquote>

And it may be called like this:

<verbatim style="color:green;">
my $webs   = $inParams->{'web'}   || $inWeb   || '';
my $topics = $inParams->{'topic'} || $inTopic || '';
my $excludeTopics = $inParams->{'excludetopic'} || '';
my $excludeWebs   = $inParams->{'excludeweb'}   || '';

my $topicData =
  Foswiki::Plugins::TopicDataHelperPlugin::createTopicData(
    $webs, $excludeWebs, $topics, $excludeTopics );
</verbatim>

The resulting hash looks like this:

<verbatim>
%topicData = (
   Web1 => {
      Topic1 => 1,
      Topic2 => 1,
      ...
   }
   Web2 => {
      Topic1 => 1,
      Topic2 => 1,
      ...
   }
)
</verbatim>

The =1= values are placeholders for now.

At this stage you will have filtered out unwanted webs and topics as passed in the parameters =$webs, $excludewebs, $topics, $excludetopics=.



---+++ Collecting data from the topics

To store the data we will retrieve from the topics need a separate data structure. I find it useful to create a data class. For <nop>AttachmentListPlugin I have used the class =FileData=:

=package Foswiki::Plugins::AttachmentListPlugin::FileData;=

To filter and sort on object data properties, these properties must be accessible as instance members.

For instance, to filter =FileData= objects on attachment date, we create a FileData =date= property that we fill with the =$attachment->{'date'}= value:

<verbatim>
sub new {
    my ( $class, $web, $topic, $attachment ) = @_;
    my $this = {};
   $this->{'attachment'} = $attachment;
   $this->{'date'} = $attachment->{'date'} || 0;
   ...
   bless $this, $class;
}
</verbatim>

... to be able to write:

<verbatim>
my $fd =
   Foswiki::Plugins::AttachmentListPlugin::FileData->new( $inWeb,
   $inTopic, $attachment );
my $date = $fd->{date};
</verbatim>

To add our data objects to the web-topic hash, we call =insertObjectData=:

<blockquote style="margin-left:0;">
==insertObjectData( $topicData, $createObjectDataFunc, $properties )==
</blockquote>

Where =$topicData= is our hash reference, and =$createObjectDataFunc= is a reference to a function that will create data objects. You will write that function.
Parameter =$properties= is optional. You may pass a hash reference with custom data to your object creation function.

For <nop>AttachmentListPlugin that function looks like:

<verbatim>
sub _createFileData {
    my ( $inTopicHash, $inWeb, $inTopic ) = @_;

    my $attachments = _getAttachmentsInTopic( $inWeb, $inTopic );
    if ( scalar @$attachments ) {
        $inTopicHash->{$inTopic} = ();
        foreach my $attachment (@$attachments) {
            my $fd =
              Foswiki::Plugins::AttachmentListPlugin::FileData->new(
                $inWeb, $inTopic, $attachment );
            my $fileName = $fd->{name};
            $inTopicHash->{$inTopic}{$fileName} = \$fd;
        }
    }
    else {
        # no META:FILEATTACHMENT, so remove from hash
        delete $inTopicHash->{$inTopic};
    }
}
</verbatim>

And it is called with:

<verbatim style="color:green;">
Foswiki::Plugins::TopicDataHelperPlugin::insertObjectData(
   $topicData, \&_createObjectData
);
</verbatim>

Now your hash will have the structure:

<verbatim>
%topicData = (
   Web1 => {
      Topic1 => {
         'key a' => object,
         'key b' => object,      
         ...
      },
   },
)
</verbatim>



---+++ Removing unwanted data

<nop>TopicDataHelperPlugin provides 4 filter functions:

<blockquote style="margin-left:0;">
==filterTopicDataByViewPermission( $topicData, $wikiUserName )==
</blockquote>

Filters topic data objects by checking if the user =$wikiUserName= has view access permissions.

Removes topic data if the user does not have permission to view the topic.

Example:
<verbatim style="color:green;">
# filter topics by view permission
my $user = Foswiki::Func::getWikiName();
my $wikiUserName = Foswiki::Func::userToWikiName( $user, 1 );
Foswiki::Plugins::TopicDataHelperPlugin::filterTopicDataByViewPermission(
   \%topicData, $wikiUserName );
</verbatim>

<blockquote style="margin-left:0;">
==filterTopicDataByDateRange( $topicData, $fromDate, $toDate, $dateKey )==
</blockquote>

Filters topic data objects by date range, from =$fromDate= to =$toDate= (both in epcoh seconds).

Removes topic data if:
   * the value of the object attribute =$dateKey= is earlier than =$fromDate=
   * the value of the object attribute =$dateKey= is later than =$toDate=

Use either =$fromDate= or =toDate=, or both.

Example:
<verbatim style="color:green;">
# filter attachments by date range
if ( defined $inParams->{'fromdate'} || defined $inParams->{'todate'} ) {
   Foswiki::Plugins::TopicDataHelperPlugin::filterTopicDataByDateRange(
      \%topicData, $inParams->{'fromdate'},
      $inParams->{'todate'} );
}
</verbatim>

<blockquote style="margin-left:0;">
==filterTopicDataByProperty( $topicData, $propertyKey, $isCaseSensitive, $includeValues, $excludeValues )==
</blockquote>

Filters topic data objects by matching an object property with a list of possible values.

Removes topic data if:
   * the object attribute =$propertyKey= is not in =$includeValues=
   * the object attribute =$propertyKey= is in =$excludeValues=

Use either =$includeValues= or =$excludeValues=, or both.

For example, <nop>AttachmentListPlugin uses this function to filter attachments by extension. %BR%
=extension="gif, jpg"= will find all attachments with extension 'gif' OR 'jpg'. OR 'GIF' or 'JPG', therefore =$isCaseSensitive= is set to =0=.

Example:
<verbatim style="color:green;">
# filter included/excluded field VALUES
my $values        = $inParams->{'includevalue'} || undef;
my $excludeValues = $inParams->{'excludevalue'} || undef;
if ( defined $values || defined $excludeValues ) {
   Foswiki::Plugins::TopicDataHelperPlugin::filterTopicDataByProperty(
      \%topicData, 'value', 1, $values, $excludeValues );
}
</verbatim>

<blockquote style="margin-left:0;">
==filterTopicDataByRegexMatch( $topicData, $propertyKey, $includeRegex, $excludeRegex )==
</blockquote>

Filters topic data objects by matching an object property with a regular expression.

Removes topic data if:
- the object attribute =$propertyKey= does not match =$includeRegex=
- the object attribute =$propertyKey= matches =$excludeRegex=

Use either =$includeRegex= or =$excludeRegex=, or both.

Example:
<verbatim style="color:green;">
# filter filenames by regular expression
if (   defined $inParams->{'includefilepattern'}
   || defined $inParams->{'excludefilepattern'} )
{
   Foswiki::Plugins::TopicDataHelperPlugin::filterTopicDataByRegexMatch(
      \%topicData, 'name',
      $inParams->{'includefilepattern'},
      $inParams->{'excludefilepattern'}
   );
}
</verbatim>

After using the filter functions, your topic data hash will probably be quite some smaller. Next step is sorting the data. Limiting the result set will come after sorting.


---+++ Sorting the data

Before sorting, we must make an array from our hash. This is what =getListOfObjectData= does:

<blockquote style="margin-left:0;">
==getListOfObjectData( $topicData ) -> \@objects==
</blockquote>

Example:
<verbatim style="color:green;">
my $objects =
  Foswiki::Plugins::TopicDataHelperPlugin::getListOfObjectData($topicData);
</verbatim>

Now we can sort the list of data objects with =sortObjectData=:

<blockquote style="margin-left:0;">
==sortObjectData( $objectData, $sortOrder, $sortKey, $compareMode, $nameKey ) -> \@objects==
</blockquote>

Function parameters:
   * =\@objectData= (array reference) - list of data objects (NOT the topic data!)
   * =$sortOrder= (int) - either =$Foswiki::Plugins::TopicDataHelperPlugin::sortDirections{'ASCENDING'}=, =$Foswiki::Plugins::TopicDataHelperPlugin::sortDirections{'DESCENDING'}= or =$Foswiki::Plugins::TopicDataHelperPlugin::sortDirections{'NONE'}=
   * =$inSortKey= (string) - primary sort key; this will be a property of your data object
   * =$compareMode= (string) - sort mode of primary key, either 'numeric' or 'alphabetical'
   * =$nameKey= (string) - to be used as secondary sort key; must be alphabetical; this will be a property of your data object

This function returns a reference to an sorted array of data objects.

To get the primary sort key and the kind of data (alphabetical or integer) we can create a lookup table in our data class:

<verbatim>
my %sortKeys = (
    '$fileDate'      => [ 'date',      'integer' ],
    '$fileSize'      => [ 'size',      'integer' ],
    '$fileUser'      => [ 'user',      'string' ],
    '$fileExtension' => [ 'extension', 'string' ],
    '$fileName'      => [ 'name',      'string' ],
    '$fileTopic'     => [ 'topic',     'string' ]
);
sub getSortKey {
    my ($inRawKey) = @_;
    return $sortKeys{$inRawKey}[0];
}
sub getCompareMode {
    my ($inRawKey) = @_;
    return $sortKeys{$inRawKey}[1];
}
</verbatim>

This can be used as follows:
<verbatim style="color:green;">
my $sortKey =
  &Foswiki::Plugins::AttachmentListPlugin::FileData::getSortKey(
   $inParams->{'sort'} );
my $compareMode =
  &Foswiki::Plugins::AttachmentListPlugin::FileData::getCompareMode(
   $inParams->{'sort'} );
</verbatim>   

Similarly we can create a mapping between user input (for instance the =sortorder= parameter) and the =$sortOrder= value we pass to <nop>TopicDataHelperPlugin:

<verbatim>
my %sortInputTable = (
    'none' => $Foswiki::Plugins::TopicDataHelperPlugin::sortDirections{'NONE'},
    'ascending' =>
      $Foswiki::Plugins::TopicDataHelperPlugin::sortDirections{'ASCENDING'},
    'descending' =>
      $Foswiki::Plugins::TopicDataHelperPlugin::sortDirections{'DESCENDING'},
);

# translate input to sort parameters
my $sortOrderParam = $inParams->{'sortorder'} || 'none';
my $sortOrder = $sortInputTable{$sortOrderParam}
  || $Foswiki::Plugins::TopicDataHelperPlugin::sortDirections{'NONE'};
</verbatim>

Now we sort the data:
<verbatim style="color:green;">
$objects =
  Foswiki::Plugins::TopicDataHelperPlugin::sortObjectData( $objects, $sortOrder,
   $sortKey, $compareMode, 'name' );
</verbatim>   



---+++ Limiting the amount of data to show

With the final data in the right order, we can simply shorten the array:

<verbatim>
splice @$objects, $inParams->{'limit'}
  if defined $inParams->{'limit'};
</verbatim>



---+++ Formatting the data

Formatting is not provided by <nop>TopicDataHelperPlugin, but your formatting function would logically have this setup:

<verbatim>
sub _formatData {
    my ( $inObjects, $inParams ) = @_;

    my @objects = @$inObjects;
   my $format = $inParams->{'format'} || $defaultFormat;
   my $separator = $inParams->{'separator'} || "\n";
   my @formattedData = ();
   foreach my $object (@object) {
      my $s = $format;
      ... perform string substitutions
      push @formattedData, $s;
   }
   my $outText = join $separator, @formattedData;
   return $outText;
}
</verbatim>


---+++ Additional functions

A useful utility function when you need to match values to a comma-separated string, is =makeHashFromString=.

<blockquote style="margin-left:0;">
==makeHashFromString( $text, $isCaseSensitive ) -> \%hash==
</blockquote>

For example:
<verbatim style="color:green;">
my $excludeTopicsList = 'WebHome, WebPreferences';
my $excludeTopics = makeHashFromString( $excludeTopicsList, 1 );
</verbatim>

... will create:

<verbatim>
$hashref = {
   'WebHome'        => 1,
   'WebPreferences' => 2,
};
</verbatim>

To store object data in a file, you may use =stringifyTopicData=:

<blockquote style="margin-left:0;">
==stringifyTopicData( $topicData ) -> \@objects==
</blockquote>

This function creates an array of strings from topic data objects, where each string is generated by the object's method =stringify= (to be implemented by your object's data class). 

For example, <nop>FormFieldData's =stringify= method looks like this:

<verbatim>
sub stringify {
    my $this = shift;
    return
      "1.0\t$this->{web}\t$this->{topic}\t$this->{name}\t$this->{value}\t$this->{date}";
}
</verbatim>

Call this method with:

<verbatim style="color:green;">
my $list = Foswiki::Plugins::TopicDataHelperPlugin::stringifyTopicData($topicData);
my $text = join "\n", @$list;
</verbatim>


---++ Plugin Installation Instructions
You do not need to install anything in the browser to use this extension. The following instructions are for the administrator who installs the extension on the server.

Open configure, and open the "Extensions" section. Use "Find More Extensions" to get a list of available extensions. Select "Install".

If you have any problems, or if the extension isn't available in =configure=, then you can still install manually from the command-line. See http://foswiki.org/Support/ManuallyInstallingExtensions for more help.



---++ Plugin Info
<!--
   * Set SHORTDESCRIPTION = helper plugin for collecting, filtering and sorting data objects
-->

|  Authors: | Foswiki:Main.ArthurClemens |
|  Copyright &copy;: | Foswiki:Main.ArthurClemens |
|  License: | [[http://www.gnu.org/copyleft/gpl.html][GPL]] |
|  Version: | 7039 (2010-04-01) |
|  Release: | 1.1.2 |
|  Change History: | <!-- versions below in reverse order --> |
|  01 Apr 2010 | V.1.1.2 Arthur Clemens: Allow topic data to be used without data structure. |
|  20 Jun 2009 | V.1.1.1 Arthur Clemens: Fixed reading of =web.topic= notation in =createTopicData=. |
|   8 Jun 2009 | V.1.1: Foswiki:Main.WillNorris: Ported to Foswiki |
|  24 Oct 2008 | V.1.0: Initial version |
|  Dependencies: | None |
|  Perl Version: | 5.005 |
|  Home: | http://foswiki.org/Extensions/%TOPIC% |
|  Support: | http://foswiki.org/Support/%TOPIC% |
Topic revision: r1 - 12 Apr 2005, ProjectContributor
System
Copyright © by the contributing authors. All material on this site is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback