Difference: DBCacheContrib (r2 vs. r1)

DBCacheContrib

Reusable code that caches Foswiki topics, and provides fast searches of the content.

Summary of Contents

This module supports fast structured queries over topics in an arbitrarily-sized web. It uses a database to cache topic data to deliver much more scaleable search performance. Different back-end techologies can be used to implement the cache, to allow simple tradeoffs between scaling and raw performance.

Historically this module was designed to be used with a companion plugin, such as Foswiki:Extensions.FormQueryPlugin or Foswiki:Extensions.DBCachePlugin, which support queries and the display of query results. When used this way, the DBCacheContrib supports queries made using a simple query language.

The module can also be used in "standard schema mode" to support plugins that accelerate core functions, such as the QueryAcceleratorPlugin.

Features

  • Perform efficient structured queries on data in forms

Using the built-in query language

The query language supported by DBCacheContrib is very similar to the query language used with %SEARCH, which was derived from it. The contrib can be used in two modes; compatible schema mode (as used by FormQueryPlugin? and DBCachePlugin) and standard schema mode (as described in QuerySearch).

In standard schema mode the schema of the DB is as described in QuerySearch. In compatible schema mode, the underlying schema is somewhat different, to support the extended requirements of the plugins that use it.

The Compatible Schema

You can think of the database as an map of all the topics in a web. Each entry is itself a map (or hash, in perl terms) that maps a set of field names to values.

Each topic in the web automatically gets a number of standard fields, generated by reading the metadata from the topic (see MetaData)

  • name - name of the topic
  • parent - name of parent topic
  • _up attachments - array of maps, each of which contains: reference
    • name - attachment name
    • attr - e.g hidden
    • comment - attachment comment
    • path - client path used to upload attachment
    • size - size in Kb
    • user - who uploaded the attachment
    • version - e.g. 1.3
    to the Map of the parent topic, if it exists
  • attachments info - array of map containing: Maps, each of which contains:
    • _up author - most recent author reference to the Map for the topic
    • name date - attachment name date of last change
    • attr format - e.g hidden topic format version
    • comment version - attachment comment topic version number
    • path - client path used to upload attachment
    • size - size in Kb
    • user - who uploaded the attachment
    • version - e.g. 1.3
  • info moved - Map map containing:
    • _up by - reference to the who moved it Map for the topic
    • author date - most recent author when they moved it
    • date from - date of last change where they moved it from
    • format to - topic format version where they moved it to
    • version - topic version number
  • moved preferences - Map containing: array of maps, each of which contains:
    • _up name - reference to preference name the Map for the topic
    • by type - who moved it either Set or Local
    • date value - when they moved it the value of the named preference
    • from - where they moved it from
    • to - where they moved it to
  • form - form type
  • form name - e.g. if a "MyForm" is attached, this will be MyForm. This is a reference to a Map map containing a key for each field in the form. Each key maps to the value in the form data for that key. The Map will also have an _up reference to the Map for the topic.
  • text - raw text of the topic)

The sub-maps created for info, form name, moved, and each row in attachments also have a reference back to the topic Map, called _up.

Other fields may be added by subclasses. Refer to the documentation for the plugin that is using the DBCache for more details.

Query operators

Fields are given by name, and values by strings or numbers. Strings should always be surrounded by 'single-quotes'. Strings which are regular expressions (RHS of =, != =~ operators) use 'perl' regular expression syntax (google for perlre for help). Numbers can be signed integers or decimals. Single quotes in values may be escaped using backslash (\).

The following operators are available:

OperatorResultMeaning
= Boolean LHS exactly matches the regular expression on the RHS. The expression must match the whole string.
!= Boolean Inverse of =
=~ Boolean LHS contains RHS i.e. the RHS is found somewhere in the field value.
< Boolean Numeric <
> Boolean Numeric >
>= Boolean Numeric >=
<= Boolean Numeric <=
lc String Unary lower case
uc String Unary UPPER CASE
IS_DATE Boolean Compare two dates e.g. '1 Apr 2003' IS_DATE '1 Apr 2004'
EARLIER_THAN Boolean Date is earlier than the given date
EARLIER_THAN_OR_ON Boolean Date is earlier than, or on, the given date
LATER_THAN Boolean LHS is later than the given date
LATER_THAN_OR_ON Boolean LHS is later than the given date
WITHIN_DAYS Boolean Date (which must be in the future) is within n working days of todays date
! Boolean Unary NOT
AND Boolean AND
OR Boolean OR
() any Bracketed subexpression

Dates for the date operators (IS_DATE, EARLIER_THAN etc) must be dates in the format expected by Time::ParseDate (like the ActionTrackerPlugin). WITHIN_DAYS works out the number of working days assuming a 5 day week (i.e. excluding Saturday and Sunday). Apologies in advance if your weekend is offset ± a day! Integers will automatically be converted to dates, by assuming they represent a number of seconds since midnight GMT on 1st January 1970. You can also use the d2n operator to convert a date string to such an integer.

The cache

To achieve best perfomance the plugin caches the data read from topics in a database. The database is stored in the work area for the DBCacheContrib (see {WorkAreaDir} in configure). If any topic changes in the web, this cache is automatically rebuilt updated. (subject to configuration options). The cache can be deleted at any point with no ill effects.

Detailed Documentation

Clients use the DBCache by defining a subclass of the Foswiki::Contrib::DBCacheContrib class. Implementors are stongly recommended to read the POD documentation in the code:

Installation Instructions

You do not need to install anything in the browser to use this extension. The following instructions are for the administrator who installs the extension on the server.

Open configure, and open the "Extensions" section. Use "Find More Extensions" to get a list of available extensions. Select "Install".

If you have any problems, or if the extension isn't available in configure, then you can still install manually from the command-line. See http://foswiki.org/Support/ManuallyInstallingExtensions for more help.

Contrib Info

Author: Foswiki:Main/CrawfordCurrie http://c-dot.co.uk, Foswiki:Main/MichaelDaum http://michaeldaumconsulting.com
Copyright ©: This code is based on an original development of Motorola Inc. and is protected by the following copyrights:
Copyright @copy 2002-2003 Motorola Inc. All Rights Reserved.
Portions copyright © 2004. 2004 Crawford Currie http://www.c-dot.co.uk.
Portions copyright © 2013 Michael Daum http://michaeldaumconsulting.com
License: GPL
Release: 17 Nov 2009 30 Jun 2013
Version: 5538 v3.0.0 (2009-11-17)
Change History:  
17 Nov 2009 10 Jul 2013 Foswikitask:Item8327 Foswikitask:Item12542 : series of robustness fixes (MD) cache non-standard %META data and make it searchable
30 Jun 2009 28 Mar 2013 Foswikitask:Item8153 Foswikitask:Item12458 : make dirs for path to fix op_ref in search queries cache; Foswikitask:Item8194: incoporated patch from Foswiki:Main.MichaelDaum Foswikitask:Item8195: extract and cache permissions settings
18 Jun 2009 14 Mar 2013 Foswikitask:Item8183 Foswikitask:Item12425 : fixed problem with Scalar::Util::weaken that was causing DBCachePlugin problems make preferences searchable by caching them into a map instead of an array
6 Jun 2009 25 Jan 2013 Foswikitask:Item1691 Foswikitask:Item12369 : changes to support fixed loading cache from disk on a change Foswiki:Extensions.QueryAcceleratorPlugin
7 Apr 2009 07 Jan 2013 Foswikitask:Item5440 Foswikitask:Item8195 : fixed negative values in SUMFIELD extract and cache preference settings; Foswikitask:Item8106 Foswikitask:Item12333 : add back implement an archivist caching a web in the segments; Map methods to the main class, as they are used by subclasses Foswikitask:Item8063: fix the accidental encoding of field values in the cache
28 Jan 2009 01 Oct 2012 Foswikitask:Item453 Foswiktask:Item11752: don't fail to build the cache for formfield names with dots in it : Foswiki version; added Berkeley DB support, dropped plain-file support. Added Michael Daum's EARLIER_THAN_OR_ON and LATER_THAN_OR_ON ops. Fixed a number of bugs.
12346 10 Jan 2012 fixing uptodate() for Andrew File Systems; fixed memory leak on persistent perl Foswikitask:Item11406: remove redundant reference to archivist from all stored values
16347 25 Aug 2011 remove META data from text hash; include META data in all hash. Foswikitask:Item11070Foswiki:Main.MichaelDaum: working around odd defaults of normalizeWebTopicName
16346 28 Mar 2011 caching all topic elements to an Foswikitask:Item9375all: disabling {AlwaysUpdateCache} by default and making an expert option with appropriate warnings field to allow th search in all of the text and the formfields like the normal grep-based SEARCH does. Foswiki:Main.MichaelDaum
15868 17 Nov 2009 fixed WITHIN_DAYS and EARLIER_THAN. Foswikitask:Item8327: series of robustness fixes (Foswiki:Main.MichaelDaum)
15583 30 Jun 2009 made query parser pluggable so that other plugins can implement their own predicates. Foswikitask:Item8153: make dirs for path to cache; Foswikitask:Item8194: incoporated patch from Foswiki:Main.MichaelDaum Foswikitask:Item8195: extract and cache permissions settings
15019 18 Jun 2009 added {DBCache}{AlwaysUpdateCache} to remove the updateCache from every operation. Foswikitask:Item8183Foswiki:Main.SvenDowideit: fixed problem with Scalar::Util::weaken that was causing DBCachePlugin problems
13562 6 Jun 2009 Bugs:Item3985 - fixed failures with hierarchical webs Foswikitask:Item1691: changes to support Foswiki:Extensions.QueryAcceleratorPlugin
13527 7 Apr 2009 Moved the cache into the extensions work areas, instead of the web directory Foswikitask:Item5440: fixed negative values in SUMFIELD Foswikitask:Item8106: add back in the Map methods to the main class, as they are used by subclasses Foswikitask:Item8063: fix the accidental encoding of field values in the cache
12943 28 Jan 2009 Bugs:Item3659: added automatic conversion of integers to dates Foswikitask:Item453: Foswiki version; added Berkeley DB support, dropped plain-file support. Added Michael Daum's EARLIER_THAN_OR_ON and LATER_THAN_OR_ON ops. Fixed a number of bugs.
12923 12346 added REF operator; added link to fixing uptodate() for Andrew File Systems; web object to hashes; fixed parent relation to end in memory leak on persistent perl System; added "web" property to topic hashes; caching META:PREFERENCES now
11537 16347 Added lc and uc operators for case-insensitive searches remove META data from text hash; include META data in all hash. Foswiki:Main.MichaelDaum
9303 16346 TWikibug:Item1844 caching all topic elements to an - don't die on broken symlinks all field to allow th search in all of the text and the formfields like the normal grep-based SEARCH does. Foswiki:Main.MichaelDaum
8682 15868 TWikibug:Item1580 fixed WITHIN_DAYS and EARLIER_THAN. - one-char fix that makes the difference Foswiki:Main.MichaelDaum
8110 15583 TWikibug:Item663 made query parser pluggable so that other plugins can implement their own predicates. - formatting and text fixes Foswiki:Main.MichaelDaum
7552 15019 TWikibug:Item997 added {DBCacheContrib}{AlwaysUpdateCache} to remove the updateCache from every operation. - test update Foswiki:Main.SvenDowideit
7274 13562 TWikibug:Item719 Bugs:Item3985 - fixed failures with hierarchical webs - onReload() is not a static method.
7262 13527 TWikibug:Item719 Moved the cache into the extensions work areas, instead of the web directory - Foswiki:Main.MichaelDaum's patch (almost) to correct parameters to onReload
7260 12943 TWikibug:Item727 Bugs:Item3659: added automatic conversion of integers to dates - made it clean the form name using normaliseWebTopicName
6353 12923 TWikibug:Item380 added REF operator; added link to web object to hashes; fixed parent relation to end in - do as the man says; make all $/ local System; added "web" property to topic hashes; caching META:PREFERENCES now
5720 11537 Updated tests Added lc and uc operators for case-insensitive searches
5719 9303 Fix for correct handling of parent relations TWikibug:Item1844 - don't die on broken symlinks
5229 8682 Small improvement to the way it handles errors from Storable and Archive TWikibug:Item1580 - one-char fix that makes the difference
5223 8110 Documentation fixes, adding gifs. TWikibug:Item663 - formatting and text fixes
5048 7552 Cairo readiness TWikibug:Item997 - test update
5036 7274 Split from SharedCode TWikibug:Item719 - onReload() is not a static method.
5031 7262 Moving to new name TWikibug:Item719 - Foswiki:Main.MichaelDaum's patch (almost) to correct parameters to onReload
5030 7260 About to rename TWikibug:Item727 - made it clean the form name using normaliseWebTopicName
5019 6353 Improved topic data model, cleaned up tests TWikibug:Item380 - do as the man says; make all $/ local
5008 5720 Added extended Updated tests access syntax, [?], [*] etc.
5006 5719 Doc fixes Fix for correct handling of parent relations
5005 5229 Poddified documentation Small improvement to the way it handles errors from Storable and Archive
5003 5223 Initial version Documentation fixes, adding gifs.
8 5048 Jul 2004 Initial version, Cairo readiness split out from FormQueryPlugin
Dependencies: 5036
NameVersionDescription
Time::ParseDate>=2003.0211Required. Available from CPAN.
Storable>=2.07Required. available from CPAN
BerkeleyDB>=0Recommended for sites with many topics. Available from CPAN
Split from SharedCode
Perl 5031 Version: 5.0 Moving to new name
5030 About to rename
5019 Improved topic data model, cleaned up tests
5008 Added extended access syntax, [?], [*] etc.
5006 Doc fixes
5005 Poddified documentation
5003 Initial version
8 Jul 2004 Initial version, split out from FormQueryPlugin
Dependencies:
NameVersionDescription
Time::ParseDate>=2003.0211Required.
Storable>=2.07Required.
BerkeleyDB>=0Optional, still experimental.
Home: http://foswiki.org/Extensions/DBCacheContrib
Support: http://foswiki.org/Support/DBCacheContrib

View topic | View difference side by side | History: r4 < r3 < r2 < r1 | More topic actions
 
This site is powered by FoswikiCopyright © by the contributing authors. All material on this site is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback