Reusable code that caches Foswiki topics, and provides fast searches of the content.
This module supports fast structured queries over topics in an arbitrarily-sized web. It uses a database to cache topic data to deliver much more scaleable search performance. Different back-end techologies can be used to implement the cache, to allow simple tradeoffs between scaling and raw performance.
Historically this module was designed to be used with a companion plugin, such as Foswiki:Extensions.FormQueryPlugin or Foswiki:Extensions.DBCachePlugin, which support queries and the display of query results. When used this way, the DBCacheContrib supports queries made using a simple query language.
The module can also be used in "standard schema mode" to support plugins that accelerate core functions, such as the QueryAcceleratorPlugin.
The query language supported by DBCacheContrib is very similar to the query language used with %SEARCH, which was derived from it. The contrib can be used in two modes; compatible schema mode (as used by FormQueryPlugin? and DBCachePlugin) and standard schema mode (as described in QuerySearch).
In standard schema mode the schema of the DB is as described in QuerySearch. In compatible schema mode, the underlying schema is somewhat different, to support the extended requirements of the plugins that use it.
You can think of the database as an map of all the topics in a web. Each entry is itself a map (or hash, in perl terms) that maps a set of field names to values.
Each topic in the web automatically gets a number of standard fields, generated by reading the metadata from the topic (see MetaData)
name
- name of the topic parent
- name of parent topic _up attachments
- array of maps, each of which contains: referencename
- attachment name attr
- e.g hidden comment
- attachment comment path
- client path used to upload attachment size
- size in Kb user
- who uploaded the attachment version
- e.g. 1.3 attachments info
- array of map containing: Maps, each of which contains: _up author
- most recent author reference to the Map for the topic name date
- attachment name date of last change attr format
- e.g hidden topic format version comment version
- attachment comment topic version number path
- client path used to upload attachment size
- size in Kb user
- who uploaded the attachment version
- e.g. 1.3 info moved
- Map map containing: _up by
- reference to the who moved it Map for the topic author date
- most recent author when they moved it date from
- date of last change where they moved it from format to
- topic format version where they moved it to version
- topic version number moved preferences
- Map containing: array of maps, each of which contains: _up name
- reference to preference name the Map for the topic by type
- who moved it either Set or Local date value
- when they moved it the value of the named preference from
- where they moved it from to
- where they moved it to form
- form type MyForm
. This is a reference to a Map map containing a key for each field in the form. Each key maps to the value in the form data for that key. The Map will also have an _up
reference to the Map for the topic. text
- raw text of the topic) The sub-maps created for info
, form name, moved
, and each row in attachments
also have a reference back to the topic Map, called _up
.
Other fields may be added by subclasses. Refer to the documentation for the plugin that is using the DBCache for more details.
Fields are given by name, and values by strings or numbers. Strings should always be surrounded by 'single-quotes'. Strings which are regular expressions (RHS of =, != =~ operators) use 'perl' regular expression syntax (google for perlre
for help). Numbers can be signed integers or decimals. Single quotes in values may be escaped using backslash (\).
The following operators are available:
Operator | Result | Meaning |
---|---|---|
= | Boolean | LHS exactly matches the regular expression on the RHS. The expression must match the whole string. |
!= | Boolean | Inverse of = |
=~ | Boolean | LHS contains RHS i.e. the RHS is found somewhere in the field value. |
< | Boolean | Numeric < |
> | Boolean | Numeric > |
>= | Boolean | Numeric >= |
<= | Boolean | Numeric <= |
lc | String | Unary lower case |
uc | String | Unary UPPER CASE |
IS_DATE | Boolean | Compare two dates e.g. '1 Apr 2003' IS_DATE '1 Apr 2004' |
EARLIER_THAN | Boolean | Date is earlier than the given date |
EARLIER_THAN_OR_ON | Boolean | Date is earlier than, or on, the given date |
LATER_THAN | Boolean | LHS is later than the given date |
LATER_THAN_OR_ON | Boolean | LHS is later than the given date |
WITHIN_DAYS | Boolean | Date (which must be in the future) is within n working days of todays date |
! | Boolean | Unary NOT |
AND | Boolean | AND |
OR | Boolean | OR |
() | any | Bracketed subexpression |
Dates for the date operators (IS_DATE
, EARLIER_THAN
etc) must be dates in the format expected by Time::ParseDate
(like the ActionTrackerPlugin). WITHIN_DAYS
works out the number of working days assuming a 5 day week (i.e. excluding Saturday and Sunday). Apologies in advance if your weekend is offset ± a day! Integers will automatically be converted to dates, by assuming they represent a number of seconds since midnight GMT on 1st January 1970. You can also use the d2n
operator to convert a date string to such an integer.
To achieve best perfomance the plugin caches the data read from topics in a database. The database is stored in the work area for the DBCacheContrib (see {WorkAreaDir}
in configure
). If any topic changes in the web, this cache is automatically rebuilt updated. (subject to configuration options). The cache can be deleted at any point with no ill effects.
Clients use the DBCache by defining a subclass of the Foswiki::Contrib::DBCacheContrib
class. Implementors are stongly recommended to read the POD documentation in the code:
You do not need to install anything in the browser to use this extension. The following instructions are for the administrator who installs the extension on the server.
Open configure, and open the "Extensions" section. Use "Find More Extensions" to get a list of available extensions. Select "Install".
If you have any problems, or if the extension isn't available in configure
, then you can still install manually from the command-line. See http://foswiki.org/Support/ManuallyInstallingExtensions for more help.
Author: | Foswiki:Main/CrawfordCurrie http://c-dot.co.uk, Foswiki:Main/MichaelDaum http://michaeldaumconsulting.com | ||||||||||||
Copyright ©: | This code is based on an original development of Motorola Inc. and is protected by the following copyrights: Copyright @copy 2002-2003 Motorola Inc. All Rights Reserved. Portions copyright © 2004. 2004 Crawford Currie http://www.c-dot.co.uk. Portions copyright © 2013 Michael Daum http://michaeldaumconsulting.com | ||||||||||||
License: | GPL | ||||||||||||
Release: | 17 Nov 2009 30 Jun 2013 | ||||||||||||
Version: | 5538 v3.0.0 (2009-11-17) | ||||||||||||
Change History: | |||||||||||||
17 Nov 2009 10 Jul 2013 | Foswikitask:Item8327 Foswikitask:Item12542 : series of robustness fixes (MD) cache non-standard %META data and make it searchable | ||||||||||||
30 Jun 2009 28 Mar 2013 | Foswikitask:Item8153 Foswikitask:Item12458 : make dirs for path to fix op_ref in search queries cache; Foswikitask:Item8194: incoporated patch from Foswiki:Main.MichaelDaum Foswikitask:Item8195: extract and cache permissions settings | ||||||||||||
18 Jun 2009 14 Mar 2013 | Foswikitask:Item8183 Foswikitask:Item12425 : fixed problem with Scalar::Util::weaken that was causing DBCachePlugin problems make preferences searchable by caching them into a map instead of an array | ||||||||||||
6 Jun 2009 25 Jan 2013 | Foswikitask:Item1691 Foswikitask:Item12369 : changes to support fixed loading cache from disk on a change Foswiki:Extensions.QueryAcceleratorPlugin | ||||||||||||
7 Apr 2009 07 Jan 2013 | Foswikitask:Item5440 Foswikitask:Item8195 : fixed negative values in SUMFIELD extract and cache preference settings; Foswikitask:Item8106 Foswikitask:Item12333 : add back implement an archivist caching a web in the segments; Map methods to the main class, as they are used by subclasses Foswikitask:Item8063: fix the accidental encoding of field values in the cache | ||||||||||||
28 Jan 2009 01 Oct 2012 | Foswikitask:Item453 Foswiktask:Item11752: don't fail to build the cache for formfield names with dots in it : Foswiki version; added Berkeley DB support, dropped plain-file support. Added Michael Daum's EARLIER_THAN_OR_ON and LATER_THAN_OR_ON ops. Fixed a number of bugs. | ||||||||||||
12346 10 Jan 2012 | fixing uptodate() for Andrew File Systems; fixed memory leak on persistent perl Foswikitask:Item11406: remove redundant reference to archivist from all stored values | ||||||||||||
16347 25 Aug 2011 | remove META data from text hash; include META data in all hash. Foswikitask:Item11070Foswiki:Main.MichaelDaum: working around odd defaults of normalizeWebTopicName | ||||||||||||
16346 28 Mar 2011 | caching all topic elements to an Foswikitask:Item9375all : disabling {AlwaysUpdateCache} by default and making an expert option with appropriate warnings field to allow th search in all of the text and the formfields like the normal grep-based SEARCH does. Foswiki:Main.MichaelDaum | ||||||||||||
15868 17 Nov 2009 | fixed WITHIN_DAYS and EARLIER_THAN. Foswikitask:Item8327: series of robustness fixes (Foswiki:Main.MichaelDaum) | ||||||||||||
15583 30 Jun 2009 | made query parser pluggable so that other plugins can implement their own predicates. Foswikitask:Item8153: make dirs for path to cache; Foswikitask:Item8194: incoporated patch from Foswiki:Main.MichaelDaum Foswikitask:Item8195: extract and cache permissions settings | ||||||||||||
15019 18 Jun 2009 | added {DBCache}{AlwaysUpdateCache} to remove the updateCache from every operation. Foswikitask:Item8183Foswiki:Main.SvenDowideit: fixed problem with Scalar::Util::weaken that was causing DBCachePlugin problems | ||||||||||||
13562 6 Jun 2009 | Bugs:Item3985 - fixed failures with hierarchical webs Foswikitask:Item1691: changes to support Foswiki:Extensions.QueryAcceleratorPlugin | ||||||||||||
13527 7 Apr 2009 | Moved the cache into the extensions work areas, instead of the web directory Foswikitask:Item5440: fixed negative values in SUMFIELD Foswikitask:Item8106: add back in the Map methods to the main class, as they are used by subclasses Foswikitask:Item8063: fix the accidental encoding of field values in the cache | ||||||||||||
12943 28 Jan 2009 | Bugs:Item3659: added automatic conversion of integers to dates Foswikitask:Item453: Foswiki version; added Berkeley DB support, dropped plain-file support. Added Michael Daum's EARLIER_THAN_OR_ON and LATER_THAN_OR_ON ops. Fixed a number of bugs. | ||||||||||||
12923 12346 | added REF operator; added link to fixing uptodate() for Andrew File Systems; web object to hashes; fixed parent relation to end in memory leak on persistent perl System; added "web" property to topic hashes; caching META:PREFERENCES now | ||||||||||||
11537 16347 | Added lc and uc operators for case-insensitive searches remove META data from text hash; include META data in all hash. Foswiki:Main.MichaelDaum | ||||||||||||
9303 16346 | TWikibug:Item1844 caching all topic elements to an - don't die on broken symlinks all field to allow th search in all of the text and the formfields like the normal grep-based SEARCH does. Foswiki:Main.MichaelDaum | ||||||||||||
8682 15868 | TWikibug:Item1580 fixed WITHIN_DAYS and EARLIER_THAN. - one-char fix that makes the difference Foswiki:Main.MichaelDaum | ||||||||||||
8110 15583 | TWikibug:Item663 made query parser pluggable so that other plugins can implement their own predicates. - formatting and text fixes Foswiki:Main.MichaelDaum | ||||||||||||
7552 15019 | TWikibug:Item997 added {DBCacheContrib}{AlwaysUpdateCache} to remove the updateCache from every operation. - test update Foswiki:Main.SvenDowideit | ||||||||||||
7274 13562 | TWikibug:Item719 Bugs:Item3985 - fixed failures with hierarchical webs - onReload() is not a static method. | ||||||||||||
7262 13527 | TWikibug:Item719 Moved the cache into the extensions work areas, instead of the web directory - Foswiki:Main.MichaelDaum's patch (almost) to correct parameters to onReload | ||||||||||||
7260 12943 | TWikibug:Item727 Bugs:Item3659: added automatic conversion of integers to dates - made it clean the form name using normaliseWebTopicName | ||||||||||||
6353 12923 | TWikibug:Item380 added REF operator; added link to web object to hashes; fixed parent relation to end in - do as the man says; make all $/ local System; added "web" property to topic hashes; caching META:PREFERENCES now | ||||||||||||
5720 11537 | Updated tests Added lc and uc operators for case-insensitive searches | ||||||||||||
5719 9303 | Fix for correct handling of parent relations TWikibug:Item1844 - don't die on broken symlinks | ||||||||||||
5229 8682 | Small improvement to the way it handles errors from Storable and Archive TWikibug:Item1580 - one-char fix that makes the difference | ||||||||||||
5223 8110 | Documentation fixes, adding gifs. TWikibug:Item663 - formatting and text fixes | ||||||||||||
5048 7552 | Cairo readiness TWikibug:Item997 - test update | ||||||||||||
5036 7274 | Split from SharedCode TWikibug:Item719 - onReload() is not a static method. | ||||||||||||
5031 7262 | Moving to new name TWikibug:Item719 - Foswiki:Main.MichaelDaum's patch (almost) to correct parameters to onReload | ||||||||||||
5030 7260 | About to rename TWikibug:Item727 - made it clean the form name using normaliseWebTopicName | ||||||||||||
5019 6353 | Improved topic data model, cleaned up tests TWikibug:Item380 - do as the man says; make all $/ local | ||||||||||||
5008 5720 | Added extended Updated tests access syntax, [?], [*] etc. | ||||||||||||
5006 5719 | Doc fixes Fix for correct handling of parent relations | ||||||||||||
5005 5229 | Poddified documentation Small improvement to the way it handles errors from Storable and Archive | ||||||||||||
5003 5223 | Initial version Documentation fixes, adding gifs. | ||||||||||||
8 5048 Jul 2004 | Initial version, Cairo readiness split out from FormQueryPlugin | ||||||||||||
Dependencies: 5036 |
| ||||||||||||
Perl 5031 Version: | 5.0 Moving to new name | ||||||||||||
5030 | About to rename | ||||||||||||
5019 | Improved topic data model, cleaned up tests | ||||||||||||
5008 | Added extended access syntax, [?], [*] etc. | ||||||||||||
5006 | Doc fixes | ||||||||||||
5005 | Poddified documentation | ||||||||||||
5003 | Initial version | ||||||||||||
8 Jul 2004 | Initial version, split out from FormQueryPlugin | ||||||||||||
Dependencies: |
| ||||||||||||
Home: | http://foswiki.org/Extensions/DBCacheContrib | ||||||||||||
Support: | http://foswiki.org/Support/DBCacheContrib |