Site Tools

Utilities for searching, navigation, and monitoring site activity

Site Tools include utilities for navigating, searching and keeping up with site activity.

In particular, you have two highly configurable, automated site monitoring tools, WebNotify, to e-mail alerts when topics are edited, and WebStatistics, to generate detailed activity reports.

WebNotify - recent changes alert

About the Web changes notification service

Each Foswiki web has an automatic e-mail notification service that sends you an e-mail with links to all of the topics modified since the last alert.

Users subscribe to email notifications using their WikiName or an alternative email address, and can specify the webs/topics they wish to track. Whole groups of users can also be subscribed for notification.

The general format of a subscription is:

three spaces * subscriber [ : topics ]

Where subscriber can be a WikiName, an email address, or a group name. If subscriber contains any characters that are not legal in an email address, then it must be enclosed in 'single' or "double" quotes. ALERT! Note: The guest user WikiGuest does not have an email address mapped to it, and will never receive email regardless of the configuraion of that user.

topics is an optional space-separated list of topics:
  • ... without a Web. prefix
  • ...that exist in this web.
Users may further customize the specific content they will receive using the following controls:
  • Using wild-card character in topic names - You can use * in a topic name, where it is treated as a wildcard character. A * will match zero or more other characters - so, for example, Fred* will match all topic names starting with Fred, *Fred will match all topic names ending with Fred, and * will match all topic names.
  • Unsubscribing to specific topics - Each topic may optionally be preceded by a '+' or '-' sign. The '+' sign means "subscribe to this topic". The '-' sign means "unsubscribe" or "don't send notifications regarding this particular topic". This allows users to elect to filter out certain topics. Topic filters ('-') take precedence over topic includes ('+') i.e. if you unsubscribe from a topic it will cancel out any subscriptions to that topic.
  • Including child-topics in subscription - Each topic may optionally be followed by an integer in parentheses, indicating the depth of the tree of children below that topic. Changes in all these children will be detected and reported along with changes to the topic itself. Note This uses the Foswiki "Topic parent" feature.
  • Subscribing to entire topic ("news mode") - Each topic may optionally be immediately followed by an exclamation mark ! and/or a question mark ? with no intervening spaces, indicating that the topic (and children if there is a tree depth specifier as well) should be mailed out as complete topics instead of change summaries. ! causes the full topic to be mailed every time even if there have been no changes, and ? will mail the full topic only if there have been changes. One can limit the content of the subscribed topic to send out by inserting %STARTPUBLISH% and %STOPPUBLISH% markers within the topic.

Examples: Subscribe Daisy to all changes to topics in this web.
Subscribe Daisy to all changes to topics that start with Web.
   * : Web*
Subscribe Daisy to changes to topics starting with Petal, and their immediate children, WeedKillers and children to a depth of 3, and all topics that match start with Pretty and end with Flowers e.g. PrettyPinkFlowers.
   * DaisyCutter: Petal* (1) WeedKillers (3) Pretty*Flowers
Subscribe StarTrekFan to changes to all topics that start with Star except those that end in Wars, sInTheirEyes or shipTroopers.
   * StarTrekFan: Star* - *Wars - *sInTheirEyes - *shipTroopers
Subscribe Daisy to the full content of NewsLetter whenever it has changed.
   * NewsLetter?
Subscribe buttercup to NewsLetter and its immediate children, even if it hasn't changed.
   * NewsLetter! (1)
Subscribe GardenGroup (which includes Petunia) to all changed topics under AllNewsLetters to a depth of 3. Then unsubscribe Petunia from the ManureNewsLetter, which she would normally get as a member of GardenGroup:
   * GardenGroup: AllNewsLetters? (3)
   * - ManureNewsLetter
Subscribe IT:admins (a non-Foswiki group defined by an alternate user mapping) to all changes to Web* topics.
   * 'IT:admins' : Web*

A user may be listed many times in the WebNotify topic. Where a user has several lines in WebNotify that all match the same topic, they will only be notified about changes in that topic once (though they will still receive individual mails for news topics).

If a group is listed for notification, the group will be recursively expanded to the email addresses of all members.

ALERT! Warning: Because an email address is not linked to a user name, there is no way for Foswiki to check access controls for subscribers identified by email addresses. A subscriber identified by an email address alone will only be sent change notifications if the topic they are subscribed to is readable by guest users. You can limit which email addresses can be used in WebNotify, or even block use of emails altogther, using the {MailerContrib}{EmailFilterIn} setting in configure.

TIP Tip: List names in alphabetical order to make it easier to find the names.

Note for System Administrators: Notification is supported by an add-on to the Foswiki kernel called the MailerContrib. See the MailerContrib topic for details of how to set up this service.

Note: If you prefer a news feed, point your reader to WebRss (for RSS 1.0 feeds) or WebAtom (for ATOM 1.0 feeds). Learn more at WebRssBase and WebAtomBase, respectively.

Related topics: Main.WikiUsers, UserRegistration

WebSearch - search the site

WebSearch is a flexible search facility, part of the core feature set. WebSearchAdvanced offers more options, including:
  • topic title or full-text search
  • regular expressions
  • query search over form data
  • search within web or site-wide
  • index-style A-Z alphabetical listing sorted topic title
  • many more

See also: SearchHelp for help; Macros and FormattedSearch for including hard-coded searches in text.

WebChanges - what's new

To check for the most recently edited topics while on-site, use the WebChanges link, usually located in the toolbar. It lists the most recently modified topics, newest first, along with the first couple of lines of the page content.

This is simply a preset SEARCH. The number of topics listed by the limit parameter.:
%SEARCH{ "1" web="%BASEWEB%" type="query" nosearch="on"
  order="modified" reverse="on" limit="50"

WebRss and WebAtom - news feeds on recent changes

You can point your news reader at WebRss and WebAtom to find out what is new in a web. WebRssBase and WebAtomBase have the details. Like WebChanges, this is based on a %SEARCH{}%.

WebIndex - list of topics

WebIndex lists all web topics in alphabetical order, with the first couple of lines of text. This is simply a preset SEARCH:
%SEARCH{"1" scope="topic" type="query" nosearch="on" }%

WebStatistics - site statistics

Statistics of visits to pages in a web can be generated manually or automatically, on a per-web basis. Statistics are compiled as a running total for each month. They include totals for Topic Views, Topic Saves, Attachment Uploads, Most Popular Topics with number of views, and Top Contributors showing total of saves and attachment uploads. Statistics from previous months are saved, and a new row is written to the table at the beginning of each month.

Configuring for automatic operation

  • You can automatically generate usage statistics for all webs. To enable this:
    • The WebStatistics topic must be present in all webs where you want to have statistics. You can use the topic in the Main web as a template.
    • Call the bin/statistics script from a cron job, once a day is recommended. This will update the WebStatistics topics in all webs.
    • Attention: The script must run as the same user as the CGI scripts are running, which is user nobody on many systems. Example crontab entry:
      0 0 * * * (cd /path/to/bin; ./statistics >/dev/null 2>&1)
    • There is a workaround in case you can't run the script as user nobody : Run the utility tools/ in your cron job and specify the URL of the bin/statistics script as a parameter. Example:
      0 0 * * * (cd /path/to/tools; ./ /urlpath/to/bin/statistics >/dev/null 2>&1)
    • NOTE: will do a CGI request as the WikiGuest user, so if you use this workaround, the WebStatistics topics you are updating will have to be writable by WikiGuest.

When running from the command line or a cron job, you can pass parameters to the script like this:
./statistics -logdate 200605 -webs Userweb,Sandbox

Generating statistics manually by URL

  • The bin/statistics script can also be executed as a CGI script, just enter the URL in your browser. Examples:
    • Update current month for all webs you have access to:
    • Update current month for Main web only:
    • Update current month for a SubWeb:
      Note: Trailing slash is required for Subweb to be recognized! Otherwise the script will assume Subweb is a topic and update the Parentweb web
    • Update May 2020 for Main web:
    • Update May 2020 for the ProjectX, ProjectY and ProjectZ webs:

Tailoring the WebStatistics layout

The Statistics topics %INCLUDE% a header topic System.DefaultWebStatistics. To tailor the help text or column heading, copy System.DefaultWebStatistics to Main.DefaultWebStatistics? and tailor as desired. This prevents your changes from being lost when Foswiki is updated.

If a WebStatistics topic does not exist, it will not be created automatically. Removal of the topic from a web will prevent statistics from being generated.

Statistics are written into the WebStatistics in the following order.
  • If a marker line is found in the topic, statistics are inserted into the topic after the marker line.
  • If no marker line is present, but the topic contains existing statistics, the current month is update, or a new month is written after the previous month.
  • If no marker line is present and no previous statistics exist, a new line is added to the end of the topic.

Removal of the marker line causes statistics to be gathered in chronological order (oldest first). By default, statistics are reported in reverse chronological order.

The marker line defines the order of the columns collected by the statistics script. The order of the columns should correspond to the table heading in the DefaultWebStatistics. Default marker line is:`;
| <!--statDate--> | <!--statViews--> | <!--statSaves--> | <!--statUploads--> | <!--statTopViews--> | <!--statTopContributors--> |

Managing the WebStatistics

WebStatistics will continue to grow. In addition every run of the statistics script will create a new entry in the %STATISTICSTOPIC% topic revision history. These can become extremely large over time and Foswiki does not provide any mechanism for automatically archiving statistics topics. For good ongoing performance, it is recommended to archive statistics annually:
  • Rename the WebStatistics to WebStatisticsYYYY
  • Restore the default topic with a copy of the data/_default/WebStatistics
Following these steps will start a fresh statistics topic with a clean revision history.

Log Files

Foswiki generates an event log which is used by the statistics script
  • The directory for the log file is defined by the {Log}{Dir} setting in configure
  • The file name is events.log Old events are archived as events.<year><month>
  • Example path name: working/logs/events.log
  • Each access gets logged as:
    | <time> | <wikiusername> | <action> | <web>.<topic> | <extra info> | <IP address> |
  • Example log entry:
    | 29 May 2020 - 06:37 | WikiGuest | view | WebRss |  | |
  • By default the following actions are logged:
    Script Action name Extra info
    attach attach when viewing attach screen of previous uploaded attachment: filename
    changes changes  
    edit edit when editing non-existing topic: (not exist)
    login login Authentication failure or success
    manage rename when moving topic: moved to Newweb.NewTopic
    manage move when moving attachment: Attachment filename moved to Newweb.NewTopic
    manage renameweb when renaming a web: oldweb moved to newweb
    rdiff rdiff higher and lower revision numbers: 4 3
    register regstart WikiUserName, e-Mail address, LoginName: user attempts to register
    register register E-mail address: user successfully registers
    register bulkregister WikiUserName of new, e-mail address, admin ID
    save save when replacing existing revision: repRev 3
    when user checks the minor changes box: dontNotify
    when user changes attributes to an exising attachment: filename.ext
    save cmd special admin parameter used when saving
    search search search string
    upload upload filename
    view view when viewing non-existing topic: (not exist)
    when viewing previous topic revision: r3
    n/a logout When authentication information is cleared
  • You can disable the logging (and therefore the reporting) of individual actions using the {Log}{Action} EXPERT setting in configure.


Configuring outgoing mail

Outgoing mail is required for UserRegistration and for recent changes alert.

The preferred place to configure e-mail is in the bin/configure tool (LocalSite.cfg). The configure tool fully documents these setting. Mail settings are found in the "Mail and Proxies" tab. Foswiki will use the Net::SMTP module if it is installed on your system.

The notify e-mail uses the default changes.tmpl template, or a skin if activated by a preference setting.

mailnotify also relies on a hidden file in each data/Web directory: .changes and a file with the Web name (one per web) in the working/work_areas/MailerContrib/ directory. Make sure both are writable by your web server process. .changes contains a list of changes; The file in the working/work_areas/MailerContrib/ directory contains a timestamp of the last time notification was done for the web. Both files are automatically created as needed.

Legacy and current configuration

ALERT! Caution: Older versions of Foswiki, and especially TWiki migrations might configure e-mail in the preference settings topic. Make sure you delete that setting if you are using a SitePreferences? topic from a previous release of Foswiki.
Parameter configure
preference settings
Default Description
Global enable {EnableEmail} n/a enabled Globally enable/disable email support.
Extermal mail program {MailProgram} n/a '/usr/sbin/sendmail -t -oi -oeq' E-mail program used when Net:SMTP is unavailable
Mail relay host {SMTP}{MAILHOST} SMTPMAILHOST not set Destination SMTP Server used to receive and relay email. Net::SMTP is disabled if this is not set.
Mail sending hostname {SMTP}{SENDERHOST} SMTPSENDERHOST not set Hostname used to identify sender. Some SMTP configurations will require this.
SMTP Password {SMTP}{Password} not supported not set

Setting the automatic e-mail schedule

For Unix platforms: Edit the cron table so that mailnotify is called in an interval of your choice. Please consult man crontab of how to modify the table that schedules program execution at certain intervals. Example:
% crontab -e
0 1 * * * (cd /path/to/bin; ./mailnotify -q)
The above line will run mailnotify nightly at 01:00. The -q switch suppresses all normal output.

For ISP installations: Many ISPs don't allow hosted accounts direct cron access, as it's often used for things that can heavily load the server. Workaround scripts are available.

On Windows: You can use a scheduled task if you have administrative privileges.

Site Permissions

  • AccessControl describes how to restrict read and write access to topics and webs, by users and groups
  • SitePermissions lists the permissions settings of the webs on this site

Help with crontab

The crontab command is used to schedule commands to be executed periodically.

Related Topics: AdminDocumentationCategory, AdminToolsCategory
Topic revision: r1 - 20 Sep 2010 - 08:56:42 - ProjectContributor

This site is powered by FoswikiCopyright © by the contributing authors. All material on this site is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback