Foswiki for non-wiki web sites

My experience

I have been using Foswiki to build a small scale personal website, https://colas.nahaboo.net (not a lot of contents yet) open to the web, but only edited by me. Although at the opposite of Foswiki original natural ecosystem of collaborative intranets, it is an interesting experience. I will dump here what are my ideas about this use case, and star () what I think could be important feature of a "public web site mode".

• Motivation: it was just that I knew Foswiki TML (Syntax) well, and it was the easiest path for me. However, as I use more and more the Markdown syntax elsewhere, I think that being able to write topics in Markdown will be more and more important. The MarkdownPlugin is a very good start, and I plan to see how I can edit topics in Markdown directly from a desktop MD editor like Marktext, as I do not have the constraints of concurrent editing, and editors unwilling/unable to set up desktop tools.
  • I think being able to write Markdown natively in Foswiki, with a way to use existing Foswiki Macros could be important for the future of Foswiki
  • Desktop editing (as for blogs) would be nice
• Robots: The first thing that hits you on the web are robots, especially if, like me, your web site have been on the web for some time. Dozens of evil crawlers ignoring the robots.txt, or even masquerading their User-Agent string to avoid detection will bring your server to its knees in no time. thus:
  • You will be monitoring your logs often to detect bad bots, and add them to the blockAccess section of the server configuration
  • A honeypot is a very simple and effective tool to fight bad bots
  • In the public view, there should be no links that robots can follow and trigger thousands of inane requests with just GET requests. E.g., disable the sorting of table rows by clicking on table headers, the links to previous revisions, the various views (raw, print, ...)
  • As a simple solution, these "internal" links that do not link to new contents, but the same contents presented differently should not ne real links, but HTML elements with a javascript "onclick" handler, so robots cannot follow them, only humans. But care should be taken to use a method usable for people with accessibility issues (e.g. browsers for blind people)
  • anonymous (guest) browsing should not consume resources, e.g allocate cookies
• Public/Private: I ended up having a skin that is widely different if you are logged in (me) or not (the rest of the world). For security reasons you do not want to allow strangers to log in on your site, as Foswiki offers too much power to a first-class user, for instance to leave comments. Plus there are parts you want to hide. If you edit an article to remove some wrong information, you do not want the old version be still viewable by the public or Google. I guess "being logged in" could be a scope/context for PreferenceValuesInDifferentContexts
  • The Foswiki engine should offer the option to behave differently if the user is logged in or not, at least by having two different skins (SKIN + PUBLICSKIN).
    Actually it is doable already! Just set the public skin in a "* Set SKIN = publicskin" in the WikiGuest topic!
  • The public skin should remove all elements that make no sense if you are not able to log in: Edit buttons, but also everything that can confuse the user by showing lots of info irrelevant for his case: history, subscribe, ... Ideally there should subsist only a tiny hidden link somewhere to log in.
• Contents/Engine: users are only interested in your actual contents. They do not want to see the "wiki plumbing". I ended up using TopicTitlePlugin to separate Contents topics (the ones having a TOPICTITLE variable defined) from the rest. And the main search engine in the banner only searches these Contents topics, the topic lists (by date or name) only show the Contents topics, etc.. not WebChanges, for instance. The Jump to topic autocomplete only lists Contents topics.
  • Topics should be divided in two classes: Contents and System, and tools such as search should be able to operate on only one class, they could have different skins, etc...
• Comments: Users should not have to log in to comment. Use of a 3rd-party somment system added client-side by Javascript on the Contents pages (like Discus) is thus a good solution. I use remark42, but many others exist: Commento, Isso, Schnack,Staticman,Valine to speak only of the self-hostable ones
  • Use an external comment system
• Tags: are mandatory. But they must be simple and fast tags, global and not per-user, as they will be set by the site editor but used by readers. Not the current tag plugins, so I had to just use a basic form field + %SEARChes for this.
  • A simple fast universal tag system
• Minimalism: Foswiki feels slow to the users, because of the huge number of auxillary pages (CSS, JS, images, libs (jquery+widgets)...). Also, it is mandatory for a good SEO that the page loads fast enough for Google. But this is currently impossible to obtain with Foswiki, as even with a bare minimal skin, plugins used in standard topics (such as WebSearch) force tons of JQuery code into the headers, in a way that blocks the rendering and lowers dramatically the compliance score with Google.
  
  • The public skin should ideally include one simple CSS and JS page, without the bloat induced to trying to work on old browsers (e.g: IE) by using all the features of HTML 5 and CSS 3 (or even 4). Datasets, CSS variables, etc...
  • but it should work fine on smartphones, at least in public view mode.
  • the core foswiki should not depend on JQuery, and not inject JS in a way that blocks the rendering

• Caching: Caching is mandatory, both for performance (e.g. RSS feeds), and get a good ranking on Google. We should ditch all the various caches to focalise on using only an efficient one on the front end (I use Apache mod_disk_cache, but Varnish is also nice), and for this I had to patch Foswiki (disabling setCacheControl in lib/Foswiki.pm) to stop emitting a Cache-Control header, setting the Cache-Control header via the skin and HttpHeaderPlugin to no-cache in logged in mode, and some hours in public (guest) mode.
  • emit a long Cache-Control: max-age %CACHEMAXAGE% if user is not logged in, and Cache-Control: no-cache if logged in

I think that Foswiki thus can make a very decent, powerful and pleasant small-to medium scale CMS. However, I will probably now begin to write my own system from scratch, to go further in the Desktop editing + minimalism directons. But I will continue to use Foswiki for my internal famillial collaborative site, but maybe with some experiments to develop Markdown use + desktop editing + Content topics.

Alternatives

There could be however very different ways to tackle this issue.

• Publishing: One could use Foswiki as a totally hidden backoffice system, with a publishing system, similar to PublishPlugin, to deploy the site as simple HTML pages, or onto a static web site engine, or an efficient CMS

-- ColasNahaboo, 2020-12-05

Example

Here is a view on my Public skin (the plus signs "+" link to more detailed pages, eg "Advanced search" for search). I got rid of the bottom toolbar entierely, and I am even looking at removing the side bar totally.

Here the logged-in view, with all Foswiki power:

Comments

Markdown: I am quite ambivalent about markdown. Pro: lots of projects are actively using it and developing software for it. i.e. CKEdit 5 has got a markdown build that lets you edit markdown in wysiwyg mode directy, that is you spare the bl*dy WysiwygPlugin having to translate HTML to markdown. Cons: markdown is not standardized despite all of the efforts. Most importantly in the area images and tables. MediaWiki markup and Foswiki TML are more stable/standardized/powerfull compared to markdown.

Robots: arent such a problem anymore as far as I see. You need fail2ban or CrowdSec for one. I tend to hide all of System using AutoRedirectPlugin. PatternSkin is quite bad with regards to robots as you already mentioned. So is TablePlugin.

SEO: is more important than robots. You need proper meta data to make pages share-able on social networks

Mobile: This is a must-have. You definitely need to provide a good user experience on mobile deviecs. SEO depends on it too.

Comments: There are two commenting systems available that let you implement anonymous commenting: MetaCommentPlugin and DisqusPlugin

Tags: The best tagging ... and category system available is ClassificationPlugin

Content: BlogPlugin lets you cover most of what you would do with WordPress in that area, such as target audience, scheduled publishing/unpublishing, multi-authored blog postings, categories, tags, commenting, seo metadata, etc. See our own Foswiki blog for instance.

Caching: using Varnish is definitely something I always wanted to do as well. However Foswiki's own PageCaching was already good enough lowering the pain points below a certain degree that I would feel the urge to go for Varnish or the like.

Design: It is mandatory to come up with a modern design. Not funky, just modern. Definitely not PatternSkin-ish as thats too 80ies.

There are a couple of good examples using Foswiki in that direction:

• https://rocketboards.org
• https://atomikos.com
• https://blog.foswiki.org

Lots more behind firewalls. I am running it as a non-wiki site for my own homepage since the beginning. SEO is the only real problem ... and probably always will be given the moving standards in this area.

-- MichaelDaum - 05 Dec 2020

Very interesting comments, Michael!
What I would add:

Markdown is just there, I now use it all the time (with marktext). And I only use the GitHub version (or rather, the subset understood by pandoc gfm and marktext) which has become the de facto standard. This is the one of the main reasons I am considering moving away from Foswiki.

Robots: I use fail2ban, quite powerful. But removing all uneeded links beforehand is safer than having to dwelve into complex fail2ban rules.

Comments: Disqus is not reliable, it can disappear at any time, self hosted comment engines are much better. I was using JS-KIT, a predecessor of Discus, but it disappeared with all my comments.

Tags: Your plugin system is nice, I started to use it, but has it had too much dependencies on your underlying framework and was harder to cache or integrate into a faceted search than a simpler system. I wanted to be keep open the possibility to migrate out of Foswiki some day.

Blog: the previous version of my site (last 10 years... time flies) was actually using a variant of the BlogPlugin

Caching: Varnish is nice, but you cannot use it on only some Virtual hosts. The Apache disk cache is both ultra efficient (at the very front end), simple and standard. PHP devs tend to use varnish because often their PHP systems (Wordpress) are not good web citizens and cannot be efficiently cached by Apache

Basically, I have been in a trip to get rid of everything and go back to the basics: just Apache, Linux, and HTML5 and CSS4. And it is incredible how this is powerful in 2020 and you can get rid of 99% of the recent web frameworks that have grown organically over the years to remedy to the braindeadness of IE. For instance, I will base myself on the Apache cache, and ditch technologies that do not work well with it or try to re-invent this wheel, but less efficiently. And my use of Foswiki does not fit anymore in this personal crusade

-- ColasNahaboo - 05 Dec 2020

Maybe it is time to make Foswiki's markup engine configurable in order to be able to adopt markdown alongside TML. Anybody interested diving into it?

-- MichaelDaum - 05 Dec 2020

ClassificationPlugin is indeed integrated into SolrPlugin to implement a facetted tag search.

-- MichaelDaum - 05 Dec 2020

Edited my top post to mention:

• that the public skin can already be set as a SKIN preference setting in the WikiGuest topic.
• that linking-via-javascript can be a solution to prevent crawlers to follow some links

Also I am now experimentally writing new articles in Markdown, with a desktop editor, with the following hack, which seems to work nicely for my use case (always editing from the same computer). This is what I do now ,and plan to do:

• by using the MarkdownPlugin and writing inside %<nop>STARTMARKDOWN% / %<nop>STOPMARKDOWN% so that Foswiki search can find the contents
• by remote-editing topic with a desktop MD editor such as Marktext: in the text have a edit link to a server CGI script that prints the url of the topic ($REQUEST_URI) with Content-Type:foswiki/md
• have this link only show if user is logged in (it could be set in the private skin)
• tell Firefox to run the downloaded files of type foswiki/md with alocal client script that fetches the topic via rsync on the server, edit it locally with marktext, and re-upload it via rsync (+ a call to bin/save to version it). Auth is then handled by rsync/ssh, no need to take care in the cgi script.
• todo: the scripts should extract the MD part to give it to marktext, and re-inject it in the topic after save
• todo: the server script should lock the topic (".lease") to prevent concurrent editing
• todo: do the same for included SVG drawings attachments, by calling a desktop editor on their attached source files (draw.io, Libreoffice, inkscape, ..), exporting the save source format it to svg and re-upload them (source + svg) as attachment

I realised that these Foswiki/MD hybrid topics were actually comprised of the contents in MD and metadata (topicinfo, form meta, topictitle prref var...). And it is suspiciously similar to what most web engines using markdown are doing: they manage not pure MD pages, but pages with a "front matter", a header of metadata written most frequently in YAML, but also TOML, JSON, ... (see for instance Hugo). I wonder if Foswii could become an engine with MD pages as contents but TML for the "front matter" and "glue" or "engine" content-less pages such as search, backlinks, changes, ...

-- ColasNahaboo - 07 Dec 2020

Maybe WebDAVContrib could be interesting for your: it allows to mount Foswiki as a network drive and edit files directly in your favorite editor. Deals with locking as well.

-- MichaelDaum - 07 Dec 2020

Yes Webdav is interesting, but rsync is lighter and faster as it only transmits the diffs. Plus my scripts can then trigger side effects like rebuilding some search indexes, invalidating some caches, ...

Also, with my solution I have the convenience of just clicking a "edit" button on the page itself, rather than navigating a filesystem to find back my topic. Incidentally this is seen as the major drawback of the current static website generators, and that make some people revert back to dynamic sites: being able to "edit in place".

But it is an interesting idea, I will try it!

-- ColasNahaboo - 07 Dec 2020

I now use a preference variable on topics TOPICFORMAT = MD, and display on them an "Edit MD" button instead of the normal Wysiwyg "Edit" (*) button. Next step is to tweak the Markdown plugin to render topics as MD if they have this pref set, minus any %META: lines, so that I do not have to add the START/STOPMARKDOWN manually to it.

(*) via code like: %IF{"context authenticated and $TOPICFORMAT='MD'"...

-- ColasNahaboo - 07 Dec 2020

I now use for all anchors ("A" elements) that link to a different way to present the information, but not to information not already available in the sitemap, a "defanged" version that crawlers cannot follow, e.g:

<a class=href data-href="url...">

Then, after the page loaded, I have some javascript code that copies the data-href attribute into an href attribute, thus making the link cliquable for humans. It also works for accessible browsers, if you use a <body aria-live=off directive. The code:

function anchors_hrefs() {
  var anchors = document.getElementsByClassName('href');
  for(var i = 0; i < anchors.length; i++) {
    var anchor=anchors[i];
    if (anchor.hasAttribute('data-href')) {
      anchor.href = anchor.dataset.href;
    }
  }
};

For instance, I have defined a searchformat.tmpl for my skin that uses non-crawlable anchors for the prev/next arrows to paginate search results:

%TMPL:DEF{SEARCH:pager_previous}%<a class=href data-href='$previousurl'>< %MAKETEXT{"Previous"}%</a>%TMPL:END%
%TMPL:DEF{SEARCH:pager_next}%<a class=href data-href='$nexturl'>%MAKETEXT{"Next"}% ></a>%TMPL:END%

PS: my site is now live at https://colas.nahaboo.net Contents is ridiculously poor now, I am prepping the engine to start writing contents when I retire in March. The goal for me is to be able to write contents in Markdown contents + frontmatter yaml metadata, so that I can transition to another web engine in the future, most probably one that I will write myself, as it has been a itching me for a very long time. Ideally I could thus reuse the contents nearly as-is on this future engine. And I may even keep Foswiki as the backoffice, publishing on the web directly or via a static web generator.

-- ColasNahaboo - 12 Dec 2020

Commenting is disabled while not logged in