You are here: Foswiki>Tasks Web>Item13442 (05 Jun 2015, CrawfordCurrie)Edit Attach

Item13442: Convert all Foswiki.org content to UTF-8

Priority: Urgent
Current State: Closed
Released In: n/a
Target Release: n/a

Applies To: Engine
Component:
Branches: master

Reported By: CrawfordCurrie
Waiting For:
Last Change By: CrawfordCurrie

As part of the forthcoming upgrade to Foswiki 1.2.0 on Foswiki.org, we need to convert existing content to UTF-8 and set {Site}{CharSet} appropriately.

It's clear from simple analysis that there are a mixture of encodings in use on the site, mainly where people have pasted encoded content into the text editor and the bytes have been trivially saved without re-encoding. The encodings detected are:

1464 strings are recognised as cp-1252
72 as UTF-8
30 are recognised as a different encoding
- Big5
- EUC-JP
- EUC-KR
- IBM866
- ISO-8859-5
- ISO-8859-7
- ISO-8859-8
- KOI8-R
- Shift_JIS
- gb18030
- windows-1251
- windows-1255
- x-mac-cyrillic

To support this change I have added a "repair" option to the CharSetConverterContrib.

-- CrawfordCurrie - 01 Jun 2015

Looking good (George did most of the work)

-- CrawfordCurrie - 05 Jun 2015

ItemTemplate edit

Summary	Convert all Foswiki.org content to UTF-8
ReportedBy	CrawfordCurrie
Codebase
SVN Range
AppliesTo	Engine
Component
Priority	Urgent
CurrentState	Closed
WaitingFor
Checkins	CharsetConverterContrib:6b98521f7ef4 CharsetConverterContrib:30521c49fffa CharsetConverterContrib:f14e681b9ed1 CharsetConverterContrib:b128469e88bd distro:15e9c34285f2 CharsetConverterContrib:30426f7109fb distro:d9d5dea27daa
TargetRelease	n/a
ReleasedIn	n/a
CheckinsOnBranches	master
trunkCheckins
masterCheckins	CharsetConverterContrib:6b98521f7ef4 CharsetConverterContrib:30521c49fffa CharsetConverterContrib:f14e681b9ed1 CharsetConverterContrib:b128469e88bd distro:15e9c34285f2 CharsetConverterContrib:30426f7109fb distro:d9d5dea27daa
ItemBranchCheckins
Release01x01Checkins