Is this being done?

---

Not for each individual file, no. Though the whole release package has md5sums generated for it and uploaded, as do all plugins.

CC

---

Thanks.

(This was added in 6512)

MC

---

Oh. I wanted one for every file. Like this:

c238ca9a9048732e0d065a17b31ee7f1 data/.htpasswd
6f5e1f146240b668978ceca85b4d3638 pub/TWiki/TWikiDocGraphics/arrowbup.gif
b9cb41cf8d8b5c564e6ec81beb8ae430 lib/TWiki/Plugins.pm

http://twiki.org/p/pub/Plugins/TWikiReleaseTrackerPluginDev/betas.md5 has an extension of this - its a combo file for all files on all packages:

c238ca9a9048732e0d065a17b31ee7f1 TWiki20020803beta=twiki/data/.htpasswd
c238ca9a9048732e0d065a17b31ee7f1 TWiki20021210beta=twiki/data/.htpasswd
c238ca9a9048732e0d065a17b31ee7f1 TWiki20021217beta=twiki/data/.htpasswd
c238ca9a9048732e0d065a17b31ee7f1 TWiki20021229beta=twiki/data/.htpasswd
c238ca9a9048732e0d065a17b31ee7f1 TWiki20021230beta=twiki/data/.htpasswd
6f5e1f146240b668978ceca85b4d3638 TWiki20040507beta=twiki/pub/TWiki/TWikiDocGraphics/arrowbup.gif
b9cb41cf8d8b5c564e6ec81beb8ae430 TWiki20021230beta=twiki/lib/TWiki/Plugins.pm

This lets the TWiki:Plugins.TWikiReleaseTracker work out whether anyone has fiddled with any file in the install.

It keeps a central database of correct MD5s, of all releases, betas and all plugins, so you know exactly where your files come from.

• It knows which files have been tampered with.
• It knows what files should be there, so you know whether files have been added or deleted.
• It also matches by MD5, so you know which files have been renamed.

That all said, I don't particularly need it to be shipped with every install. I just need to keep a central database updated, so having a trigger notifying me is sufficient.

It took about a month's solid effort to build - it would be good to put it to use. I think it could really help with security.

MC

---

I know what you wanted, but unless someone else puts in the effort to

1. write a proper spec for this, and
2. code up building such a file from the build scripts,

it won't happen, because I'm not going to do it.

CC

---

The spec: very roughly, it looks like what I wrote above. A line for each file. Filename MD5. Optional packagename= before the file name. Not much more to say.

I built the code about a year ago. TWikiReleaseTrackerPlugin/lib/TWiki/Plugins/TWikiReleaseTrackerPlugin/IndexDistributions.pm, called from ./build.pl

After I've squashed the user Item452 I'll see if I can make it work again.

But where in the distro should I put this package? TWiki::Contrib? It creates an MD5 list for any tree. It seems a shame to turn it back into a script just so to squish it into the tools dir.

MC

---

We are not shipping the TWikiReleaseTracker, or any part thereof, in the release. The maximum that should be added to the release package is one file, that contains the md5sums for the entire release tree. I don't see a need for additional code to generate that file, since it can easily be generated using md5sum e.g.

md5sum `find . -type f -print` > <project>md5sums.md5

this file needs to be generated after the release is compiled into the staging area, and before the zips are built. The file name must be chosen not to conflict with the .md5 file, that contains the md5s for the zip and tgz.

Note that there are two places the sums could be generated; in tools/build.pl, in which case a single file is generated for the entire release; or in Build.pm in the BuildContrib, where they will automatically be generated for any plugin built using the BuildContrib (which these days is all of them). In either case, the generated filename must be chosen so that the md5sums from different packages don't overwrite eachother. My preference is for the latter (I prefer generic solutions).

CC

I don't care at all how the md5sums are made. I was simply pointing out a pure perl way to generate them.

Manic projects permitting, I'll take a look over the next couple of days.

MC

Ok. I've generated an index against every package in twikiplugins, these are saved in /md5sums.

If you think these are useful we can work out where to best put them and how to keep them up to date.

MC

If useful, I suggest we move the md5sum generation code into its own contrib package. It provides a facility to filter which files get in the file, it does this by walking a tree. Arguably we'd want to have it read the initial list from the MANIFEST file.

I've not generated for TWiki install packages though I have done so in the past.

results are in r6728

My code to build the indices is an uncommitted change to the TWikiReleaseTrackerPlugin. (the Config for which for DakarRelease can be markedly simplified).

MC

Hi Crawford, I've made those sys_action changes to tools/build.pl, plus done a lot to tidy the messaging.

More importantly I've built the MD5 stuff I wanted into BuildContrib.pm and tidied the resulting /tmp directory, but am hestitant to check it in as it is a critical part of all builds.

I currently generate a package name.md5 that contains, e.g.

e72d14c9c2df3d2c154c054b06f4e803  AUTHORS
e90b8036109dad942de7cd6812ec0e12  bin/attach
3536ef15c1d7e54eb43e95986becefb2  bin/changes
148c4b341707e5b02a8f8fe7d4c7391f  bin/configure

And a DEPS.md5 (temporary name) that contains MD5s from dependencies, e.g.

b5a91d7a2755198b2eb729541ad3288c  pub/TWiki/EditTablePlugin/menuarrow.gif
1ac9bfd77362c9a08a11ba9badd4d676  pub/TWiki/EditTablePlugin/release-notes.html
71fe0d007640c41d407d2d4bcc1b1c16  data/TWiki/HeadlinesPlugin.txt
41fc152eaf0e16aa2a6912b8205dbb4b  lib/TWiki/Plugins/HeadlinesPlugin.pm
e486af332bc66096a410ded634c840ea  lib/TWiki/Plugins/InterwikiPlugin.pm
c9440ec913b46caf49c498b0dceea3e6  data/TWiki/InterwikiPlugin.txt

As you know, the current SVN HEAD has simply a TWiki.md5 containing, e.g.:

b5a91d7a2755198b2eb729541ad3288c  TWiki.tgz
1ac9bfd77362c9a08a11ba9badd4d676  TWiki.zip

I think I need two names to for the MD5s, one for your package level security, and another for my file level security (containing both the package files and its deps). Do you agree?

If so, what should these two be called for any given package, p?

p.package.md5 and p.files.md5?

Please advise.

Alternatively I could check my changes in. I'd put it in scratch but I am too tired right now.

Cheers, M.

PS. It seems that it builds up the hierarchy of deps and then unlocks the files listed in the main manifest. I suspect this means that the plugins topics are not unlocked.

--

After checking things manually, things seem to work. No test case I could see. So...

twiki$ svn commit -m "Item437: Revised BuildContrib to build MD5s. Cr to ditch anything you don't like. (I sent you email about this a couple of days ago). MD5 files are generated for each package and then aggregated during a hands-off install to collect all dependent MD5s into package/DEPS.md5" twikiplugins/BuildContrib/lib/TWiki/Contrib/Build.pm
 tools/build.pl
Sending        tools/build.pl
Sending        twikiplugins/BuildContrib/lib/TWiki/Contrib/Build.pm
Transmitting file data ..
Committed revision 7207.

-- MC

Reverted in 7214 and 7215.

-- MC

Undeferred, post Dakar CC

---

There is no infrastructure to do this, or use the sums if they exist, so that would have to be developed.

CC

Cleared checkins - they are all against the TWiki SVN repo. Otherwise shows up as a work in process.