This is the mail archive of the
cygwin-apps-cvs
mailing list for the cygwin-apps project.
[calm - Cygwin server-side packaging maintenance script] branch master, updated. ac6a34563121d96f19e7f4c88ea40c203ea84c19
- From: jturney at sourceware dot org
- To: cygwin-apps-cvs at sourceware dot org
- Date: 29 Mar 2016 11:28:26 -0000
- Subject: [calm - Cygwin server-side packaging maintenance script] branch master, updated. ac6a34563121d96f19e7f4c88ea40c203ea84c19
https://sourceware.org/git/gitweb.cgi?p=cygwin-apps/calm.git;h=ac6a34563121d96f19e7f4c88ea40c203ea84c19
commit ac6a34563121d96f19e7f4c88ea40c203ea84c19
Author: Jon Turney <jon.turney@dronecode.org.uk>
Date: Tue Mar 29 12:21:56 2016 +0100
Don't emit warnings about ignored upload files on every run
Periodically emit warnings about apparently forgotten upload files which are
being ignored, but not on every run
https://sourceware.org/git/gitweb.cgi?p=cygwin-apps/calm.git;h=83dc7d12d415f8c2c2d941acc5b145bb8cbb28b8
commit 83dc7d12d415f8c2c2d941acc5b145bb8cbb28b8
Author: Jon Turney <jon.turney@dronecode.org.uk>
Date: Tue Mar 29 12:15:59 2016 +0100
Add an 'X-Calm' header to mark mails from calm
https://sourceware.org/git/gitweb.cgi?p=cygwin-apps/calm.git;h=71791009a682a83ca273c3c58998891d7052ea1d
commit 71791009a682a83ca273c3c58998891d7052ea1d
Author: Jon Turney <jon.turney@dronecode.org.uk>
Date: Wed Mar 23 21:22:55 2016 +0000
Add a testing facility to Bcc: all emails sent
https://sourceware.org/git/gitweb.cgi?p=cygwin-apps/calm.git;h=b245856fe5e286431d75bab81e8ce68786b8b2d3
commit b245856fe5e286431d75bab81e8ce68786b8b2d3
Author: Jon Turney <jon.turney@dronecode.org.uk>
Date: Tue Mar 22 18:48:12 2016 +0000
Improve checking sdesc for package name
- match case-insensitively
- try harder to manufacture a 'package basename'
- ignore whitespace before ':'
- look for 'packagename -' as well
https://sourceware.org/git/gitweb.cgi?p=cygwin-apps/calm.git;h=95ff7fea38070dbe9219172aecb043ddea659ead
commit 95ff7fea38070dbe9219172aecb043ddea659ead
Author: Jon Turney <jon.turney@dronecode.org.uk>
Date: Tue Mar 22 23:50:40 2016 +0000
Fix sdesc which end with '.'
Don't just warn, also fix any sdesc which end with '.'
Update tests
https://sourceware.org/git/gitweb.cgi?p=cygwin-apps/calm.git;h=13d133e800946d7b339f2e3bf82b36e12ef686ca
commit 13d133e800946d7b339f2e3bf82b36e12ef686ca
Author: Jon Turney <jon.turney@dronecode.org.uk>
Date: Tue Mar 22 18:47:00 2016 +0000
Warn if sdesc contains ' '
Some sdesc erroneously contained multiple spaces between words.
https://sourceware.org/git/gitweb.cgi?p=cygwin-apps/calm.git;h=35dcccec440f79e6f86bdaa7098ed922859de94b
commit 35dcccec440f79e6f86bdaa7098ed922859de94b
Author: Jon Turney <jon.turney@dronecode.org.uk>
Date: Wed Mar 23 21:27:41 2016 +0000
Sort requires: when read
This avoids having to treat requires: specially when showing differences,
which wasn't doing quite the right thing anyhow, as that was writing
a setup.ini based on the sorted requires, leading to a differences of
ordering on the next run...
https://sourceware.org/git/gitweb.cgi?p=cygwin-apps/calm.git;h=8713d4e83fe3b9646366073f84c970c4f61c64cf
commit 8713d4e83fe3b9646366073f84c970c4f61c64cf
Author: Jon Turney <jon.turney@dronecode.org.uk>
Date: Mon Mar 21 20:14:06 2016 +0000
Add .xz compression of setup.ini
Diff:
---
TODO | 1 -
buffering_smtp_handler.py | 2 +
calm.py | 4 ++-
common_constants.py | 3 ++
hint.py | 18 ++++++++--
package.py | 22 ++++++------
testdata/.gitignore | 1 +
.../x86/base-cygwin/base-cygwin-3.6-1 | 2 +-
.../x86/base-cygwin/base-cygwin-3.8-1 | 2 +-
testdata/htdocs.expected/x86/packages.inc | 2 +-
testdata/inifile/setup.ini.expected | 2 +-
testdata/x86.hints/release/base-cygwin/expected | 2 +-
.../release/libtextcat/libtextcat-devel/expected | 2 +-
.../release/mingw64-i686-binutils/expected | 2 +-
.../mingw64-i686-binutils-debuginfo/expected | 2 +-
testdata/x86.hints/release/splint/expected | 2 +-
uploads.py | 36 +++++++++++++++++--
17 files changed, 75 insertions(+), 30 deletions(-)
diff --git a/TODO b/TODO
index 246aac2..f2c7bc7 100644
--- a/TODO
+++ b/TODO
@@ -1,6 +1,5 @@
* per-package-version hint files (e.g. named <pvr>.hint, or /.setup.hint inside tar file)
* more than 2 versions possible (and automatically vault 'old' versions, where we have some mechanism to explicity say what is old)
-* enable .xz compressed setup.ini (setup.exe checks for it and presumably can handle it)
* automatically suppress 'empty source' packages from setup.ini ?
* run more often, option to not do anything if no uploads (to avoid reading the release area if we don't need to), lockfile to avoid colliding runs
* work out some way to package it as a replacement for genini
diff --git a/buffering_smtp_handler.py b/buffering_smtp_handler.py
index 7720588..2bcffcc 100644
--- a/buffering_smtp_handler.py
+++ b/buffering_smtp_handler.py
@@ -55,7 +55,9 @@ class BufferingSMTPHandler(logging.handlers.BufferingHandler):
m = email.message.Message()
m['From'] = self.fromaddr
m['To'] = ','.join(self.toaddrs)
+ m['Bcc'] = common_constants.ALWAYS_BCC
m['Subject'] = self.subject
+ m['X-Calm'] = '1'
# use utf-8 only if the message can't be ascii encoded
charset = 'ascii'
diff --git a/calm.py b/calm.py
index f1b6b2d..0dd001a 100755
--- a/calm.py
+++ b/calm.py
@@ -188,7 +188,7 @@ def main(args):
shutil.move(tmpfile.name, inifile)
# compress and re-sign
- for ext in ['.ini', '.bz2']:
+ for ext in ['.ini', '.bz2', '.xz']:
try:
os.remove(os.path.join(basedir, 'setup' + ext + '.sig'))
except FileNotFoundError:
@@ -196,6 +196,8 @@ def main(args):
if ext == '.bz2':
os.system('/usr/bin/bzip2 <%s >%s' % (inifile, os.path.splitext(inifile)[0] + ext))
+ elif ext == '.xz':
+ os.system('/usr/bin/xz -6e <%s >%s' % (inifile, os.path.splitext(inifile)[0] + ext))
os.system('/usr/bin/gpg --batch --yes -b ' + os.path.join(basedir, 'setup' + ext))
diff --git a/common_constants.py b/common_constants.py
index 7d031c9..e40c346 100644
--- a/common_constants.py
+++ b/common_constants.py
@@ -38,6 +38,9 @@ FTP = '/var/ftp/pub/cygwin'
# logs are always emailed to these addresses
EMAILS = ','.join(map(lambda m: m + '@sourceware.org', ['corinna', 'yselkowitz', 'jturney']))
+# for testing purposes, every email we send is bcc'd to these addresses
+ALWAYS_BCC = ''
+
# these maintainers can upload orphaned packages as well
ORPHANMAINT = "Yaakov Selkowitz"
diff --git a/hint.py b/hint.py
index 4751466..2579fad 100755
--- a/hint.py
+++ b/hint.py
@@ -180,9 +180,6 @@ def setup_hint_parse(fn):
if key in hints:
errors.append('duplicate key %s' % (key))
- # store the key:value
- hints[key] = value
-
# check the value meets any key-specific constraints
if (key in valkeys) and (len(value) == 0):
errors.append('%s has empty value' % (key))
@@ -202,10 +199,16 @@ def setup_hint_parse(fn):
if not (value.startswith('"') and value.endswith('"')):
errors.append("%s value '%s' should be quoted" % (key, value))
- # warn if sdesc ends with a '.'
+ # if sdesc ends with a '.', warn and fix it
if key == 'sdesc':
if re.search(r'\."$', value):
warnings.append("sdesc ends with '.'")
+ value = re.sub(r'\."$', '"', value)
+
+ # warn if sdesc contains ' '
+ if key == 'sdesc':
+ if ' ' in value:
+ warnings.append("sdesc contains ' '")
# only 'ldesc' and 'message' are allowed a multi-line value
if (key not in multilinevalkeys) and (len(value.splitlines()) > 1):
@@ -219,6 +222,9 @@ def setup_hint_parse(fn):
# warn if value starts with a quote followed by whitespace
if re.match(r'^"[ \t]+', value):
warnings.append('value for key %s starts with quoted whitespace' % (key))
+
+ # store the key:value
+ hints[key] = value
else:
errors.append("unknown setup construct '%s' at line %d" % (item, i))
@@ -239,6 +245,10 @@ def setup_hint_parse(fn):
if len(hints['sdesc']) > 2*len(hints['ldesc']):
warnings.append('sdesc is much longer than ldesc')
+ # sort requires: as differences in ordering are uninteresting
+ if 'requires' in hints:
+ hints['requires'] = ' '.join(sorted(hints['requires'].split()))
+
except UnicodeDecodeError:
errors.append('invalid UTF-8')
diff --git a/package.py b/package.py
index 0b1313a..b1ca42e 100755
--- a/package.py
+++ b/package.py
@@ -203,13 +203,18 @@ def read_package(packages, basedir, dirpath, files, strict=False):
if p in past_mistakes.self_source:
packages[p].hints['self-source'] = ''
- # don't allow a redundant 'package-initial-substring:' at start of sdesc
+ # don't allow a redundant 'package:' or 'package - ' at start of sdesc
+ #
+ # match case-insensitively, and use a base package name (trim off any
+ # leading 'lib' from package name, remove any soversion or 'devel'
+ # suffix)
+ #
if 'sdesc' in hints:
- sdesc = re.sub(r'^"|"$', '', hints['sdesc'])
- colon = sdesc.find(':')
- if colon > -1:
- if sdesc[:colon] == p[:colon]:
- logging.warning("package '%s' sdesc starts with '%s:'; this is redundant as the UI will show both the package name and sdesc" % (p, sdesc[:colon]))
+ colon = re.match(r'^"(.*?)(\s*:|\s+-)', hints['sdesc'])
+ if colon:
+ package_basename = re.sub(r'^lib(.*?)(|-devel|\d*)$', r'\1', p)
+ if package_basename.upper().startswith(colon.group(1).upper()):
+ logging.warning("package '%s' sdesc starts with '%s'; this is redundant as the UI will show both the package name and sdesc" % (p, ''.join(colon.group(1, 2))))
warnings = True
elif (len(files) > 0) and (relpath.count(os.path.sep) > 0):
@@ -591,11 +596,6 @@ def merge(a, b):
if a[p].hints != b[p].hints:
c[p].hints = b[p].hints
- # sort requires: as differences in ordering are uninteresting
- for hints in [a[p].hints, b[p].hints]:
- if 'requires' in hints:
- hints['requires'] = ' '.join(sorted(hints['requires'].split()))
-
diff = '\n'.join(difflib.ndiff(
pprint.pformat(a[p].hints).splitlines(),
pprint.pformat(b[p].hints).splitlines()))
diff --git a/testdata/.gitignore b/testdata/.gitignore
index 22f46b8..2dada27 100644
--- a/testdata/.gitignore
+++ b/testdata/.gitignore
@@ -4,3 +4,4 @@ results
setup.ini
sha512.sum
\!ready
+\!reminder-timestamp
diff --git a/testdata/htdocs.expected/x86/base-cygwin/base-cygwin-3.6-1 b/testdata/htdocs.expected/x86/base-cygwin/base-cygwin-3.6-1
index 42b3a8e..a6c9d6c 100644
--- a/testdata/htdocs.expected/x86/base-cygwin/base-cygwin-3.6-1
+++ b/testdata/htdocs.expected/x86/base-cygwin/base-cygwin-3.6-1
@@ -1,5 +1,5 @@
<html>
-<h1>base-cygwin: Initial base installation helper script. (installed binaries and support files)</h1>
+<h1>base-cygwin: Initial base installation helper script (installed binaries and support files)</h1>
<tt><pre>
2015-10-11 14:45 26 test/test.1
2015-10-11 14:45 31 test/test.2
diff --git a/testdata/htdocs.expected/x86/base-cygwin/base-cygwin-3.8-1 b/testdata/htdocs.expected/x86/base-cygwin/base-cygwin-3.8-1
index 42b3a8e..a6c9d6c 100644
--- a/testdata/htdocs.expected/x86/base-cygwin/base-cygwin-3.8-1
+++ b/testdata/htdocs.expected/x86/base-cygwin/base-cygwin-3.8-1
@@ -1,5 +1,5 @@
<html>
-<h1>base-cygwin: Initial base installation helper script. (installed binaries and support files)</h1>
+<h1>base-cygwin: Initial base installation helper script (installed binaries and support files)</h1>
<tt><pre>
2015-10-11 14:45 26 test/test.1
2015-10-11 14:45 31 test/test.2
diff --git a/testdata/htdocs.expected/x86/packages.inc b/testdata/htdocs.expected/x86/packages.inc
index 484f797..cd5e235 100755
--- a/testdata/htdocs.expected/x86/packages.inc
+++ b/testdata/htdocs.expected/x86/packages.inc
@@ -7,7 +7,7 @@
<br>
<table class="pkglist">
<tr><td><a href="x86/arc">arc</a></td><td>The ARC archive utility</td></tr>
-<tr><td><a href="x86/base-cygwin">base-cygwin</a></td><td>Initial base installation helper script.</td></tr>
+<tr><td><a href="x86/base-cygwin">base-cygwin</a></td><td>Initial base installation helper script</td></tr>
<tr><td><a href="x86/cygwin">cygwin</a></td><td>The UNIX emulation engine</td></tr>
<tr><td><a href="x86/cygwin-debuginfo">cygwin-debuginfo</a></td><td>Debug info for cygwin</td></tr>
<tr><td><a href="x86/cygwin-devel">cygwin-devel</a></td><td>Core development files</td></tr>
diff --git a/testdata/inifile/setup.ini.expected b/testdata/inifile/setup.ini.expected
index 6140c5d..6b241ef 100644
--- a/testdata/inifile/setup.ini.expected
+++ b/testdata/inifile/setup.ini.expected
@@ -23,7 +23,7 @@
'6de201dfed1d45412509c65deb34690dc2d09c6aafccfe491fd2f440f92842b9c755b61dc7bcdd4cc0c9f18cf46c2b3a1241e99c4c2a33fff5555e7b2f0b6348\n'
'\n'
'@ base-cygwin\n'
- 'sdesc: "Initial base installation helper script."\n'
+ 'sdesc: "Initial base installation helper script"\n'
'ldesc: "Initial base installation helper script."\n'
'category: Base\n'
'version: 3.8-1\n'
diff --git a/testdata/x86.hints/release/base-cygwin/expected b/testdata/x86.hints/release/base-cygwin/expected
index e3409a1..7505632 100644
--- a/testdata/x86.hints/release/base-cygwin/expected
+++ b/testdata/x86.hints/release/base-cygwin/expected
@@ -1,4 +1,4 @@
-{'sdesc': '"Initial base installation helper script."',
+{'sdesc': '"Initial base installation helper script"',
'ldesc': '"Initial base installation helper script."',
'category': 'Base',
'requires': '',
diff --git a/testdata/x86.hints/release/libtextcat/libtextcat-devel/expected b/testdata/x86.hints/release/libtextcat/libtextcat-devel/expected
index f60d34d..aca0db6 100644
--- a/testdata/x86.hints/release/libtextcat/libtextcat-devel/expected
+++ b/testdata/x86.hints/release/libtextcat/libtextcat-devel/expected
@@ -7,6 +7,6 @@
'BSD License.\n'
'http://software.wise-guys.nl/libtextcat/";',
'category': 'Devel Text',
- 'requires': 'libtextcat0 libtextcat',
+ 'requires': 'libtextcat libtextcat0',
'external-source': 'libtextcat',
'parse-errors': ['embedded quote at line 7']}
diff --git a/testdata/x86.hints/release/mingw64-i686-binutils/expected b/testdata/x86.hints/release/mingw64-i686-binutils/expected
index bc0368d..114e53c 100644
--- a/testdata/x86.hints/release/mingw64-i686-binutils/expected
+++ b/testdata/x86.hints/release/mingw64-i686-binutils/expected
@@ -1,6 +1,6 @@
{'category': 'Devel',
'requires': 'cygwin libintl8 zlib0',
- 'sdesc': '"The GNU Binutils are a collection of binary tools. This package\n is capable of targeting win32."',
+ 'sdesc': '"The GNU Binutils are a collection of binary tools. This package\n is capable of targeting win32"',
'ldesc': '"Binutils for MinGW-w64 Win32 toolchain"',
'parse-errors': ['key sdesc has multi-line value'],
'parse-warnings': ["sdesc ends with '.'", 'sdesc is much longer than ldesc']}
diff --git a/testdata/x86.hints/release/mingw64-i686-binutils/mingw64-i686-binutils-debuginfo/expected b/testdata/x86.hints/release/mingw64-i686-binutils/mingw64-i686-binutils-debuginfo/expected
index d925953..c936ff6 100644
--- a/testdata/x86.hints/release/mingw64-i686-binutils/mingw64-i686-binutils-debuginfo/expected
+++ b/testdata/x86.hints/release/mingw64-i686-binutils/mingw64-i686-binutils-debuginfo/expected
@@ -2,6 +2,6 @@
'requires': 'cygwin-debuginfo',
'external-source': 'mingw64-i686-binutils',
'ldesc': '"Debug info for mingw64-i686-binutils"',
- 'sdesc': '"This package contains files necessary for debugging the\nmingw64-i686-binutils package with gdb."',
+ 'sdesc': '"This package contains files necessary for debugging the\nmingw64-i686-binutils package with gdb"',
'parse-errors': ['key sdesc has multi-line value'],
'parse-warnings': ["sdesc ends with '.'", 'sdesc is much longer than ldesc']}
diff --git a/testdata/x86.hints/release/splint/expected b/testdata/x86.hints/release/splint/expected
index 8c0b6c4..5667e4b 100644
--- a/testdata/x86.hints/release/splint/expected
+++ b/testdata/x86.hints/release/splint/expected
@@ -1,4 +1,4 @@
-{'sdesc': '"Check C programs for security vulnerabilities and programming\nmistakes."',
+{'sdesc': '"Check C programs for security vulnerabilities and programming\nmistakes"',
'ldesc': '"Program does many of the traditional lint checks including\n'
'unused declarations, type inconsistencies, use before definition,\n'
'unreachable code, ignored return values, execution paths with no\n'
diff --git a/uploads.py b/uploads.py
index 050c494..e3f6d78 100644
--- a/uploads.py
+++ b/uploads.py
@@ -30,9 +30,13 @@ import filecmp
import os
import logging
import re
+import time
import package
+# reminders will be issued daily
+REMINDER_INTERVAL = 60*60*24
+
#
#
@@ -60,6 +64,14 @@ def scan(m, all_packages, args):
logging.info('processing files with mtime older than %d' % (mtime))
remove.append(ready)
+ # the mtime of this file indicates when 'ignoring as there is no !ready'
+ # warnings were last emitted
+ reminder_file = os.path.join(basedir, '!reminder-timestamp')
+ if os.path.exists(reminder_file):
+ reminder_time = os.path.getmtime(reminder_file)
+ else:
+ reminder_time = 0
+
# scan package directories
for (dirpath, subdirs, files) in os.walk(os.path.join(basedir, 'release')):
relpath = os.path.relpath(dirpath, basedir)
@@ -83,10 +95,9 @@ def scan(m, all_packages, args):
# shortest-to-longest order, since os.walk() walks the tree
# top-down), and use the mtime of the first (longest) matching path.
while True:
- (path, time) = mtimes[-1]
+ (path, mtime) = mtimes[-1]
if relpath.startswith(path):
- logging.info("using mtime %d from subpath '%s' of '%s'" % (time, path, relpath))
- mtime = time
+ logging.info("using mtime %d from subpath '%s' of '%s'" % (mtime, path, relpath))
break
else:
mtimes.pop()
@@ -116,7 +127,15 @@ def scan(m, all_packages, args):
# only process files newer than !ready
if os.path.getmtime(fn) > mtime:
if mtime == 0:
- logging.warning("ignoring %s as there is no !ready" % fn)
+ lvl = logging.INFO
+
+ # if more than REMINDER_INTERVAL has elapsed since we warned
+ # about files being ignored, warn again
+ if time.time() > (reminder_time + REMINDER_INTERVAL):
+ lvl = logging.WARNING
+ touch(reminder_file)
+
+ logging.log(lvl, "ignoring %s as there is no !ready" % fn)
else:
logging.warning("ignoring %s as it is newer than !ready" % fn)
files.remove(f)
@@ -159,6 +178,15 @@ def scan(m, all_packages, args):
#
#
+def touch(fn, times=None):
+ with open(fn, 'a'):
+ os.utime(fn, times)
+
+
+#
+#
+#
+
def remove(args, remove):
for f in remove:
logging.info("rm %s", f)