Merge branch 'trunk' into release-22.1.0-10296

README.rst

@@ -2,6 +2,7 @@ Twisted
 =======
 
 |gitter|_
+|rtd|_
 |pypi|_
 |mypy|_
 
@@ -116,3 +117,6 @@ Again, see the included `LICENSE <LICENSE>`_ file for specific legal details.
 
 .. |mypy| image:: https://github.com/twisted/twisted/workflows/mypy/badge.svg
 .. _mypy: https://github.com/twisted/twisted
+
+.. |rtd| image:: https://readthedocs.org/projects/twisted/badge/?version=latest&style=flat
+.. _rtd: https://docs.twistedmatrix.com

docs/core/development/policy/release-process.rst

@@ -13,7 +13,7 @@ Outcomes
 By the end of a Twisted release we'll have:
 
 - Wheel and sdist package published on `PyPI Twisted project <https://pypi.org/project/Twisted/>`_.
-- Updated documentation (API & howtos) on `Twisted Read The Docs <https://twisted.readthedocs.io/en/latest/>`_
+- Updated documentation (API & howtos) on `Twisted Read The Docs <https://docs.twistedmatrix.com/>`_ for `stable` and `$RELEASE` versions.
 - Announcement email sent to Twisted main list
 - A `GitHub Release <https://github.com/twisted/twisted/releases>`_ with the associated tag in our Git repository
 
@@ -26,6 +26,23 @@ To release Twisted, you will need:
 - Commit privileges to Twisted GitHub repository.
 
 
+Dependencies
+------------
+
+Below is the list of moving parts and web services used by the release process.
+For day to day operation, you should not need management access to them.
+If things go wrong, you should be aware of them and get administration access.
+
+* Release tag is automatically created via the GitHub Release GUI.
+* PyPi file publishing is done via GitHub Actions workflow when a tag is created.
+  Any Twisted contributor in GitHub should have access to modify the workflow.
+* docs.twistedmatrix.com is a CNAME and you will need access to Twisted DNS server to modify it.
+* Documentation is published via `Read The Docs Twisted project <https://readthedocs.org/dashboard/twisted/edit/>`_.
+  There is an `automated rule <https://readthedocs.org/dashboard/twisted/rules/regex/1057/>` to activate the documentation for every tag matching ``^twisted-\d+\.\d+\.\d+$`` (release candidates are excluded)
+  From RTD `Advanced Settings <https://readthedocs.org/dashboard/twisted/advanced/>`_ the branch named `stable` is configured as the default branch.
+  There is also a "active" documentation version for the branch named `stable`.
+
+
 Version numbers
 ---------------
 
@@ -69,28 +86,38 @@ To release Twisted, we
 Prepare for a release
 ---------------------
 
-#. Check for any ​regressions using `Trac regression report <https://twistedmatrix.com/trac/report/26>`_
+#. Check for any regressions using `Trac regression report <https://twistedmatrix.com/trac/report/26>`_
 
-#. Any regression should be fixed and merged trunk before making the release branch
+#. Any regression should be fixed and merged into trunk before making the release branch
 
 #. Choose a version number.
+   $RELEASE will be something like 21.7.0 for a major release.
+   $RELEASE will be something like 21.7.1 for a bugfix release.
 
 #. File a ticket in Trac called "Release $RELEASE" and assign it to yourself.
 
-#. Make a branch for the release and include the ticket number in the name (4290 is Trac ticket ID):
+#. Make a branch for the release.
+   It's very important to use `release-$RELEASE-$TRAC_ID` as the branch name (4290 is Trac ticket ID, 21.7.0 is the release number) as this is used as a hint for CI:
 
    - ``git fetch origin``
    - ``git checkout origin/trunk``
-   - ``git checkout -b release-$RELEASE-4290``
+   - ``git checkout -b release-21.7.0-4290``
 
 
 How to do a release candidate
 -----------------------------
 
 
+This section describes the steps and requirements for creating the first release candidate.
+
+
 Prepare the branch
 ~~~~~~~~~~~~~~~~~~
 
+#. Check that all the CI tests on the main branch (trunk) pass.
+   Failing tests on the main branch should be considered release blocker.
+   They should be fixed in separate ticket/PR.
+   The release can continue once the main branch is green again.
 #. In your Git repo, fetch and check out the new release branch.
 #. Run ``python -m incremental.update Twisted --rc``
 #. Commit the changes made by Incremental.
@@ -99,13 +126,31 @@ Prepare the branch
 #. Bump copyright dates in ``LICENSE``, ``src/twisted/copyright.py``, and ``README.rst`` if required
 #. Push the changes up to GitHub and create a new release PR.
 #. The GitHub PR is dedicated to the final release and the same PR is used to release the candidate and final version.
+#. Wait for all the PR checks to pass.
+#. If a check fails investigate it.
+   If is just a flaky tests, retry the run.
+   Any serious error should be considered a blocker and should be
+   fixed in a separate ticket/PR.
+   Avoid making non-release changes (even minor one) as part of the release branch.
 #. Use the `GitHub Create Release UI <https://github.com/twisted/twisted/releases/new>`_ the make a new release.
-#. Create a tag using the format `twisted-VERSION` based on the latest commit on the release branch that was approved after the review.
+#. Create a tag using the format `twisted-VERSION` based on the latest commit on the release branch.
 #. Use `Twisted VERSION` as the name of the release.
 #. Add the release NEWS to GitHub Release page.
 #. Make sure 'This is a pre-release` is checked.
 #. Github Actions will upload the dist to PyPI when a new tag is pushed to the repo.
-#. Read the Docs hooks will publish a new version of the docs.
+#. You can check the status of the automatic upload via `GitHub Action <https://github.com/twisted/twisted/actions/workflows/test.yaml?query=event%3Apush>`_
+#. Read the Docs hooks not have version for the release candidate.
+   Use the Read the Docs published for the pull request.
+#. The review for the PR will be requested after the files are on PyPI so that a full review and manual test can be done.
+#. Most probably there will be some minor comments received via email or GitHub regarding the final content of the release notes.
+   It's OK to make those changes as part of the release branch.
+   It's OK to update the text of the candidate release notes,
+   in the final NEWS file the release candidate version is removed and replaced with the final version.
+   No need for a new ticket or separate pull request.
+   These changes will be reviewed as part of the final release review process.
+#. While the final public release is not made and the release tag created
+   the release branch will not be kept up to date with trunk.
+
 
 Announce
 ~~~~~~~~
@@ -114,36 +159,48 @@ Announce
 
 #. Announce the release candidate on
 
-   - the twisted-python mailing list by sending the content of latest release NEWS
-   - on IRC in the ``#twisted-dev`` topic by sending the version number
+   - the twisted-python mailing list by sending the an email with the subject: Twisted $RELEASE Pre-Release Announcement
+   - on IRC in the ``#twisted-dev`` topic by sending the version number or pip install command
 
 The release candidate announcement might mention the important changes since the last release, and ask readers to test this release candidate.
 
-Here's what the $RELEASErc1 release announcement might look like::
+Here's what the $RELEASE release candidate announcement might look like::
 
-    Live from PyCon Atlanta, I'm pleased to herald the approaching
-    footsteps of the $API release.
+   On behalf of the Twisted contributors I announce the release candidate of Twisted $RELEASE
 
-    Wheels for Twisted $RELEASE release candidate are now available on PyPI.
+   Short summary of the release.
+   For example:
+   Python 3.5 is no longer a supported platform.
+   The minimum supported platform is Python 3.6.7.
 
-    Highlights include:
 
-     * Improved documentation, including "Twisted Web in 60 seconds"
+   The notable changes are:
 
-     * Faster Perspective Broker applications
+   * Mention the main new features.
+   * As well as important bug fixes
+   * Or deprecation/removals
 
-     * A new Windows installer that ships without zope.interface
+   The release and NEWS file is available for review at
 
-     * Twisted no longer supports Python 2.3
+      https://github.com/twisted/twisted/pull/PRID/files
 
-     * Over one hundred closed tickets
+   Release candidate documentation is available at
 
-    For more information, see the NEWS file.
+      https://twisted--PRID.org.readthedocs.build/en/PRID/
 
-    Please download the tarballs and test them as much as possible.
+   Wheels for the release candidate are available on PyPI
 
-    Thanks,
-    jml
+      https://pypi.org/project/Twisted/$RELEASErc1
+
+      python -m pip install Twisted==$RELEASErc1
+
+   Please test it and report any issues.
+   If nothing comes up in one week,
+   $RELEASE will be released based on the latest release candidate.
+
+   Many thanks to everyone who had a part in Twisted
+   the supporters of the Twisted Software Foundation,
+   the developers, and all the people testing and building great things with Twisted!
 
 A week is a generally good length of time to wait before doing the final release.
 
@@ -157,9 +214,11 @@ Prepare the branch
 
 #. Have the release branch, previously used to generate a release candidate, checked out
 #. Run ``python -m incremental.update Twisted --newversion $RELEASE``
-#. Manually update the release date if necessary.
+#. Manually update the release version and date inside the NEWS file.
+   The release candidate notes will be removed from the final NEWS file.
+   Manually move all the release notes from the release candidates to the notes for the final version.
 #. Commit and push.
-#. Submit the ticket for review
+#. Submit the ticket for the final review.
 #. Pause until the ticket is reviewed and accepted.
 #. Use the `GitHub Create Release UI <https://github.com/twisted/twisted/releases/new>`_ the make a new release.
 #. Create a tag using the format `twisted-VERSION` based on the latest commit on the release branch that was approved after the review.
@@ -167,7 +226,7 @@ Prepare the branch
 #. Add the release NEWS to GitHub Release page.
 #. Make sure 'This is a pre-release` is not checked.
 #. Github Actions will upload the dist to PyPI when a new tag is pushed to the repo. PyPI is the only canonical source for Twisted packages.
-#. Read the Docs hooks will publish a new version of the docs.
+#. Read the Docs hooks will publish a new version of the docs for the tag.
 
 
 Announce
@@ -177,29 +236,39 @@ Announce
 
 #. Announce the release
 
-   - Send a text version of the announcement to: twisted-python@twistedmatrix.com, python-announce-list@python.org, twisted-web@twistedmatrix.com
-   - ​http://labs.twistedmatrix.com (Post a web version of the announcements, with links instead of literal URLs)
-   - Twitter, if you feel like it
-   - ``#twisted`` topic on IRC (you'll need ops)
+   - Send a text version of the announcement to: twisted@python.org
+   - Twitter, TikTok, Instagram, Snapchat if you feel like it :)
+   - ``#twisted`` message on IRC
 
 
 Post release
 ~~~~~~~~~~~~
 
-#. Run ``python -m incremental.update Twisted --post`` to add a `post` postfix.
+#. Run ``python -m incremental.update Twisted --post`` to add a `post` version number.
 
 #. Commit the post0 update change.
 
-#. Merge the release branch into trunk, closing the release ticket at the same time.
+#. Update the trunk into the release branch, resolving any possible conflicts.
+
+#. No need to request another review.
+
+#. Merge the release branch into trunk (via GitHub PR UI),
+   closing the release ticket at the same time.
 
 
-When things go wrong
---------------------
+Release candidate fixes
+-----------------------
 
-If you discover a showstopper bug during the release process, you have three options.
+This section described the steps to follow when after a release candidate is published, critical or regression defects are found.
 
-1. Abort the release, make a new point release (e.g. abort 10.0.0, make 10.0.1 after the bug is fixed)
-2. Abort the release, make a new release candidate (e.g. abort 10.0.0, make 10.0.0pre3 after the bug is fixed)
+If a defect is found after the final release is published, check the next section: `Bug fix releases`.
+
+1. Pause the release process.
+2. Separate tickets should be files for each defect.
+3. The defect should be fixed, reviewed and merged in trunk.
+4. On the release branch, cherry-pick the merges from trunk that merges the fixes `git cherry-pick -m 1 TRUNK_MERGE_SHA`.
+5. Follow the same steps as for any release candidate, with the exception that a new branch is not created.
+   Use the same `python -m incremental.update Twisted --rc` command to increment the release candidate version.
 
 Don't delete a tag that was already pushed for a release.
 Create a new tag with incremented version.
@@ -213,7 +282,7 @@ This section goes over doing these "bugfix" releases.
 
 1. Ensure all bugfixes are in trunk.
 
-2. Make a branch off the affected version.
+2. Make a branch off the affected released version (not from trunk HEAD).
 
 3. Cherry-pick the merge commits that merge the bugfixes into trunk, onto the new release branch.
 
@@ -221,3 +290,8 @@ This section goes over doing these "bugfix" releases.
 
    - Instead of just ``--rc`` when running the change-versions script, add the patch flag, making it ``--patch --rc``.
    - Instead of waiting a week, a shorter pause is acceptable for a patch release.
+     You can do the release as soon as you get the confirmation from the original bug reports that the release candidate fixes the issues.
+
+5. If you are doing a security release for an older release,
+   the automated release will overwrite the `stable` branch and consider it as the latest release.
+   You will need to manually reset/rebase the `stable` branch to point to the actual latest release.

src/twisted/conch/test/test_cftp.py

@@ -1412,6 +1412,7 @@ exit
 
 
 @skipIf(skipTests, "don't run w/o spawnProcess or cryptography or pyasn1")
+@skipIf(not which("ssh"), "no ssh command-line client available")
 @skipIf(not which("sftp"), "no sftp command-line client available")
 class OurServerSftpClientTests(CFTPClientTestBase):
     """
@@ -1433,6 +1434,10 @@ class OurServerSftpClientTests(CFTPClientTestBase):
         This test is mainly here to check that
         L{filetransfer.FILEXFER_ATTR_EXTENDED} has the correct value.
         """
+        # Get the current environment to pass along so that `ssh` and `sftp`
+        # can be found on our PATH.
+        env = dict(os.environ)
+
         fn = self.mktemp()
         with open(fn, "w") as f:
             f.write("ls .\nexit")
@@ -1452,7 +1457,7 @@ class OurServerSftpClientTests(CFTPClientTestBase):
         # first need to check if we can set it. If we can, -V will just print
         # the version without doing anything else; if we can't, we will get a
         # configuration error.
-        d = getProcessValue("ssh", ("-o", "PubkeyAcceptedKeyTypes=ssh-dss", "-V"))
+        d = getProcessValue("ssh", ("-o", "PubkeyAcceptedKeyTypes=ssh-dss", "-V"), env)
 
         def hasPAKT(status):
             if status == 0:
@@ -1480,7 +1485,7 @@ class OurServerSftpClientTests(CFTPClientTestBase):
             return args
 
         def check(result):
-            self.assertEqual(result[2], 0)
+            self.assertEqual(result[2], 0, result[1].decode("ascii"))
             for i in [
                 b"testDirectory",
                 b"testRemoveFile",
@@ -1490,5 +1495,5 @@ class OurServerSftpClientTests(CFTPClientTestBase):
                 self.assertIn(i, result[0])
 
         d.addCallback(hasPAKT)
-        d.addCallback(lambda args: getProcessOutputAndValue("sftp", args))
+        d.addCallback(lambda args: getProcessOutputAndValue("sftp", args, env))
         return d.addCallback(check)

src/twisted/newsfragments/10270.doc

@@ -0,0 +1,2 @@
+Add type annotations for twisted.web.http.Request.getHeader.
+

src/twisted/python/test/test_release.py

@@ -49,10 +49,16 @@ if sys.platform != "win32":
 else:
     skip = "Release toolchain only supported on POSIX."
 
-# This should match the GitHub Actions environment used by pre-comnmit.ci to push changes to the auto-updated branches.
-PRECOMMIT_CI_ENVIRON = {"GITHUB_HEAD_REF": "pre-commit-ci-update-config"}
+# This should match the GitHub Actions environment used by pre-commit.ci to push changes to the auto-updated branches.
+PRECOMMIT_CI_ENVIRON = {
+    "GITHUB_HEAD_REF": "pre-commit-ci-update-config",
+    "PATH": os.environ["PATH"],
+}
 # This should match the GHA environment for non pre-commit.ci PRs.
-GENERIC_CI_ENVIRON = {"GITHUB_HEAD_REF": "1234-some-branch-name"}
+GENERIC_CI_ENVIRON = {
+    "GITHUB_HEAD_REF": "1234-some-branch-name",
+    "PATH": os.environ["PATH"],
+}
 
 
 class ExternalTempdirTestCase(TestCase):

src/twisted/test/test_process.py

@@ -980,15 +980,20 @@ class PosixProcessBase:
         Return the path of the shell command named C{commandName}, looking at
         common locations.
         """
+        for loc in procutils.which(commandName):
+            return FilePath(loc).asBytesMode().path
+
         binLoc = FilePath("/bin").child(commandName)
         usrbinLoc = FilePath("/usr/bin").child(commandName)
 
         if binLoc.exists():
-            return binLoc._asBytesPath()
+            return binLoc.asBytesMode().path
         elif usrbinLoc.exists():
-            return usrbinLoc._asBytesPath()
+            return usrbinLoc.asBytesMode().path
         else:
-            raise RuntimeError(f"{commandName} not found in /bin or /usr/bin")
+            raise RuntimeError(
+                f"{commandName} found in neither standard location nor on PATH ({os.environ['PATH']})"
+            )
 
     def test_normalTermination(self):
         cmd = self.getCommand("true")

src/twisted/trial/_dist/test/test_workertrial.py

@@ -69,7 +69,7 @@ class MainTests(TestCase):
         the stdin fd and C{self.writeStream} for the stdout fd.
         """
         if fd == _WORKER_AMP_STDIN:
-            self.assertIdentical("rb", mode)
+            self.assertEqual("rb", mode)
             return self.readStream
         elif fd == _WORKER_AMP_STDOUT:
             self.assertEqual("wb", mode)

src/twisted/trial/newsfragments/10302.bugfix

@@ -0,0 +1 @@
+_dist.test.test_workertrial now correctly compare strings via assertEqual() and pass on PyPy3

src/twisted/web/http.py

@@ -108,7 +108,7 @@ import tempfile
 import time
 import warnings
 from io import BytesIO
-from typing import Callable
+from typing import AnyStr, Callable, Optional
 from urllib.parse import (
     ParseResultBytes,
     unquote_to_bytes as unquote,
@@ -815,7 +815,7 @@ class Request:
         self.client = self.channel.getPeer()
         self.host = self.channel.getHost()
 
-        self.requestHeaders = Headers()
+        self.requestHeaders: Headers = Headers()
         self.received_cookies = {}
         self.responseHeaders = Headers()
         self.cookies = []  # outgoing cookies
@@ -1051,7 +1051,7 @@ class Request:
 
     # The following is the public interface that people should be
     # writing to.
-    def getHeader(self, key):
+    def getHeader(self, key: AnyStr) -> Optional[AnyStr]:
         """
         Get an HTTP request header.
 
@@ -1066,6 +1066,7 @@ class Request:
         value = self.requestHeaders.getRawHeaders(key)
         if value is not None:
             return value[-1]
+        return None
 
     def getCookie(self, key):
         """