Merge branch 'jc/doc-maintainer'

Describe tools for automation that were invented since this
document was originally written.

* jc/doc-maintainer:
  howto/maintain: document "### match next" convention in jch/pu branch
  howto/maintain: mark titles for asciidoc
  Documentation: update "howto maintain git"

1 files changed, 256 insertions, 91 deletions

diff --git a/Documentation/howto/maintain-git.txt b/Documentation/howto/maintain-git.txt
index ea6e4a52c..816c79150 100644
--- a/Documentation/howto/maintain-git.txt
+++ b/Documentation/howto/maintain-git.txt
@@ -10,35 +10,42 @@ Content-type: text/asciidoc
 How to maintain Git
 ===================
 
+Activities
+----------
+
 The maintainer's git time is spent on three activities.
 
- - Communication (60%)
+ - Communication (45%)
 
    Mailing list discussions on general design, fielding user
    questions, diagnosing bug reports; reviewing, commenting on,
    suggesting alternatives to, and rejecting patches.
 
- - Integration (30%)
+ - Integration (50%)
 
    Applying new patches from the contributors while spotting and
    correcting minor mistakes, shuffling the integration and
    testing branches, pushing the results out, cutting the
    releases, and making announcements.
 
- - Own development (10%)
+ - Own development (5%)
 
    Scratching my own itch and sending proposed patch series out.
 
+The Policy
+----------
+
 The policy on Integration is informally mentioned in "A Note
 from the maintainer" message, which is periodically posted to
 this mailing list after each feature release is made.
 
-The policy.
-
  - Feature releases are numbered as vX.Y.Z and are meant to
    contain bugfixes and enhancements in any area, including
    functionality, performance and usability, without regression.
 
+ - One release cycle for a feature release is expected to last for
+   eight to ten weeks.
+
  - Maintenance releases are numbered as vX.Y.Z.W and are meant
    to contain only bugfixes for the corresponding vX.Y.Z feature
    release and earlier maintenance releases vX.Y.Z.V (V < W).
@@ -62,12 +69,15 @@ The policy.
  - 'pu' branch is used to publish other proposed changes that do
    not yet pass the criteria set for 'next'.
 
- - The tips of 'master', 'maint' and 'next' branches will always
-   fast-forward, to allow people to build their own
-   customization on top of them.
+ - The tips of 'master' and 'maint' branches will not be rewound to
+   allow people to build their own customization on top of them.
+   Early in a new development cycle, 'next' is rewound to the tip of
+   'master' once, but otherwise it will not be rewound until the end
+   of the cycle.
 
- - Usually 'master' contains all of 'maint', 'next' contains all
-   of 'master' and 'pu' contains all of 'next'.
+ - Usually 'master' contains all of 'maint' and 'next' contains all
+   of 'master'.  'pu' contains all the topics merged to 'next', but
+   is rebuilt directly on 'master'.
 
  - The tip of 'master' is meant to be more stable than any
    tagged releases, and the users are encouraged to follow it.
@@ -77,14 +87,22 @@ The policy.
    are found before new topics are merged to 'master'.
 
 
+A Typical Git Day
+-----------------
+
 A typical git day for the maintainer implements the above policy
 by doing the following:
 
- - Scan mailing list and #git channel log.  Respond with review
-   comments, suggestions etc.  Kibitz.  Collect potentially
-   usable patches from the mailing list.  Patches about a single
-   topic go to one mailbox (I read my mail in Gnus, and type
-   \C-o to save/append messages in files in mbox format).
+ - Scan mailing list.  Respond with review comments, suggestions
+   etc.  Kibitz.  Collect potentially usable patches from the
+   mailing list.  Patches about a single topic go to one mailbox (I
+   read my mail in Gnus, and type \C-o to save/append messages in
+   files in mbox format).
+
+ - Write his own patches to address issues raised on the list but
+   nobody has stepped up solving.  Send it out just like other
+   contributors do, and pick them up just like patches from other
+   contributors (see above).
 
  - Review the patches in the saved mailboxes.  Edit proposed log
    message for typofixes and clarifications, and add Acks
@@ -100,40 +118,32 @@ by doing the following:
    - Obviously correct fixes that pertain to the tip of 'master'
      are directly applied to 'master'.
 
+   - Other topics are not handled in this step.
+
    This step is done with "git am".
 
      $ git checkout master    ;# or "git checkout maint"
-     $ git am -3 -s mailbox
+     $ git am -sc3 mailbox
      $ make test
 
- - Merge downwards (maint->master):
-
-     $ git checkout master
-     $ git merge maint
-     $ make test
+   In practice, almost no patch directly goes to 'master' or
+   'maint'.
 
  - Review the last issue of "What's cooking" message, review the
-   topics scheduled for merging upwards (topic->master and
-   topic->maint), and merge.
+   topics ready for merging (topic->master and topic->maint).  Use
+   "Meta/cook -w" script (where Meta/ contains a checkout of the
+   'todo' branch) to aid this step.
+
+   And perform the merge.  Use "Meta/Reintegrate -e" script (see
+   later) to aid this step.
+
+     $ Meta/cook -w last-issue-of-whats-cooking.mbox
 
      $ git checkout master    ;# or "git checkout maint"
-     $ git merge ai/topic     ;# or "git merge ai/maint-topic"
+     $ echo ai/topic | Meta/Reintegrate -e ;# "git merge ai/topic"
      $ git log -p ORIG_HEAD.. ;# final review
      $ git diff ORIG_HEAD..   ;# final review
      $ make test              ;# final review
-     $ git branch -d ai/topic ;# or "git branch -d ai/maint-topic"
-
- - Merge downwards (maint->master) if needed:
-
-     $ git checkout master
-     $ git merge maint
-     $ make test
-
- - Merge downwards (master->next) if needed:
-
-     $ git checkout next
-     $ git merge master
-     $ make test
 
  - Handle the remaining patches:
 
@@ -142,9 +152,9 @@ by doing the following:
      and not in 'master') is applied to a new topic branch that
      is forked from the tip of 'master'.  This includes both
      enhancements and unobvious fixes to 'master'.  A topic
-     branch is named as ai/topic where "ai" is typically
-     author's initial and "topic" is a descriptive name of the
-     topic (in other words, "what's the series is about").
+     branch is named as ai/topic where "ai" is two-letter string
+     named after author's initial and "topic" is a descriptive name
+     of the topic (in other words, "what's the series is about").
 
    - An unobvious fix meant for 'maint' is applied to a new
      topic branch that is forked from the tip of 'maint'.  The
@@ -162,7 +172,8 @@ by doing the following:
 
    The above except the "replacement" are all done with:
 
-     $ git am -3 -s mailbox
+     $ git checkout ai/topic ;# or "git checkout -b ai/topic master"
+     $ git am -sc3 mailbox
 
    while patch replacement is often done by:
 
@@ -170,93 +181,170 @@ by doing the following:
 
    then replace some parts with the new patch, and reapplying:
 
+     $ git checkout ai/topic
      $ git reset --hard ai/topic~$n
-     $ git am -3 -s 000*.txt
+     $ git am -sc3 -s 000*.txt
 
    The full test suite is always run for 'maint' and 'master'
    after patch application; for topic branches the tests are run
    as time permits.
 
- - Update "What's cooking" message to review the updates to
-   existing topics, newly added topics and graduated topics.
+ - Merge maint to master as needed:
 
-   This step is helped with Meta/cook script (where Meta/ contains
-   a checkout of the 'todo' branch).
-
- - Merge topics to 'next'.  For each branch whose tip is not
-   merged to 'next', one of three things can happen:
+     $ git checkout master
+     $ git merge maint
+     $ make test
 
-   - The commits are all next-worthy; merge the topic to next:
+ - Merge master to next as needed:
 
      $ git checkout next
-     $ git merge ai/topic     ;# or "git merge ai/maint-topic"
+     $ git merge master
      $ make test
 
+ - Review the last issue of "What's cooking" again and see if topics
+   that are ready to be merged to 'next' are still in good shape
+   (e.g. has there any new issue identified on the list with the
+   series?)
+
+ - Prepare 'jch' branch, which is used to represent somewhere
+   between 'master' and 'pu' and often is slightly ahead of 'next'.
+
+     $ Meta/Reintegrate master..pu >Meta/redo-jch.sh
+
+   The result is a script that lists topics to be merged in order to
+   rebuild 'pu' as the input to Meta/Reintegrate script.  Remove
+   later topics that should not be in 'jch' yet.  Add a line that
+   consists of '### match next' before the name of the first topic
+   in the output that should be in 'jch' but not in 'next' yet.
+
+ - Now we are ready to start merging topics to 'next'.  For each
+   branch whose tip is not merged to 'next', one of three things can
+   happen:
+
+   - The commits are all next-worthy; merge the topic to next;
    - The new parts are of mixed quality, but earlier ones are
-     next-worthy; merge the early parts to next:
+     next-worthy; merge the early parts to next;
+   - Nothing is next-worthy; do not do anything.
+
+   This step is aided with Meta/redo-jch.sh script created earlier.
+   If a topic that was already in 'next' gained a patch, the script
+   would list it as "ai/topic~1".  To include the new patch to the
+   updated 'next', drop the "~1" part; to keep it excluded, do not
+   touch the line.  If a topic that was not in 'next' should be
+   merged to 'next', add it at the end of the list.  Then:
+
+     $ git checkout -B jch master
+     $ Meta/redo-jch.sh -c1
+
+   to rebuild the 'jch' branch from scratch.  "-c1" tells the script
+   to stop merging at the first line that begins with '###'
+   (i.e. the "### match next" line you added earlier).
+
+   At this point, build-test the result.  It may reveal semantic
+   conflicts (e.g. a topic renamed a variable, another added a new
+   reference to the variable under its old name), in which case
+   prepare an appropriate merge-fix first (see appendix), and
+   rebuild the 'jch' branch from scratch, starting at the tip of
+   'master'.
+
+   Then do the same to 'next'
 
      $ git checkout next
-     $ git merge ai/topic~2   ;# the tip two are dubious
-     $ make test
+     $ sh Meta/redo-jch.sh -c1 -e
 
-   - Nothing is next-worthy; do not do anything.
+   The "-e" option allows the merge message that comes from the
+   history of the topic and the comments in the "What's cooking" to
+   be edited.  The resulting tree should match 'jch' as the same set
+   of topics are merged on 'master'; otherwise there is a mismerge.
+   Investigate why and do not proceed until the mismerge is found
+   and rectified.
 
- - [** OBSOLETE **] Optionally rebase topics that do not have any commit
-   in next yet, when they can take advantage of low-level framework
-   change that is merged to 'master' already.
+     $ git diff jch next
 
-     $ git rebase master ai/topic
+   When all is well, clean up the redo-jch.sh script with
 
-   This step is helped with Meta/git-topic.perl script to
-   identify which topic is rebaseable.  There also is a
-   pre-rebase hook to make sure that topics that are already in
-   'next' are not rebased beyond the merged commit.
+     $ sh Meta/redo-jch.sh -u
 
- - [** OBSOLETE **] Rebuild "pu" to merge the tips of topics not in 'next'.
+   This removes topics listed in the script that have already been
+   merged to 'master'.  This may lose '### match next' marker;
+   add it again to the appropriate place when it happens.
 
-     $ git checkout pu
-     $ git reset --hard next
-     $ git merge ai/topic     ;# repeat for all remaining topics
-     $ make test
+ - Rebuild 'pu'.
 
-   This step is helped with Meta/PU script
+     $ Meta/Reintegrate master..pu >Meta/redo-pu.sh
 
- - Push four integration branches to a private repository at
-   k.org and run "make test" on all of them.
+   Edit the result by adding new topics that are not still in 'pu'
+   in the script.  Then
 
- - Push four integration branches to /pub/scm/git/git.git at
-   k.org.  This triggers its post-update hook which:
+     $ git checkout -B pu jch
+     $ sh Meta/redo-pu.sh
 
-    (1) runs "git pull" in $HOME/git-doc/ repository to pull
-        'master' just pushed out;
+   When all is well, clean up the redo-pu.sh script with
 
-    (2) runs "make doc" in $HOME/git-doc/, install the generated
-        documentation in staging areas, which are separate
-        repositories that have html and man branches checked
-        out.
+     $ sh Meta/redo-pu.sh -u
 
-    (3) runs "git commit" in the staging areas, and run "git
-        push" back to /pub/scm/git/git.git/ to update the html
-        and man branches.
+   Double check by running
 
-    (4) installs generated documentation to /pub/software/scm/git/docs/
-        to be viewed from http://www.kernel.org/
+     $ git branch --no-merged pu
 
- - Fetch html and man branches back from k.org, and push four
-   integration branches and the two documentation branches to
-   repo.or.cz and other mirrors.
+   to see there is no unexpected leftover topics.
 
+   At this point, build-test the result for semantic conflicts, and
+   if there are, prepare an appropriate merge-fix first (see
+   appendix), and rebuild the 'pu' branch from scratch, starting at
+   the tip of 'jch'.
+
+ - Update "What's cooking" message to review the updates to
+   existing topics, newly added topics and graduated topics.
+
+   This step is helped with Meta/cook script.
+
+     $ Meta/cook
+
+   This script inspects the history between master..pu, finds tips
+   of topic branches, compares what it found with the current
+   contents in Meta/whats-cooking.txt, and updates that file.
+   Topics not listed in the file but are found in master..pu are
+   added to the "New topics" section, topics listed in the file that
+   are no longer found in master..pu are moved to the "Graduated to
+   master" section, and topics whose commits changed their states
+   (e.g. used to be only in 'pu', now merged to 'next') are updated
+   with change markers "<<" and ">>".
+
+   Look for lines enclosed in "<<" and ">>"; they hold contents from
+   old file that are replaced by this integration round.  After
+   verifying them, remove the old part.  Review the description for
+   each topic and update its doneness and plan as needed.  To review
+   the updated plan, run
+
+     $ Meta/cook -w
+
+   which will pick up comments given to the topics, such as "Will
+   merge to 'next'", etc. (see Meta/cook script to learn what kind
+   of phrases are supported).
+
+ - Compile, test and install all four (five) integration branches;
+   Meta/Dothem script may aid this step.
+
+ - Format documentation if the 'master' branch was updated;
+   Meta/dodoc.sh script may aid this step.
+
+ - Push the integration branches out to public places; Meta/pushall
+   script may aid this step.
+
+Observations
+------------
 
 Some observations to be made.
 
- * Each topic is tested individually, and also together with
-   other topics cooking in 'next'.  Until it matures, none part
-   of it is merged to 'master'.
+ * Each topic is tested individually, and also together with other
+   topics cooking first in 'pu', then in 'jch' and then in 'next'.
+   Until it matures, no part of it is merged to 'master'.
 
  * A topic already in 'next' can get fixes while still in
    'next'.  Such a topic will have many merges to 'next' (in
    other words, "git log --first-parent next" will show many
-   "Merge ai/topic to next" for the same topic.
+   "Merge branch 'ai/topic' to next" for the same topic.
 
  * An unobvious fix for 'maint' is cooked in 'next' and then
    merged to 'master' to make extra sure it is Ok and then
@@ -278,3 +366,80 @@ Some observations to be made.
  * Being in the 'next' branch is not a guarantee for a topic to
    be included in the next feature release.  Being in the
    'master' branch typically is.
+
+
+Appendix
+--------
+
+Preparing a "merge-fix"
+~~~~~~~~~~~~~~~~~~~~~~~
+
+A merge of two topics may not textually conflict but still have
+conflict at the semantic level. A classic example is for one topic
+to rename an variable and all its uses, while another topic adds a
+new use of the variable under its old name. When these two topics
+are merged together, the reference to the variable newly added by
+the latter topic will still use the old name in the result.
+
+The Meta/Reintegrate script that is used by redo-jch and redo-pu
+scripts implements a crude but usable way to work this issue around.
+When the script merges branch $X, it checks if "refs/merge-fix/$X"
+exists, and if so, the effect of it is squashed into the result of
+the mechanical merge.  In other words,
+
+     $ echo $X | Meta/Reintegrate
+
+is roughly equivalent to this sequence:
+
+     $ git merge --rerere-autoupdate $X
+     $ git commit
+     $ git cherry-pick -n refs/merge-fix/$X
+     $ git commit --amend
+
+The goal of this "prepare a merge-fix" step is to come up with a
+commit that can be squashed into a result of mechanical merge to
+correct semantic conflicts.
+
+After finding that the result of merging branch "ai/topic" to an
+integration branch had such a semantic conflict, say pu~4, check the
+problematic merge out on a detached HEAD, edit the working tree to
+fix the semantic conflict, and make a separate commit to record the
+fix-up:
+
+     $ git checkout pu~4
+     $ git show -s --pretty=%s ;# double check
+     Merge branch 'ai/topic' to pu
+     $ edit
+     $ git commit -m 'merge-fix/ai/topic' -a
+
+Then make a reference "refs/merge-fix/ai/topic" to point at this
+result:
+
+     $ git update-ref refs/merge-fix/ai/topic HEAD
+
+Then double check the result by asking Meta/Reintegrate to redo the
+merge:
+
+     $ git checkout pu~5 ;# the parent of the problem merge
+     $ echo ai/topic | Meta/Reintegrate
+     $ git diff pu~4
+
+This time, because you prepared refs/merge-fix/ai/topic, the
+resulting merge should have been tweaked to include the fix for the
+semantic conflict.
+
+Note that this assumes that the order in which conflicting branches
+are merged does not change.  If the reason why merging ai/topic
+branch needs this merge-fix is because another branch merged earlier
+to the integration branch changed the underlying assumption ai/topic
+branch made (e.g. ai/topic branch added a site to refer to a
+variable, while the other branch renamed that variable and adjusted
+existing use sites), and if you changed redo-jch (or redo-pu) script
+to merge ai/topic branch before the other branch, then the above
+merge-fix should not be applied while merging ai/topic, but should
+instead be applied while merging the other branch.  You would need
+to move the fix to apply to the other branch, perhaps like this:
+
+      $ mf=refs/merge-fix
+      $ git update-ref $mf/$the_other_branch $mf/ai/topic
+      $ git update-ref -d $mf/ai/topic