[H-GEN] gnu make command-line target ordering problems?

[Ben Carlyle]

[ Humbug *General* list - semi-serious discussions about Humbug and
Unix-related topics. ]

Anthony Towns wrote:

> > I think the simplest way to state my case is this:
> > The parallel execution of command-line targets violates the make
> > commands are processed in order.

> Similary, the parallel execution of dependent targets violates the
> rule that dependents are processed in order. The real question is
> whether this is particularly useful.

Is your in-order processing rule documented?  <curious>

> > If the parallel make does not
> > process the command-line in order, then it has different behaviour
> > characteristics to the sequential make.
> Similarly, if the parallel make does not process the dependent targets
> inorder, then it has different behavioural characteristics to the
> sequential make.

The principal behind parallel builds is that in a Makefile with fully
expressed dependancies, the behavioural characteristics of sequential
and parallel make are identical.  Modern makefiles are to maximise
visibility of the dependancy tree to make in order to allow maximum
concurrent processing.

> > The command-line should
> > not specify a list of depedancies, but a list of targets to be
> > executed one after the other.
> Or, alternately, it should specify a list of goals, all of which have
> to be made. If you want to build your system, you should have a single
> goal that does just that. If you want to clean your system, you should
> have a single goal that does that.

As I do have.

> Having a goal `build' that only works when you've already done a `clean',
> is broken.

In my own defense I ask you to note what happens when you modify a
makefile.  Suddenly your targets are not identical to those that
correspond to that makefile.  Do you have every single target in
your system with a dependancy listed on the makefile?
Personally I do not.  In some cases I prefer clarity in makefiles over
fully-defined dependancies.
There are often reasons to do a make clean, then make build, especially
when makefile development is being performed.

> > Executing the command-line parameters in parallel is equivalent
> > to adding a rule
> > commandline: arg1 arg2 arg3
> > to the makefile.
> > Executing the command-line parameters sequentially is equivalent
> > to adding the rules
> > commandline: arg3
> > arg3: arg2
> > arg2: arg1

> Note that both `arg1 arg2 arg3' is much more similar to what you actually
> write on the command line.

I conceed this.

> > It is this second rule that best represents the behaviour of a
> > sequential make,

> Note that the first rule exactly represents the behaviour of both
> sequential and parallel (GNU) make.

If your rule stated at the top of this email is documented and
in fact the dependancies are guranteed to be exectued in order,
then I conceed this point also.  I believe in this case, however,
that your rule cannot apply in a distributed build environment.
Since it does not apply in the distributed build, consistency
demands that it not apply to in the sequential build which is
a special case of the distributed build.  Only the second rule
can be applied to both targets regardless of the order of
dependancies.

> > The statement that make
> > processes its targets in order must be deleted from all
> > documentation, for in fact it executes them in a random order that
> > just so happens usually runs from first to last.
> Imagine. A bug in the documentation. (Furthermore, I suspect you're
> guaranteed to have them started in the order given)

In fact I conceeded this point in my first email.  I also noted
that this gurantee is meaningless due to the nature of a unix system
which places no gurantee on the execution order of concurrently
executing processes.

> > Parallelism should be left to the defined make dependancies, not
> > assumed from the commandline.  make clean should run parallel
> > clean targets.  make build should run parallel build targets.
> > make clean build should run parallel clean targets, then parallel
> > build targets.

> Nonsense. You should always tell make about *every* dependency in your
> build environment. You shouldn't leave it up to chance, or implicit
> ordering based on which one you happen to specify first. If your template
> libraries depend on every single .c and .h file, you should tell make
> that, and have it rebuild them itself when it thinks it needs to.

Nonsense yourself.

This case is perhaps more complicated than you realise.

Firstly, I am controlling a centralised rule set that is
included by each makefile.  The rule set has to be strict
enough to keep the makefiles simple, but also versitile
enough to make everything from C and C++ code to gperf
and yacc and shared libraries and static libraries to
relocatable objects that are linked at runtime.  I'm currently
looking at around 3000 lines of included makefile.  The makefiles
themselves add up to an additional 30klocs.  It's a complicated
system with complicated dependancy structures that can either
be expressed in every rule at a large software engineering
and documentation cost, or the make system can be encapsulated
into "super-targets" with simple interdependancies.  My system
is an attempt almost at namespaces in Makefiles.  By wrapping
up dependancies into blocks which have clear orderings I can
simplify the overall structures.

Now you'll argue that I should be running these as shell scripts,
and that could work at the top-most level... in fact that is
truely what I am doing.  I am encapsualting shell scripts into
phoney targets to force the system to behave in simple ways.
On lower levels dependancies are clear and simple.  At higher
levels dependancies are handled through shell scripts that
call targets in particular orders.  At every level life is
simple, despite the size.

I call into question your expertise with systems of this size
and complexity.  `Always' is a very strong word in software
engineering.

> Personally, I think `make clean' isn't an ideal thing anyway --- make's
> good at building stuff, having it destroy stuff too isn't Right. Much
> better is the way Aegis (a source control tool, a la CVS) does it: by
> having two separate tools: aec (aegis clean) and aeb (aegis build). The
> former deletes any files in the source tree that aren't in the repository
> (ie, anything that's an object file and any other trash lying around),
> and the latter calls make (or an equivalent).

That's not a bad idea.  In my CM tool the only simple way to do that
is to completely delete the work areas, then sync them back out of
the database.  This takes an hour or two for a system of this size,
so until I have a better solution I'll continue to use make for this
purpose.


Benjamin.

--
This is list (humbug) general handled by majordomo at lists.humbug.org.au .
Postings only from subscribed addresses of lists general or general-post.