| | |
- add_rule(rule_fn, phony=False, root=None, autocmds=None, autodeps=None, internal=0, always=False, state=None, targetinfo=None, categories=[], lockfilename=None)
- Adds a new Yabs rule.
<rule_fn>
<rule_fn> should be a function that takes a target filename and a
context (an object with a .state member that is the Yabs state, and a
.out member to which text output should be sent). If the rule cannot
create the target, it should return None. Otherwise, it should return a
tuple with an initial portion of the following list:
<commands>
A string consisting of individual commands separated by
newlines, or an empty string if no command is necessary.
<prerequisites>
A list of prerequisites (files that are required to build the
target).
<semi_prerequisites>:
A list of files that don't have to exist, but will cause a
remake of <target> if they can't be built or are newer than
<target>.
<extraprereq_fn>
A function that will be called after all prerequisites and
semi-prerequisites have been created, which should return
any additional prereqistes that can only be specified by
examining the existing prerequisites. This function takes a
single State parameter, and should return a list of additional
prerequisites. It is repeatedly called until it returns an
empty list.
<internal>
A number that is added to the rule's <internal> value. This is
useful if a rule applies to different types of targets which
require different levels of diagnostics.
<resource>
Resource required by the rule when it runs. Only used if Yabs
is concurrent. <resource> must provide two methods:
tryacquire():
Returns True if resource is acquired. Otherwise returns
False or returns the absolute time at which the
resource may become available - Yabs will check again
after this time.
If an exception is raised, Yabs assumes that the
resource will never become available, and all targets
that require that resource are marked as having failed.
release():
Releases resource.
*Disabled*: if <rule_fn> is a string, it will be compiled into an
anonymous function using yabs.fn().
If a command is prefixed with a '-', any error it returns is ignored.
Rules are inserted at front of <state>'s rules, so rules added later
will be found first by the yabs.make() function.
The returned prerequisites and semi-prerequisites can instead be
functions, which take a single state parameter, which return a list of
filenames when called. Alternatively, they can be a string, which is
converted into a single (semi-)prerequisite.
Similarly, the returned command can be a function taking a single
state parameter that returns a command-string or None, or raises an
exception. This function can generate the target itself directly, but
it is usually better to return a command-string to allow autocmds and
autodeps to work.
A prerequisite or semi-prerequisite that is None is treated specially -
it ensures that a target is always remade.
If a prerequisite or semi-prerequisite is RemakeIfZeroLength, the
target will always be remade if it exists but has zero length.
<phony>
If <phony> is True, Yabs never looks at <target>'s datestamp, and
doesn't require that <target> exists after the rule's commands are
run. The target is still treated as a real filename though - it will be
converted to an absolute filename using <root> (if specified) or the
current directory.
<root>
If <root> is None, the rule is always passed absolute filenames,
and must return absolute filenames for both prerequisites and
semi-prerequisites.
Otherwise, <root> can be an integer, which identifies a function in
the backtrace, whose module's directory is taken as the root for the
rule. E.g. root=yabs.caller will use the module of the function that
called yabs.add_rule(). root=yabs.caller+1 will use this function's
caller's module etc.
Otherwise, <root> must be an absolute path, (optionally ending
with os.sep) and all targets will be given relative to <root>. Any
prerequisite or semi-prerequisite returned from <rule-function> that
are not an absolute path will have <root> prepended. The command
returned from the rule will be run with the current directory set to
<root>. Alternatively, <root> can be yabs.caller, in which case yabs
will replace it with the directory containing the file that calls
yabs.add_rule().
<autodeps>
If not None, <autodeps> is treated as a filename suffix that is
appended to a target filename to form a dependency filename. If
the dependency file exists, its contents are treated like extra
semi-prerequisites. When the rule's commands are run, all files that
are opened for reading by these commands are logged to the dependency
file. Note that this system has only been implemented for Unix systems,
using the $LD_PRELOAD environmental variable with a special shared
library, yabs.autodeps_so, that is built automatically. This shared
library has only been built for OpenBSD and Linux at the time of
writing; other Unix systems may require slightly different build
parameters.
Also, files that the command attempts, but fails, to open, are also
logged. They are treated as a third form of prerequisite, where
inability to remake the prerequisite doesn't force a remake of the
target; this is the correct behaviour - if such a prerequisites exists,
then the command will always be re-run, but if it still doesn't
exist, it is safe to not re-run the command. [At the moment, this is
done using a hack - the semi-prerequisite is prefixed with a `-';
ultimately, it would make more sense to recognise the three forms of
prerequisites that seem to be relevent to Yabs, perhaps using simple
`', `-' and `--' prefixes.] This is necessary in situations involving
multiple include-paths, such as the following:
compiler include path is: /usr/local/include:/usr/include
/usr/local/include/foo.h does not exist. /usr/include/foo.h exists.
source file main.c contains <code>#include "foo.h"</code> Yabs is
asked to build main.o, so compiles main.c /usr/local/include/foo.h
is created. Yabs is asked to build main.o again.
The creation of the file /usr/local/include/foo.h should force a
recompilation of main.c, even though the attempt to open that file
will have failed previously, and the filename will not be listed in
tradional auto-dependency information such as the output from gcc
-M. Note that gcc doesn't seem to attempt to open the header file when
the parent directory doesn't exist, so this system isn't bomb-proof.
<autocmds>
If not None, <autocmds> is treated as a filename suffix that is
appended to target filenames, and used to store the command(s) that
<rule_fn> returns when asked how to create a target. If the commands
are changed (e.g. <rule_fn> is modified), Yabs will detect this and
force a remake of the target, even if it is newer than all of its
prerequisites.
The <autocmds> system works in the following way: yabs.add_rule()
makes a call to yabs._add_autocmds_rule(), so that Yabs will have a
rule available that will write the commands for a target <foo> to a
file <foo><autocmds>. Also, Yabs ensures that <target><autocmds> is
always added to the prerequisites returned from <rule_fn>. Note that
this mechanism results in an extra file per target; if there are many
targets, it might be better to use the technique in yabs3's compilation
rules where command files are per-directory.
<internal>
<internal> is used to reduce the diagnostics when this rule is
used. For example, setting internal=1 will reduce the effective
state.debug level by 1 when outputing diagnostics in connection with
this rule. [this is currently broken].
<always>
If True, this rule's command(s) will be run each time the target is
required. Usually, a target's commands are run only once.
<state>
If <state> is None, yabs.default_state is used.
<targetinfo>
If not None, is yielded by known_targets(). E.g. could be a
regex used by the rule when matching targets.
<categories>
A list of categories, e.g. the objects returned by
make_rulecategory_suffix() or make_rulecategory_root(). Categories are
used purely as a speed optimisation, allowing Yabs to avoid calling
rules for targets that the rule will not match. Yabs remembers the
results of calling a category for a particular target so if (as is the
intention) many rules use the same category objects, Yabs will be able
to detect when it doesn't have to call a rule, with very little cost.
A category should be callable, taking a single <target> - an absolute
target name. It should return True if the category matches (indicating
that rules that include the category may match <target>) or False
(indicating that rules that include the category will always return
None for this target).
<lockfilename>
If specified, should be either a string or a ( string, timeout)
pair. The specified file is locked while the rule runs. This can be
used to avoid concurrency problems between different invocations of
Yabs programmes. If <lockfilename> contains '%s', it is replaced by the
name of the target. If <timeout> is specified, the attempted lock will
time out in the specified number of seconds.
We return the created _Rule object. This is useful because it contains the
rule's root if applicable.
Example usage:
def myrule( target, state):
if target!='myprog.exe': return None
prereqs = [ 'foo.o', 'bar.o']
semiprereqs = [ 'foo.h', 'bar.h']
command = 'link -o ' + target + ' ' + string.join( prereqs, ' ')
return prereqs, semiprereqs, command
yabs.add_rule( myrule)
- callers()
- cancel_targets(targets, state)
- command_run(command, state=None, cwd=None, autodeps_filename=None, fn=None, endtime=None, prefixes=None)
- Runs newline-separated commands in <command>, prefixing each with `cd <cwd>
&& ' if <cwd> is specified. Also prefixes with settings for $LD_PRELOAD
and $YABS_AUTODEPS_FILENAME if <autodeps_filename> is set, so that auto
dependencies are written to <autodeps_filename>.
If <state>.silent is False, commands are written to stdout before being
run. If <state>.dryrun is True, command are not actually run. If
<state>.echo is True (the default), output from commands is written to
stdout as the command executes; see yabs2's -e option.
Returns ( e, outputtext). e is 0 if no error occurred, else a
yabserrors.command_error, which will also contain the output text from the
last sub-command.
If state.use_os_system is True, always calls commands using os.system. This
is to overcome problems with child tasks not responding properly to stdin
etc. When python 2.4 is widespred, will hopefully be able to rely on the
subprocess module.
Absorbs exceptions.
- command_run_text(command, state=None, cwd=None, autodeps_filename=None, fn=None, endtime=None)
- As command_run, but returns text, or raises exception.
- current_exception_as_string(limit=None, prefix='')
- Returns string representation of the current exception.
- debug(*args)
- debug1(*args)
- #if 'server-41' in os.uname()[1]:
- exception_as_string(etype, value, tb, limit=None, prefix='')
- Returns string representation of the exception specified by (etype, value,
tb).
- fn(text, globals_=None, locals_=None, up=None)
- Returns an anonymous function created by calling exec on <text>.
<text> should be a string containing a Python function definition,
ommiting the initial `def <fnname>' (actually, leading `def' can be
retained, for clarity). Any leading/trailing white space is removed using
str.strip(). Leading white space on all lines will be handled automatically
by exec, so you can use an indented python triple-quoted string.
In addition, newlines and backslashes are re-escaped inside
single/double-quoted strings. this enables nested use of fn().
e.g.:
yabs.fn(
"""
def ( target, state):
if target != 'foo': return None
return [], None, 'touch foo'
""")
If globals_/locals_ are not specified, fn() will find its caller's
globals/locals using yabs._upglobals()/yabs._uplocals(). If <up> is
specified, yabs.fn() will look at the stack frame <up> levels up.
Unfortunately the use of globals_/locals_ doesn't work in exactly the way
one would hope - the resultant function doesn't seem to be able to access
locals variables. E.g.:
import yabs
def a():
x = 42
f = yabs.fn( """
def ():
print x # fails
def g():
print x # succeeds
f()
g()
a()
See make.py:test22().
The created function will have a .func_code attribute like conventional
functions. Unfortunately this attribute is read-only, and the only
element that we can control is .func_code.co_name, which we set to
encode our caller's filename, function and line number (using '_' for
non-alphanumberic characters). An example of a .func_code passed through
str() is:
import yabs
f = yabs.fn( """
def():
pass
""")
print f.func_code
- which outputs:
<code object _yabs_fn_temp___foo_py___4 at 0xb7be9b20, file "<string>", line 1>
- get_caller_directory(n=1)
- Returns absolute path of the directory containing the python module
corresponding to the function <n> levels up the call stack. Directory's
trailing os.sep is retained.
E.g. yabs.get_caller_directory(0) returns the directory containing the
yabs.py module (which contains the get_caller_directory() function itself).
get_caller_directory(1) returns the directory containing the caller's
module. get_caller_directory(2) returns the directory containing the
caller's caller's module.
Usually get_caller_directory(1) is the desired call. yabs.caller is a
constant that gives the same effect, e.g.: yabs.get_caller_directory(
yabs.caller).
- handle_sighup(state=None)
- adds handler for SIGHUP that sets <state>.received_sighup.
start_make() polls this when it is called, and outputs a diagnostic
about the current state.
- known_targets(state=None)
- Yields (<root>, <targetinfo>) for each rule that as a non-None
<targetinfo>. E.g. rules registered with yabs2.add_patternrule()
and yabs3.add_exe(), yabs3.add_so() etc. Intended for use as a
diagnostic, e.g. to show the user what targets are available.
- make(target, prefix=None, echo_prefix=None, state=None)
- If <target> is a relative path, <target> is converted to an absolute path
using os.path.abspath().
Uses the registered rules in <state> to try to remake <target>'s
prerequisites, semi-prerequisites and <target> itself, returning one of the
following:
yabs.unchanged - <target>'s rule's command was not run because <target>
was found to be up-to-date, or the command was run but the timestamp of
<target> was unchanged.
yabs.changed - <target>'s rule's command was run and it updated
<target>, or it was run and was a phony rule.
A list containing information about why <target> could not be
made. Each item in the list is a 3-tuple ( <rule>, <prerequisite>,
<error>), where <rule> is an instance of the yabs._Rule class,
describing a rule that claimed to be able to remake <target>, but which
required a prerequisite <prerequisite> that could not be made. The
<error> item is the error returned from make() when it was recursively
called to remake <prerequisite>, so the list returned from make() is
actually a tree-like structure. Note that <target> does not appear in
the list. The function yabs.print_failures() can be used to display the
list in a human readable format.
All diagnostics are prefixed with <prefix>. <prefix> can be a function
taking no params, in which case it should return a prefix string.
Note that make() does not first check that rules exist for all of a rule's
prerequisites before attempting to create the first prerequisite.
This has the disadvantage that it means that rules could be run even when
they ultimately don't work because some other prerequisites cannot be made.
It has the advantage that the rules can depend on files created by earlier
rules; for example a rule could untar a .tar file, and subsequent rules
will be able to respond to the extracted files. It might even be possible
to import extra rules from these files into the build programme itself. See
http://tinyurl.com/yqk6l for a usenet posting that mentions this issue.
- make_rulecategory_root(root)
- make_rulecategory_suffix(suffix)
- mt_exit(state)
- mtime(filename, mtimecache=None)
- Returns modification time of <filename>, or 0 if <filename> doesn't exist,
using caching.
- mtime_add_oldprefix(tree, mtimecache=None)
- mtime_flush(filename, mtimecache=None)
- Removes any cached modification time for <filename>, including entries from
calls to mtime_markold() and mtime_marknew().
For example, this is called after a command is run that remakes a target.
- mtime_mark_nonexistent(filename, mtimecache=None)
- Future calls to mtime( <filetime>, <mtimecache>) will return 0.
- mtime_marknew(filename, mtimecache=None)
- Future calls to mtime( <filetime>, <mtimecache>) will return fixed value
that is a long time in the future.
- mtime_markold(filename, mtimecache=None)
- Future calls to mtime( <filetime>, <mtimecache>) will return 1.
- mtime_raw(filename, mtimecache=None)
- Returns modification time of <filename>, or 0 if it doesn't exist.
- periodic_debug(text, state=None)
- place(n=1)
- Useful debugging function - returns representation of source position of
caller.
- print_exc(limit=None, file=None, prefix='')
- Like traceback.print_exc(), except uses print_exception, so customised
backtrace formatting will be used.
- print_exception = _traceback_print_exception_wrapper(etype, value, tb0, limit=None, file=None, prefix='')
- Ignores the <prefix> param.
- print_exception_compact(etype, value, tb0, limit=None, file=None, prefix='')
- Alternative to traceback.print_exception; prints a more compact backtrace,
with one line per frame, each line looking like:
<filename>:<line>:<functionname>():<codetext>
- print_failures(whereto, prefixes, target, rule, e, depth, state, format, recurse)
- Outputs human-readable information about a failed call to yabs.make().
<e> should be an error returned from make() - see yabs.make() for a
description of the format.
- relativepath(from0, to_, sep='/', up='../')
- relativepath() is rather heavily used, often with identical parameters, so
we use a cache - significantly improves speed.
- relativepath_simple(from0, to_, sep='/', up='../')
- Constructs relative path, from directory <from0> to path <to_>. This has be
optimised for speed - see alternative implementations below.
- relativepath_user(user, to, sep='/', up='../')
- If <to> is within ~<user>, returns representation of <to> starting with
`~<user>'. Otherwise returns absolute path of <to>.
- set_file_rule(rulefn, state=None)
- start_make(target, state, prefix=None, echo_prefix=None, prefixes=None, nesting=0, nesting_targets=[])
- This is the heart of Yabs. We are a Python generator. We start
a builds of <target>, then repeatedly yield a dummy value until
the build has completed, leaving the result in state.targetcache[
target]. We use yield in this way so that concurrent builds can
schedule nested prerequisites without blocking. We never block; the
caller is expected to block with state._wait() during iteration.
We call ourselves recursively to build prerequisites.
- subprocess(command, echo=None, state=None, fn=None, endtime=None, prefix=None)
- Runs <command> using <fn>, which defaults to <state>.subprocessfn, which in
turn defaults to _subprocess_popen4.
If specified, <endtime> should be an absolute time in seconds since
epoc, as returned by time.time(), and the command is terminated at this
time. This is not supported by all _subprocess_* functions.
<prefix> can be a string, or a function that returns a string; it will be
used to prefix all lines returned in <text> (and echoed to screen if <echo>
is set). If None, <state.echo_prefix> is used.
Returns (e, text), where <e> is the integer return code, and <text> is the
text output by the command.
- subprocess_text(command, echo=None, state=None, fn=None, endtime=None, prefix=None)
- Returns just the text part of the return from subprocess() function. Raises
exception if the subprocess() function indicates an error.
- test_flock()
- Test flock works with multiple processes and threads.
- tstats_print()
- updatediff(filename, new_contents, always=False)
- Writes <new_contents> to <filename> if <filename> doesn't exist, or
<filename>'s existing contents differ from <new_contents>.
File's parent directory is always made if not already present.
Returns True if file has been changed, or False if file has been left
unchanged.
If <always> is true, always overwrites file, but still returns False if
contents are unchanged.
- xtermtitle_usertarget(target, state)
- writes <user>@<host>:<target> into titlebar of xterm. intended for use as
<state>.premake_fn.
|