| | |
- add_autocmds_rule(autocmds, state=None)
- This is normally called only internally by Yabs (when add_rule() is
called with a <autocmds> parameter).
Adds a rule for filenames ending with <autocmds>. Given a target
foo<autocmds>, this rule looks in Yabs's rule cache for the
command for target foo, and writes it to foo<autocmds> using
yabs.updatediff().
This function can be called multiple times with the same <autocmds>
- it checks for multiple registration.
- add_rule(rule_fn, phony=False, root=None, autocmds=None, autodeps=None, internal=0, state=None)
- Adds a new Yabs rule.
<rule_fn>
<rule_fn> should be a function that takes a target filename and a
context (an object with a .state member that is the Yabs state, and a
.out member to which text output should be sent). If the rule cannot
create the target, it should return None. Otherwise, it should return a
tuple with an initial portion of the following list:
<commands>
A string consisting of individual commands separated by
newlines, or an empty string if no command is necessary.
<prerequisites>
A list of prerequisites (files that are required to
build the target).
<semi_prerequisites>:
A list of files that don't have to exist, but will cause
a remake of <target> if they can't be built or are newer
than <target>.
<extraprereq_fn>
A function that will be called after all prerequisites
and semi-prerequisites have been created, which should
return any additional prereqistes that can only be
specified by examining the existing prerequisites. This
function takes a single State parameter, and should
return a list of additional prerequisites. It is
repeatedly called until it returns an empty list.
<internal>
A number that is added to the rule's <internal>
value. This is useful if a rule applies to different
types of targets which require different levels of
diagnostics.
<resource>
Resource required by the rule when it runs. Only used if
Yabs is concurrent. <resource> must provide two methods:
tryacquire():
Returns True if resource is acquired, or False.
release():
Releases resource.
*Disabled*: if <rule_fn> is a string, it will be compiled into
an anonymous function using yabs.fn().
If a command is prefixed with a '-', any error it returns is
ignored.
Rules are inserted at front of <state>'s rules, so rules added
later will be found first by the yabs.make() function.
The returned prerequisites and semi-prerequisites can instead
be functions, which take a single state parameter, which return
a list of filenames when called. Alternatively, they can be a
string, which is converted into a single (semi-)prerequisite.
Similarly, the returned command can be a function taking
a single state parameter that returns a command-string or
None, or raises an exception. This function can generate the
target itself directly, but it is usually better to return a
command-string to allow autocmds and autodeps to work.
A prerequisite or semi-prerequisite that is None is treated
specially - it ensures that a target is always remade.
<phony>
If <phony> is True, Yabs never looks at <target>'s datestamp,
and doesn't require that <target> exists after the rule's
commands are run. The target is still treated as a real filename
though - it will be converted to an absolute filename using
<root> (if specified) or the current directory.
<root>
If <root> is None, the rule is always passed absolute filenames,
and must return absolute filenames for both prerequisites and
semi-prerequisites.
Otherwise, <root> can be an integer, which identifies a function in
the backtrace, whose module's directory is taken as the root for the
rule. E.g. root=yabs.caller will use the module of the function that
called yabs.add_rule(). root=yabs.caller+1 will use this function's
caller's module etc.
Otherwise, <root> must be an absolute path, (optionally
ending with os.sep) and all targets will be given relative to
<root>. Any prerequisite or semi-prerequisite returned from
<rule-function> that are not an absolute path will have <root>
prepended. The command returned from the rule will be run with
the current directory set to <root>. Alternatively, <root> can
be yabs.caller, in which case yabs will replace it with the
directory containing the file that calls yabs.add_rule().
<autodeps>
If not None, <autodeps> is treated as a filename suffix that is
appended to a target filename to form a dependency filename. If
the dependency file exists, its contents are treated like extra
semi-prerequisites. When the rule's commands are run, all files
that are opened for reading by these commands are logged to the
dependency file. Note that this system has only been implemented
for Unix systems, using the $LD_PRELOAD environmental variable
with a special shared library, yabs.autodeps_so, that is built
automatically. This shared library has only been built for
OpenBSD and Linux at the time of writing; other Unix systems may
require slightly different build parameters.
Also, files that the command attempts, but fails, to open, are
also logged. They are treated as a third form of prerequisite,
where inability to remake the prerequisite doesn't force a
remake of the target; this is the correct behaviour - if
such a prerequisites exists, then the command will always be
re-run, but if it still doesn't exist, it is safe to not re-run
the command. [At the moment, this is done using a hack - the
semi-prerequisite is prefixed with a `-'; ultimately, it would
make more sense to recognise the three forms of prerequisites
that seem to be relevent to Yabs, perhaps using simple `', `-'
and `--' prefixes.] This is necessary in situations involving
multiple include-paths, such as the following:
compiler include path is: /usr/local/include:/usr/include
/usr/local/include/foo.h does not exist.
/usr/include/foo.h exists.
source file main.c contains <code>#include "foo.h"</code>
Yabs is asked to build main.o, so compiles main.c
/usr/local/include/foo.h is created.
Yabs is asked to build main.o again.
The creation of the file /usr/local/include/foo.h should force
a recompilation of main.c, even though the attempt to open that
file will have failed previously, and the filename will not be
listed in tradional auto-dependency information such as the
output from gcc -M. Note that gcc doesn't seem to attempt to
open the header file when the parent directory doesn't exist, so
this system isn't bomb-proof.
<autocmds>
If not None, <autocmds> is treated as a filename suffix that is
appended to target filenames, and used to store the command(s)
that <rule_fn> returns when asked how to create a target. If
the commands are changed (e.g. <rule_fn> is modified), Yabs will
detect this and force a remake of the target, even if it is
newer than all of its prerequisites.
The <autocmds> system works in the following way:
yabs.add_rule() makes a call to yabs.add_autocmds_rule(),
so that Yabs will have a rule available that will write the
commands for a target <foo> to a file <foo><autocmds>. Also,
Yabs ensures that <target><autocmds> is always added to the
prerequisites returned from <rule_fn>. Note that this mechanism
results in an extra file per target; if there are many targets,
it might be better to use the technique in yabs3's compilation
rules where command files are per-directory.
<internal>
<internal> is used to reduce the diagnostics when this rule is
used. For example, setting internal=1 will reduce the effective
state.debug level by 1 when outputing diagnostics in connection
with this rule. [this is currently broken].
<state>
If <state> is None, yabs.default_state is used.
We return the created Rule object. This is useful because it contains the
rule's root if applicable.
Example usage:
def myrule( target, state):
if target!='myprog.exe': return None
prereqs = [ 'foo.o', 'bar.o']
semiprereqs = [ 'foo.h', 'bar.h']
command = 'link -o ' + target + ' ' + string.join( prereqs, ' ')
return prereqs, semiprereqs, command
yabs.add_rule( myrule)
- autodeps_get(autodeps_name, state)
- A generator that yields prerequisites from an autodeps file.
- call_rule(rule, target, state, prefix)
- callers()
- cancel_targets(targets, state)
- command_gettext(command, state, prefix=None)
- This is normally called only internally by Yabs.
Calls <command> if it is callable, to get the text of the
command. Return text command, or raises exception containing any
output from the callable command.
If <state>.echo is False, (the default is True), the callable
command will be given dummy sys.stdout and sys.stderr which write to
an internal buffer. This buffer is included in any exception, but is
otherwise discarded. See yabs2's -e option. Note that this facility
is turned off when yabs is concurrent.
- command_run(command, state=None, cwd=None, autodeps_filename=None, fn=None, endtime=None)
- Runs newline-separated commands in <command>, prefixing each
with `cd <cwd> && ' if <cwd> is specified. Also prefixes
with settings for $LD_PRELOAD and $YABS_AUTODEPS_FILENAME if
<autodeps_filename> is set, so that auto dependencies are written to
<autodeps_filename>.
If <state>.silent is False, commands are written to stdout before
being run. If <state>.dryrun is True, command are not actually
run. If <state>.echo is True (the default), output from commands is
written to stdout as the command executes; see yabs2's -e option.
Returns ( e, outputtext). e is 0 if no error occurred, else a
yabserrors.command_error, which will also contain the output text
from the last sub-command.
If state.use_os_system is True, always calls commands using
os.system. This is to overcome problems with child tasks not
responding properly to stdin etc. When python 2.4 is widespred,
will hopefully be able to rely on the subprocess module.
Absorbs exceptions.
- command_run_text(command, state=None, cwd=None, autodeps_filename=None, fn=None, endtime=None)
- As command_run, but returns text, or raises exception.
- findrule_fromcache(target, state=None)
- Finds matching rule for target, from <state>'s rule cache.
This assumes that the rule has already been found by make(). Returns
a tuple: ( rule, ruleresult, phony), where <phony> is True/False,
and <ruleresult> is the tuple returned from the rule function.
If rule not found, raises exception.
- fn(text, globals_=None, locals_=None, up=None)
- Returns an anonymous function created by calling exec on <text>.
<text> should be a string containing a Python function definition,
ommiting the initial `def <fnname>' (actually, leading `def' can
be retained, for clarity). Any leading/trailing white space is
removed using str.strip(). Leading white space on all lines will be
handled automatically by exec, so you can use an indented python
triple-quoted string.
In addition, newlines and backslashes are re-escaped inside
single/double-quoted strings. this enables nested use of fn().
e.g.:
yabs.fn(
"""
def ( target, state):
if target != 'foo': return None
return [], None, 'touch foo'
""")
If globals_/locals_ are not specified, fn() will find its caller's
globals/locals using yabs._upglobals()/yabs._uplocals(). If <up>
is specified, yabs.fn() will look at the stack frame <up> levels
up.
Unfortunately the use of globals_/locals_ doesn't work in exactly
the way one would hope - the resultant function doesn't seem to be
able to access locals variables. E.g.:
import yabs
def a():
x = 42
f = yabs.fn( """
def ():
print x # fails
def g():
print x # succeeds
f()
g()
a()
See make.py:test22().
The created function will have a .func_code attribute like
conventional functions. Unfortunately this attribute is read-only,
and the only element that we can control is .func_code.co_name,
which we set to encode our caller's filename, function and line
number (using '_' for non-alphanumberic characters). An example of a
.func_code passed through str() is:
import yabs
f = yabs.fn( """
def():
pass
""")
print f.func_code
- which outputs:
<code object _yabs_fn_temp___foo_py___4 at 0xb7be9b20, file "<string>", line 1>
- get_caller_directory(n=1)
- Returns absolute path of the directory containing the python
module corresponding to the function <n> levels up the call
stack. Directory's trailing os.sep is retained.
E.g. yabs.get_caller_directory(0) returns the directory containing
the yabs.py module (which contains the get_caller_directory()
function itself). get_caller_directory(1) returns the directory
containing the caller's module. get_caller_directory(2) returns
the directory containing the caller's caller's module.
Usually get_caller_directory(1) is the desired call. yabs.caller is
a constant that gives the same effect, e.g.:
yabs.get_caller_directory( yabs.caller).
- make(target, prefix=None, state=None)
- If <target> is a relative path, <target> is converted to an absolute
path using os.path.abspath().
Uses the registered rules in <state> to try to remake <target>'s
prerequisites, semi-prerequisites and <target> itself, returning one
of the following:
yabs.unchanged - <target>'s rule's command was not run because
<target> was found to be up-to-date, or the command was run but
the timestamp of <target> was unchanged.
yabs.changed - <target>'s rule's command was run and it updated
<target>, or it was run and was a phony rule.
A list containing information about why <target> could
not be made. Each item in the list is a 3-tuple ( <rule>,
<prerequisite>, <error>), where <rule> is an instance of
the yabs.Rule class, describing a rule that claimed to be
able to remake <target>, but which required a prerequisite
<prerequisite> that could not be made. The <error> item
is the error returned from make() when it was recursively
called to remake <prerequisite>, so the list returned
from make() is actually a tree-like structure. Note
that <target> does not appear in the list. The function
yabs._print_failed_prerequisites() can be used to display the
list in a human readable format.
All diagnostics are prefixed with <prefix>. <prefix> can be a
function taking no params, in which case it should return a prefix
string.
Note that make() does not first check that rules exist for all
of a rule's prerequisites before attempting to create the first
prerequisite.
This has the disadvantage that it means that rules could be run even
when they ultimately don't work because some other prerequisites
cannot be made.
It has the advantage that the rules can depend on files created
by earlier rules; for example a rule could untar a .tar file, and
subsequent rules will be able to respond to the extracted files.
It might even be possible to import extra rules from these files
into the build programme itself. See http://tinyurl.com/yqk6l for a
usenet posting that mentions this issue.
- mt_exit(state)
- mt_spawntasks(state)
- Repeatedly attempts to claim resources required to build targets in
<state>.mt_targets. When a target's resource is acquired, we spawn a
new thread to build that target.
We wait on state.mt_condition.
We always scan targets from the first target, so if more than target
share a resource, the earliest target in <state>.mt_targets will
always be built first.
Currently there is no attempt to reuse threads. We return when
<state>.mt_exited is set.
- mt_target_fn(ruleresult, target, state)
- run in separate thread by mt_spawntasks(). builds <target>,
then releases the <ruleresult>'s resource and calls
state.mt_condition.notifyAll().
- mtime(filename, mtimecache=None)
- Returns modification time of <filename>, or 0 if <filename> doesn't
exist, using caching.
- mtime_add_oldprefix(tree, mtimecache=None)
- mtime_flush(filename, mtimecache=None)
- Removes any cached modification time for <filename>, including
entries from calls to mtime_markold() and mtime_marknew().
For example, this is called after a command is run that remakes a
target.
- mtime_marknew(filename, mtimecache=None)
- Future calls to mtime( <filetime>, <mtimecache>) will return
fixed value that is a long time in the future.
- mtime_markold(filename, mtimecache=None)
- Future calls to mtime( <filetime>, <mtimecache>) will return 1.
- mtime_raw(filename, mtimecache=None)
- Returns modification time of <filename>, or 0 if it doesn't exist.
- place(n=1)
- Debugging fn; returns representation of source position of caller.
- prefixfn(prefix0, nesting, state)
- relativepath(from0, to_, sep='/', up='../')
- constructs relative path, from directory <from0> to path <to_>.
this has be optimised for speed.
- relativepath_user(user, to, sep='/', up='../')
- If <to> is within ~<user>, returns representation of <to> starting
with `~<user>'. Otherwise returns absolute path of <to>.
- run_ruleresult(ruleresult, state)
- Runs the command/fn returned from a rule, by calling
command_run(). Unlocks Yabs's lock while doing so.
- show_result(target, rule, result, prefix, state, nesting)
- start_make(target, prefix0, state, nesting=0)
- This is the heart of Yabs. Builds <target>, returning same as
make(). Can also return <pending> if <state>.mt is set, in which
case one can use wait_result() to wait for <target> to be built.
Calls itself recursively to build prerequisites.
The returning of <pending> is very useful when building
prerequisites recursively - we start building all of a target's
prerequisites, then repeatedly wait for any of them to finish using
wait_result().
- subprocess(command, echo=None, state=None, fn=None, endtime=None, prefix=None)
- Runs <command> using <fn>, which defaults to <state>.subprocessfn, which in
turn defaults to _subprocess_popen.
If specified, <endtime> should be an absolute time in seconds since
epoc, as returned by time.time(), and the command is terminated at this
time. This is not supported by all _subprocess_* functions.
<prefix> can be a string, or a function that returns a string; it will be
used to prefix all lines returned in <text> (and echoed to screen if <echo>
is set). If None, <state.echo_prefix> is used.
Returns (e, text), where <e> is the integer return code, and <text> is the
text output by the command.
- subprocess_text(command, echo=None, state=None, fn=None, endtime=None, prefix=None)
- Returns just the text part of the return from subprocess()
function. Raises exception if the subprocess() function indicates an
error.
- tdb_acquire(mutex)
- acquires <mutex>, asserting if there would be a deadlock.
- tdb_release(mutex)
- releases <mutex>.
- tdb_status(state)
- tdb_str()
- tdb_wait(mutex, condition)
- does a wait on <mutex>,<condition>.
- tdb_willdeadlock(m)
- if attempting to acquire mutex <m> will deadlock, returns list of
<mutex>,<thread>,<mutex>,<thread>... that describe the deadlock.
Otherwise returns False.
- try_get_result(targets, state, prefix, nesting)
- Looks in <state>.targetcache for result for any target in <targets>.
if found, removes from <targets> and returns ( <target>, result),
else returns None.
- updatediff(filename, new_contents)
- Writes <new_contents> to <filename> if <filename> doesn't exist, or
<filename>'s existing contents differ from <new_contents>.
File's parent directory is always made if not already present.
Returns True if file has been changed, or False if file has been
left unchanged.
- wait_result(targets, state, prefix, nesting)
- Waits for any item in <targets> to be made. Removes the item from
<targets>, and returns ( <target>, <result>,).
Relies on state.mt_condition.notifyAll() being called whenever
<state>.targetcache is changed. This is done automatically -
<state>.targetcache is a NotifyingDict.
- xtermtitle_usertarget(target, state)
- writes <user>@<host>:<target> into titlebar of xterm. intended for
use as <state>.premake_fn.
|