Not all needs can be addressed by the default values used inside bzr and bzrlib, no matter how well they are chosen (and they are ;).
Options that are rarely used don’t deserve a corresponding command line switch in one or several commands.
Many parts of bzrlib depends on some constants though and the user should be able to customize the behavior to suit his needs so these constants need to become configuration options or more generally, be easier to set.
These options can be set from the command-line, acquired from an environment variable or recorded in a configuration file.
To simplify writing (and reading), this document refers to the old and new config designs: * the old design is using Config as a base class for all config files, * the new design use ConfigStacks of Section from config Stores.
Many parts of bzrlib declare constants and there is no way for the user to look at or modify them (see http://pad.lv/832061).
The old design requires a configuration object to create, modify or delete a configuration option in a given configuration file. bzr config makes it almost transparent for the user. Internally though, not all cases are handled: only BranchConfig implements chained configs, nothing is provided at the repository level and too many plugins define their own section or even their own config file. (config.Stack now provides a way to chain config files, BranchStack properly implements the desired behavior, bzr config uses the new design).
locations.conf defines the options that need to override any setting in branch.conf for both local and remotes branches (but some remote branch options just ignore locations.conf). Many users want a way to define default values for options that are not defined in branch.conf (and even more users think that locations.conf provide default values, see also http://pad.lv/843211 and http://pad.lv/832046). This could be approximated today by not defining these options in branch.conf but in locations.conf instead. This workaround doesn’t allow a user to define defaults in locations.conf and override them in branch.conf. (Allowing sections in ‘bazaar.conf’ (or introducing a new defaults.conf’ allowing sections) can now address that. Defining and using a new file is easier as it avoids handling a migration path for bazaar.conf and doesn’t require banning the use of sections for special purpose needs (ALIASES and BOOKMARKS for example)).
Defining a new option requires adding a new method in the Config object to get access to features like:
A related problem is that, in the actual implementation, some configuration options have defined methods, others don’t and this is inconsistent. (Using only Stacks addresses that).
Access to the ‘active’ configuration option value from the command line doesn’t give access to the specific section. This is a concern if the user has no other way to address a specific configuration option including Store and Section when using bzr config (see http://pad.lv/725234). Plugins defining their own stacks and/or stores also have no way to properly plug into bzr config (see http://pad.lv/788991).
Rules for configuration options are not clearly defined for remote branches (they may differ between dumb and smart servers the former will use the local bazaar.conf and locations.conf files while the later will use (or ignore ?) the remote ones).
The features offered by the Bazaar configuration files should be easily accessible to plugin authors either by supporting plugin configuration options in the configuration files or allowing the plugins to define their own configuration files. (Separating Section, Store and Stack starts addressing that, a stack registry should complete the needed means http://pad.lv/832036).
While the actual configuration files support sections, they are used in mutually exclusive ways that make it impossible to offer the same set of features to all configuration files:
This is addressed by defining different stacks selecting the relevant sections from the stores involved. ALIASES for example can define a stack that select only the ALIASES section from bazaar.conf.
There is no easy way to get configuration options for a given repository or an arbitrary path. Working trees and branches are generally organized in hierarchies and being able to share the option definitions is an often required feature. This can also address some needs exhibited by various branch schemes like looms, pipeline, colocated branches and nested trees. Being able to specify options in a working tree could also help support conflict resolution options for a given file, directory or subtree (see http://pad.lv/359320).
Since sections allow different definitions for the same option (in the same store), a total order should be defined between sections to select the right definition for a given context (paths or globs for locations.conf but other schemes can be used, window names for qbzr, repository UUIDs for bzr-svn for example). Allowing globs for section names is harmful in this respect since the order is currently defined as being based on the number of path components. The caveat here is that if the order is always defined for a given set of sections it can change when one or several globs are modified and the user may get surprising and unwanted results in these cases. The lexicographical order is otherwise fine to define what section is more specific than another. (This may not be a problem in real life since longer globs are generally more specific than shorter ones and explicit paths should also be longer than matching globs. That may leave a glob and a path of equal length in a gray area but in practice using bzr config should give enough feedback to address them. See also http://pad.lv/832046 asking for a less magical section matcher).
Internally, configuration files (and their fallbacks, bazaar.conf and locations.conf for branch.conf) are read every time one option is queried. Likewise, setting or deleting a configuration option implies writing the configuration file immediately after re-reading the file to avoid racing updates (see http://pad.lv/832042).
The current implementation use a mix of transport-based and direct file systems operations (Addressed by Store implementation relying on transports only and the hpss implementing the corresponding verbs).
While the underlying ConfigObj implementation provides an interpolation feature, the bzrlib implementation doesn’t provide an easy handling of templates where other configuration options can be interpolated. Instead, locations.conf (and only it) allows for appendpath and norecurse. (Cross-section, cross-file interpolation and section local options are now implemented in the new design).
Inherited list values can’t be modified, a more specific configuration can only redefine the whole list.
There is no easy way to define dicts (the most obvious one being to use a dedicated section which is already overloaded). Using embedded sections for this would not be practical either if we keep using a no-name section for default values. In a few known cases, a bencoded dict is stored in a config value, so while this isn’t user-friendly, not providing a better alternative shouldn’t be a concern. A possible, limited, implementation can be envisioned: limiting the dict to a single level only, with simple names as keys and unicode strings as values. The keys can then be mapped to options prefixed with the dict name.
While of course configurations files can be versioned they are not intended to be accessed in sync with the files they refer to (one can imagine handling versioned properties this way but this is not what the bazaar configuration files are targeted at). bzr will always refer to configuration files as they exist on disk when an option is queried or set.
The configuration files are generally local to the file system but some of them can be accessed remotely (branch.conf, repo.conf).
Option names are organized into a name space for a given stack. One such set includes bazaar.conf, locations.conf, branch.conf, etc. Plugins can define their own sets for their own needs. While it is conceivable that the same option name can be used in unrelated configuration stacks, it seems better to define a single name space for all options if only to avoid ambiguities.
Using a name space is meant to help:
Using valid python identifiers is recommended but not enforced (but we may do so in the future).
The option name space is organized by topic:
All option values are text. They are provided as Unicode strings to API users with some refinements:
Since the configuration files can be edited by the user, bzr doesn’t expect their content to be valid at all times. Instead, the code using options should be ready to handle invalid values by warning the user and falling back to a default value.
Likely, if an option is not defined in any configuration file, the code should fallback to a default value (helpers should be provided by the API to handle common cases: warning the user, getting a particular type of value, returning a default value)(most of that is now handled at Option definition).
This also ensures compatibility with values provided via environment variables or from the command line (where no validation can be expected either)(done in the new design).
Some option values can be templates and contain references to other options. This is especially useful to define URLs in sections shared for multiple branches for example. It can also be used to describe commands where some parameters are set by bzrlib at runtime.
Since option values are text-only, and to avoid clashing with other option expansion (also known as interpolation) syntaxes, references are enclosed with curly brackets:
Homepush_location = lp:~{launchpad_username}/bzr/{nick}
In the example above, launchpad_username is an already defined configuration option while nick is the branch nickname and is set when a configuration applies to a given branch.
The interpolation implementation should accept an additional dict so that bzrlib or plugins can define references that can be expanded without being existing configuration options:
diff_command={cmd} {cmd_opts} {file_a} {file_b}
There are two common errors that should be handled when handling interpolation:
The loop handling can be modified to allow cross-sections and cross-files interpolation: if an option refers to itself (directly or indirectly) during an expansion, the fallback sections or files can be queried for its value.
This allows list values to refer to the definition in the less specific configurations:
bazaar.conf:
debug_flags = hpss
branch.conf for mybranch:
debug_flags = {debug_flags}, hpssdetail
$ bzr -d mybranch config debug_flags
hpss, hpssdetail
Undefined references are detected if they are not defined in any configuration. This will trigger errors while displaying the value. Diagnosing typos should be doable in this case.
The configuration file is mostly an ini-file. It contains name = value lines grouped in sections. A section starts with a string enclosed in squared brackets (‘[section_name]`), this string uniquely identifies the section in the file. Comments are allowed by prefixing them with the ‘#’ character.
A section is named by the path (or some other unuique identifier) it should apply to (more examples below).
When sections are used, they provide a finer grain of configuration by defining option sets that apply to some working trees, branches, repositories (or any kind of context) or part of them. The relationship between a given context and the sections it applies to is defined by the config file.
So far, Bazaar uses a glob in locations.conf and select the sections that apply to a given url (or a local path).
The subset is defined by the common leading path or a glob.
Different kinds of section names can be defined:
Section names define another name space (than the option names one) with an entirely different purpose: in a given configuration file, for a given context, only some sections will be relevant and will be queried in a specific order.
This matching is specific to each config file and implemented by the SectionMatcher objects.
Whatever this matching does, the user can observe the results with the bzr config command which displays the sections in the order they are queried.
The context here is either:
Note that for both the provided context and the section names, if an URL uses a file:/// form, it is converted to a local path.
The sections names can use globs for each path component (i.e. /dir/*/subdir is allowed but /dir/\*\*/subdir will match /dir/a/subdir but not /dir/a/b/subdir.
The reason is that the ordering is defined by sorting the section names matching the context on the number of path components followed by the path itself in lexicographical order. This results in most specific sections being queried before the more generic ones.
LocationMatcher has some obscure (for unaware users) edge cases and limitations that can be surprising. PathMatcher aims at addressing these issues by providing simpler rules while still giving full control to the user (http://pad.lv/832046).
The context here is a local path, absolute or relative. If the path is relative it is interpreted from the file base directory.
Note that ‘base directory’ for configuration files in Bazaar directories is really:
The order is the one observed in the file so most generic values are specified first and most specific ones last. As such, the order in the file is the opposite of the one displayed by bzr config which displays most specific values first. This seems to be the most natural order in both cases.
A section matches if the section name is a prefix of the context path (relative paths being converted to absolute on the fly).
(copied from a recent version of bzr.dev for easier reading, refer to the original for an up to date version)
The Option object is used to define its properties:
Options are grouped into sections which share some properties with the well known dict objects:
MutableSection is needed to set or remove an option, ReadOnlySection should be used otherwise.
This is an implementation-level object that should rarely be used directly.
it can be local or remote
locking
All lock operations should be implemented via transport objects. (True for Store).
option life cycle
Working trees, branches and repositories should define a config attribute following the same life cycle as their lock: the associated config file is read once and written once if needed. This should minimize the file system accesses or the network requests. There is no known racing scenarios for configuration options, changing the existing implementation to this less constrained one shouldn’t introduce any. Yet, in order to detect such racing scenarios, we can add a check that the current content of the configuration file is the expected one before writing the new content and emit warnings if differences occur. The checks should be performed for the modified values only. As of today (and in the foreseeable future), the size of the configuration files are small enough to be kept in memory (see http://pad.lv/832042).
This the object that provides access to the needed features:
Depending on the files involved, a working tree, branch or repository object (or more generally a context) should be provided to access the corresponding configuration files. Note that providing a working tree object also implicitly provides the associated branch and repository object so only one of them is required (or none for configuration files specific to the user like bazaar.conf).
Depending on the option, there are various places where it can be defined and several ways to override these settings when needed.
The following lists all possible places where a configuration option can be defined, but some options will make sense in only some of them. The first to define a value for an option wins (None is therefore used to express that an option is not set).
command-line -Ooption=value see http://pad.lv/491196.
~/.bazaar/locations.conf
When an option is set in locations.conf it overrides any other configuration file. This should be used with care as it allows setting a different value than what is recommended by the project
tree (Not Implemented Yet)
The options related to the working tree.
This includes all options related to commits, ignored files, junk files, etc.
Note that the sections defined there can use relative paths if some options should apply to a subtree or some specific files only.
See http://pad.lv/430538 and http://pad.lv/654998.
branch located in .bzr/branch/branch.conf
The options related to the branch.
Sections can be defined for colocated branches or loom threads.
repository (Not Implemented Yet)
The options related to the repository.
Using an option to describe whether or not a repository is shared could help address http://pad.lv/342119 but this will probably requires a format bump).
project (Not Implemented Yet)
The options common to all branches and working trees for a project.
organization (Not Implemented Yet)
The options common to all branches and working trees for an organization.
See http://pad.lv/419854.
system (Not Implemented Yet but see http://pad.lv/419854 and https://code.launchpad.net/~thomir/bzr/add-global-config/+merge/69592)
The options common to all users of a system (may be /etc/bzr/defaults or /usr/local/etc/bzr/defaults or /Library/Preferences/com.canonical.defaults or c:windowsbazaar.conf (someone fix this one please ;) depending on the OS).
bazaar.conf
The options the user has selected for the host he is using.
Sections can be defined for both remote and local branches to define default values (i.e. the most common use of locations.conf today).
default (implemented by the OptionRegistry)
The options defined in the bzr source code.
This will be implemented via the Option objects.
Plugins can define additional configuration files as they see fit and insert them in this list, see their documentation for details.
There are ways to keep the same files while ensuring compatibility via various tricks but there are cases where using new files to replace the old ones is definitely easier:
The known cases so far are described below.
Obvious at this point:
Unclear at this point:
[BOOKMARKS] section can be replaced by bookmarks.xxx options (the bookmarks plugins already uses bookmarks_xxx in branch.conf since no sections were supported there). The easiest here is probably to just merge the plugin into core and use the appropriate option names consistently. A config: directory service may even be better as any option can be used as a bookmark. This allows things like:
[/whatever/path]
my_push = lp:<launchpad.login>/xxx/{nick}
web_site=ftp://example.com/
bzr push config:web_site
Which means we completely replace the plugin and don’t need to care about migrating the section.
[ALIASES] section can be replaced by corresponding bzr.alias.xxx options. This could be automated by creating the corresponding options ?
I don’t know about other sections, feedback welcome. Plugin authors are encouraged to migrate to the new name space scheme by prefixing their options with their plugin name.
These are random notes about concepts, ideas or issues not implemented yet.
An alternative is to just let the devs decide which stack they use for a given option, stacked_on_location for example is said to relate to the branch only and changing it or setting it in a different config file may not be appropriate. This may not be a good example as there is also the default_stack_on option which can be set only in control.conf though...
This is relevant for http://pad.lv/832042.
bzr behavior, as well as the objects it acts upon, is configured via a set of so-called configuration files.
These files allow to define working trees, branches and repositories, their relationships and how bzr should handle them.
The default behavior of bzr is aimed at making this configuration as transparent as possible by keeping track of how these objects are created and modified when they are used. In short, they are useless until you want to change the default behavior in some specific context.
We mostly read config options. Therefore all we care about is to guarantee that:
From there, conceptually, all operations can clearly define whether or not they need to modify a config file and do so only when they succeed. All modifications occurring during such an operation are delayed until the very end of the operation.
Now, we want to minimize the overlapping times where one bzr operation has changed a value and another concurrent operation is unaware of this modification.
These overlapping periods are as of today rare.
The only known case, http://pad.lv/525571 has been fixed in bzr-2.1.3. The bug there was triggered when two processes tried to write the same config file at the same time leaving an invalid file in the end.
Such a period can be recognized and detected though: when changing an option value, if the preserved original value is different in the config file, someone else modified it and the operation can be invalid because it relied on the original value.
For the sake of the example, if an option value represent a global unique ID via a simple counter (very bad idea), if two operations try to increment it, both will use the same value that won’t be unique anymore. Checking the value present in the file when trying to save the updated value with identify such a collision.
An assumption is floating around: it should be enough to report when an operation is modifying an already modified option and observe that no-one reports such occurrences.
Note that this assumption is made in a context where no known scenarios exist in the bzr code base not in any plugin (for a best effort value of ‘any’, feedback highly welcome, bug reports even ;)
With this in mind, we can change the definition of config options, stores and stacks to ensure that:
A reader can then safely assume that reading a config file gives it a valid (and coherent) definition of the configuration when the operation starts. All the operation has to do is to declare which config files may be modified by an operation (whether or not we can be liberal on this ‘may be’ is yet to be defined).