The configuration handling in SyncEvolution was originally designed for a SyncML client. This article explains how it is to be extended to cover SyncML servers and other use cases.
Terminology and Items
- A data source is a local database which is to be synchronized
- Access to data sources in SyncEvolution is done via backends which are either compiled into the main executable or loaded dynamically. Exactly one backend is responsiple for each data source.
- A SyncML server implements the more complex role in a SyncML session. It has to remember meta information about all its clients.
- A SyncML client typically has to store less or at least, different meta information. Client and server cannot switch their roles without starting over from scratch.
- Used instead of client and server in cases when the specific role does not matter.
- A single configuration value. Identified by its location in the configuration tree and its key. A property has a value, a comment and an attribute which determines whether the attribute is internal (not meant to be edited by users) or user-configurable.
- configuration tree
- A hierarchy of nodes. Each node contains a set of configuration properties with unique keys plus one or more additional key/value stores. Many different implementations of a configuration tree are possible; the one currently used in SyncEvolution uses plain .ini files inside a directory hierarchy that is rooted in $XDG_CONFIG_ROOT/syncevolution (typically ~/.config/syncevolution). User-visible properties are stored in config.ini, hidden ones in .internal.ini. Each backend gets access to a key/value store for its own, for example change tracking (.other.ini). In a server, the glue code for a backend and the Synthesis engine uses .server.ini for the Synthesis meta information.
- Utility code in a Synthesis client engine which stores meta information automatically in a configured directory. This code simplifies change tracking (no need to report deleted items; no extra work for suspend/resume state). This data is stored in a .synthesis directory at the root of the configuration tree of a client.
The initial implementation of the SyncEvolution configuration was provided by the Funambol C++ client library. It was not meant to be accessed by users directly and therefore was not very user-friendly (deep directory hierarchy with unusual names). SyncEvolution 0.8 reimplemented this using its own code and a simpler file layout. All data was stored as properties or in the key/value store.
In SyncEvolution 0.9, the configuration was kept more or less the same despite the migration to the Synthesis engine. The .synthesis directory was added.
The main new feature in SyncEvolution 1.0, the support for running as a SyncML server, requires more fundamental changes. So far, each configuration contained a complete set of source settings for each server. Different configurations were completely independent, despite accessing the same local data. Because a client typically only talks to one server, this was intuitive and acceptable despite duplicating some settings when configuring multiple servers. A server talks to many different clients. We have operations on the local data (backup/restore, viewing logs) which are independent of the peer. Given these new use cases, the main drawbacks of the old scheme are:
- Settings for sources and logging are duplicated and might become intentionally or unintentionally inconsistent between peers.
- Finding all old sessions involving a certain local data source is hard: one has to check multiple different peer configurations and cannot be sure that source "addressbook" of peer A really is the same "addressbook" as in peer B.
- When we are contacted for the first time by a peer, it is unclear which local source and logging configuration we are supposed to use.
Configuration Handling SyncEvolution >= 1.0
Before 1.0, users and GUIs only had to deal with a very simple configuration schema: a server configuration had sync properties describing the server and parameterizing the sync and one set of source properties for each data source. This directly mapped to user-visible config files like this:
We decided to keep it that simple as far as the user and the APIs are concerned. SyncEvolution >= 1.0 provides a view which only distinguishes between sync properties and source properties. Some sync properties are shared between these views. Changing them in one view also updates all other views. Each property is stored in exactly one config node.
Sync properties are now
- the same in all configurations, for example "which peer is the default"
- per source set
- apply to all peer configurations using the same set of local data sources, for example logging settings
- per peer
- specific to a certain peer, like device ID or sync URL
Source properties are:
- per source
- the same for all users of a certain source, like backend and database selection
- per peer and source
- meta information, URIs at the peer's side
A view is selected via a config name. A fuzzy matching is used to select the relevant properties:
- empty string ""
- A view with global and per source properties and no properties specific to a peer.
- <peer name>
- A view with global and per source properties and properties specific to the peer. This is the same as the traditional "server configuration" and can be used like one.
- <peer name>@<source set<
- Instead of the default set of sources and peers, a completely different configuration can be used. It only has the global sync properties in common with other configurations.
The layout looks like this:
config.ini global sync properties
config.ini source set config
.synthesis shared (!) Synthesis binfiles
config.ini source config
config.ini peer configuration
config.ini per peer source config
.other.ini per peer and per source storage for backend
.... a complete set of configuration files as above
For each config.ini there's also an .internal.ini which is not exposed via command line or D-Bus API. The exact location of each property is defined in this table:
|per source set||logdir, maxlogdirs|
|peer||syncURL, username, password, useProxy, proxyHost, proxyUsername, proxyPassword, RetryDuration, RetryInterval, deviceId, enableWBXML, maxMsgSize, maxObjSize, SSLServerCertificates, SSLVerifyServer, SSLVerifyHost, loglevel, printChanges, WebURL, ConsumerReady, IconURI|
|source||type, evolutionsource, evolutionuser, evolutionpassword|
|per peer source properties||sync, uri|
In the new scheme the Synthesis binfiles are shared between peers, as intended by Synthesis. This is not necessary for us at the moment, but some of the binfile features (simplified change tracking via a single "updated" flag in the local database) might be useful for other backends.
When asking for a peer template, that template is already populated with existing shared properties, so that it can be written back without unintentionally overwriting those.
Synthesis Data Storage
Meta data for sync clients is stored in the .synthesis directory using the binfile support which is compiled into the engine. The only configurable parameter here is the location of these files.
Meta data for sync servers is handled differently:
- Per source local to remote ID mapping is written into the .server.ini key/value stores if a backend decides to use the SyncSourceAdmin class. A backend can decide to provide a different implementation, but one has to be provided.
- Per source admin data (for example, sync anchors) are written as a text blob into the internal, per source, per peer "adminData" property by SyncSourceAdmin. The content of this blob is opaque for SyncEvolution.
- Per peer data is stored as internal sync properties by SyncContext: lastNonce, remoteDeviceID, adminData. SyncContext also handles passwords and session login.
A drawback of changing the configuration layout is the question of migrating old configs and changing back and forth between new and old SyncEvolution releases. Ideally migration should be automatic and still allow going back to an older SyncEvolution seamlessly.
With two tricks this is feasible:
- move existing properties into the new layout and *duplicate* the shared properties inside the per-peer .ini files
- recreate the traditional layout with symlinks
So we would end up with:
.synthesis/ -> ~/.config/default/.synthesis
config.ini -> .../peers/scheduleworld/config.ini
sources/ -> .../peers/scheduleworld/sources/
config.ini files are updated by creating a new file and then renaming it to config.ini. This replaces the config.ini symlink. Also, if a host's source property is changed using an old SyncEvolution, the change won't be noticed by a new SyncEvolution because although the changed config.ini is read, the newer SyncEvolution doesn't take the host properties from there. This is so advanced that we don't have to support it.
The automatic migration also doesn't work if the user has multiple server configs because we cannot "join" different .synthesis binfile dirs. In such a situation the user has to force a migration and be prepared for slow syncs. Without such a forced migration, the new SyncEvolution should continue using the old layout. This is easily implemented by using the same config node twice.
Finally, the instructions for synchronizing multiple ScheduleWorld calendars have to be updated. The ScheduleWorld Wiki says:
- create normal "scheduleworld" config
- create second config with
--template scheduleworld_home calendar
In the old scheme, this creates two server configs with different syncURL and two different "calendar/config.ini" files, accessing different local data.
In the new scheme, there is only one "calendar/config.ini:evolutionSource", so the second invocation of syncevolution changes the local data accessed by the normal "scheduleworld" config, which is both unexpected and (in this special case) undesirable. The second configuration must use a different source name. Then the normal "scheduleworld" config uses the default "calendar", and the second "scheduleworld_home" the other. This complicated setup would not be necessary if ScheduleWorld selected the calendar via a parameter attached to the URI. At least it supports multiple calendars at all.
Creating these configurations from the command line could be made simpler if it was possible to clone source configurations or even whole existing configurations.
Multiple Transports per Peer
In "Outgoing obex connection for SyncEvolution" we touched on some configuration related aspects. In particular we talked about how to specify multiple ways of contacting a peer. This is still a bit in flux, but in general the idea is that one peer configuration documents all methods available for contacting a peer.
There's one problem with that: how does the GUI create such a configuration correctly when it only knows that "Bluetooth device with MAC address xyz is available for sync"? There might be a peer configuration for this device already, using USB as transport. But because the only common element between the two peers (the SyncML Device ID) isn't available for Bluetooth at this point, the existing peer configuration cannot be identified automatically.
Do we rely on the user to add the Bluetooth transport to the right existing configuration? If he gets that wrong, he will end up with two different local sets of meta data (source/addressbook/.other.ini), which will break syncs when switching back and forth between the two transports.
The new configuration handling should be tolerant of this. This means that the location of the .other.ini files can only be determined once the peer's Device ID is known (internal backend API change).
A separate directory for .other.ini files would make the configuration layout more complex in all cases. Therefore the .other.ini file is kept in the normal/peer//sources/ directories and duplicated peer configurations are handled as follows:
- when we learn about the Device ID of a peer, we check which of the currently configured peers has the same ID and use an existing .other.ini file for it
- when removing a peer configuration, we have to check whether the .other.ini is still needed and if so, move the file because it is still needed