OData Synchronization with WCF Data Services

by

Over the last couple of months I’ve been doing quite a bit of work with OData, being one of the easiest ways to consume data with a Windows Phone application. One of the things that has frustrated me is that there isn’t any built in synchronization components for Windows Phone (there isn’t any database persist technology either but I won’t digress…). However, because of the way that OData exposes data, it isn’t too difficult to come up with scheme for doing synchronization.

Background

Before we get into how I’ve been doing data synchronization over OData I want to take this opportunity to point out that there are different approaches to synchronization. If you look at technologies such as RDA and Merge Replication these are clearly targeting the synchronization of relational data. Change tracking is typically at a per row level for each table that you choose to synchronize. Alternatively if you take Sync Framework this takes a much more generic approach to synchronization where by you could be synchronizing anything whether they be files, rss content or data. For most data driven applications this isn’t particularly useful since all you want to be able to do is say “synchronize this data set between the server and client”. You end up using the default sql providers in which case you might as well have just used Merge Replication.

One of the challenges of building a good synchronization strategy is getting the granularity of change tracking right. I don’t mean change tracking at the database level (after all that can be done by simply enabling change tracking on Sql Server 2008), I mean change tracking between the client and server. Take the example of a Person entity which can be broken down into a Peron table with joins to an Address table (1 to many, since they might have a work and home address).

1) Table/Row Change Tracking

You could track changes on both the Person and Address tables, in which case the client could request all changes to the Person and Address table since the last sync.

Pro: Change sets are small as only the data changed is synchronized (eg just the Address data if a change to an Address)

Con: More change tracking data (you need to track last updated timestamp and deleted flag as a minimum)

2) Entity Change Tracking

Alternatively you could track changes at just the Person table level. In this case changes to any Address would result in the corresponding Person table too.

Pro: Less change tracking data (you only need to track at the entity level, in this case the Person level)

Cons: Change sets are larger as the entire entity needs to be synchronized (eg if an Address changes, the entire Person entity needs to be synchronized).

If you decide to go with the first option then you should consider either Sync Framework or Merge Replication (for platforms where it is supported). Alternatively, if you don’t have a client side relational database (eg Silverlight or Windows Phone) then you might want to consider synchronizing whole entities and tracking at the entity level.

Change Tracking with OData

Ok, lets return to doing synchronization with OData. There are a number of different strategies for tracking changes to data on the server. As mentioned earlier you could go with SQL Server Change Tracking which once enabled allows you to request changes to a table between two timestamps. For simplicity we’re going to go with a much simpler approach of having two additional fields, LastUpdated and IsDeleted, on the tables that we want to track changes on. This gives us the flexibility of either tracking on a table-by-table basis (1) or at an entity level (2). LastUpdated is a server date stamp indicating when the data was last modified. We’re not tracking all changes sets, only the cumulative effect of all changes – in other words if you do multiple changes to an entity they will all get synchronized as a single change based on the LastUpdated value. The IsDeleted boolean value is required to ensure clients get notified when an entity is deleted. The upshot is that you should never actually delete an entity from the database that is synchronized because you don’t know whether a client still has that entity in their offline cache.

Publishing OData

We’re going to continue with the previous example of a Person entity which can have a number of associated addresses. The basic data model would look like ( Person > PersonId, Name; Address> AddressId, PersonId, StreetAddress):

image

To enable change tracking you need to add the LastUpdated and IsDeleted fields. You can either add these fields to both tables (option 1 from earlier) or just the Person table (option 2 since the Peron entity is effectively a row in the Person table coupled with the associated rows from the Address table).

image image
The next thing to do is to start building out the server components:

– New project called PeopleServer based on the ASP.NET Empty Web Application project template

– Add a new item called PeopleModel.edmx based on the ADO.NET Entity Data Model. Follow the steps to setup a connection to your database and add the Person and Address tables to your model.

image

– Add a new item called PeopleDataService.svc based on the WCF Data Service item template. Update the template to use the MyPeopleEntities (this name may vary depending on what you called your entity set – open the edmx file, and look at the Properties window. This name corresponds to the Entity Container Name)

public class PeopleDataService : DataService<MyPeopleEntities>
{
    public static void InitializeService(DataServiceConfiguration config)
    {
        config.SetEntitySetAccessRule("*", EntitySetRights.All);
        config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2;
    }
}

Server Change Tracking

This is enough to publish your data using odata, allowing read/write access to all the entities (ie both Person and Address tables). However, it doesn’t enforce any behavior regarding setting the LastUpdated or the IsDeleted flag. The LastUpdated should be set to the server time of any updates (rather than a client date stamp) and the IsDeleted flag should be set to true instead of actually deleting an entity. We can add this functionality in the SaveChanges method on the MyPeopleEntities. Rather than having to write code for each type of entity we want to track changes for, we’ll also extend our entities to implement the IEntityTracking interface.

– Add a new item called IEntityTracking.cs based on the Interface item template. This will define the LastUpdated and IsDeleted properties.

public interface IEntityTracking
{
    DateTime LastUpdated { get; set; }
    bool IsDeleted { get; set; }
}

– Add a new item called PeopleModel.Custom.cs based on the Code File item template. Extend the Person partial class to implement the IEntityTracking interface (note that you don’t need to add any properties since there are already properties which match those required by the interface). We’re only going to track changes against the whole Person entity here but if you want to track changes at the Address level as well you can extend that class with this interface too.

public partial class Person : IEntityTracking
{
}
    
public partial class MyPeopleEntities
{
    public override int SaveChanges(SaveOptions options)
    {
        var changes = this.ObjectStateManager.GetObjectStateEntries(
                                 EntityState.Modified |
                                 EntityState.Added |
                                 EntityState.Deleted);
        foreach (var change in changes)
        {
            var entity = change.Entity as IEntityTracking;
            if (entity != null)
            {
                if (change.State == EntityState.Deleted)
                {
                    change.ChangeState(EntityState.Modified);
                    entity.IsDeleted = true;
                }
                entity.LastUpdated = DateTime.Now;
            }
        }

        return base.SaveChanges(options);
    }
}

In the SaveChanges method you simply iterate through the changes, detect whether the item being modified implements IEntityTracking. If it does you reverse any deletion attempts (replacing it with setting the IsDeleted flag to true) and set the LastUpdated date to the current server time.

Consuming The Data

OK, we done configuring the server side. Let’s look at how you consume this from the client. Rather than going through an actual client implementation (which will vary depending on the technology stack you’re using). I’ll walk through this using Fiddler to send GET/POST/PUT/DELETE messages.

Let’s start by  retrieving all the entries in the Person table. The following image shows how to build a simple GET request in Fiddler. Note that we’re specifying that we want the results in JSON (Set the header “Accept: application/json”). You have to love ADO.NET Entity Framework – when you have a table called Person, it knows that the plural (ie the set name) should be People!

image

Ok, so what does the response look like:

HTTP/1.1 200 OK
Server: ASP.NET Development Server/10.0.0.0
Date: Wed, 04 Aug 2010 04:39:12 GMT
X-AspNet-Version: 4.0.30319
DataServiceVersion: 1.0;
Content-Length: 3611
Cache-Control: no-cache
Content-Type: application/json;charset=utf-8
Connection: Close

{“d” : [{
“__metadata”: {“uri”: “<a href=”http://localhost:16896/PeopleDataService.svc/People(guid’2b64d1ca-5802-4937-8134-0ad978415117′)””>http://localhost:16896/PeopleDataService.svc/People(guid’2b64d1ca-5802-4937-8134-0ad978415117′)”</a>, “type”: “MyPeopleModel.Person”
}, “PersonId”: “2b64d1ca-5802-4937-8134-0ad978415117”, “Name”: “Mary”, “LastUpdated”: “/Date(1280932624977)/”, “IsDeleted”: false, “Addresses”: {
“__deferred”: {
“uri”: “<a href=”http://localhost:16896/PeopleDataService.svc/People(guid’2b64d1ca-5802-4937-8134-0ad978415117′)/Addresses””>http://localhost:16896/PeopleDataService.svc/People(guid’2b64d1ca-5802-4937-8134-0ad978415117′)/Addresses”</a>}}}, {
“__metadata”: {
“uri”: “<a href=”http://localhost:16896/PeopleDataService.svc/People(guid’f1c245d0-2262-43e0-b78b-3086068a72fa’)””>http://localhost:16896/PeopleDataService.svc/People(guid’f1c245d0-2262-43e0-b78b-3086068a72fa’)”</a>, “type”: “MyPeopleModel.Person”
}, “PersonId”: “f1c245d0-2262-43e0-b78b-3086068a72fa”, “Name”: “Ken”, “LastUpdated”: “/Date(1280932635510)/”, “IsDeleted”: false, “Addresses”: {
“__deferred”: {
“uri”: “<a href=”http://localhost:16896/PeopleDataService.svc/People(guid’f1c245d0-2262-43e0-b78b-3086068a72fa’)/Addresses””>http://localhost:16896/PeopleDataService.svc/People(guid’f1c245d0-2262-43e0-b78b-3086068a72fa’)/Addresses”</a>
}…

Hmmm, that’s not particularly easy to read. The JSON Viewer plugin for Fiddler makes it much easier to see what’s going on.

image

As you can see the data returned is a collection of objects that represent the data in the Person table. The __medadata node specifies the uri of the individual entities, which in this case specifies the Person to return by the PersonId (which is a Guid). Unfortunately this isn’t all the information about the Person entities since it doesn’t include the Address information. At the end of the Person object there is a node called Addresses but it has a _deferred node. If you follow that you can access the collection of addresses associated with the Person. This requires further round trips in order to get the data for a single entity. There is a work around, which is to include the $expand parameter in the query:

http://localhost:16896/PeopleDataService.svc/People?$expand=Addresses

image

You should note that the value of the $expand is the name of the Navigation property (see the entity model that you added to your project). Some other points to note:

– If your Person has other linked tables (eg PhoneNumber) you can expand multiple tables by separating them with a comma.

http://localhost:16896/PeopleDataService.svc/People?$expand=Addresses,PhoneNumbers

– If the Address table links to another table (eg AddressType) you can expand to that table as well by adding a slash.

http://localhost:16896/PeopleDataService.svc/People?$expand=Addresses/AddressType,PhoneNumbers

Filtering the Data For Synchronization

Coming back to how we use this to do synchronization. Well the basic principle of synchronization is to download changes from the server and upload changes you’ve made locally. The changes to download should be all entities that have changed between the last time you sync (which might be never) and now. To do this you can apply the $filter parameter to the query. For the initial sync you should pick sometime a long time in the past:

http://localhost:16896/PeopleDataService.svc/People?$expand=Addresses&filter=(LastUpdated%20gt%20datetime’1900-00-00T00:00:00.0000000′)

The entities that are returned will determine what the next filter value is. Select the most recent LastUpdated value for the entities received and use this as the next filter value.

Now, if your database is of a reasonable size you will probably want to employ paging and rather than allowing the client to specify this you probably want to enforce this on the server. You can do this by updating the InitializeService method on your data service:

public static void InitializeService(DataServiceConfiguration config)
{
    config.SetEntitySetAccessRule("*", EntitySetRights.All);
    config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2;
    config.SetEntitySetPageSize("People", 50);
}

Here we’re applying paging on the People entity set. You can apply it to all entities by using the “*” instead of “People”. This introduces some extra work that you have to do on the client. If there are multiple pages then you will need to follow the “__next” link that will be included in the returned result set.

image

If you want some indication of how many entities there are in the total change set (for example if you want to track percentage complete) you can request that the total number of items is included across all pages. This is done by including the $inlinecount=allpages parameter.

http://localhost:16896/PeopleDataService.svc/People?$expand=Addresses&filter=(LastUpdated%20gt%20datetime’1900-00-00T00:00:00.0000000′)&$inlinecount=allpages

image

Saving Changes

If you want to save changes back to the server you need to issue a POST (new entity), PUT (modified) or DELETE (you guessed it – delete) request to the data service. Here’s an example of each:

Modify an item by issuing a PUT to the address specified in the __metadata.uri element.

PUT http://localhost:16896/PeopleDataService.svc/People(guid’2b64d1ca-5802-4937-8134-0ad978415117′) HTTP/1.1
User-Agent: Fiddler
Host: localhost:16896
Accept: application/json
Content-Type: application/json
Content-Length: 71

{ “PersonId”: “2b64d1ca-5802-4937-8134-0ad978415117”, “Name”: “Mary”}

Insert an item by issuing a POST to the entity set url (

POST http://localhost:16896/PeopleDataService.svc/People HTTP/1.1
Accept: application/json
Content-Type: application/json
Host: localhost:16896
Content-Length: 77

{ “PersonId”: “C5AC406D-2278-48D7-91BF-BDFF53ECD9F6”, “Name”: “New Person”}

Delete an item by issuing a DELETE to the address specified in the __metadata.uri element

DELETE http://localhost:16896/PeopleDataService.svc/People(guid’2b64d1ca-5802-4937-8134-0ad978415117′) HTTP/1.1
Accept: application/json
Host: localhost:16896
Content-Length: 2

In the case of the delete this request should return a valid success response (ie a 204) but in fact it won’t have deleted the item, only set the IsDeleted flag to true.

Completing the Loop

Bringing this all together you should iterate over the download-upload cycle until the client and server are in sync (ie no changes pending in either direction).

Hopefully this will have given you an idea of how to perform synchronization using odata in a way that is not reliant on a particular client technology. Feel free to post comments on how you think this could be improved or if you see any issues with this approach.

Leave a comment