[External] together with [Include] doesn't work for me

09-22-2009 10:47 AM |

Hi,

I'm experimenting with [External] associations.
I'm using separate assemblies for my domain services (for domain contexts respectively).
To avoid an extra round-trip to the server I'm trying to get around the loading of the referenced entities on the client-side (see [1]).

I tried to add an [Include] to my [External] associations.

Section [2] doesn't take this scenario into account.

I'm getting the following error: "Entity types cannot be defined in multiple assemblies"

I assume that the code-generator tries to re-construct the referenced type and misses the information that it is [External], thus not to be re-constructed.

Am I missing something?
If not:
  - Are reasons known for this scenario not to be supported?
  - Is this scenario planned to be supported?

(RIA Services overview document)
[1] Section 20.2.2 "Client Project"
[2] Section 4.8.1  "Returning Related Entities"

Regards,
Kasimier

Re: [External] together with [Include] doesn't work for me

09-22-2009 1:05 PM |

Hmm, seems like I don't want to use [Include] in conjunction with [External].

Somehow [External] for me currently seems to be just a missing flag on [Include].

1) Both attributes will generated a property on the client side.

[edited]
Actually the Association attribute is triggering the generation of the corresponding property. But you can't have [External] or [Include] without an [Association] anyway.
[/edited]

2) Different serialization semantics:
   [Include] implicitely sends the complex type over the wire (if available).
   [External] implicitely doesn't send the complex type over the wire.

3) Similar tracking semantics:
   [Include] keeps track of its entities in the including domain context.
   [External] dispatches tracking of its entities to the external domain context.
  
  
So what about adding the following 2 properties to the [Include] attribute?

[Include( IsExternal = true, IsTransferred = true )]

1) IsExternal is false by default.
   If IsExternal == true, then just dispatch tracking of the entity to the external domain context, otherwise not.
  
2) IsTransferred is true by default.
   If IsTransferred == true, then send the complex type over the wire, otherwise not.

3) If (IsExternal == true) and (IsTransferred == true), then just materialize the complex type
   and add it (or let the merge operation be specifyable) to the registered external domain context.


Or simply add a flag to [External]:

[External(IsTransferred = true)]


This way you leave the decision to us, whether we need complex types be send over the wire directly, or be acquired via custom code on the client-side.


Comments are very appreciated.

Regards,
Kasimier

Re: [External] together with [Include] doesn't work for me

09-22-2009 2:00 PM |

Include means that during serialization go ahead and include the associated object during serialization (aka load that associated object). External means that the entity is going to be tracked (and therefore loaded) by a different DomainService/DomainContext and that the association should be wired up client side against that other DomainContext's EntityList. To do the include the DomainContext needs to generate that Entity and the EntityList. That is causing a conflict with the other DomainContext which is also generating the same Entity and since you can't have two DomainContext's with the same Entity you are getting the error. The External flag is a bit of a red herring here as it has no involvement in the error you are getting.

I would guess this scenario isn't planned to be supported. You are trying to get two DomainContexts to load like a single DomainContext and I can't understand why you wouldn't just use the single DomainContext in the first place.

Re: [External] together with [Include] doesn't work for me

09-22-2009 2:12 PM |

To add what I already said, according to the RIA Services overview the reason that the External attribute exists is for situations where we have two seperate data sources server side and we want to create a relationship between them. An example would be where you have some data coming from two seperate databases. What you are doing with External is taking a single server side data source and splitting into two seperate client side data sources which is pretty much the exact opposite of the original intent.

Re: [External] together with [Include] doesn't work for me

09-22-2009 6:05 PM |

@ColinBlair:
"Include means that during serialization go ahead and include the associated object during serialization"


Ah, you're right. I assumed that it also defined wheter the property will actually be generated.
The [Association] seems to be the one that triggers the generation of the property.

This means that [External] could have a IsIncluded property, defining whether we want to send the external entity over the wire.


@ColinBlair:
"I would guess this scenario isn't planned to be supported. You are trying to get two DomainContexts to load like a single DomainContext and I can't understand why you wouldn't just use the single DomainContext in the first place"

About why this might not be supported
-------------------------------------

First, as mentioned [External] is not that different from [Include].

[External] makes sure that the entity is handled by an EntityList of the *external* domain-context and not the referencing domain-context.

Included complex objects (Entities) can already be serialized and deserialized when using [Include], so where's the problem with doing this for external complex objects (also Entities)?

What information is not available during serialization/deserialization to support this?
I'm assuming the follwing:: the *external* domain-context is not available during deserialization-time.

When you do a AddReference(...), the referenced domain-context is never stored anywhere during that registration, only the *external* EntityList is stored.

I think this is a flaw.

I currently cannot think of an other reason why the following could not work:

1) Serialize the external complex object.
2) Send the stuff over the wire.
3) Deserialize whatever you can.
4) Let the domain-context (or is it the DomainClient) get hold of the *external* domain-context and dispatch deserialization of external entities to this *external* domain-context.
5) Put the deserialized entity into the EntityContainer of the *external* domain-context.


The reasons for multiple domain-contexts are modularity and dependencies
------------------------------------------------------------------------

1) You have domain A.
2) You have domain B which uses parts of domain A.

You seem to assume that everything in domain B should be accessible to domain A and vice versa.
I think this would be a very limited scenario.

Modularity
----------

I read that you're using Prism, so you are aware of the aspects of mudularity.
By using a single all-for-all domain-context you effectively lose modularity and plugability.
One cannot plug-in new features without extensibility. For every new plug-in, the all-for-all domain-context needs to be changed and updated from a central place. It becomes the sum of all features.
3rd party creators of modules would not be able to plug into your architecture this way.
Additionally you need to load this single domain-context even if only fraction of the modules you provide are actually used.

Dependencies
------------

Using a single all-for-all domain-context becomes problematic:
(E.g. see http://agileinaflash.blogspot.com/2009/04/principles-of-package-coupling.html)

Consider a service which lets you query for all country names.
If this would sit in a all-for-all domain-context, then any module using any other services in this domain-context, would depend on that tiny country-service, making it highly responsible and stable (in the bad sense).
Any changes to that tiny country-service now become problematic.


@ColinBlair:
"...reason that the External attribute exists is for situations where we have two seperate data sources server side and we want to create a relationship between them."


If one has two data sources on the server-side an they are related, so what does hinder you to query the one inside the other? Apparently nothing does hinder you from doing that on the client-side.
I think this smells a bit. This moves complexity over from the server-side to the client-side, only because MS cannot think of a scenario for server-side aggregation?

@ColinBlair:
"An example would be where you have some data coming from two seperate databases. What you are doing with External is taking a single server side data source and splitting into two seperate client side data sources which is pretty much the exact opposite of the original intent"


No, my data does not come from a single database, no I don't have a single-server side data source.
I'm not using EF or similar.

To clear up the picture a bit: My scenario could be simplified to existing domain models; additionally there are domain models, which aggregate parts of other domain-models, where the aggregated model *must* *not* have dependencies to the aggregating model. I also have facade models with e.g. legacy models as sub-systems, wrapper models, etc.

How and what, I actually pull from a persistance layer is totally irrelevant and will differ from case to case. I think and hope, persistence issues should not relate that much to RIA Services.

The world doesn't consist only of issues about how to work with table-data from SQL Servers with EF/L2S.

This all means that what I'm able to query at server-side is not related to the structure and partitioning of domain models; thus I'm having a problem with the current, apparently short-sighted, support for cross domain activities.


This all said, I'm currently trying to inject automatic loading of missing external enties by overriding the Load(...) method of the domain-context.
That doesn't work as well. The problem is really that you don't have access to the *external* domain-contexts from within here.

The RIA Services overview document proposes the following for loading the *external* entities - section 20.2.2 "Client Project":

public void InitializeDomainContexts()
{
  MyOrdersContext myOrders = new MyOrdersContext();
  ProductCatalogContext productCatalog = new ProductCatalogContext(); 
  myOrders.AddReference<Product>(productCatalog);
 
  // Start loading data in each DomainContext...
  // so that the cross-DomainContext associations will be populated.
}

It reads:
"Start loading data in each DomainContext...
so that the cross-DomainContext associations will be populated."

Yes, but how to do this, please?

- How do we know at this point which entities shall be loaded?
- Subsequentely, if we do not know which entities will be loaded, how can we load any *external* entities at this point?
- If additional entities are loaded after this point in time (after setting a filter), when and where do we load the *external* entities? In each user-defined callback when calling Load(...) of the domain-context?

How do we load *external* entities of *external* entities?


I'm a bit clueless currently.


Regards & thanks,
Kasimier

Re: [External] together with [Include] doesn't work for me

09-22-2009 7:37 PM |

Casimodo72:

Ah, you're right. I assumed that it also defined wheter the property will actually be generated.
The [Association] seems to be the one that triggers the generation of the property.

The [Include] is involved in the code generation. If you have the association but no include and there isn't a seperate Get then the entity will not be generated.

Casimodo72:

This means that [External] could have a IsIncluded property, defining whether we want to send the external entity over the wire.

Casimodo72:

First, as mentioned [External] is not that different from [Include].

I don't think that is correct, the [External] and [Include] attributes are not alike and have nothing to do with each other. One clue is the difference in namespaces. IncludeAttribute is in the System.Web.DomainService namespace. ExternalAttribute is in the System.Web.Ria.Data namespace. The IncludeAttribute belongs to the DomainService, it doesn't even exist on the client side. It is purely a directive to the DomainService on what related entities should be serialized. If you are connecting your DomainContext to a non DomainService based server (which you can do) then there is no IncludeAttribute. As I stated earlier, the code generator does use the IncludeAttribute as a hint when generating the DomainContext but the code generator and the DomainContext are not the same thing.

(Edit: The previous paragraph was orignally more forceful then I meant it to be. I have modified it to be closer to my intent.)

I understand what you are saying in the rest of your post about modularity, but what you are asking for makes modularity impossible and that is my central problem with what you are asking for.

You are saying that you want seperate DomainContext's for modularity reasons but you want the related DomainContexts to be able to serialize and deserialize each others entities. If the DomainContext's could do that then they would need to have direct references to each others entity types which means they are strongly attached to each other which means they are not modular and cannot be seperated. If you are going to do that, then you may as well have a single DomainContext.

Yes, I have a highly modular system using Prism and I have lots of DomainContexts, one for each module. None of the DomainContexts and DomainServices know about each other, none of them use the External attribute. All of my entities have interfaces attached to them so that I can share code between the modules. I could setup one External attribute on each to my central object ( the root table of the database) but I don't need access to the root table from the modules so I just pass the key information. That keeps everything very neat, tidy, and really easy to extend.

Casimodo72:

No, my data does not come from a single database, no I don't have a single-server side data source.
I'm not using EF or similar.

But you do have a single persistance layer, from the RIA Services perspective that is what matters. If you persistance layer can aggregate the data from multiple databases, Excel files, whatever that that is great for your persistance layer. My point is that you are trying to get two seperate DomainServices and DomainContexts attached to the same Domain (aka persistance layer) which doesn't match with the design of the External attribute. If you didn't have a single persistance layer then you couldnt' try to load the data using the Include.

Re: [External] together with [Include] doesn't work for me

09-23-2009 4:22 PM |

Hi,

(Quoting mis-formats somehow my other text, so I'm not using the standard quotation mechanism)

---
Quote user="ColinBlair"
     "If the DomainContext's could do that then they would need to have direct references to each others entity types which means they are strongly attached to each other which means they are not modular and cannot be seperated."
End Quote
---

Please consider the following:
1) domain-context A
2) domain-context B, which depends on domain-context A

The same applies to domain-services respectively.

I think what you are missing in the picture you describe is that domain-context A is completely independent.
It is only domain-context B which is dependent.
If you merge both contexts then you lose the independecy of domain-context A; this is an important difference.
Modularity does not mean that every module must be completely isolated.
Please consider re-examining your definition of modularity.

---
Quote user="ColinBlair"
     "My point is that you are trying to get two seperate DomainServices and DomainContexts attached to the same Domain (aka persistance layer) which doesn't match with the design of the External attribute. If you didn't have a single persistance layer then you couldnt' try to load the data using the Include."
End Quote
---

Please explain why a Domain is the same as the persistence layer.
This would mean that I should get rid of the AuthenticationContext, since it is fed via my persistence layer.

In my eyes a "Domain" in the RIA Services sense is a group of domain classes which are related to a specific task/field/domain (e.g. authentication with the AuthenticationContext).

You definition of a "Domain" seems to be directly related to the physical ability of gaining access to some data.

I'm not saying that you're wrong and I'm right, but that I would be very disappointed if your definition is the way to go for RIA Services.

I understand that if one really doesn't have access to data for a specific domain class on the server-side, then that would be a problem, and it can be solved by using [External] which moves the problem over to the client-side. Although, I still don't know why access to aggregated data is expected to be on the client-side but not on the server-side; it this a restriction of the persistence layer MS is designing against?

I think, the current behaviour of [External] is nice for scenarios, where e.g. master data is cached on the client, so data is not trasferred redundantly over the wire.


Again: Not allowing to use [External] domain classes to be transferred directly over the wire together with the aggregating domain class produces complexity on the client side, plus extra round-trips to the server.
[External] is currently the only way to allow for aggregation accross domain-contexts (domain-services respectively).
Please MS, when a server-side transmission is needed, consider to allow for a way to opt-in.


Here a bit of a possible way to load external entities on the client-side:

1) load entities

2) foreach distinct type of loaded entities:
      foreach entity:
           foreach external association:
               add a Where clause to an EntityQuery using the foreign key values
              
3) foreach distinct type of loaded entities:
      load externals based on the built EntityQuery
     
4) proceed further at #1 in order to load subsequent externals of externals
  
(I've coded that today and it seems to work fine)

Now those operations will be asynchronous. How do we handle that? How will I notify the UI? Will I pop-in and pop-out a loading-screen? I'm done... oh, wait, no, I'm not done... I'm done... oh, wait, no, I'm not done... etc, etc.
This would be easier if we had a composite LoadOperation.


Regards,
Kasimier

Re: [External] together with [Include] doesn't work for me

09-24-2009 12:57 PM |

I will get back to this, probably this weekend. I realized that I had managed to trap myself into advocating a position I didn't actually hold and got this completely sidetracked from where this conversation needed to be and I apologize for that. For example, no the definition of a domain is not limited to being a persistance layer. However, having two seperate persistance layers (which are considered two seperate domains within the business problem) was the situation that the External attribute was designed to solve. I let my original point, that the External and Include attributes are not related to each other and are probably not the solution to the problem you are trying to solve get switched to a debate on if your problem should be solved at all which is a stupid position for anyone to hold.

So, let me start again with the positive goal on how do we solve the problem of hierarchial DomainContexts where we want DomainContext's lower down the reference tree to be able to load the entities of DomainContext's higher up tree. I need to do a lot more thinking about this before I can continue this converation. I think the solution is going to involve a shared EntityContainer but I think there needs to be some allowance for multiple DomainContexts attached to a single DomainService all saved at once so that there is a single transaction. I know that V1 of RIA Services is pretty much code complete at this point and there may or may not be changes in the next release that make this problem easier or harder but I think any real solution is probably going to be targeted at V.Next of RIA Services.

Re: [External] together with [Include] doesn't work for me

09-24-2009 7:17 PM |

Colin, I thank you very much indeed for those wise and reflective words.

Regarding the transactional issue you mentioned: The scenario I have in mind does not imply that the referencing domain-context gains authority over update/insert/delete activities wrt the [External] entities. I think the external EntityList which is registered with AddReference(...) still should be the authority for such functionality (actually the external domain-context, which is - and that's the flaw - not reachable via the external EntityList).

If you find time to look at the description of my approach of automatic loading of [External] entities on the client side, you'll find that is raises some issues, which expose the client-side loading as being quite a hairy endeavour.
See http://forums.silverlight.net/forums/t/130729.aspx.

The essence of my request is to be able to opt-in in order to serialize the damn [External] thingy and send it over the wire, then merge it into the already available (since registered with AddReference(...)) external EntityList.

Regards & thanks for the patience,

Kasimier