An educated guess (Timur Fanshteyn): Extending the Data Grid - LocalStorage=False and Extend connectivity

Oracle coherence is an excellent technology. It is extremely powerful in distributing data and processing across multiple physical and virtual servers. This functionality is extremely simple to achieve, you simply start another machine on coherence JVM on a connected network, and the two nodes will find each other and start talking.

Well… with great power comes great responsibility. Here are a few important things you must be aware of and keep in mind when designing and deploying your distributed coherence applications.

All Coherence Nodes act as active members of the grid. This statement is true even if LocalStorage is set to false. Setting LocalStorage to false only removes the node from the list of places where distributed data can be located. However, for replicated data, invocation services, node membership, etc the node is still a fully capable node. See: http://forums.oracle.com/forums/message.jspa?messageID=3664294
Extend is the only way to make Coherence Client-Server. This is very closely related to the previous point. It is also very easy to overlook. More explanation below.
All coherence nodes must have access to all classes included in POF Configuration. Default POF Configuration Context initializes all types included in the POF configuration even if they will not be used. That means that you must provide access to the classes that define them even though you will not be using them. If you don’t, the cache will fail to initialize
You DO NOT need to have identical configuration between all nodes in the grid. There is a benefit to keeping a different set of Cache and POF Configurations (see previous point) However, there is also a complexity of managing multiple configuration files.
Extend clients require an extra network hop for working with distributed data located in the grid. However, the overhead is small. Yes – there is an overhead of an extra network hop of working with over extend client. This is something that has to be profiled and waited against the benefits of terminating the extend of the grid to at the Extend Proxy. Consider it over making a client a member of the grid, and therefore making it take role in the cluster management.
Extend clients require proxy to pre/post process invocation requests. Complexity for processing invocation service requests is bigger for extend clients. By default, invocation request will be processed by the proxy node, instead of being distributed to the multiple nodes. http://coherence.oracle.com/display/COH35UG/Configuring+and+Using+Coherence+Extend

Extend is the only way to make Coherence Client-Server

Here’s an example where this becomes critically important:

Imagine a 2 node coherence grid that holds some transaction data, and a web site that provides views against that data. Fairly simple environment.

As you see, the connections are between every machine. Every machine is responsible for other members in the grid. Now, we create another website with different functionality, but utilizing the data in the same coherence cache.

Every member is again talking to every other member. Pay attention to the connectivity between Web1 and Web2. Creating second website, not only requires connectivity between the new webserver and data nodes, but also between web2 and web1. If running from different DMZ sites, this might mean a new set of rules in the firewalls. If Web2 is a disaster recovery site that is running in warm state, that will mean that connection DR will actively participate in actions within the Production grid. A different layout of the same problem is when you have separate data entry service, and website for data display

Connectivity between Data Entry Service and Website might not bet apparent upfront, however, without it, the cluster will not start.

Even without physical connectivity concerns, the extend of the grid is something to think extra hard about. Coherence includes the concept of Senior Member. From http://wiki.tangosol.com/display/COH34UG/Security+Framework: “Each clustered service in Coherence maintains a concept of a "senior" service member (cluster node), which serves as a controlling agent for a particular service. While the senior member does not have to consult anyone when accessing a clustered resource, any junior node willing to join that service has to request and receive a confirmation from the senior member, which in turn notifies all other cluster nodes about the joining node.” After running the grid for a while, the coherence node running within Web site, might be the oldest running member of the grid, and therefore will become the Senior Member. It will no longer be a “client” of the grid as you might believe, but it will become critical actor during grid membership resolution.