As for your second image, yes, the architecture will look like this.
If so, how will the Infinispan cache interact with each other?
Please look here: https://docs.jboss.org/author/display/ISPN/Getting+Started+Guide#GettingStartedGuide-UsingInfinispanasanembeddeddatagridinJavaSE Infinispan will manage it using the JGroups protocol and send messages between nodes. A cluster will be formed, and the nodes will be grouped. After that, you can experience the expected behavior of replicating records to specific nodes.
And here we go to the next question:
It is possible that at some point we will look for data for those users who will be on another Infinispan Service Instance cache. Right? So what will happen in this scenario? Will infinispan take care of these things?
Infinispan was designed for this scenario, so you don't need to worry about it at all. If you have, for example, 4 nodes and setting the distribution mode using numberOfOwners = 2, your cached data will be displayed exactly on 2 nodes at every moment. When you issue a GET command for the owner of a NON node, the record will be received from the owner .
You can also set clustering mode for replication, where all nodes contain all records. Please read more about the modes here: https://docs.jboss.org/author/display/ISPN/Clustering+modes and choose what works best for your use case.
In addition, when you add a new node to the cluster, StateTransfer will be executed and synchronize / rebalance records in the cluster. NonBlockingStateTransfer is already implemented, so your cluster will still be able to serve responses at this join stage. See: https://community.jboss.org/wiki/Non-BlockingStateTransferV2 Similarly for deleting / crashing nodes in your cluster. There will be an automatic rebalancing process , therefore, for example, some records (numOwners = 2), which after a failure live on only one node, will be replicated accordingly to live on 2 nodes in accordance with the numberOfOwners property in distribution mode.
To summarize, your cluster will still be updated, and it does not matter which node you request for a specific record . If it does not contain it, a record will be received from the owner.
if so, what kind of configuration setting do I need to make sure this thing works fine.
The above start guide is full of examples plus you can find examples of configuration files in the Infinispan distribution: ispn / etc / config-samples / *
I would also suggest you take a look at this source: http://refcardz.dzone.com/refcardz/getting-started-infinispan , where you can find even simpler and very quick configuration examples.
This source also contains information related to the solution for your first question: "Should I use the built-in mode or the remote client server mode?" From my point of view, using a remote cluster is a more ready-to-use solution (see http://howtojboss.com/2012/11/07/data-grid-why/ ). Your caching level scales very easily, is highly accessible and fault tolerant, and does not depend on the level of your database or the level of the application, because it is simply located between them.
And you may also be interested in this feature: https://docs.jboss.org/author/display/ISPN/Cache+Loaders+and+Stores