Introduction

Distributed caching allows you to harness additional benefits of horizontal scale-out, without losing on low latency offered by local on-heap tiers.

diag 023dc90b3469ccf84002e716bfe382de

  • Hot data is cached locally, hotter data in faster tiers

  • Data cached by one application instance is available to all cluster members.

  • Full data is available in the cluster.

  • One or more mirror servers may be deployed to provide HA

To enable clustering with Terracotta, you will have to deploy a Terracotta server configured with clustered cache storage. For convenience Ehcache 3.1 introduced a downloadable kit that contains the Terracotta Server and also the required client libraries.

You will then need to configure a cache manager to have clustering capabilities such that the caches it manages can utilize the clustered storage. Finally, any caches which should be distributed should be configured with a clustered storage tier.

Clustering Concepts

In this section we discuss some Terracotta clustering terms and concepts that you need to understand before creating cache managers and caches with clustering support.

Server off-heap resource

Server off-heap resources are storage resources defined at the server. Caches can reserve a storage area for their cluster tiers within these server off-heap resources.

Cluster Tier Manager

The Ehcache Cluster Tier Manager is the server-side component that gives clustering capabilities to a cache manager. Cache managers connect to it to get access to the server’s storage resources so that the clustered tiers of caches defined in them can consume those resources. An Ehcache cluster tier manager at the server side is identified by a unique identifier. Using the unique identifier of any given cluster tier manager, multiple cache managers can connect to the same cluster tier manager in order to share cache data. The cluster tier manager is also responsible for managing the storage of the cluster tier of caches, with the following different options.

Dedicated pool

Dedicated pools are a fixed-amount of storage pools allocated to the cluster tiers of caches. A dedicated amount of storage is allocated directly from server off-heap resources to these pools. And this storage space is used exclusively by a given cluster tier.

Shared pool

Shared pools are also fixed-amount storage pools, but can be shared by the cluster tiers of multiple caches. As in the case of dedicated pools, shared pools are also carved out from server off-heap resources. The storage available in these shared pools is strictly shared. In other words, no cluster tier can ask for a fixed-amount of storage from a shared pool.

Sharing of storage via shared pools does not mean that the data is shared. That is, if two caches are using a shared pool as their clustered tier, the data of each cache is still isolated but the underlying storage is shared. Consequently, when resource capacity is reached and triggers eviction, the evicted mapping can come from any of the cluster tiers sharing the pool.

Here is a pictorial representation of the concepts explained above:

diag 7066c19a85be7f4b557b92d7002a3c17

Starting the Terracotta Server

The snippet below defines two offheap resources named primary-server-resource and secondary-server-resource having sizes 128MB and 96MB respectively:

offheap-resources=primary-server-resource:128MB,secondary-server-resource:96MB

This can either be defined in config properties file or during server startup.

Assuming that you have the clustered Ehcache kit available locally, start with extracting the ehcache-clustered kit. Change to your extracted directory and then execute the start-tc-server script located under $KIT_DIR/server/bin to start the Terracotta server. You will then need to activate the cluster using activate command of config tool which is located under $KIT_DIR/tools/bin.

You will need to have JAVA_HOME point to a Java 8 installation while starting the Terracotta server.

Check for the below INFO log to confirm if the server started successfully, Terracotta Server instance has started up as ACTIVE node on 0:0:0:0:0:0:0:0:9410 successfully, and is now ready for work.

Creating a Cache Manager with Clustering Capabilities

After starting the Terracotta server, as described in the previous section, you can now proceed to create the cache manager. For creating the cache manager with clustering support you will need to provide the clustering service configuration. Here is a code sample that shows how to configure a cache manager with clustering service.

CacheManagerBuilder<PersistentCacheManager> clusteredCacheManagerBuilder =
    CacheManagerBuilder.newCacheManagerBuilder() (1)
        .with(ClusteringServiceConfigurationBuilder.cluster(URI.create("terracotta://localhost/my-application")) (2)
            .autoCreateOnReconnect(c -> c)); (3)
PersistentCacheManager cacheManager = clusteredCacheManagerBuilder.build(true); (4)

cacheManager.close(); (5)
1 Returns the org.ehcache.config.builders.CacheManagerBuilder instance.
2 Use the ClusteringServiceConfigurationBuilder static method .cluster(URI) for connecting the cache manager to the clustered storage at the URI specified that returns the clustering service configuration builder instance. Sample URI provided in the example is pointing to the clustered storage instance named "my-application" on the Terracotta server (assuming the server is running on localhost and port 9410).
3 Auto-create the clustered storage if it doesn’t already exist. We also allow auto-create on reconnection since the cluster is not persistent.
4 Returns a fully initialized cache manager that can be used to create clustered caches.
5 Close the cache manager.

Cache Manager Configuration and Usage of Server Side Resources

This code sample demonstrates the usage of the concepts explained in the previous section in configuring a cache manager and clustered caches by using a broader clustering service configuration:

CacheManagerBuilder<PersistentCacheManager> clusteredCacheManagerBuilder =
    CacheManagerBuilder.newCacheManagerBuilder()
        .with(ClusteringServiceConfigurationBuilder.cluster(URI.create("terracotta://localhost/my-application")).autoCreateOnReconnect(server -> server
            .defaultServerResource("primary-server-resource") (1)
            .resourcePool("resource-pool-a", 8, MemoryUnit.MB, "secondary-server-resource") (2)
            .resourcePool("resource-pool-b", 10, MemoryUnit.MB))) (3)
        .withCache("clustered-cache", CacheConfigurationBuilder.newCacheConfigurationBuilder(Long.class, String.class, (4)
            ResourcePoolsBuilder.newResourcePoolsBuilder()
                .with(ClusteredResourcePoolBuilder.clusteredDedicated("primary-server-resource", 8, MemoryUnit.MB)))) (5)
        .withCache("shared-cache-1", CacheConfigurationBuilder.newCacheConfigurationBuilder(Long.class, String.class,
            ResourcePoolsBuilder.newResourcePoolsBuilder()
                .with(ClusteredResourcePoolBuilder.clusteredShared("resource-pool-a")))) (6)
        .withCache("shared-cache-2", CacheConfigurationBuilder.newCacheConfigurationBuilder(Long.class, String.class,
            ResourcePoolsBuilder.newResourcePoolsBuilder()
                .with(ClusteredResourcePoolBuilder.clusteredShared("resource-pool-a")))); (7)
PersistentCacheManager cacheManager = clusteredCacheManagerBuilder.build(true); (8)

cacheManager.close();
1 defaultServerResource(String) on ClusteringServiceConfigurationBuilder instance sets the default server off-heap resource for the cache manager. From the example, cache manager sets its default server off-heap resource to primary-server-resource in the server.
2 Adds a resource pool for the cache manager with the specified name (resource-pool-a) and size (28MB) consumed out of the named server off-heap resource secondary-server-resource. A resource pool at the cache manager level maps directly to a shared pool at the server side.
3 Adds another resource pool for the cache manager with the specified name (resource-pool-b) and size (32MB). Since the server resource identifier is not explicitly passed, this resource pool will be consumed out of default server resource provided in Step 3. This demonstrates that a cache manager with clustering support can have multiple resource pools created out of several server off-heap resources.
4 Provide the cache configuration to be created.
5 ClusteredResourcePoolBuilder.clusteredDedicated(String, long, MemoryUnit) allocates a dedicated pool of storage to the cache from the specified server off-heap resource. In this example, a dedicated pool of 32MB is allocated for clustered-cache from primary-server-resource.
6 ClusteredResourcePoolBuilder.clusteredShared(String), passing the name of the resource pool specifies that shared-cache-1 shares the storage resources with other caches using the same resource pool (resource-pool-a).
7 Configures another cache (shared-cache-2) that shares the resource pool (resource-pool-a) with shared-cache-1.
8 Creates fully initialized cache manager with the clustered caches.
When a cache is allocated a block of memory from a shared pool, it is retained forever and would never get reallocated to another cache sharing the pool.

Ehcache Cluster Tier Manager Lifecycle

When configuring a cache manager to connect to a cluster tier manager there are three possible connection modes:

CacheManagerBuilder<PersistentCacheManager> autoCreate = CacheManagerBuilder.newCacheManagerBuilder()
        .with(ClusteringServiceConfigurationBuilder.cluster(URI.create("terracotta://localhost/my-application"))
            .autoCreate(server -> server (1)
              .resourcePool("resource-pool", 8, MemoryUnit.MB, "primary-server-resource")))
        .withCache("clustered-cache", CacheConfigurationBuilder.newCacheConfigurationBuilder(Long.class, String.class,
            ResourcePoolsBuilder.newResourcePoolsBuilder()
                .with(ClusteredResourcePoolBuilder.clusteredShared("resource-pool"))));

CacheManagerBuilder<PersistentCacheManager> autoCreateOnReconnect = CacheManagerBuilder.newCacheManagerBuilder()
        .with(ClusteringServiceConfigurationBuilder.cluster(URI.create("terracotta://localhost/my-application"))
            .autoCreateOnReconnect(server -> server (2)
              .resourcePool("resource-pool", 8, MemoryUnit.MB, "primary-server-resource")))
        .withCache("clustered-cache", CacheConfigurationBuilder.newCacheConfigurationBuilder(Long.class, String.class,
            ResourcePoolsBuilder.newResourcePoolsBuilder()
                .with(ClusteredResourcePoolBuilder.clusteredShared("resource-pool"))));

CacheManagerBuilder<PersistentCacheManager> expecting = CacheManagerBuilder.newCacheManagerBuilder()
        .with(ClusteringServiceConfigurationBuilder.cluster(URI.create("terracotta://localhost/my-application"))
            .expecting(server -> server (3)
              .resourcePool("resource-pool", 8, MemoryUnit.MB, "primary-server-resource")))
        .withCache("clustered-cache", CacheConfigurationBuilder.newCacheConfigurationBuilder(Long.class, String.class,
            ResourcePoolsBuilder.newResourcePoolsBuilder()
                .with(ClusteredResourcePoolBuilder.clusteredShared("resource-pool"))));

CacheManagerBuilder<PersistentCacheManager> configless = CacheManagerBuilder.newCacheManagerBuilder()
        .with(ClusteringServiceConfigurationBuilder.cluster(URI.create("terracotta://localhost/my-application")))
            (4)
        .withCache("clustered-cache", CacheConfigurationBuilder.newCacheConfigurationBuilder(Long.class, String.class,
            ResourcePoolsBuilder.newResourcePoolsBuilder()
                .with(ClusteredResourcePoolBuilder.clusteredShared("resource-pool"))));
1 In auto create mode if no cluster tier manager exists then one is created with the supplied configuration. If it exists and its configuration matches the supplied configuration then a connection is established. If the supplied configuration does not match then the cache manager will fail to initialize.
2 In auto create on reconnect mode we additionally support auto creation of any necessary entities when reconnecting to a cluster. This behavior is useful in an non-persistent cluster in case the cluster loses its state due to a restart (planned or accidental).
3 In expected mode if a cluster tier manager exists and its configuration matches the supplied configuration then a connection is established. If the supplied configuration does not match or the cluster tier manager does not exist then the cache manager will fail to initialize.
4 In config-less mode if a cluster tier manager exists then a connection is established without regard to its configuration. If it does not exist then the cache manager will fail to initialize.

Configuring a Clustered Cache

Clustered Storage Tier

CacheConfiguration<Long, String> config = CacheConfigurationBuilder.newCacheConfigurationBuilder(Long.class, String.class,
    ResourcePoolsBuilder.newResourcePoolsBuilder()
        .heap(2, MemoryUnit.MB) (1)
        .with(ClusteredResourcePoolBuilder.clusteredDedicated("primary-server-resource", 8, MemoryUnit.MB))) (2)
    .withService(ClusteredStoreConfigurationBuilder.withConsistency(Consistency.STRONG))
    .build();

Cache<Long, String> cache = cacheManager.createCache("clustered-cache-tiered", config);
cache.put(42L, "All you need to know!");
1 Configuring the heap tier for cache.
2 Configuring the cluster tier of dedicated size from server off-heap resource using ClusteredResourcePoolBuilder.

The equivalent XML configuration is as follows:

<cache alias="clustered-cache-tiered">
  <resources>
    <heap unit="MB">10</heap> (1)
    <tc:clustered-dedicated unit="MB">50</tc:clustered-dedicated> (2)
  </resources>
  <tc:clustered-store consistency="strong"/>
</cache>
1 Specify the heap tier for cache.
2 Specify the cluster tier for cache through a custom service configuration from the clustered namespace.

Specifying consistency level

Ehcache offers two levels of consistency:

Eventual

This consistency level indicates that the visibility of a write operation is not guaranteed when the operation returns. Other clients may still see a stale value for the given key. However this consistency level guarantees that for a mapping (K, V1) updated to (K, V2), once a client sees (K, V2) it will never see (K, V1) again.

Strong

This consistency level provides strong visibility guarantees ensuring that when a write operation returns other clients will be able to observe it immediately. This comes with a latency penalty on the write operation required to give this guarantee.

CacheConfiguration<Long, String> config = CacheConfigurationBuilder.newCacheConfigurationBuilder(Long.class, String.class,
        ResourcePoolsBuilder.newResourcePoolsBuilder()
                .with(ClusteredResourcePoolBuilder.clusteredDedicated("primary-server-resource", 2, MemoryUnit.MB)))
    .withService(ClusteredStoreConfigurationBuilder.withConsistency(Consistency.STRONG)) (1)
    .build();

Cache<Long, String> cache = cacheManager.createCache("clustered-cache", config);
cache.put(42L, "All you need to know!"); (2)
1 Specify the consistency level through the use of additional service configuration, using strong consistency here.
2 With the consistency used above, this put operation will return only when all other clients have had the corresponding mapping invalidated.

The equivalent XML configuration is as follows:

<cache alias="clustered-cache">
  <resources>
    <tc:clustered-dedicated unit="MB">50</tc:clustered-dedicated>
  </resources>
  <tc:clustered-store consistency="strong"/> (1)
</cache>
1 Specify the consistency level through a custom service configuration from the clustered namespace.

Clustered Cache Expiry

Expiry in clustered caches work with an exception that Expiry#getExpiryForAccess is handled on a best effort basis for cluster tiers. It may not be as accurate as in the case of local tiers.

Clustered Unspecified Inheritance

We have included an option which allows a cache to be created inside the cache manager without having to explicitly define its cluster tier resource pool allocation. In order to use this feature the cluster tier must already have been created with either a shared or dedicated resource pool.

In this case the definition of the cluster resource is done simply with a clustered() resource pool. This effectively means unspecified and indicates you expect it to exist already. It will then inherit the clustered resource pool as it was configured when creating the cluster tier.

This option provides many benefits. The main benefit is it simplifies clustered configuration by allowing clustered resource pool configuration to be handled by one client, then all subsequent clients can inherit this configuration. In addition, it also reduces clustered pool allocation configuration errors. More importantly, sizing calculations only need to be done by one person and updated in one location. Thus any programmer can use the cache without having to worry about using matching resource pool allocations.

The example code below shows how this can be implemented.

CacheManagerBuilder<PersistentCacheManager> cacheManagerBuilderAutoCreate = CacheManagerBuilder.newCacheManagerBuilder()
        .with(ClusteringServiceConfigurationBuilder.cluster(URI.create("terracotta://localhost/my-application"))
          .autoCreateOnReconnect(server -> server  (1)
            .resourcePool("resource-pool", 8, MemoryUnit.MB, "primary-server-resource")));

PersistentCacheManager cacheManager1 = cacheManagerBuilderAutoCreate.build(false);
cacheManager1.init();
try {
  CacheConfiguration<Long, String> cacheConfigDedicated = CacheConfigurationBuilder.newCacheConfigurationBuilder(Long.class, String.class,
    ResourcePoolsBuilder.newResourcePoolsBuilder()
      .with(ClusteredResourcePoolBuilder.clusteredDedicated("primary-server-resource", 8, MemoryUnit.MB)))  (2)
    .withService(ClusteredStoreConfigurationBuilder.withConsistency(Consistency.STRONG))
    .build();

  Cache<Long, String> cacheDedicated = cacheManager1.createCache("my-dedicated-cache", cacheConfigDedicated);  (3)

  CacheManagerBuilder<PersistentCacheManager> cacheManagerBuilderExpecting = CacheManagerBuilder.newCacheManagerBuilder()
    .with(ClusteringServiceConfigurationBuilder.cluster(URI.create("terracotta://localhost/my-application"))
      .expecting(server -> server  (4)
        .resourcePool("resource-pool", 8, MemoryUnit.MB, "primary-server-resource")));

  PersistentCacheManager cacheManager2 = cacheManagerBuilderExpecting.build(false);
  cacheManager2.init();
  try {
    CacheConfiguration<Long, String> cacheConfigUnspecified = CacheConfigurationBuilder.newCacheConfigurationBuilder(Long.class, String.class,
      ResourcePoolsBuilder.newResourcePoolsBuilder()
        .with(ClusteredResourcePoolBuilder.clustered()))  (5)
      .withService(ClusteredStoreConfigurationBuilder.withConsistency(Consistency.STRONG))
      .build();

    Cache<Long, String> cacheUnspecified = cacheManager2.createCache("my-dedicated-cache", cacheConfigUnspecified); (6)
  } finally {
    cacheManager2.close();
  }
} finally {
  cacheManager1.close();
}
1 Configure the first cache manager with auto create on reconnect
2 Build a cache configuration for a clustered dedicated resource pool
3 Create cache my-dedicated-cache using the cache configuration
4 Configure the second cache manager as expecting
5 Build a cache configuration for a clustered unspecified resource pool, which will use the previously configured clustered dedicated resource pool.
6 Create cache with the same name my-dedicated-cache and use the clustered unspecified cache configuration