CacheManager Configuration

The attributes of ehcache are:

• name

  an optional name for the CacheManager. The name is optional and primarily used for documentation or to distinguish Terracotta clustered cache state. With Terracotta clustered caches, a combination of CacheManager name and cache name uniquely identify a particular cache store in the Terracotta clustered memory.

  The name will show up in the Developer Consoles.

• updateCheck

  an optional boolean flag specifying whether this CacheManager should check for new versions of Ehcache over the Internet. If not specified, updateCheck="true".

• monitoring

  an optional setting that determines whether the CacheManager should automatically register the SampledCacheMBean with the system MBean server. Currently, this monitoring is only useful when using Terracotta and thus the "autodetect" value will detect the presence of Terracotta and register the MBean. Other allowed values are "on" and "off". The default is "autodetect".

  <Ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:noNamespaceSchemaLocation="ehcache.xsd"
               updateCheck="true" monitoring="autodetect">
  

Terracotta Server Configuration

Note: You need to install and run one or more Terracotta servers to use Terracotta clustering.

See http://www.terracotta.org/web/display/orgsite/Download.

With a server/servers up and running you need to specify the location of the servers.

Configuration can be specified in two main ways: by reference to a source of configuration or by use of an embedded Terracotta configuration file.

    <terracottaConfig>
        <tc-config>
            <servers>
                <server host="server1" name="s1"/>
                <server host="server2" name="s2"/>
            </servers>
            <clients>
                <logs>app/logs-%i</logs>
            </clients>
        </tc-config>
    </terracottaConfig>

Enabling Terracotta clustering per cache

Cache elements can also contain information about whether the cache can be clustered with Terracotta.

The terracotta sub-element has the following attributes:

• clustered=true|false

  Indicates whether this cache should be clustered with Terracotta. By default, if the terracotta element is included, clustered=true.

• valueMode=serialization|identity

  Indicates whether this cache should be clustered with serialized copies of the values or using Terracotta identity mode. By default, values will be cached in serialization mode which is similar to other replicated Ehcache modes. The identity mode is only available in certain Terracotta deployment scenarios and will maintain actual object identity of the keys and values across the cluster. In this case, all users of a value retrieved from the cache are using the same clustered value and must provide appropriate locking for any changes made to the value (or objects referred to by the value).

• coherentReads=true|false

  Indicates whether this cache should have coherent reads with guaranteed consistency across the cluster. By default, this setting is true. If you set this property to false, reads are allowed to check the local value without locking, possibly seeing stale values.

  This is a performance optimization with weaker concurrency guarantees and should generally be used with caches that contain read-only data or where the application can tolerate reading stale data.

  Note that from Ehcache 2.0, this setting has the same affect as coherent.

• copyOnRead=true|false

  Indicates whether cache values are deserialized on every read or if the materialized cache value can be re-used between get() calls.

  This setting is useful if a cache is being shared by callers with disparate classloaders or to prevent local drift if keys/values are mutated locally w/o putting back to the cache. i.e. if set to true then each thread has its own copy and cannot affect other threads.

  NOTE: This setting is only relevant for caches with valueMode=serialization

• coherent=true|false

  Indicates whether this cache should have coherent reads and writes with guaranteed consistency across the cluster. By default, its value is true. If this attribute is set to false (or "incoherent" mode), values from the cache are read without locking, possibly yielding stale data. Writes to a cache in incoherent mode are batched and applied without acquiring cluster-wide locks, possibly creating inconsistent values across cluster. Incoherent mode is a performance optimization with weaker concurrency guarantees and should generally be used for bulk-loading caches, for loading a read-only cache, or where the application that can tolerate reading stale data.

  This setting overrides coherentReads, which is deprecated. For backward compatibility any configurations setting a value for coherentReads will apply to coherent.

• synchronousWrites=true|false

  When set to true, clustered caches use Terracotta SYNCHRONOUS WRITE locks. Asynchronous writes(synchronousWrites="false") maximize performance byallowing clients to proceed without waiting for a "transaction received" acknowledgement from the server. Synchronous writes (synchronousWrites="true") maximizes data safety by requiring that a client receive server acknowledgement of a transaction before that client can proceed. If coherence mode is disabled using configuration (coherent="false") or through the coherence API, only asynchronous writes can occur (synchronousWrites="true" is ignored). By default this value is false (i.e. clustered caches use normal Terracotta WRITE locks).

  The simplest way to enable clustering is to add:

          <terracotta/>

  To indicate the cache should not be clustered (or remove the terracotta element altogether):

          <terracotta clustered="false"/>

  To indicate the cache should be clustered using identity mode:

          <terracotta clustered="true" valueMode="identity"/>

  Following is an example Terracotta clustered cache named sampleTerracottaCache.

      <cache name="sampleTerracottaCache"
             maxElementsInMemory="1000"
             eternal="false"
             timeToIdleSeconds="3600"
             timeToLiveSeconds="1800"
             overflowToDisk="false">
          <terracotta/>
      </cache>

Cache listeners

Cache listeners listen for changes, including replicated cluster changes, made through the Cache API. Because Terracotta cluster changes happen transparently directly to the MemoryStore a listener will not be invoked when an event occurs out on the cluster. If it occurs locally, then it must have occurred through the Cache API, so a local event will be detected by a local listener.

A common use of listeners is to trigger a reload of a just invalidated Element. In Terracotta clustering this is avoided as a change in one node is always coherent to the other nodes.

To send out cache change events across the cluster, you need to set up a local listener on the relevant cache so that these events can be propagated to other nodes. This is done by adding the following cacheEventListenerFactory tag to the cache:

    <cacheEventListenerFactory class="net.sf.ehcache.event.TerracottaCacheEventReplicationFactory"/>

Overflow to Disk

Overflow to Disk is ignored when using Terracotta Clustering. However it is not needed because the Terracotta server has its own overflow to disk. Once the maxElementsInMemory limit is reached Elements will be evicted to the cluster.

Example Code

This example prints out the cluster nodes and then registers a ClusterTopologyListener which prints out events as they happen.

    CacheManager mgr = ...

    CacheCluster cluster = mgr.getCluster("Terracotta");



    // Get current nodes

    Collection<ClusterNode> nodes = cluster.getNodes();

    for(ClusterNode node : nodes) {

      System.out.println(node.getId() + " " + node.getHostname() + " " + node.getIp());

    }



    // Register listener

    cluster.addTopologyListener(new ClusterTopologyListener() {

      public void nodeJoined(ClusterNode node) { System.out.println(node + " joined"); }

      public void nodeLeft(ClusterNode node) { System.out.println(node + " left"); }

      public void clusterOnline(ClusterNode node) { System.out.println(node + " enabled"); }

      public void clusterOffline(ClusterNode node) { System.out.println(node + " disabled"); }

    });

Uses for Cluster Events

If Ehcache got disconnected from the Terracotta Server Array say due to a network issue, then in Ehcache 2.0 each cache operation will block indefinitely. In other words it is configured for fail-fast to protect the ACIDity of the cluster.

However this approach will also cause processing of requests to the cache to stop likely causing the outage to cascade. In some cases graceful degradation may be more appropriate. When the clusterOffline events fires you could call Cache.setDisabled() which will cause puts and gets to bypass the cache. Your application would then degrade to operating without a cache, but might be able to do useful work.

You could also take the whole application off-line.

When connectivity is restored you could then reverse the action, taking the cache back online or the application back on line as the case may be.

We expect to add some of these common behaviours into configuration in the next release:

• blocking - block until connectivity is restored
• disable cache - cache cleared, puts ignored, gets return null
• local mode - continue but use local data only. This last one presents problems on reconnect because local changes will not be consistent with the cluster.

Maven

Terracotta has a Maven plugin available which makes this very simple.

    <pluginRepositories>
        <pluginRepository>
            <id>terracotta-snapshots</id>
            <url>http://www.terracotta.org/download/reflector/maven2</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </pluginRepository>
    </pluginRepositories>

    <plugin>
        <groupId>org.terracotta.maven.plugins</groupId>
        <artifactId>tc-maven-plugin</artifactId>
        <version>1.5.1</version>
        <executions>
            <execution>
                <id>run-integration</id>
                <phase>pre-integration-test</phase>
                <goals>
                    <goal>run-integration</goal>
                </goals>
            </execution>
            <execution>
                <id>terminate-integration</id>
                <phase>post-integration-test</phase>
                <goals>
                    <goal>terminate-integration</goal>
                </goals>
            </execution>
        </executions>
    </plugin>

    mvn tc:start

    mvn tc:stop

I get a net.sf.ehcache.CacheException: Terracotta cache classes are not available, you are missing jar(s) most likely

You need to include the ehcache-terracotta jar in your classpath.

When I start ehcache I get "WARN - Can't connect to server[localhost:9510:s1]. Retrying..."

You have not configured a Terracotta server for Ehcache to connect to.

Is Expiry the same in Terracotta?

timeToIdle and timeToLive work as usual. Ehcache 1.7 introduced a less fine grained age recording in Element which rounds up to the nearest second. Some APIs may be sensitive to this change.

In Ehcache Elements can have overridden TTI and TTLs. Terracotta supports this functionality.

What Eviction strategies are supported?

Ehcache supports LRU, LFU and FIFO eviction strategies.

Terracotta supports LRU and LFU eviction from the local node. Not FIFO and not custom evictors.

What Stores are available and how are they configured?

The Terracotta server provides an additional store, generally referred to as the Level 2 or L2 store.

The MemoryStore in JVM in the local node is referred to as the L1 Store.

maxElementsInMemory - the maximum number of elements in the local L1 store.

maxElementsOnDisk - is overridden when using Terracotta to provide the L2 size. The L2 size is effectively the maximum cache size.

overflowToDisk normally controls whether to overflow to the DiskStore. This is ignored when using Terracotta - the DiskStore is never used. When the store gets full, elements will always overflow to the Terracotta L2 Store running on the server. The L2 can be further configured with the tcconfig.

When do Elements overflow?

Two things to cause elements to be flushed from L1 to L2.

• the L1 store exceeding maxElementsInMemory
• When the local JVM exceeds 70% of Old Generation. This can be turned off in the TC Config. By default it is on (in 1.7).

How does Element equality work in Serialization mode?

An Element, key and value in Ehcache is guaranteed to .equals() another as it moves between stores.

In the Express install or Serialization mode of Terracotta, which is the default, Terracotta is the same. Elements will not == each other as they move between stores.

How does Element equality work in Identity mode?

An Element in Ehcache is guaranteed to .equals() another as it moves between stores.

However in Identity mode, Terracotta makes a further guarantee that they key and the value ==. This is achieved using extensions to the Java Memory Model.

What is the recommended way to write to a database?

Terracotta' non Ehcache API offers an async writethrough to the database which is guaranteed. It uses the TIM Async module and works by putting the database update in a clustered queue. It guarantees that a node, even if the local node fails, will take it out and process it.

That option is not available with Ehcache although it may get added.

If updates to a database bypass the Terracotta clustered application, then how is that coherent?

It isn't. This is a problem with using a database as an integration point. Integration via a message queue, with a Terracotta clustered application acting as a message queue listener and updating the database avoids this. As would The application receiving a REST or SOAP call and writing to the database.

AQ can have DB trigger put in a poll. Or AQ can push it up.

Do CacheEventListeners work?

A local CacheEventListener will work locally, but other nodes in a Terracotta cluster are not notified unless the TerracottaCacheEventReplicationFactory event listener is registered for the cache.

What are the JMX statistics available in the Terracotta Developer Console?

SampledCache and SampledCacheManager MBeans are made available in the Terracotta Developer Console.

These are time based gauges, based on once per second measurements. These are different to the JMX MBeans available through the ManagementService.

What classloader is used to deserialize clustered objects in valueMode="serialization"?

The following classloaders are tried in this order:

• Thread.currentThread().getContextClassLoader() (so set this to override)
• The classloader used to define the Ehcache CacheManager - the one that loaded the ehcache-terracotta.jar

What versions of Ehcache and Terracotta work together?

For full compatibility Ehcache 2.0 should be used with Terracotta Server 3.2.1.

Ehcache 1.7 works with TC 3.1.1.

Ehcache Core 1.7.3 core works with TC 3.2.

What happens when an L1 (i.e. an Ehcache node) disconnects from the L2 (i.e. the Terracotta Server Array)?

• The L1 receives an operations disabled event. Then spawns a thread with a timer waiting for an operations-enabled event. How long to wait depends on heart beating settings. 65 seconds should work for the default assuming l2.l1.reconnect is enabled with 5s timeout.
• If in this time there is no Operations-Enabled event, then the L1 can conclude that it is disconnected and flip a Boolean in the application, which instructs incoming requests to not access the Terracotta Shared space.

  Note that existing threads doing I/O against the TC server (whether for data or for locks) are stuck.

• If the application desires timeouts then you have to employ a proxy collection - e.g. a wrapper around a Queue or Map, (in DSO) - where the Queue APIs are proxied through to a thread pool - so that you can try/join out in "timeout" seconds and throw a timeout to the application. This is however somewhat messy - since the application threads may receive a timeout, but the "terracotta transaction" may still make it through to the L2.