How Startup and Shutdown Work with Disk Stores

When you shut down a member that is persisting data, the data remains in the disk store files. When the member starts up again, the data is reloaded and reinitializes the member's persistent regions.

Shutdown: Most Recent Data from the Last Run

If more than one member has the same persistent region or queue, the last member to exit leaves the most up-to-date data on disk.

GemFire stores information on member exit order in the disk stores, so it can start your members with the most recent data set:
  • For a persistent replicate, the last member to exit leaves the most recent data on disk.
  • For a partitioned region, where the data is split into buckets:
    • If you use gfsh shutdown command, all online data stores are synchronized before shutting down so all hold the most recent data copy.
    • Otherwise, different members might host different most recent buckets.

Startup Process

When you start a member with disk stores, the stores are loaded back into the cache to initialize the member’s persistent regions.
  • If any region does not hold all most recent data in the system:
    1. Region creation is blocked, waiting for the members with the most recent data.
      If your log level is info or below, the system provides messaging about the wait. Here, the disk store for hostA has the most recent data for the region and the hostB member is waiting for it.
      Region /people has potentially stale data. 
      It is waiting for another member to recover the latest data.
      My persistent id:
        DiskStore ID: 6893751ee74d4fbd-b4780d844e6d5ce7
        Name: server1
        Location: /
      Members with potentially new data:
        DiskStore ID: 160d415538c44ab0-9f7d97bae0a2f8de
        Name: server2
        Location: /
      Use the "gemfire list-missing-disk-stores" command to see all disk stores 
      that are being waited on by other members.

      During normal startup, especially with partitioned regions that were not shut down using the gemfire shut-down-all command, you can expect to see some waiting messages.

    2. When the most recent data is available, the system updates the local region as needed, logs a message like this, and continues with startup.
      [info 2010/04/09 10:52:13.010 PDT CacheRunner <main> tid=0x1]    
         Done waiting for the remote data to be available.
  • If the disk store has data for a region that is never created, the data remains in memory.
Each member’s persistent regions load and go online as quickly as possible, not waiting unnecessarily for other members to complete. For performance reasons, several actions occur asynchronously:
  • If both primary and secondary buckets are persisted, data will be available when the primary buckets are loaded without waiting for the secondary buckets to load; the secondary buckets will load asynchronously.
  • Entry keys first get loaded from the key file if this file is available (see information about the krf file in Disk Store File Names and Extensions). Once all keys are loaded, GemFire loads the entry values asynchronously. If a value is requested before it is loaded, the value will immediately be fetched from the disk store.

Example Startup Scenarios

The following lists scenarios for starting up a system with disk stores after a shutdown:
  • Stop order for a replicate persistent region
    1. Member A (MA) exits first, leaving persisted data on disk for RegionP.
    2. MB continues to run operations on RegionP, which update its disk store and leave the disk store for MA in a stale condition.
    3. MB exits, leaving the most up-to-date data on disk for RegionP.
  • Restart order Scenario 1
    1. MB is started first. GemFire recognizes MB as having the most recent disk data for RegionP and initializes it from disk.
    2. MA is started, recovers its data from disk, and updates it as needed from the data in MB.
  • Restart order Scenario 2
    1. MA is started first. GemFire recognizes that MA does not have the most recent disk data and waits for MB to start before creating RegionP in MA.
    2. MB is started. GemFire recognizes MB as having the most recent disk data for RegionP and initializes it from disk.
    3. MA recovers its RegionP data from disk and updates it as needed from the data in MB.