There are several types of GemFire files that are critical for troubleshooting.
GemFire logs and statistics are the two most important artifacts used in troubleshooting.
In addition, they are required for GemFire system health verification and performance
analysis. For these reasons, logging and statistics should always be enabled, especially
in production. Save the following files for troubleshooting purposes:
- Log files. Even at the default logging
level, the log contains data that may be important. Save the whole log, not just the
stack. For comparison, save log files from before, during, and after the problem
- Statistics archive files.
- Core files or stack traces.
- For Linux, you can use gdb to extract a
stack from a core file.
- Crash dumps.
- For Windows, save the user mode dump files.
Some locations to check for these files:
When a problem arises that involves more than one process, a network problem is the most
likely cause. When you diagnose a problem, create a log file for each member of all the
distributed systems involved. If you are running a client/server architecture, create
log files for the clients.
Note: You must run a time synchronization service on all hosts for troubleshooting.
Synchronized time stamps ensure that log messages on different hosts can be merged
to accurately reproduce a chronological history of a distributed run.
For each process, complete these steps:
- Make sure the host’s clock is
synchronized with the other hosts. Use a time synchronization tool such as
Network Time Protocol (NTP).
- Enable logging to a file instead of
standard output by editing gemfire.properties to include this
- Keep the log level at
config to avoid filling up the disk while including
configuration information. Add this line to gemfire.properties:
Note: Running with the log level at
fine can impact system performance and fill up your
- Enable statistics gathering for the distributed system either by modifying
or by using the gfsh alter rutime
alter runtime --group=myMemberGroup --enable-statistics=true --statistic-archive-file=StatisticsArchiveFile.gfs
statistics at the default sample rate frequency of 1000 milliseconds does
not incur performance overhead.
- Run the application again.
- Examine the log files. To get the
clearest picture, merge the files. To find all the errors in the log file,
search for lines that begin with these strings:
For details on merging log files,
see the --merge-log argument for the export logscommand.
- Export and analyze the stack traces on the member or member group where the
application is running. Use the gfsh export stack-traces
gfsh> export stack-traces --file=ApplicationStackTrace.txt --member=member1