Jgroups join error: "...sent to <GROUP> timed out (after 3000ms), on try..."

During Wildfly startup, you may see a significant portion of the startup time allotted to the JVM attempting to join a jgroups cluster. Note that this is a separate "cluster" than your application server cluster, as this is setup as a cluster-of-one in single server deployments.

Wildfly will attempt to join each jgroups cluster ten times, with a three second delay between each attempt. Depending on the amount of cached clusters, it will amplify the delays since each jgroups cluster will cycle through its own set of ten attempts.

The issue can be diagnosed by the following Wildfly server log message repeating:

2021-03-22 16:11:51,527 WARN [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 86) GROUP-0: JOIN(GROUP-0) sent to GROUP-0 timed out (after 3000 ms), on try 6
2021-03-22 16:11:54,530 WARN [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 86) GROUP-0: JOIN(GROUP-0) sent to GROUP-0 timed out (after 3000 ms), on try 7
2021-03-22 16:11:57,531 WARN [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 86) GROUP-0: JOIN(GROUP-0) sent to GROUP-0 timed out (after 3000 ms), on try 7
2021-03-22 16:12:00,531 WARN [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 86) GROUP-0: JOIN(GROUP-0) sent to GROUP-0 timed out (after 3000 ms), on try 7
2021-03-22 16:12:03,532 WARN [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 86) GROUP-0: JOIN(GROUP-0) sent to GROUP-0 timed out (after 3000 ms), on try 7
2021-03-22 16:12:06,532 WARN [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 86) GROUP-0: JOIN(GROUP-0) sent to GROUP-0 timed out (after 3000 ms), on try 7
2021-03-22 16:12:09,533 WARN [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 86) GROUP-0: JOIN(GROUP-0) sent to GROUP-0 timed out (after 3000 ms), on try 7

Due to the startup delays introduced by the jgroup timeouts, it is common that the WFLYCTL0348 error will occur due to slow startup.

The jgroups issue can be resolved by clearing up the cluster cache:

1. Shutting down the JVM
2. Clear out the contents of the following directory:

   <install home>/igx-data/<config>/<product>/data/cluster

3. Restart JVM