Maven site "too many open files"

It strikes me that things "building up" on the filesystem would require the filesystem to be persistent. Obviously most of the instances Jenkins fires up are transient, and are terminated after some idle period. Stuff can't build up in that case.

I see, though, that the i-05e533608386d1b78 instance:

• Is stopped, not terminated

• Is shown by Jenkins as a CentOS7-nightly node that is offline

• Has been around for something like 8 hours

• Has a very similar instance ID to the one I poked several days ago (I didn't write down the whole ID, though, so this is probably coincidence rather than a really long-running instance)

I am wondering if the rules for this node type are different, and allow instances to be kept around and reused. That might allow grunge to accumulate.

After moving the *-site jobs to AWS (CentOS7-nightly EC2 instance) from master (shibboleth.net) as part of INFRA-246, we had several successful builds using the default ulimit of 1024.

My guess is that something was "building up" on the filesystem causing too many file handles to be used.

The only relevant files I could think of are the net.shibboleth and org.opensaml snapshots, so I added a pre-build step to the java-identity-provider-v4-site job to delete the net/shibboleth and org/opensaml directories from the local Maven repository :

-Pcentral-disabled dependency:purge-local-repository -DreResolve=false -Dverbose=true -DmanualInclude=org.opensaml,net.shibboleth

Bumped ulimit for jenkins user to 30720.

I managed to get this to work on my development machine (macOS Catalina) after using ulimit -n 30000. The default on that system is only 256.

I'm not sure that's repeatable, though, as other things may have stopped using file descriptors on my system in the meanwhile, so it's not necessarily the ulimit command that made the difference.

A useful command to find out what's holding open file descriptors on macOS:

Added this to the list for V4.0.0 on the basis that we have to have some idea of what we're going to do, even if it's that we have to do a manual site build on release. Having said which, I'm having the same result on a manual build on my Mac under default configuration, so that might not be a simple matter either...