Jenkins job with many downstream jobs struggles to load

Question

I have a Jenkins job which often fails to load when I am trying to browse to it in the Jenkin's control panel. I get a 502 error. Other times it loads slowly, and other times it loads quickly with no error. Sometimes other parts of the control panel struggle to load on occasion, but it feels like it's mostly this page.

The job's function is to queue a dynamic set of roughly 300 downstream jobs, using groovy, and then aggregate their test results. The list of sub-projects in the /job/Job_Name page is unresolved because it is a variable, so I think that (probably) isn't causing the slowness.

The jenkins process is typically consuming a max of 24% of machine memory ( ~2 out of 8 GB ) but up to 350% of CPU ( 4 core system ) even when Jenkins has no builds running. The instance has ~25 slave nodes and ~350 jobs. I have already configured the views to include a minimum of information about the jobs, and I am only keep 1 day of build history.

Is it the test result aggregation that causes the slow load speeds, or is there something else that may be the root of the issue?

Thank you, Josh

Answer 1

The issue turned out to be copying and fingerprinting artifacts. Once I removed the creation, fingerprinting, and copying of the artifacts, the performance issue completely disappeared. The artifacts were a necessary step in order to aggregate test results, but as I had 300+ downstream jobs copying the artifacts, it turned out to be too much of a performance drag.

The solution I ended up implementing was to create a MySQL database on the Jenkin's master, and then use results from each job to insert, aggregate, and search results. As Jenkins is slow at formatting TAP, I ended up creating my own little website to display the results.