Server load average
| Server | 1 min | 5 min | 15 min | Top CPU Hogs |
|---|
| bugs-vm1 | 0.04 | 0.06 | 0.07 | |
| bugs-vm2 | 0.20 | 0.13 | 0.05 | |
| build | 1.19 | 0.88 | 0.81 | 2.1 mysql /usr/sbin/mysqld
1.0 root /sbin/bacula-fd
0.4 root [kworker/10:2] |
| centslave1.eclipse.org | 0.01 | 0.04 | 0.05 | 0.5 genie.cbi /shared/common/jdk1.8.0_x64-latest/bin/java
0.5 genie.releng /shared/common/jdk1.8.0_x64-latest/bin/java
0.5 genie.reddeer /shared/common/jdk1.8.0_x64-latest/bin/java |
| db2master | 0.12 | 0.21 | 0.22 | 28.0 mysql /usr/sbin/mysqld
0.1 root /usr/sbin/haveged
0.0 root /usr/lib/systemd/systemd |
| db2slave | 0.13 | 0.13 | 0.14 | 9.3 mysql /usr/sbin/mysqld
0.0 root /usr/lib/systemd/systemd
0.0 root [kthreadd] |
| dbapi | 0.29 | 0.18 | 0.19 | 5.5 mysql /usr/sbin/mysqld
0.1 root [kworker/15:0]
0.0 root /usr/lib/systemd/systemd |
| dbapislave | 0.07 | 0.05 | 0.05 | 0.4 mysql /usr/sbin/mysqld
0.0 root /usr/lib/systemd/systemd
0.0 root [kthreadd] |
| dbmaster | 0.01 | 0.02 | 0.05 | 95.1 mysql /usr/sbin/mysqld
0.0 root /usr/lib/systemd/systemd
0.0 root [kthreadd] |
| dbslave2 | | | | DEAD, offline or not reporting |
| dev1 | 0.27 | 0.34 | 0.34 | 0.0 root [md0_raid1]
0.0 root /usr/sbin/xinetd
0.0 root [kjournald] |
| dev2 | 0.65 | 0.31 | 0.26 | 1.7 cvs /usr/local/libexec/git-core/git
0.5 cvs /usr/local/libexec/git-core/git
0.0 root /usr/sbin/xinetd |
| dev3 | 0.45 | 0.55 | 0.49 | 41.0 cvs /usr/local/libexec/git-core/git
2.6 cvs /usr/local/libexec/git-core/git
0.1 mailman /usr/bin/python |
| download1 | 0.73 | 0.72 | 1.01 | 28.1 nobody /usr/sbin/rsyncd
7.0 nobody /usr/sbin/rsyncd
1.5 nobody /usr/sbin/rsyncd |
| download2 | 0.16 | 0.79 | 0.82 | 8.6 nobody /usr/sbin/rsyncd
1.3 wwwrun /usr/sbin/httpd-worker
1.3 wwwrun /usr/sbin/httpd-worker |
| econ-staging | 0.00 | 0.03 | 0.06 | 0.0 root /usr/lib/systemd/systemd
0.0 root [kthreadd]
0.0 root [ksoftirqd/0] |
| econ-vm1 | 0.43 | 0.23 | 0.13 | 0.3 wwwrun /usr/sbin/httpd-prefork
0.3 wwwrun /usr/sbin/httpd-prefork
0.3 wwwrun /usr/sbin/httpd-prefork |
| econ-vm2 | 0.39 | 0.17 | 0.11 | 0.3 wwwrun /usr/sbin/httpd-prefork
0.3 wwwrun /usr/sbin/httpd-prefork
0.3 wwwrun /usr/sbin/httpd-prefork |
| econ-vm3 | | | | DEAD, offline or not reporting |
| foundation | 0.76 | 0.49 | 0.44 | 5.2 root puppet
1.4 wwwrun php
1.2 mysql /usr/sbin/mysqld |
| fred | 2.80 | 3.28 | 2.96 | 5.2 dirsrv /usr/sbin/ns-slapd
5.0 root [nfsd]
3.8 root /usr/bin/rsync |
| gerrit-vm1 | 0.46 | 0.52 | 0.43 | 38.9 gerrit GerritCodeReview
0.8 gerrit GerritCodeReview
0.7 root /usr/lib/sssd/sssd_be |
| hipp10 | 1.38 | 1.13 | 0.82 | 2.9 genie.omr /shared/common/jdk1.8.0_x64-latest/bin/java
1.5 genie.objectteams /shared/common/jdk1.8.0_x64-latest/bin/java
1.1 genie.webtools /shared/common/jdk1.8.0_x64-latest/bin/java |
| hipp6 | | | | DEAD, offline or not reporting |
| hipp7 | 0.03 | 0.06 | 0.05 | 0.6 genie.kitalpha /shared/common/jdk1.8.0_x64-latest/bin/java
0.6 root /usr/lib/sssd/sssd_be
0.4 root [btrfs-transacti] |
| hipp8 | 2.33 | 3.11 | 3.63 | 75.2 genie.hono /tmp/genie.hono/extract-69785af9-3364-4b64-a686-566baad5102aextractmongod
74.9 genie.hono /tmp/genie.hono/extract-00e12228-6a13-4177-b020-5448cfa5b763extractmongod
49.7 genie.packaging /shared/common/java/openjdk/jdk-11_x64-latest/bin/java |
| hipp9 | 0.01 | 0.05 | 0.11 | 7.9 genie.shared /shared/common/jdk1.8.0_x64-latest/bin/java
0.7 genie.pdt /shared/common/jdk1.8.0_x64-latest/bin/java
0.6 genie.papyrus /shared/common/jdk1.8.0_x64-latest/bin/java |
| hippcentos.eclipse.org | 0.00 | 0.01 | 0.05 | 0.5 genie.releng /shared/common/jdk1.8.0_x64-latest/bin/java
0.0 root /usr/lib/systemd/systemd
0.0 root [kthreadd] |
| php-vm1 | 2.24 | 2.17 | 2.17 | |
| php-vm2 | 1.68 | 1.80 | 1.90 | |
| php-vm3 | 1.96 | 1.89 | 1.98 | |
| php-vm5 | | | | DEAD, offline or not reporting |
| varn-vm1 | 0.10 | 0.11 | 0.09 | 8.6 solr /usr/lib64/jvm/java-1_5_0-ibm-1.5.0/jre/bin/java
0.0 root [pdflush]
0.0 root /usr/sbin/nscd |
| www-staging | 0.74 | 0.75 | 0.75 | |
| www-vm1 | 1.53 | 1.44 | 1.45 | |
| www-vm2 | 1.49 | 1.31 | 1.29 | |
| www-vm3 | 1.42 | 1.20 | 1.16 | |
Server load indicates how busy our servers are at any point in time.
"main" and "main2" are backend file/database/authentication servers,
whereas node1 to node4 and utils are load-balanced web/cvs servers.
For an overall picture of system load, look at the CPU graph, above.
Server load will rarely be the cause of a performance slowdown,
but can affect certain system maintenance jobs that are
scheduled to run throughout the day.
Load average is displayed for the last minute,
the last 5 minutes and the last 15 minutes.
Which of those servers are you using? I don't know.
Your web pages, bugzilla pages and CVS connections are being
processed by any one of the "node" servers.
- 0.00 to 1.99 : no delay in processing requests. This is what you should see most of the time.
- 2.00 to 4.99: server is busy, but nothing major.
- 5.00 to 9.99: the house is rockin' - expect short processing delays.
- 10.00 and up: general server overload. It could be just a spike due to replication or disk activity, because this isn't normal.
Bandwidth usage and connection count
| Server | Mbps out | Mbps in | HTTP Conn | RSYNCd Conn | git Procs |
|---|
| bugs-vm1 | 0.07 | 0.01 | 8
| 0
| 0
|
| bugs-vm2 | 0.19 | 0.01 | 10
| 0
| 0
|
| build | 2.26 | 20.83 | 37
| 0
| 0
|
| centslave1.eclipse.org | 0.00 | 0.00 | 0
| 0
| 0
|
| db2master | 67.62 | 5.34 | 0
| 0
| 0
|
| db2slave | 2.11 | 0.63 | 0
| 0
| 0
|
| dbapi | 0.00 | 0.00 | 0
| 0
| 0
|
| dbapislave | 0.14 | 0.11 | 0
| 0
| 0
|
| dbmaster | 0.01 | 0.00 | 0
| 0
| 0
|
| dbslave2 | 0 | 0 | 0 | 0 | 0 |
| dev1 | 0.17 | 0.01 | 31
| 0
| 0
|
| dev2 | 6.55 | 0.12 | 31
| 0
| 3
|
| dev3 | 59.97 | 0.86 | 31
| 0
| 3
|
| download1 | 132.57 | 0.64 | 224 | 6
| 0
|
| download2 | 113.35 | 0.56 | 224 | 2
| 0
|
| econ-staging | 0.00 | 0.00 | 11
| 0
| 0
|
| econ-vm1 | 0.12 | 0.01 | 51
| 0
| 0
|
| econ-vm2 | 0.15 | 0.01 | 51
| 0
| 0
|
| econ-vm3 | 0 | 0 | 0 | 0 | 0 |
| foundation | 0.06 | 0.06 | 16
| 0
| 0
|
| gerrit-vm1 | 0.83 | 0.40 | 0
| 0
| 0
|
| hipp10 | 0.01 | 0.21 | 0
| 0
| 0
|
| hipp6 | 0 | 0 | 0 | 0 | 0 |
| hipp7 | 0.00 | 0.00 | 0
| 0
| 0
|
| hipp8 | 0.00 | 0.00 | 0
| 0
| 0
|
| hipp9 | 0.00 | 0.00 | 0
| 0
| 0
|
| hippcentos.eclipse.org | 0.00 | 0.00 | 0
| 0
| 0
|
| php-vm1 | 0.42 | 0.03 | 101
| 0
| 0
|
| php-vm2 | 0.39 | 0.04 | 101
| 0
| 0
|
| php-vm3 | 0.37 | 0.03 | 101
| 0
| 0
|
| php-vm5 | 0 | 0 | 0 | 0 | 0 |
| proxy-vm1 | 0.00 | 0.00 | 64
| | |
| proxy-vm2 | 0.01 | 0.01 | 75
| | |
| varn-vm1 | 0.00 | 0.00 | 1
| 0
| 0
|
| www-staging | 0.00 | 0.00 | 11
| 0
| 0
|
| www-vm1 | 0.91 | 0.08 | 101
| 0
| 1
|
| www-vm2 | 0.80 | 0.08 | 101
| 0
| 2
|
| www-vm3 | 1.06 | 0.09 | 101
| 0
| 1
|
| Total | 390.14 | 30.17 | 1482 | 8 | 10 |
Bandwidth usage indicates the overall usage of our Internet link.
When usage is at, or near 100%, the following services get priority access to a portion of our bandwidth, and are
generally not affected by a saturated internet connection (in no particular order): CVS, Bugzilla,
RSYNC, SSH (including SFTP and SCP), www.eclipse.org and newsgroups.
When our connection is saturated, downloading from download.eclipse.org will
be slower than CVS or our website.
MySQL server metrics
| | dbmaster | dbslave2 | dbapi | dbapislave | db2master | db2slave |
|---|
| Threads | 2 | NR | 11 | 1 | 31 | |
| Queries_Per_Second | 0.97 | NR | 115.77 | 91.83 | 1981.13 | |
| Query_Cache_Hit_Rate(%) | 31.03 | NR | 52.72 | 4.25 | 74.33 | |
| Slave_running | OFF | NR | OFF | ON | OFF | |
| Slave_IO_Running | No | NR | | Yes | | |
| Slave_SQL_Running | No | NR | | Yes | | |
| Seconds_Behind_Master |
| NR |
| 0
|
|
|
The database servers, a master (main) and a slave (main2) handle all database traffic: Bugzilla, eclipse.org search engine,
eclipse.org website, the Wiki, and others. All write operations (committing a bug, indexing the site, tracking a download) go to the master
and are replicated immediately to the slave. Some applications, such as Bugzilla, Download Stats, the Website and Search, use the slave server for
searches and lookups for increased performance.
A high load average on main or main2 will cause database operations to be slower. The Queries_Per_Second and Threads metrics are good indicators of
how busy the database servers are.
Legend
- Orange indicates dev/download general cluster node.
- Green indicates Bugzilla cluster node.
- Blue indicates PHP/Wiki cluster node.
- Gray indicates www.eclipse.org cluster node.
- Light red indicates Storage/database backend node.
Currently running on accounts.eclipse.org at 172.30.4.71 on Thursday, May 28, 2020 - 16:45.
- CPU: 0.9%
- Bandwidth: 111.5%
- HTTPConn: 14.8%
