Trace number 3321831

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (MO) 11741 1484.01

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v35000-c147000-S648642831-094-UNKNOWN.cnf
MD5SUM4a57833314362e0939d4095b3206ad82
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark144.491
Satisfiable
(Un)Satisfiability was proved
Number of variables35000
Number of clauses147000
Sum of the clauses size441000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3147000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.39/4.45	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.39/4.45	c INFO: solving HOME/instance-3321831-1307741589.cnf
0.39/4.45	c INFO: computing task features
0.84/4.96	c INFO: computed variable-clause graph statistics
0.84/4.98	c INFO: computed clause balance statistics
3.44/7.54	c INFO: computed variable balance statistics
5.54/9.65	c INFO: computed small-clause counts
5.54/9.66	c INFO: computed Horn-clause counts
7.14/11.22	c INFO: computed variable graph statistics
7.14/11.22	c INFO: collected features for HOME/instance-3321831-1307741589.cnf in 6.76 s
7.14/11.26	c INFO: running hybridGM3@0 for 97 with 39993 remaining (b = 0.36)
7.25/11.30	c INFO: running hybridGM3@0 for 97 with 39993 remaining (b = 0.22)
7.25/11.34	c INFO: running hybridGM3@0 for 97 with 39993 remaining (b = 0.14)
7.25/11.38	c INFO: running hybridGM3@0 for 194 with 39993 remaining (b = 0.18)
7.34/11.42	c INFO: running hybridGM3@0 for 194 with 39993 remaining (b = 0.14)
7.34/11.46	c INFO: running hybridGM3@0 for 194 with 39993 remaining (b = 0.11)
7.34/11.49	c INFO: running TNM@0 for 97 with 39992 remaining (b = 0.04)
7.45/11.53	c INFO: running hybridGM3@0 for 194 with 39992 remaining (b = 0.07)
807.36/111.52	c INFO: running hybridGM3@0 for 97 with 39892 remaining (b = 0.03)
807.36/111.56	c INFO: running hybridGM3@0 for 97 with 39892 remaining (b = 0.03)
808.10/111.60	c INFO: running hybridGM3@0 for 97 with 39892 remaining (b = 0.03)
808.91/111.70	c INFO: running hybridGM3@0 for 194 with 39892 remaining (b = 0.06)
1566.63/206.68	c INFO: running hybridGM3@100 for 97 with 39797 remaining (b = 0.03)
1567.45/206.73	c INFO: running hybridGM3@100 for 97 with 39797 remaining (b = 0.03)
1567.45/206.77	c INFO: running hybridGM3@99 for 97 with 39797 remaining (b = 0.03)
1569.77/207.03	c INFO: running hybridGM3@100 for 97 with 39797 remaining (b = 0.03)
1575.35/207.76	c INFO: running hybridGM3@0 for 97 with 39796 remaining (b = 0.03)
1575.35/207.80	c INFO: running hybridGM3@0 for 97 with 39796 remaining (b = 0.03)
1576.14/207.84	c INFO: running hybridGM3@0 for 97 with 39796 remaining (b = 0.03)
2374.55/307.84	c INFO: running hybridGM3@0 for 194 with 39696 remaining (b = 0.04)
2374.55/307.90	c INFO: running TNM@0 for 97 with 39696 remaining (b = 0.02)
2375.36/307.96	c INFO: running hybridGM3@294 for 97 with 39696 remaining (b = 0.02)
2376.69/308.20	c INFO: running hybridGM3@296 for 97 with 39696 remaining (b = 0.02)
2382.28/308.93	c INFO: running TNM@0 for 1940 with 39695 remaining (b = 0.24)
2382.28/308.98	c INFO: running hybridGM3@0 for 194 with 39695 remaining (b = 0.03)
2384.03/309.03	c INFO: running hybridGM3@101 for 97 with 39695 remaining (b = 0.02)
2416.72/313.17	c INFO: running hybridGM3@95 for 97 with 39691 remaining (b = 0.02)
3135.29/403.13	c INFO: running TNM@0 for 97 with 39601 remaining (b = 0.01)
3136.75/403.37	c INFO: running hybridGM3@0 for 194 with 39601 remaining (b = 0.03)
3144.06/404.20	c INFO: running hybridGM3@300 for 97 with 39600 remaining (b = 0.01)
3168.82/407.34	c INFO: running hybridGM3@190 for 97 with 39597 remaining (b = 0.01)
3174.45/408.09	c INFO: running TNM@0 for 97 with 39596 remaining (b = 0.01)
3903.31/499.38	c INFO: running hybridGM3@395 for 97 with 39505 remaining (b = 0.01)
3935.19/503.32	c INFO: running hybridGM3@395 for 97 with 39501 remaining (b = 0.01)
3943.17/504.32	c INFO: running hybridGM3@100 for 97 with 39500 remaining (b = 0.01)
3951.94/505.43	c INFO: running hybridGM3@100 for 97 with 39499 remaining (b = 0.01)
3968.64/507.53	c INFO: running hybridGM3@394 for 97 with 39496 remaining (b = 0.01)
3974.97/508.32	c INFO: running TNM@0 for 97 with 39496 remaining (b = 0.01)
4678.33/596.57	c INFO: running hybridGM3@0 for 194 with 39407 remaining (b = 0.02)
4710.26/600.51	c INFO: running TNM@0 for 97 with 39403 remaining (b = 0.01)
4743.65/604.73	c INFO: running adaptg2wsat2009++@0 for 194 with 39399 remaining (b = 0.02)
4750.02/605.51	c INFO: running hybridGM3@296 for 97 with 39399 remaining (b = 0.01)
4752.40/605.83	c INFO: running hybridGM3@93 for 97 with 39398 remaining (b = 0.01)
4758.77/606.63	c INFO: running TNM@0 for 97 with 39397 remaining (b = 0.01)
4766.71/607.61	c INFO: running TNM@0 for 97 with 39396 remaining (b = 0.01)
5509.11/700.72	c INFO: running TNM@0 for 97 with 39303 remaining (b = 0.01)
5509.77/700.81	c INFO: running hybridGM3@0 for 194 with 39303 remaining (b = 0.02)
5510.98/701.05	c INFO: running TNM@0 for 97 with 39303 remaining (b = 0.01)
5557.55/706.87	c INFO: running TNM@0 for 97 with 39297 remaining (b = 0.01)
5565.46/707.85	c INFO: running TNM@101 for 1940 with 39296 remaining (b = 0.12)
6291.84/799.09	c INFO: running hybridGM3@97 for 97 with 39205 remaining (b = 0.01)
6306.94/800.96	c INFO: running hybridGM3@0 for 194 with 39203 remaining (b = 0.01)
6308.38/801.10	c INFO: running adaptg2wsat2009++@100 for 97 with 39203 remaining (b = 0.01)
6309.77/801.30	c INFO: running adaptg2wsat2009++@0 for 194 with 39203 remaining (b = 0.01)
6356.24/807.12	c INFO: running hybridGM3@0 for 194 with 39197 remaining (b = 0.01)
7050.74/894.34	c INFO: running adaptg2wsat2009++@0 for 194 with 39110 remaining (b = 0.01)
7066.52/896.36	c INFO: running FH@0 for 97 with 39108 remaining (b = 0.01)
7067.25/896.48	c INFO: running hybridGM3@95 for 97 with 39108 remaining (b = 0.01)
7857.96/995.76	c INFO: running hybridGM3@193 for 97 with 39008 remaining (b = 0.01)
7872.25/997.54	c INFO: running hybridGM3@195 for 291 with 39007 remaining (b = 0.02)
7905.68/1001.72	c INFO: running hybridGM3@95 for 97 with 39002 remaining (b = 0.01)
7906.41/1001.83	c INFO: running hybridGM3@194 for 291 with 39002 remaining (b = 0.02)
7912.79/1002.66	c INFO: running hybridGM3@196 for 291 with 39001 remaining (b = 0.01)
8614.81/1090.88	c INFO: running adaptg2wsat2009++@100 for 97 with 38913 remaining (b = 0.00)
8616.37/1091.02	c INFO: running hybridGM3@390 for 97 with 38913 remaining (b = 0.01)
8750.83/1107.99	c INFO: running hybridGM3@294 for 194 with 38896 remaining (b = 0.01)
9388.95/1188.14	c INFO: running adaptg2wsat2009++@95 for 97 with 38816 remaining (b = 0.00)
9421.60/1192.29	c INFO: running adaptg2wsat2009++@0 for 194 with 38812 remaining (b = 0.01)
10162.96/1285.41	c INFO: running TNM@0 for 97 with 38719 remaining (b = 0.00)
10175.72/1287.10	c INFO: running hybridGM3@389 for 97 with 38717 remaining (b = 0.01)
10209.06/1291.22	c INFO: running hybridGM3@392 for 97 with 38713 remaining (b = 0.01)
10209.88/1291.39	c INFO: running hybridGM3@296 for 194 with 38713 remaining (b = 0.01)
10345.88/1308.43	c INFO: running hybridGM3@391 for 97 with 38696 remaining (b = 0.01)
10933.73/1382.36	c INFO: running TNM@0 for 97 with 38622 remaining (b = 0.00)
10966.31/1386.49	c INFO: running TNM@0 for 97 with 38618 remaining (b = 0.00)
10984.60/1388.72	c INFO: running adaptg2wsat2009++@97 for 97 with 38615 remaining (b = 0.00)
11008.44/1391.71	c INFO: running TNM@288 for 1843 with 38612 remaining (b = 0.08)
11143.67/1408.71	c INFO: running hybridGM3@196 for 291 with 38595 remaining (b = 0.01)
11732.28/1482.74	c INFO: running hybridGM3@290 for 291 with 38521 remaining (b = 0.01)
11733.04/1482.86	c INFO: running hybridGM3@0 for 485 with 38521 remaining (b = 0.02)
11741.04/1483.97	c INFO: running NCVWr@0 for 485 with 38520 remaining (b = 0.02)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 988)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3321831-1307741589/watcher-3321831-1307741589 -o /tmp/evaluation-result-3321831-1307741589/solver-3321831-1307741589 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3321831-1307741589.cnf -seed 951405007 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.00 0.00 0.00 2/170 24349
/proc/meminfo: memFree=31861964/32950928 swapFree=67055156/67111528
[pid=24349] ppid=24347 vsize=11804 CPUtime=0 cores=0-7
/proc/24349/stat : 24349 (runsolver) S 24347 24349 24319 0 -1 4202560 47 0 0 0 0 0 0 0 22 0 1 0 885759277 12087296 56 33554432000 4194304 5436572 140735078342160 18446744073709551615 5057063 0 2147483391 4096 24578 18446744071699199166 0 0 17 6 0 0 0
/proc/24349/statm: 2951 56 36 304 0 2643 0

[startup+0.037332 s]
/proc/loadavg: 0.00 0.00 0.00 2/170 24349
/proc/meminfo: memFree=31861964/32950928 swapFree=67055156/67111528
[pid=24349] ppid=24347 vsize=15080 CPUtime=0 cores=0-7
/proc/24349/stat : 24349 (python) S 24347 24349 24319 0 -1 4202496 651 0 0 0 0 0 0 0 22 0 1 0 885759277 15441920 545 33554432000 4194304 5683644 140735831790768 18446744073709551615 248089696256 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/24349/statm: 3770 545 312 364 0 257 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15080

[startup+0.100321 s]
/proc/loadavg: 0.00 0.00 0.00 2/170 24349
/proc/meminfo: memFree=31861964/32950928 swapFree=67055156/67111528
[pid=24349] ppid=24347 vsize=15780 CPUtime=0 cores=0-7
/proc/24349/stat : 24349 (python) S 24347 24349 24319 0 -1 4202496 897 0 0 0 0 0 0 0 22 0 1 0 885759277 16158720 777 33554432000 4194304 5683644 140735831790768 18446744073709551615 248089696256 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/24349/statm: 3945 777 343 364 0 432 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15780

[startup+0.300286 s]
/proc/loadavg: 0.00 0.00 0.00 2/170 24349
/proc/meminfo: memFree=31861964/32950928 swapFree=67055156/67111528
[pid=24349] ppid=24347 vsize=27352 CPUtime=0.03 cores=0-7
/proc/24349/stat : 24349 (python) S 24347 24349 24319 0 -1 4202496 1353 0 0 0 2 1 0 0 20 0 1 0 885759277 28008448 1197 33554432000 4194304 5683644 140735831790768 18446744073709551615 248089696256 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/24349/statm: 6838 1197 440 364 0 725 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 27352

[startup+0.700216 s]
/proc/loadavg: 0.00 0.00 0.00 2/170 24349
/proc/meminfo: memFree=31861964/32950928 swapFree=67055156/67111528
[pid=24349] ppid=24347 vsize=119076 CPUtime=0.11 cores=0-7
/proc/24349/stat : 24349 (python) S 24347 24349 24319 0 -1 4202496 3913 0 0 0 8 3 0 0 17 0 1 0 885759277 121933824 3542 33554432000 4194304 5683644 140735831790768 18446744073709551615 248089696256 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/24349/statm: 29769 3542 940 364 0 2653 0
Current children cumulated CPU time (s) 0.11
Current children cumulated vsize (KiB) 119076

[startup+1.50008 s]
/proc/loadavg: 0.00 0.00 0.00 1/171 24354
/proc/meminfo: memFree=31849060/32950928 swapFree=67055156/67111528
[pid=24349] ppid=24347 vsize=135752 CPUtime=0.21 cores=0-7
/proc/24349/stat : 24349 (python) S 24347 24349 24319 0 -1 4202496 5967 935 1 0 15 6 0 0 15 0 1 0 885759277 139010048 4151 33554432000 4194304 5683644 140735831790768 18446744073709551615 248089694213 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/24349/statm: 33938 4151 1032 364 0 3124 0
Current children cumulated CPU time (s) 0.21
Current children cumulated vsize (KiB) 135752

[startup+3.10087 s]
/proc/loadavg: 0.00 0.00 0.00 1/171 24354
/proc/meminfo: memFree=31848688/32950928 swapFree=67055156/67111528
[pid=24349] ppid=24347 vsize=202008 CPUtime=0.29 cores=0-7
/proc/24349/stat : 24349 (python) S 24347 24349 24319 0 -1 4202496 7265 935 1 0 21 8 0 0 15 0 1 0 885759277 206856192 5287 33554432000 4194304 5683644 140735831790768 18446744073709551615 248089696256 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/24349/statm: 50502 5287 1490 364 0 4011 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 202008
heavy processes:

[startup+6.30022 s]
/proc/loadavg: 0.00 0.00 0.00 2/171 24354
/proc/meminfo: memFree=31805828/32950928 swapFree=67055156/67111528
[pid=24349] ppid=24347 vsize=312976 CPUtime=2.24 cores=0-7
/proc/24349/stat : 24349 (python) R 24347 24349 24319 0 -1 4202496 21971 935 1 0 212 12 0 0 18 0 1 0 885759277 320487424 15628 33554432000 4194304 5683644 140735831790768 18446744073709551615 47853608069589 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/24349/statm: 78244 15628 1826 364 0 14168 0
Current children cumulated CPU time (s) 2.24
Current children cumulated vsize (KiB) 312976
heavy processes:

[startup+12.7011 s]
/proc/loadavg: 0.71 0.15 0.05 9/195 24378
/proc/meminfo: memFree=31650768/32950928 swapFree=67055156/67111528
[pid=24349] ppid=24347 vsize=457840 CPUtime=7.48 cores=0-7
/proc/24349/stat : 24349 (python) S 24347 24349 24319 0 -1 4202496 28191 935 1 0 733 15 0 0 23 0 9 0 885759277 468828160 15288 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101524939 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/24349/statm: 114460 15288 1889 364 0 50357 0
[pid=24355] ppid=24349 vsize=310268 CPUtime=0 cores=0-7
/proc/24355/stat : 24355 (python) S 24349 24349 24319 0 -1 4202560 734 0 0 0 0 0 0 0 17 0 1 0 885760404 317714432 13472 33554432000 4194304 5683644 140735831790768 18446744073709551615 248089718287 0 0 16781312 514 4048256682231300360 0 0 17 0 0 0 0
/proc/24355/statm: 77567 13472 353 364 0 13485 0
[pid=24357] ppid=24355 vsize=268420 CPUtime=1.42 cores=0-7
/proc/24357/stat : 24357 (hybridGM3) R 24355 24357 24357 0 -1 4202496 8085 0 0 0 142 0 0 0 25 0 1 0 885760404 274862080 7820 33554432000 134512640 135204385 4292080912 18446744073709551615 134545851 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/24357/statm: 67105 7820 64 169 0 66935 0
[pid=24358] ppid=24349 vsize=320524 CPUtime=0 cores=0-7
/proc/24358/stat : 24358 (python) S 24349 24349 24319 0 -1 4202560 742 0 0 0 0 0 0 0 17 0 1 0 885760407 328216576 13470 33554432000 4194304 5683644 140735831790768 18446744073709551615 248089718342 0 0 16781312 514 13439383407159116040 0 0 17 1 0 0 0
/proc/24358/statm: 80131 13470 347 364 0 16046 0
[pid=24360] ppid=24358 vsize=268428 CPUtime=1.38 cores=0-7
/proc/24360/stat : 24360 (hybridGM3) R 24358 24360 24360 0 -1 4202496 8079 0 0 0 137 1 0 0 25 0 1 0 885760408 274870272 7817 33554432000 134512640 135204385 4292110928 18446744073709551615 134547418 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/24360/statm: 67107 7817 64 169 0 66937 0
[pid=24361] ppid=24349 vsize=330780 CPUtime=0 cores=0-7
/proc/24361/stat : 24361 (python) S 24349 24349 24319 0 -1 4202560 748 0 0 0 0 0 0 0 17 0 1 0 885760411 338718720 13474 33554432000 4194304 5683644 140735831790768 18446744073709551615 248089718342 0 0 16781312 514 2732303991504339208 0 0 17 0 0 0 0
/proc/24361/statm: 82695 13474 347 364 0 18607 0
[pid=24363] ppid=24361 vsize=268424 CPUtime=1.34 cores=0-7
/proc/24363/stat : 24363 (hybridGM3) R 24361 24363 24363 0 -1 4202496 8082 0 0 0 133 1 0 0 25 0 1 0 885760412 274866176 7816 33554432000 134512640 135204385 4288212416 18446744073709551615 134536744 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/24363/statm: 67106 7816 64 169 0 66936 0
[pid=24364] ppid=24349 vsize=341036 CPUtime=0 cores=0-7
/proc/24364/stat : 24364 (python) S 24349 24349 24319 0 -1 4202560 747 0 0 0 0 0 0 0 17 0 1 0 885760415 349220864 13478 33554432000 4194304 5683644 140735831790768 18446744073709551615 248089718342 0 0 16781312 514 4090495520923943176 0 0 17 6 0 0 0
/proc/24364/statm: 85259 13478 347 364 0 21168 0
[pid=24366] ppid=24364 vsize=268428 CPUtime=1.3 cores=0-7
/proc/24366/stat : 24366 (hybridGM3) R 24364 24366 24366 0 -1 4202496 8077 0 0 0 130 0 0 0 25 0 1 0 885760416 274870272 7815 33554432000 134512640 135204385 4290021424 18446744073709551615 134546082 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/24366/statm: 67107 7815 64 169 0 66937 0
[pid=24367] ppid=24349 vsize=416828 CPUtime=0 cores=0-7
/proc/24367/stat : 24367 (python) S 24349 24349 24319 0 -1 4202560 721 0 0 0 0 0 0 0 17 0 1 0 885760419 426831872 13656 33554432000 4194304 5683644 140735831790768 18446744073709551615 248089718342 0 0 16781312 514 4090121686970499336 0 0 17 0 0 0 0
/proc/24367/statm: 104207 13656 348 364 0 40113 0
[pid=24369] ppid=24367 vsize=268424 CPUtime=1.26 cores=0-7
/proc/24369/stat : 24369 (hybridGM3) R 24367 24369 24369 0 -1 4202496 7921 0 0 0 126 0 0 0 25 0 1 0 885760420 274866176 7666 33554432000 134512640 135204385 4288367744 18446744073709551615 134537583 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/24369/statm: 67106 7666 64 169 0 66936 0
[pid=24370] ppid=24349 vsize=427084 CPUtime=0 cores=0-7
/proc/24370/stat : 24370 (python) S 24349 24349 24319 0 -1 4202560 729 0 0 0 0 0 0 0 16 0 1 0 885760423 437334016 13632 33554432000 4194304 5683644 140735831790768 18446744073709551615 248089718342 0 0 16781312 514 4090535103342543112 0 0 17 0 0 0 0
/proc/24370/statm: 106771 13632 348 364 0 42674 0
[pid=24372] ppid=24370 vsize=268424 CPUtime=1.23 cores=0-7
/proc/24372/stat : 24372 (hybridGM3) R 24370 24372 24372 0 -1 4202496 7923 0 0 0 122 1 0 0 25 0 1 0 885760423 274866176 7668 33554432000 134512640 135204385 4291922048 18446744073709551615 134546073 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/24372/statm: 67106 7668 64 169 0 66936 0
[pid=24373] ppid=24349 vsize=437340 CPUtime=0 cores=0-7
/proc/24373/stat : 24373 (python) S 24349 24349 24319 0 -1 4202560 733 0 0 0 0 0 0 0 18 0 1 0 885760427 447836160 13619 33554432000 4194304 5683644 140735831790768 18446744073709551615 248089718342 0 0 16781312 514 4613607969047707647 0 0 17 0 0 0 0
/proc/24373/statm: 109335 13619 348 364 0 45235 0
[pid=24375] ppid=24373 vsize=465044 CPUtime=1.16 cores=0-7
/proc/24375/stat : 24375 (TNM) R 24373 24375 24375 0 -1 4202496 6008 0 0 0 116 0 0 0 25 0 1 0 885760429 476205056 5750 33554432000 134512640 135008748 4289609552 18446744073709551615 134528940 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/24375/statm: 116261 5750 46 122 0 115161 0
[pid=24376] ppid=24349 vsize=447596 CPUtime=0 cores=0-7
/proc/24376/stat : 24376 (python) S 24349 24349 24319 0 -1 4202560 771 0 0 0 0 0 0 0 18 0 1 0 885760431 458338304 13749 33554432000 4194304 5683644 140735831790768 18446744073709551615 248089718342 0 0 16781312 514 4424940294936461311 0 0 17 7 0 0 0
/proc/24376/statm: 111899 13749 354 364 0 47796 0
[pid=24378] ppid=24376 vsize=268420 CPUtime=1.04 cores=0-7
/proc/24378/stat : 24378 (hybridGM3) R 24376 24378 24378 0 -1 4202496 4362 0 0 0 104 0 0 0 25 0 1 0 885760432 274862080 4098 33554432000 134512640 135204385 4288779792 18446744073709551615 134543975 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/24378/statm: 67105 4098 57 169 0 66935 0
[pid=24349/tid=24356] ppid=24347 vsize=457840 CPUtime=0 cores=0-7
/proc/24349/task/24356/stat : 24356 (python) S 24347 24349 24319 0 -1 4202560 26 935 0 0 0 0 0 0 25 0 9 0 885760404 468828160 15288 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 1 0 0 0
[pid=24349/tid=24359] ppid=24347 vsize=457840 CPUtime=0 cores=0-7
/proc/24349/task/24359/stat : 24359 (python) S 24347 24349 24319 0 -1 4202560 24 935 0 0 0 0 0 0 25 0 9 0 885760408 468828160 15288 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 5 0 0 0
[pid=24349/tid=24362] ppid=24347 vsize=457840 CPUtime=0 cores=0-7
/proc/24349/task/24362/stat : 24362 (python) S 24347 24349 24319 0 -1 4202560 22 935 0 0 0 0 0 0 25 0 9 0 885760412 468828160 15288 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 5 0 0 0
[pid=24349/tid=24365] ppid=24347 vsize=457840 CPUtime=0 cores=0-7
/proc/24349/task/24365/stat : 24365 (python) S 24347 24349 24319 0 -1 4202560 24 935 0 0 0 0 0 0 25 0 9 0 885760416 468828160 15288 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 4 0 0 0
[pid=24349/tid=24368] ppid=24347 vsize=457840 CPUtime=0 cores=0-7
/proc/24349/task/24368/stat : 24368 (python) S 24347 24349 24319 0 -1 4202560 88 935 0 0 0 0 0 0 19 0 9 0 885760419 468828160 15288 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 1 0 0 0
[pid=24349/tid=24371] ppid=24347 vsize=457840 CPUtime=0 cores=0-7
/proc/24349/task/24371/stat : 24371 (python) S 24347 24349 24319 0 -1 4202560 30 935 0 0 0 0 0 0 20 0 9 0 885760423 468828160 15288 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 6 0 0 0
[pid=24349/tid=24374] ppid=24347 vsize=457840 CPUtime=0 cores=0-7
/proc/24349/task/24374/stat : 24374 (python) S 24347 24349 24319 0 -1 4202560 33 935 0 0 0 0 0 0 21 0 9 0 885760427 468828160 15288 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 7 0 0 0
[pid=24349/tid=24377] ppid=24347 vsize=457840 CPUtime=0 cores=0-7
/proc/24349/task/24377/stat : 24377 (python) S 24347 24349 24319 0 -1 4202560 33 935 0 0 0 0 0 0 23 0 9 0 885760431 468828160 15288 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 17.61
Current children cumulated vsize (KiB) 5833308

[startup+25.5018 s]
/proc/loadavg: 1.83 0.41 0.13 9/195 24379
/proc/meminfo: memFree=31533560/32950928 swapFree=67055156/67111528
[pid=24349] ppid=24347 vsize=457840 CPUtime=7.48 cores=0-7
/proc/24349/stat : 24349 (python) S 24347 24349 24319 0 -1 4202496 28191 935 1 0 733 15 0 0 23 0 9 0 885759277 468828160 15288 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101524939 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/24349/statm: 114460 15288 1889 364 0 50357 0
[pid=24355] ppid=24349 vsize=310268 CPUtime=0.01 cores=0-7
/proc/24355/stat : 24355 (python) S 24349 24349 24319 0 -1 4202560 890 0 0 0 1 0 0 0 15 0 1 0 885760404 317714432 13494 33554432000 4194304 5683644 140735831790768 18446744073709551615 248089718287 0 0 16781312 514 14286033744530735368 0 0 17 0 0 0 0
/proc/24355/statm: 77567 13494 375 364 0 13485 0
[pid=24357] ppid=24355 vsize=268420 CPUtime=14.22 cores=0-7
/proc/24357/stat : 24357 (hybridGM3) R 24355 24357 24357 0 -1 4202496 8093 0 0 0 1422 0 0 0 25 0 1 0 885760404 274862080 7828 33554432000 134512640 135204385 4292080912 18446744073709551615 134537292 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/24357/statm: 67105 7828 64 169 0 66935 0
[pid=24358] ppid=24349 vsize=320524 CPUtime=0.01 cores=0-7
/proc/24358/stat : 24358 (python) S 24349 24349 24319 0 -1 4202560 906 0 0 0 0 1 0 0 15 0 1 0 885760407 328216576 13493 33554432000 4194304 5683644 140735831790768 18446744073709551615 248089718342 0 0 16781312 514 14286033744530735368 0 0 17 1 0 0 0
/proc/24358/statm: 80131 13493 370 364 0 16046 0
[pid=24360] ppid=24358 vsize=268428 CPUtime=14.18 cores=0-7
/proc/24360/stat : 24360 (hybridGM3) R 24358 24360 24360 0 -1 4202496 8086 0 0 0 1417 1 0 0 25 0 1 0 885760408 274870272 7824 33554432000 134512640 135204385 4292110928 18446744073709551615 134537573 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/24360/statm: 67107 7824 64 169 0 66937 0
[pid=24361] ppid=24349 vsize=330780 CPUtime=0.01 cores=0-7
/proc/24361/stat : 24361 (python) S 24349 24349 24319 0 -1 4202560 909 0 0 0 0 1 0 0 15 0 1 0 885760411 338718720 13497 33554432000 4194304 5683644 140735831790768 18446744073709551615 248089718342 0 0 16781312 514 14286033744530735368 0 0 17 0 0 0 0
/proc/24361/statm: 82695 13497 370 364 0 18607 0
[pid=24363] ppid=24361 vsize=268424 CPUtime=14.14 cores=0-7
/proc/24363/stat : 24363 (hybridGM3) R 24361 24363 24363 0 -1 4202496 8084 0 0 0 1413 1 0 0 25 0 1 0 885760412 274866176 7818 33554432000 134512640 135204385 4288212416 18446744073709551615 134545915 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/24363/statm: 67106 7818 64 169 0 66936 0
[pid=24364] ppid=24349 vsize=341036 CPUtime=0.01 cores=0-7
/proc/24364/stat : 24364 (python) S 24349 24349 24319 0 -1 4202560 936 0 0 0 0 1 0 0 15 0 1 0 885760415 349220864 13501 33554432000 4194304 5683644 140735831790768 18446744073709551615 248089718342 0 0 16781312 514 14286033744530735368 0 0 17 6 0 0 0
/proc/24364/statm: 85259 13501 370 364 0 21168 0
[pid=24366] ppid=24364 vsize=268428 CPUtime=14.1 cores=0-7
/proc/24366/stat : 24366 (hybridGM3) R 24364 24366 24366 0 -1 4202496 8083 0 0 0 1410 0 0 0 25 0 1 0 885760416 274870272 7821 33554432000 134512640 135204385 4290021424 18446744073709551615 134547462 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/24366/statm: 67107 7821 64 169 0 66937 0
[pid=24367] ppid=24349 vsize=416828 CPUtime=0 cores=0-7
/proc/24367/stat : 24367 (python) S 24349 24349 24319 0 -1 4202560 792 0 0 0 0 0 0 0 15 0 1 0 885760419 426831872 13679 33554432000 4194304 5683644 140735831790768 18446744073709551615 248089718342 0 0 16781312 514 14286033744530735368 0 0 17 0 0 0 0
/proc/24367/statm: 104207 13679 371 364 0 40113 0
[pid=24369] ppid=24367 vsize=268424 CPUtime=14.04 cores=0-7

################
# More data... #
################

/proc/24466/statm: 116261 5750 46 122 0 115161 0
[pid=24464/tid=24480] ppid=24349 vsize=621948 CPUtime=0 cores=0-7
/proc/24464/task/24480/stat : 24480 (python) S 24349 24349 24319 0 -1 4202560 84 0 0 0 0 0 0 0 19 0 2 0 885839979 636874752 14065 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 514 18446604453805342848 0 0 -1 4 0 0 0
[pid=24472] ppid=24349 vsize=632204 CPUtime=0.32 cores=0-7
/proc/24472/stat : 24472 (python) S 24349 24349 24319 0 -1 4202560 1178 0 0 0 18 14 0 0 16 0 2 0 885839374 647376896 14060 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/24472/statm: 158051 14060 400 364 0 93894 0
[pid=24474] ppid=24472 vsize=268424 CPUtime=196.41 cores=0-7
/proc/24474/stat : 24474 (hybridGM3) T 24472 24474 24474 0 -1 4202496 8108 0 0 0 19640 1 0 0 25 0 1 0 885839374 274866176 7826 33554432000 134512640 135204385 4293819664 18446744073709551615 134537113 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/24474/statm: 67106 7826 64 169 0 66936 0
[pid=24472/tid=24497] ppid=24349 vsize=632204 CPUtime=0 cores=0-7
/proc/24472/task/24497/stat : 24497 (python) S 24349 24349 24319 0 -1 4202560 90 0 0 0 0 0 0 0 16 0 2 0 885859020 647376896 14060 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 514 18446604453870414784 0 0 -1 4 0 0 0
[pid=24477] ppid=24349 vsize=642460 CPUtime=0.51 cores=0-7
/proc/24477/stat : 24477 (python) S 24349 24349 24319 0 -1 4202560 1184 0 0 0 26 25 0 0 18 0 2 0 885839408 657879040 14037 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/24477/statm: 160615 14037 401 364 0 96455 0
[pid=24479] ppid=24477 vsize=431704 CPUtime=296.2 cores=0-7
/proc/24479/stat : 24479 (adaptg2wsat2009) T 24477 24479 24479 0 -1 4202496 6265 0 0 0 29616 4 0 0 25 0 1 0 885839408 442064896 5982 33554432000 134512640 135107411 4293610672 18446744073709551615 134532349 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/24479/statm: 107926 5982 59 146 0 106802 0
[pid=24477/tid=24499] ppid=24349 vsize=642460 CPUtime=0 cores=0-7
/proc/24477/task/24499/stat : 24499 (python) S 24349 24349 24319 0 -1 4202560 91 0 0 0 0 0 0 0 18 0 2 0 885859447 657879040 14037 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 514 18446604471128543552 0 0 -1 0 0 0 0
[pid=24481] ppid=24349 vsize=652716 CPUtime=1.02 cores=0-7
/proc/24481/stat : 24481 (python) S 24349 24349 24319 0 -1 4202560 1204 0 0 0 59 43 0 0 15 0 2 0 885839990 668381184 14149 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/24481/statm: 163179 14149 400 364 0 99016 0
[pid=24483] ppid=24481 vsize=268428 CPUtime=587.42 cores=0-7
/proc/24483/stat : 24483 (hybridGM3) T 24481 24483 24483 0 -1 4202496 8116 0 0 0 58736 6 0 0 25 0 1 0 885839990 274870272 7828 33554432000 134512640 135204385 4294781552 18446744073709551615 134537020 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/24483/statm: 67107 7828 64 169 0 66937 0
[pid=24481/tid=24498] ppid=24349 vsize=652716 CPUtime=0 cores=0-7
/proc/24481/task/24498/stat : 24498 (python) S 24349 24349 24319 0 -1 4202560 92 0 0 0 0 0 0 0 15 0 2 0 885859437 668381184 14149 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 514 18446604468026737792 0 0 -1 2 0 0 0
[pid=24487] ppid=24349 vsize=662972 CPUtime=0.5 cores=0-7
/proc/24487/stat : 24487 (python) S 24349 24349 24319 0 -1 4202560 1225 0 0 0 26 24 0 0 18 0 2 0 885848712 678883328 14034 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/24487/statm: 165743 14034 407 364 0 101577 0
[pid=24489] ppid=24487 vsize=431704 CPUtime=293.46 cores=0-7
/proc/24489/stat : 24489 (adaptg2wsat2009) T 24487 24489 24489 0 -1 4202496 6281 0 0 0 29345 1 0 0 25 0 1 0 885848712 442064896 5984 33554432000 134512640 135107411 4289323200 18446744073709551615 134531491 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/24489/statm: 107926 5984 59 146 0 106802 0
[pid=24487/tid=24503] ppid=24349 vsize=662972 CPUtime=0 cores=0-7
/proc/24487/task/24503/stat : 24503 (python) S 24349 24349 24319 0 -1 4202560 93 0 0 0 0 0 0 0 18 0 2 0 885868354 678883328 14034 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 514 18446604468026738688 0 0 -1 1 0 0 0
[pid=24491] ppid=24349 vsize=673228 CPUtime=0.18 cores=0-7
/proc/24491/stat : 24491 (python) S 24349 24349 24319 0 -1 4202560 1220 0 0 0 10 8 0 0 19 0 2 0 885848913 689385472 14154 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/24491/statm: 168307 14154 407 364 0 104138 0
[pid=24493] ppid=24491 vsize=260404 CPUtime=101.88 cores=0-7
/proc/24493/stat : 24493 (FH) T 24491 24493 24493 0 -1 4202496 6185 0 0 0 10187 1 0 0 25 0 1 0 885848915 266653696 5897 33554432000 134512640 134936988 4290573760 18446744073709551615 134547038 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/24493/statm: 65101 5897 41 104 0 64605 0
[pid=24491/tid=24500] ppid=24349 vsize=673228 CPUtime=0 cores=0-7
/proc/24491/task/24500/stat : 24500 (python) S 24349 24349 24319 0 -1 4202560 91 0 0 0 0 0 0 0 19 0 2 0 885859532 689385472 14154 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 514 18446604468026736896 0 0 -1 4 0 0 0
[pid=24507] ppid=24349 vsize=683484 CPUtime=0.49 cores=0-7
/proc/24507/stat : 24507 (python) S 24349 24349 24319 0 -1 4202560 1243 0 0 0 24 25 0 0 18 0 2 0 885878506 699887616 14133 33554432000 4194304 5683644 140735831790768 18446744073709551615 248089718342 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/24507/statm: 170871 14133 406 364 0 106699 0
[pid=24509] ppid=24507 vsize=431704 CPUtime=291.24 cores=0-7
/proc/24509/stat : 24509 (adaptg2wsat2009) R 24507 24509 24509 0 -1 4202496 6276 0 0 0 29123 1 0 0 25 0 1 0 885878507 442064896 5983 33554432000 134512640 135107411 4293714128 18446744073709551615 134532128 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/24509/statm: 107926 5983 59 146 0 106802 0
[pid=24507/tid=24524] ppid=24349 vsize=683484 CPUtime=0 cores=0-7
/proc/24507/task/24524/stat : 24524 (python) S 24349 24349 24319 0 -1 4202560 95 0 0 0 0 0 0 0 19 0 2 0 885898137 699887616 14133 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 514 18446604453853080320 0 0 -1 5 0 0 0
[pid=24512] ppid=24349 vsize=693740 CPUtime=0.32 cores=0-7
/proc/24512/stat : 24512 (python) S 24349 24349 24319 0 -1 4202560 1245 0 0 0 16 16 0 0 18 0 2 0 885887819 710389760 14232 33554432000 4194304 5683644 140735831790768 18446744073709551615 248089718342 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/24512/statm: 173435 14232 407 364 0 109260 0
[pid=24514] ppid=24512 vsize=465044 CPUtime=193.17 cores=0-7
/proc/24514/stat : 24514 (TNM) R 24512 24514 24514 0 -1 4202496 6042 0 0 0 19315 2 0 0 25 0 1 0 885887819 476205056 5749 33554432000 134512640 135008748 4287972384 18446744073709551615 134528967 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/24514/statm: 116261 5749 46 122 0 115161 0
[pid=24512/tid=24525] ppid=24349 vsize=693740 CPUtime=0 cores=0-7
/proc/24512/task/24525/stat : 24525 (python) S 24349 24349 24319 0 -1 4202560 92 0 0 0 0 0 0 0 19 0 2 0 885898436 710389760 14232 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 514 18446604453966996736 0 0 -1 1 0 0 0
[pid=24518] ppid=24349 vsize=703996 CPUtime=0.16 cores=0-7
/proc/24518/stat : 24518 (python) S 24349 24349 24319 0 -1 4202560 1243 0 0 0 9 7 0 0 19 0 2 0 885897513 720891904 14179 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/24518/statm: 175999 14179 401 364 0 111821 0
[pid=24520] ppid=24518 vsize=465044 CPUtime=100.22 cores=0-7
/proc/24520/stat : 24520 (TNM) T 24518 24520 24520 0 -1 4202496 6042 0 0 0 10021 1 0 0 25 0 1 0 885897514 476205056 5749 33554432000 134512640 135008748 4287445440 18446744073709551615 134527773 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/24520/statm: 116261 5749 46 122 0 115161 0
[pid=24518/tid=24528] ppid=24349 vsize=703996 CPUtime=0 cores=0-7
/proc/24518/task/24528/stat : 24528 (python) S 24349 24349 24319 0 -1 4202560 95 0 0 0 0 0 0 0 19 0 2 0 885907539 720891904 14179 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 514 18446604453828441984 0 0 -1 3 0 0 0
[pid=24521] ppid=24349 vsize=714252 CPUtime=0.15 cores=0-7
/proc/24521/stat : 24521 (python) S 24349 24349 24319 0 -1 4202560 1044 0 0 0 8 7 0 0 18 0 1 0 885897927 731394048 14220 33554432000 4194304 5683644 140735831790768 18446744073709551615 248089718342 0 0 16781312 514 14286033744530735361 0 0 17 4 0 0 0
/proc/24521/statm: 178563 14220 373 364 0 114382 0
[pid=24523] ppid=24521 vsize=465044 CPUtime=92.84 cores=0-7
/proc/24523/stat : 24523 (TNM) R 24521 24523 24523 0 -1 4202496 6038 0 0 0 9284 0 0 0 25 0 1 0 885897927 476205056 5749 33554432000 134512640 135008748 4292306752 18446744073709551615 134527444 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/24523/statm: 116261 5749 46 122 0 115161 0
[pid=24529] ppid=24349 vsize=724508 CPUtime=0 cores=0-7
/proc/24529/stat : 24529 (python) S 24349 24349 24319 0 -1 4202560 918 0 0 0 0 0 0 0 19 0 1 0 885907563 741896192 14109 33554432000 4194304 5683644 140735831790768 18446744073709551615 248089718342 0 0 16781312 514 3872921686118498303 0 0 17 4 0 0 0
/proc/24529/statm: 181127 14109 345 364 0 116943 0
[pid=24531] ppid=24529 vsize=268424 CPUtime=0.93 cores=0-7
/proc/24531/stat : 24531 (hybridGM3) R 24529 24531 24531 0 -1 4202496 4247 0 0 0 93 0 0 0 23 0 1 0 885907565 274866176 3955 33554432000 134512640 135204385 4286703776 18446744073709551615 134543968 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/24531/statm: 67106 3955 57 169 0 66936 0
[pid=24349/tid=24356] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24356/stat : 24356 (python) S 24347 24349 24319 0 -1 4202560 45 935 0 0 0 0 0 0 15 0 36 0 885760404 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 1 0 0 0
[pid=24349/tid=24359] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24359/stat : 24359 (python) S 24347 24349 24319 0 -1 4202560 48 935 0 0 0 0 0 0 15 0 36 0 885760408 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 5 0 0 0
[pid=24349/tid=24362] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24362/stat : 24362 (python) S 24347 24349 24319 0 -1 4202560 39 935 0 0 0 0 0 0 18 0 36 0 885760412 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 5 0 0 0
[pid=24349/tid=24365] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24365/stat : 24365 (python) S 24347 24349 24319 0 -1 4202560 42 935 0 0 0 0 0 0 15 0 36 0 885760416 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 4 0 0 0
[pid=24349/tid=24368] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24368/stat : 24368 (python) S 24347 24349 24319 0 -1 4202560 98 935 0 0 0 0 0 0 15 0 36 0 885760419 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 6 0 0 0
[pid=24349/tid=24371] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24371/stat : 24371 (python) S 24347 24349 24319 0 -1 4202560 56 935 0 0 0 0 0 0 18 0 36 0 885760423 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 6 0 0 0
[pid=24349/tid=24374] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24374/stat : 24374 (python) S 24347 24349 24319 0 -1 4202560 40 935 0 0 0 0 0 0 18 0 36 0 885760427 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 7 0 0 0
[pid=24349/tid=24377] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24377/stat : 24377 (python) S 24347 24349 24319 0 -1 4202560 49 935 0 0 0 0 0 0 18 0 36 0 885760431 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 1 0 0 0
[pid=24349/tid=24387] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24387/stat : 24387 (python) S 24347 24349 24319 0 -1 4202560 57 935 0 0 0 0 0 0 18 0 36 0 885770448 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 7 0 0 0
[pid=24349/tid=24399] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24399/stat : 24399 (python) S 24347 24349 24319 0 -1 4202560 54 935 0 0 0 0 0 0 18 0 36 0 885790063 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 6 0 0 0
[pid=24349/tid=24402] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24402/stat : 24402 (python) S 24347 24349 24319 0 -1 4202560 40 935 0 0 0 0 0 0 18 0 36 0 885790068 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 0 0 0 0
[pid=24349/tid=24405] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24405/stat : 24405 (python) S 24347 24349 24319 0 -1 4202560 44 935 0 0 0 0 0 0 18 0 36 0 885790176 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 1 0 0 0
[pid=24349/tid=24412] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24412/stat : 24412 (python) S 24347 24349 24319 0 -1 4202560 41 935 0 0 0 0 0 0 20 0 36 0 885799591 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 2 0 0 0
[pid=24349/tid=24415] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24415/stat : 24415 (python) S 24347 24349 24319 0 -1 4202560 43 935 0 0 0 0 0 0 18 0 36 0 885799615 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 3 0 0 0
[pid=24349/tid=24419] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24419/stat : 24419 (python) S 24347 24349 24319 0 -1 4202560 42 935 0 0 0 0 0 0 20 0 36 0 885800087 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 3 0 0 0
[pid=24349/tid=24428] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24428/stat : 24428 (python) S 24347 24349 24319 0 -1 4202560 96 935 0 0 0 0 0 0 20 0 36 0 885810110 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 0 0 0 0
[pid=24349/tid=24434] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24434/stat : 24434 (python) S 24347 24349 24319 0 -1 4202560 41 935 0 0 0 0 0 0 18 0 36 0 885818935 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 0 0 0 0
[pid=24349/tid=24437] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24437/stat : 24437 (python) S 24347 24349 24319 0 -1 4202560 32 935 0 0 0 0 0 0 19 0 36 0 885819329 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 3 0 0 0
[pid=24349/tid=24440] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24440/stat : 24440 (python) S 24347 24349 24319 0 -1 4202560 40 935 0 0 0 0 0 0 18 0 36 0 885819751 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 3 0 0 0
[pid=24349/tid=24444] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24444/stat : 24444 (python) S 24347 24349 24319 0 -1 4202560 38 935 0 0 0 0 0 0 20 0 36 0 885819941 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 3 0 0 0
[pid=24349/tid=24448] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24448/stat : 24448 (python) S 24347 24349 24319 0 -1 4202560 36 935 0 0 0 0 0 0 20 0 36 0 885820039 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 3 0 0 0
[pid=24349/tid=24455] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24455/stat : 24455 (python) S 24347 24349 24319 0 -1 4202560 34 935 0 0 0 0 0 0 20 0 36 0 885829350 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 3 0 0 0
[pid=24349/tid=24458] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24458/stat : 24458 (python) S 24347 24349 24319 0 -1 4202560 39 935 0 0 0 0 0 0 15 0 36 0 885829359 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 7 0 0 0
[pid=24349/tid=24461] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24461/stat : 24461 (python) S 24347 24349 24319 0 -1 4202560 30 935 0 0 0 0 0 0 22 0 36 0 885829383 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 4 0 0 0
[pid=24349/tid=24465] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24465/stat : 24465 (python) S 24347 24349 24319 0 -1 4202560 34 935 0 0 0 0 0 0 20 0 36 0 885829965 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 4 0 0 0
[pid=24349/tid=24473] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24473/stat : 24473 (python) S 24347 24349 24319 0 -1 4202560 35 935 0 0 0 0 0 0 20 0 36 0 885839374 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 1 0 0 0
[pid=24349/tid=24478] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24478/stat : 24478 (python) S 24347 24349 24319 0 -1 4202560 41 935 0 0 0 0 0 0 18 0 36 0 885839408 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 0 0 0 0
[pid=24349/tid=24482] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24482/stat : 24482 (python) S 24347 24349 24319 0 -1 4202560 44 935 0 0 0 0 0 0 15 0 36 0 885839990 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 7 0 0 0
[pid=24349/tid=24488] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24488/stat : 24488 (python) S 24347 24349 24319 0 -1 4202560 35 935 0 0 0 0 0 0 18 0 36 0 885848712 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 7 0 0 0
[pid=24349/tid=24492] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24492/stat : 24492 (python) S 24347 24349 24319 0 -1 4202560 28 935 0 0 0 0 0 0 19 0 36 0 885848914 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 4 0 0 0
[pid=24349/tid=24508] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24508/stat : 24508 (python) S 24347 24349 24319 0 -1 4202560 45 935 0 0 0 0 0 0 18 0 36 0 885878506 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 1 0 0 0
[pid=24349/tid=24513] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24513/stat : 24513 (python) S 24347 24349 24319 0 -1 4202560 37 935 0 0 0 0 0 0 18 0 36 0 885887819 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 1 0 0 0
[pid=24349/tid=24519] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24519/stat : 24519 (python) S 24347 24349 24319 0 -1 4202560 36 935 0 0 0 0 0 0 20 0 36 0 885897514 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 1 0 0 0
[pid=24349/tid=24522] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24522/stat : 24522 (python) S 24347 24349 24319 0 -1 4202560 30 935 0 0 0 0 0 0 20 0 36 0 885897927 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 4 0 0 0
[pid=24349/tid=24530] ppid=24347 vsize=734752 CPUtime=0 cores=0-7
/proc/24349/task/24530/stat : 24530 (python) S 24347 24349 24319 0 -1 4202560 40 935 0 0 0 0 0 0 19 0 36 0 885907564 752386048 15712 33554432000 4194304 5683644 140735831790768 18446744073709551615 248101521841 0 0 16781312 2 18446604471149066944 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 11741
Current children cumulated vsize (KiB) 32596476

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 24349 and gives
#  childrusage.ru_utime.tv_sec=1659
#  childrusage.ru_utime.tv_usec=413730
#  childrusage.ru_stime.tv_sec=8
#  childrusage.ru_stime.tv_usec=374726
# CPU time returned by wait4() is 1667.79
# while last known CPU time is 11741
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1484.01
CPU time (s): 11741
CPU user time (s): 11731.5
CPU system time (s): 9.55
CPU usage (%): 791.17
Max. virtual memory (cumulated for all children) (KiB): 32596476

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1659.41
system time used= 8.37473
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 154784
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 15712
involuntary context switches= 10448

runsolver used 12.4821 second user time and 31.4662 second system time

The end

Launcher Data 

Begin job on node123 at 2011-06-10 23:33:10
IDJOB=3321831
IDBENCH=82609
IDSOLVER=1712
FILE ID=node123/3321831-1307741589
RUNJOBID= node123-1307741589-24333
PBS_JOBID= 13550783
Free space on /tmp= 74180 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v35000-c147000-S648642831-094-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3321831-1307741589/watcher-3321831-1307741589 -o /tmp/evaluation-result-3321831-1307741589/solver-3321831-1307741589 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3321831-1307741589.cnf -seed 951405007 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 4a57833314362e0939d4095b3206ad82
RANDOM SEED=951405007

node123.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      31863808 kB
Buffers:        232140 kB
Cached:         646960 kB
SwapCached:      10356 kB
Active:         398356 kB
Inactive:       498788 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      31863808 kB
SwapTotal:    67111528 kB
SwapFree:     67055156 kB
Dirty:           34548 kB
Writeback:           0 kB
AnonPages:       15364 kB
Mapped:          12308 kB
Slab:           126080 kB
PageTables:       4244 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   185436 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 74140 MiB
End job on node123 at 2011-06-10 23:58:01