Trace number 3385436

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (MO) 12528.2 1584.13

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v50000-c210000-S964643921-035-UNKNOWN.cnf
MD5SUMd8337de8a71a630ccf2dd02943cd7d7f
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark801.476
Satisfiable
(Un)Satisfiability was proved
Number of variables50000
Number of clauses210000
Sum of the clauses size630000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3210000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.32/0.47	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.32/0.47	c INFO: solving HOME/instance-3385436-1305566608.cnf
0.32/0.47	c INFO: computing task features
1.11/1.23	c INFO: computed variable-clause graph statistics
1.11/1.26	c INFO: computed clause balance statistics
4.81/4.91	c INFO: computed variable balance statistics
7.81/7.91	c INFO: computed small-clause counts
7.81/7.93	c INFO: computed Horn-clause counts
10.82/11.00	c INFO: computed variable graph statistics
10.82/11.00	c INFO: collected features for HOME/instance-3385436-1305566608.cnf in 10.52 s
10.92/11.03	c INFO: running hybridGM3@0 for 97 with 4989 remaining (b = 0.35)
10.92/11.06	c INFO: running hybridGM3@0 for 97 with 4989 remaining (b = 0.21)
10.92/11.09	c INFO: running hybridGM3@0 for 97 with 4989 remaining (b = 0.14)
11.01/11.13	c INFO: running hybridGM3@0 for 97 with 4989 remaining (b = 0.09)
11.01/11.16	c INFO: running hybridGM3@0 for 97 with 4989 remaining (b = 0.07)
11.01/11.19	c INFO: running hybridGM3@0 for 194 with 4989 remaining (b = 0.10)
11.10/11.22	c INFO: running hybridGM3@0 for 194 with 4989 remaining (b = 0.07)
11.10/11.26	c INFO: running hybridGM3@0 for 194 with 4989 remaining (b = 0.06)
810.97/111.49	c INFO: running hybridGM3@0 for 97 with 4889 remaining (b = 0.03)
811.77/111.52	c INFO: running hybridGM3@0 for 97 with 4888 remaining (b = 0.03)
811.77/111.55	c INFO: running hybridGM3@0 for 97 with 4888 remaining (b = 0.03)
811.77/111.59	c INFO: running hybridGM3@0 for 97 with 4888 remaining (b = 0.03)
812.52/111.62	c INFO: running hybridGM3@0 for 97 with 4888 remaining (b = 0.03)
1570.90/206.81	c INFO: running hybridGM3@100 for 97 with 4793 remaining (b = 0.03)
1571.70/206.84	c INFO: running hybridGM3@100 for 97 with 4793 remaining (b = 0.03)
1571.70/206.87	c INFO: running TNM@0 for 97 with 4793 remaining (b = 0.03)
1572.31/206.91	c INFO: running hybridGM3@100 for 97 with 4793 remaining (b = 0.02)
1572.31/206.94	c INFO: running hybridGM3@100 for 97 with 4793 remaining (b = 0.02)
1581.08/208.02	c INFO: running hybridGM3@94 for 97 with 4792 remaining (b = 0.02)
1628.97/214.02	c INFO: running hybridGM3@100 for 97 with 4786 remaining (b = 0.02)
1628.97/214.06	c INFO: running hybridGM3@0 for 97 with 4786 remaining (b = 0.02)
2365.10/306.40	c INFO: running hybridGM3@0 for 97 with 4694 remaining (b = 0.02)
2372.25/307.22	c INFO: running hybridGM3@293 for 97 with 4693 remaining (b = 0.02)
2379.43/308.14	c INFO: running TNM@0 for 97 with 4692 remaining (b = 0.02)
2379.43/308.18	c INFO: running hybridGM3@0 for 194 with 4692 remaining (b = 0.04)
2380.15/308.25	c INFO: running hybridGM3@0 for 194 with 4692 remaining (b = 0.03)
2380.15/308.29	c INFO: running hybridGM3@296 for 97 with 4692 remaining (b = 0.02)
2381.23/308.40	c INFO: running hybridGM3@98 for 97 with 4692 remaining (b = 0.02)
2404.95/311.38	c INFO: running hybridGM3@299 for 97 with 4689 remaining (b = 0.02)
3132.37/402.55	c INFO: running TNM@0 for 97 with 4597 remaining (b = 0.02)
3141.12/403.62	c INFO: running TNM@0 for 97 with 4596 remaining (b = 0.01)
3141.82/403.73	c INFO: running hybridGM3@395 for 97 with 4596 remaining (b = 0.01)
3141.82/403.77	c INFO: running TNM@0 for 97 with 4596 remaining (b = 0.01)
3179.99/408.51	c INFO: running hybridGM3@390 for 97 with 4591 remaining (b = 0.01)
3213.46/412.73	c INFO: running TNM@0 for 97 with 4587 remaining (b = 0.01)
3932.01/502.91	c INFO: running TNM@0 for 97 with 4497 remaining (b = 0.01)
3932.72/503.06	c INFO: running hybridGM3@394 for 97 with 4497 remaining (b = 0.01)
3941.53/504.10	c INFO: running TNM@0 for 97 with 4496 remaining (b = 0.01)
3941.53/504.17	c INFO: running TNM@94 for 1940 with 4496 remaining (b = 0.19)
3948.67/505.04	c INFO: running hybridGM3@98 for 97 with 4495 remaining (b = 0.01)
3955.07/505.88	c INFO: running hybridGM3@101 for 97 with 4494 remaining (b = 0.01)
3997.22/511.15	c INFO: running hybridGM3@100 for 97 with 4489 remaining (b = 0.01)
4005.19/512.13	c INFO: running hybridGM3@0 for 194 with 4488 remaining (b = 0.02)
4708.55/600.45	c INFO: running adaptg2wsat2009++@0 for 194 with 4400 remaining (b = 0.01)
4714.91/601.22	c INFO: running hybridGM3@299 for 97 with 4399 remaining (b = 0.01)
4722.83/602.29	c INFO: running TNM@0 for 97 with 4398 remaining (b = 0.01)
4724.32/602.42	c INFO: running TNM@0 for 97 with 4398 remaining (b = 0.01)
4741.10/604.52	c INFO: running TNM@98 for 1940 with 4395 remaining (b = 0.14)
4804.80/612.52	c INFO: running hybridGM3@0 for 194 with 4387 remaining (b = 0.02)
5520.94/702.60	c INFO: running hybridGM3@399 for 97 with 4297 remaining (b = 0.01)
5521.71/702.68	c INFO: running hybridGM3@0 for 194 with 4297 remaining (b = 0.01)
5571.00/708.85	c INFO: running hybridGM3@0 for 194 with 4291 remaining (b = 0.01)
5572.44/709.05	c INFO: running hybridGM3@96 for 97 with 4291 remaining (b = 0.01)
6273.52/797.14	c INFO: running adaptg2wsat2009++@95 for 97 with 4203 remaining (b = 0.01)
6279.88/797.96	c INFO: running adaptg2wsat2009++@0 for 194 with 4202 remaining (b = 0.01)
6371.36/809.45	c INFO: running hybridGM3@101 for 97 with 4191 remaining (b = 0.01)
6379.31/810.43	c INFO: running hybridGM3@292 for 194 with 4190 remaining (b = 0.01)
7031.76/892.50	c INFO: running adaptg2wsat2009++@190 for 97 with 4108 remaining (b = 0.01)
7130.37/904.81	c INFO: running hybridGM3@0 for 194 with 4095 remaining (b = 0.01)
7135.91/905.60	c INFO: running hybridGM3@101 for 97 with 4094 remaining (b = 0.01)
7137.51/905.79	c INFO: running adaptg2wsat2009++@0 for 194 with 4094 remaining (b = 0.01)
7838.34/993.89	c INFO: running FH@0 for 97 with 4006 remaining (b = 0.00)
7844.66/994.66	c INFO: running hybridGM3@195 for 291 with 4005 remaining (b = 0.01)
7904.47/1002.12	c INFO: running hybridGM3@388 for 194 with 3998 remaining (b = 0.01)
7942.53/1007.00	c INFO: running hybridGM3@298 for 194 with 3993 remaining (b = 0.01)
8629.87/1093.36	c INFO: running TNM@0 for 97 with 3907 remaining (b = 0.00)
8702.85/1102.51	c INFO: running adaptg2wsat2009++@101 for 97 with 3898 remaining (b = 0.00)
8793.46/1113.91	c INFO: running TNM@0 for 97 with 3886 remaining (b = 0.00)
9427.55/1193.77	c INFO: running adaptg2wsat2009++@0 for 194 with 3806 remaining (b = 0.01)
9466.43/1198.70	c INFO: running hybridGM3@397 for 97 with 3801 remaining (b = 0.01)
9475.98/1199.84	c INFO: running TNM@0 for 97 with 3800 remaining (b = 0.00)
9548.39/1208.95	c INFO: running adaptg2wsat2009++@300 for 291 with 3791 remaining (b = 0.01)
9591.25/1214.33	c INFO: running TNM@195 for 1843 with 3786 remaining (b = 0.08)
10149.79/1284.70	c INFO: running hybridGM3@385 for 194 with 3715 remaining (b = 0.01)
10273.83/1300.26	c INFO: running hybridGM3@195 for 291 with 3700 remaining (b = 0.01)
10320.59/1306.12	c INFO: running hybridGM3@195 for 291 with 3694 remaining (b = 0.01)
10975.34/1388.56	c INFO: running hybridGM3@195 for 291 with 3611 remaining (b = 0.01)
11769.16/1488.43	c INFO: running hybridGM3@485 for 97 with 3512 remaining (b = 0.01)
11869.28/1501.03	c INFO: running adaptg2wsat2009++@399 for 194 with 3499 remaining (b = 0.01)
12526.34/1583.81	c INFO: running NCVWr@0 for 485 with 3416 remaining (b = 0.02)
12528.16/1584.03	c INFO: running hybridGM3@196 for 291 with 3416 remaining (b = 0.01)
12528.16/1584.03	Traceback (most recent call last):
12528.16/1584.03	  File "HOME/solve", line 33, in <module>
12528.16/1584.03	    plac.call(main)
12528.16/1584.03	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
12528.16/1584.03	    cmd, result = parser_from(obj).consume(arglist)
12528.16/1584.03	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
12528.16/1584.03	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
12528.16/1584.03	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
12528.16/1584.03	    (_, answer, _) = solver(input_path, budget / cores, cores)
12528.16/1584.03	  File "HOME/borg/build/lib/borg/bilevel.py", line 202, in __call__
12528.16/1584.03	    solver.go(max_cost * borg.defaults.machine_speed)
12528.16/1584.03	  File "HOME/borg/build/lib/borg/solvers.py", line 179, in go
12528.16/1584.03	    self._process.start()
12528.16/1584.03	  File "/home/evaluation/evaluation/tools64/lib/python2.7/multiprocessing/process.py", line 94, in start
12528.16/1584.04	    assert self._popen is None, 'cannot start a process twice'
12528.16/1584.04	AssertionError: cannot start a process twice

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3385436-1305566608/watcher-3385436-1305566608 -o /tmp/evaluation-result-3385436-1305566608/solver-3385436-1305566608 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385436-1305566608.cnf -seed 1321995845 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.10 8.09 6.78 1/396 22614
/proc/meminfo: memFree=19233512/32951124 swapFree=63083184/67111528
[pid=22614] ppid=22612 vsize=16236 CPUtime=0 cores=0-7
/proc/22614/stat : 22614 (python) R 22612 22614 22174 0 -1 4202496 1037 0 0 0 0 0 0 0 20 0 1 0 649076469 16625664 909 33554432000 4194304 5683644 140733469874736 18446744073709551615 4571337 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/22614/statm: 4059 909 380 364 0 546 0

[startup+0.0241669 s]
/proc/loadavg: 8.10 8.09 6.78 1/396 22614
/proc/meminfo: memFree=19233512/32951124 swapFree=63083184/67111528
[pid=22614] ppid=22612 vsize=16532 CPUtime=0.01 cores=0-7
/proc/22614/stat : 22614 (python) S 22612 22614 22174 0 -1 4202496 1183 0 0 0 1 0 0 0 20 0 1 0 649076469 16928768 1045 33554432000 4194304 5683644 140733469874736 18446744073709551615 264726403072 0 2147483391 16781312 2 0 0 0 17 2 0 0 0
/proc/22614/statm: 4133 1045 417 364 0 620 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16532

[startup+0.101167 s]
/proc/loadavg: 8.10 8.09 6.78 1/396 22614
/proc/meminfo: memFree=19233512/32951124 swapFree=63083184/67111528
[pid=22614] ppid=22612 vsize=104224 CPUtime=0.07 cores=0-7
/proc/22614/stat : 22614 (python) S 22612 22614 22174 0 -1 4202496 3070 0 0 0 4 3 0 0 17 0 1 0 649076469 106725376 2751 33554432000 4194304 5683644 140733469874736 18446744073709551615 264726403072 0 2147483391 16781312 2 18446744071699207358 0 0 17 2 0 0 0
/proc/22614/statm: 26056 2751 847 364 0 1804 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 104224

[startup+0.301134 s]
/proc/loadavg: 8.10 8.09 6.78 1/396 22614
/proc/meminfo: memFree=19233512/32951124 swapFree=63083184/67111528
[pid=22614] ppid=22612 vsize=170752 CPUtime=0.23 cores=0-7
/proc/22614/stat : 22614 (python) S 22612 22614 22174 0 -1 4202496 6735 935 1 0 16 7 0 0 18 0 1 0 649076469 174850048 4806 33554432000 4194304 5683644 140733469874736 18446744073709551615 264726403072 0 2147483391 16781312 2 18446744071699207358 0 0 17 2 0 0 0
/proc/22614/statm: 42688 4806 1336 364 0 3453 0
Current children cumulated CPU time (s) 0.23
Current children cumulated vsize (KiB) 170752

[startup+0.701039 s]
/proc/loadavg: 8.10 8.09 6.78 1/396 22614
/proc/meminfo: memFree=19233512/32951124 swapFree=63083184/67111528
[pid=22614] ppid=22612 vsize=296424 CPUtime=0.6 cores=0-7
/proc/22614/stat : 22614 (python) R 22612 22614 22174 0 -1 4202496 13654 935 1 0 50 10 0 0 18 0 1 0 649076469 303538176 11336 33554432000 4194304 5683644 140733469874736 18446744073709551615 4571108 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/22614/statm: 74106 11336 1761 364 0 10030 0
Current children cumulated CPU time (s) 0.6
Current children cumulated vsize (KiB) 296424

[startup+1.50089 s]
/proc/loadavg: 8.10 8.09 6.78 2/397 22619
/proc/meminfo: memFree=19176536/32951124 swapFree=63083184/67111528
[pid=22614] ppid=22612 vsize=328000 CPUtime=1.41 cores=0-7
/proc/22614/stat : 22614 (python) R 22612 22614 22174 0 -1 4202496 27454 935 1 0 129 12 0 0 20 0 1 0 649076469 335872000 19385 33554432000 4194304 5683644 140733469874736 18446744073709551615 47377113037950 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/22614/statm: 82000 19385 1826 364 0 17924 0
Current children cumulated CPU time (s) 1.41
Current children cumulated vsize (KiB) 328000

[startup+3.1006 s]
/proc/loadavg: 8.10 8.09 6.78 2/397 22619
/proc/meminfo: memFree=19161532/32951124 swapFree=63083184/67111528
[pid=22614] ppid=22612 vsize=328000 CPUtime=3.01 cores=0-7
/proc/22614/stat : 22614 (python) R 22612 22614 22174 0 -1 4202496 27454 935 1 0 289 12 0 0 25 0 1 0 649076469 335872000 19385 33554432000 4194304 5683644 140733469874736 18446744073709551615 264726063295 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/22614/statm: 82000 19385 1826 364 0 17924 0
Current children cumulated CPU time (s) 3.01
Current children cumulated vsize (KiB) 328000

[startup+6.30098 s]
/proc/loadavg: 7.53 7.97 6.75 2/397 22619
/proc/meminfo: memFree=19162120/32951124 swapFree=63083184/67111528
[pid=22614] ppid=22612 vsize=328000 CPUtime=6.21 cores=0-7
/proc/22614/stat : 22614 (python) R 22612 22614 22174 0 -1 4202496 27455 935 1 0 609 12 0 0 25 0 1 0 649076469 335872000 19386 33554432000 4194304 5683644 140733469874736 18446744073709551615 264729813673 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/22614/statm: 82000 19386 1827 364 0 17924 0
Current children cumulated CPU time (s) 6.21
Current children cumulated vsize (KiB) 328000

[startup+12.7017 s]
/proc/loadavg: 7.01 7.86 6.72 9/421 22643
/proc/meminfo: memFree=19004640/32951124 swapFree=63083184/67111528
[pid=22614] ppid=22612 vsize=471692 CPUtime=11.17 cores=0-7
/proc/22614/stat : 22614 (python) S 22612 22614 22174 0 -1 4202496 33307 935 1 0 1102 15 0 0 21 0 9 0 649076469 483012608 18635 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738231755 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/22614/statm: 117923 18635 1889 364 0 53820 0
[pid=22620] ppid=22614 vsize=324120 CPUtime=0 cores=0-7
/proc/22620/stat : 22620 (python) S 22614 22614 22174 0 -1 4202560 734 0 0 0 0 0 0 0 18 0 1 0 649077573 331898880 16936 33554432000 4194304 5683644 140733469874736 18446744073709551615 264726425103 0 0 16781312 514 9591123221390819327 0 0 17 1 0 0 0
/proc/22620/statm: 81030 16936 353 364 0 16948 0
[pid=22622] ppid=22620 vsize=276232 CPUtime=1.65 cores=0-7
/proc/22622/stat : 22622 (hybridGM3) R 22620 22622 22622 0 -1 4202496 4997 0 0 0 164 1 0 0 25 0 1 0 649077574 282861568 4732 33554432000 134512640 135204385 4287680432 18446744073709551615 134543984 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/22622/statm: 69058 4732 57 169 0 68888 0
[pid=22623] ppid=22614 vsize=334376 CPUtime=0 cores=0-7
/proc/22623/stat : 22623 (python) S 22614 22614 22174 0 -1 4202560 744 0 0 0 0 0 0 0 18 0 1 0 649077576 342401024 16934 33554432000 4194304 5683644 140733469874736 18446744073709551615 264726425158 0 0 16781312 514 6918551027994066943 0 0 17 1 0 0 0
/proc/22623/statm: 83594 16934 347 364 0 19509 0
[pid=22625] ppid=22623 vsize=276236 CPUtime=1.61 cores=0-7
/proc/22625/stat : 22625 (hybridGM3) R 22623 22625 22625 0 -1 4202496 4973 0 0 0 161 0 0 0 25 0 1 0 649077577 282865664 4709 33554432000 134512640 135204385 4291001904 18446744073709551615 134543968 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/22625/statm: 69059 4709 57 169 0 68889 0
[pid=22626] ppid=22614 vsize=410168 CPUtime=0 cores=0-7
/proc/22626/stat : 22626 (python) S 22614 22614 22174 0 -1 4202560 702 0 0 0 0 0 0 0 18 0 1 0 649077579 420012032 17112 33554432000 4194304 5683644 140733469874736 18446744073709551615 264726425158 0 0 16781312 514 13474004004660838399 0 0 17 1 0 0 0
/proc/22626/statm: 102542 17112 348 364 0 38454 0
[pid=22628] ppid=22626 vsize=276236 CPUtime=1.58 cores=0-7
/proc/22628/stat : 22628 (hybridGM3) R 22626 22628 22628 0 -1 4202496 4942 0 0 0 158 0 0 0 25 0 1 0 649077580 282865664 4687 33554432000 134512640 135204385 4289151376 18446744073709551615 134543968 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/22628/statm: 69059 4687 57 169 0 68889 0
[pid=22629] ppid=22614 vsize=420424 CPUtime=0 cores=0-7
/proc/22629/stat : 22629 (python) S 22614 22614 22174 0 -1 4202560 716 0 0 0 0 0 0 0 18 0 1 0 649077583 430514176 17076 33554432000 4194304 5683644 140733469874736 18446744073709551615 264726425158 0 0 16781312 514 13474000156370141183 0 0 17 1 0 0 0
/proc/22629/statm: 105106 17076 348 364 0 41015 0
[pid=22631] ppid=22629 vsize=276240 CPUtime=1.55 cores=0-7
/proc/22631/stat : 22631 (hybridGM3) R 22629 22631 22631 0 -1 4202496 4922 0 0 0 155 0 0 0 25 0 1 0 649077583 282869760 4667 33554432000 134512640 135204385 4291540288 18446744073709551615 134543968 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22631/statm: 69060 4667 57 169 0 68890 0
[pid=22632] ppid=22614 vsize=430680 CPUtime=0 cores=0-7
/proc/22632/stat : 22632 (python) S 22614 22614 22174 0 -1 4202560 723 0 0 0 0 0 0 0 18 0 1 0 649077586 441016320 17112 33554432000 4194304 5683644 140733469874736 18446744073709551615 264726425158 0 0 16781312 514 4057086585236094975 0 0 17 1 0 0 0
/proc/22632/statm: 107670 17112 348 364 0 43576 0
[pid=22634] ppid=22632 vsize=276236 CPUtime=1.52 cores=0-7
/proc/22634/stat : 22634 (hybridGM3) R 22632 22634 22634 0 -1 4202496 4897 0 0 0 151 1 0 0 25 0 1 0 649077586 282865664 4640 33554432000 134512640 135204385 4289432112 18446744073709551615 134543975 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/22634/statm: 69059 4640 57 169 0 68889 0
[pid=22635] ppid=22614 vsize=440936 CPUtime=0 cores=0-7
/proc/22635/stat : 22635 (python) S 22614 22614 22174 0 -1 4202560 730 0 0 0 0 0 0 0 18 0 1 0 649077589 451518464 17116 33554432000 4194304 5683644 140733469874736 18446744073709551615 264726425158 0 0 16781312 514 4057082736945397759 0 0 17 1 0 0 0
/proc/22635/statm: 110234 17116 348 364 0 46137 0
[pid=22637] ppid=22635 vsize=276236 CPUtime=1.47 cores=0-7
/proc/22637/stat : 22637 (hybridGM3) R 22635 22637 22637 0 -1 4202496 4867 0 0 0 147 0 0 0 25 0 1 0 649077589 282865664 4607 33554432000 134512640 135204385 4294754672 18446744073709551615 134543975 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/22637/statm: 69059 4607 57 169 0 68889 0
[pid=22638] ppid=22614 vsize=451192 CPUtime=0 cores=0-7
/proc/22638/stat : 22638 (python) S 22614 22614 22174 0 -1 4202560 739 0 0 0 0 0 0 0 18 0 1 0 649077592 462020608 17103 33554432000 4194304 5683644 140733469874736 18446744073709551615 264726425158 0 0 16781312 514 4057078888654700543 0 0 17 1 0 0 0
/proc/22638/statm: 112798 17103 348 364 0 48698 0
[pid=22640] ppid=22638 vsize=276232 CPUtime=1.44 cores=0-7
/proc/22640/stat : 22640 (hybridGM3) R 22638 22640 22640 0 -1 4202496 4845 0 0 0 143 1 0 0 25 0 1 0 649077594 282861568 4585 33554432000 134512640 135204385 4287152464 18446744073709551615 134543968 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/22640/statm: 69058 4585 57 169 0 68888 0
[pid=22641] ppid=22614 vsize=461448 CPUtime=0 cores=0-7
/proc/22641/stat : 22641 (python) S 22614 22614 22174 0 -1 4202560 779 0 0 0 0 0 0 0 18 0 1 0 649077596 472522752 17096 33554432000 4194304 5683644 140733469874736 18446744073709551615 264726425158 0 0 16781312 514 4057075040364003327 0 0 17 2 0 0 0
/proc/22641/statm: 115362 17096 354 364 0 51259 0
[pid=22643] ppid=22641 vsize=276236 CPUtime=1.42 cores=0-7
/proc/22643/stat : 22643 (hybridGM3) R 22641 22643 22643 0 -1 4202496 4839 0 0 0 141 1 0 0 25 0 1 0 649077596 282865664 4572 33554432000 134512640 135204385 4293012832 18446744073709551615 134543968 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/22643/statm: 69059 4572 57 169 0 68889 0
[pid=22614/tid=22621] ppid=22612 vsize=471692 CPUtime=0 cores=0-7
/proc/22614/task/22621/stat : 22621 (python) S 22612 22614 22174 0 -1 4202560 25 935 0 0 0 0 0 0 25 0 9 0 649077573 483012608 18635 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 4 0 0 0
[pid=22614/tid=22624] ppid=22612 vsize=471692 CPUtime=0 cores=0-7
/proc/22614/task/22624/stat : 22624 (python) S 22612 22614 22174 0 -1 4202560 23 935 0 0 0 0 0 0 18 0 9 0 649077576 483012608 18635 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 3 0 0 0
[pid=22614/tid=22627] ppid=22612 vsize=471692 CPUtime=0 cores=0-7
/proc/22614/task/22627/stat : 22627 (python) S 22612 22614 22174 0 -1 4202560 84 935 0 0 0 0 0 0 19 0 9 0 649077580 483012608 18635 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 0 0 0 0
[pid=22614/tid=22630] ppid=22612 vsize=471692 CPUtime=0 cores=0-7
/proc/22614/task/22630/stat : 22630 (python) S 22612 22614 22174 0 -1 4202560 28 935 0 0 0 0 0 0 18 0 9 0 649077583 483012608 18635 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 5 0 0 0
[pid=22614/tid=22633] ppid=22612 vsize=471692 CPUtime=0 cores=0-7
/proc/22614/task/22633/stat : 22633 (python) S 22612 22614 22174 0 -1 4202560 32 935 0 0 0 0 0 0 19 0 9 0 649077586 483012608 18635 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 0 0 0 0
[pid=22614/tid=22636] ppid=22612 vsize=471692 CPUtime=0 cores=0-7
/proc/22614/task/22636/stat : 22636 (python) S 22612 22614 22174 0 -1 4202560 32 935 0 0 0 0 0 0 20 0 9 0 649077589 483012608 18635 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 7 0 0 0
[pid=22614/tid=22639] ppid=22612 vsize=471692 CPUtime=0 cores=0-7
/proc/22614/task/22639/stat : 22639 (python) S 22612 22614 22174 0 -1 4202560 34 935 0 0 0 0 0 0 19 0 9 0 649077592 483012608 18635 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 1 0 0 0
[pid=22614/tid=22642] ppid=22612 vsize=471692 CPUtime=0 cores=0-7
/proc/22614/task/22642/stat : 22642 (python) S 22612 22614 22174 0 -1 4202560 34 935 0 0 0 0 0 0 21 0 9 0 649077596 483012608 18635 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 23.41
Current children cumulated vsize (KiB) 5954920

[startup+25.5013 s]
/proc/loadavg: 7.23 7.86 6.74 9/421 22644
/proc/meminfo: memFree=18841336/32951124 swapFree=63083184/67111528
[pid=22614] ppid=22612 vsize=471692 CPUtime=11.17 cores=0-7
/proc/22614/stat : 22614 (python) S 22612 22614 22174 0 -1 4202496 33307 935 1 0 1102 15 0 0 21 0 9 0 649076469 483012608 18635 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738231755 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/22614/statm: 117923 18635 1889 364 0 53820 0
[pid=22620] ppid=22614 vsize=324120 CPUtime=0.03 cores=0-7
/proc/22620/stat : 22620 (python) S 22614 22614 22174 0 -1 4202560 897 0 0 0 1 2 0 0 15 0 1 0 649077573 331898880 16959 33554432000 4194304 5683644 140733469874736 18446744073709551615 264726425103 0 0 16781312 514 15343500047660581122 0 0 17 1 0 0 0
/proc/22620/statm: 81030 16959 376 364 0 16948 0
[pid=22622] ppid=22620 vsize=276236 CPUtime=14.44 cores=0-7
/proc/22622/stat : 22622 (hybridGM3) R 22620 22622 22622 0 -1 4202496 9522 0 0 0 1443 1 0 0 25 0 1 0 649077574 282865664 9257 33554432000 134512640 135204385 4287680432 18446744073709551615 134537510 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/22622/statm: 69059 9257 64 169 0 68889 0
[pid=22623] ppid=22614 vsize=334376 CPUtime=0.03 cores=0-7
/proc/22623/stat : 22623 (python) S 22614 22614 22174 0 -1 4202560 912 0 0 0 2 1 0 0 15 0 1 0 649077576 342401024 16957 33554432000 4194304 5683644 140733469874736 18446744073709551615 264726425158 0 0 16781312 514 15343500047660581121 0 0 17 1 0 0 0
/proc/22623/statm: 83594 16957 370 364 0 19509 0
[pid=22625] ppid=22623 vsize=276240 CPUtime=14.42 cores=0-7
/proc/22625/stat : 22625 (hybridGM3) R 22623 22625 22625 0 -1 4202496 9513 0 0 0 1441 1 0 0 25 0 1 0 649077577 282869760 9249 33554432000 134512640 135204385 4291001904 18446744073709551615 134536993 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/22625/statm: 69060 9249 64 169 0 68890 0
[pid=22626] ppid=22614 vsize=410168 CPUtime=0.03 cores=0-7
/proc/22626/stat : 22626 (python) S 22614 22614 22174 0 -1 4202560 782 0 0 0 1 2 0 0 15 0 1 0 649077579 420012032 17135 33554432000 4194304 5683644 140733469874736 18446744073709551615 264726425158 0 0 16781312 514 15343500047660581122 0 0 17 1 0 0 0
/proc/22626/statm: 102542 17135 371 364 0 38454 0
[pid=22628] ppid=22626 vsize=276240 CPUtime=14.37 cores=0-7
/proc/22628/stat : 22628 (hybridGM3) R 22626 22628 22628 0 -1 4202496 9505 0 0 0 1436 1 0 0 25 0 1 0 649077580 282869760 9250 33554432000 134512640 135204385 4289151376 18446744073709551615 134537403 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/22628/statm: 69060 9250 64 169 0 68890 0
[pid=22629] ppid=22614 vsize=420424 CPUtime=0.03 cores=0-7
/proc/22629/stat : 22629 (python) S 22614 22614 22174 0 -1 4202560 797 0 0 0 1 2 0 0 15 0 1 0 649077583 430514176 17099 33554432000 4194304 5683644 140733469874736 18446744073709551615 264726425158 0 0 16781312 514 15343500047660581122 0 0 17 1 0 0 0
/proc/22629/statm: 105106 17099 371 364 0 41015 0
[pid=22631] ppid=22629 vsize=276244 CPUtime=14.36 cores=0-7
/proc/22631/stat : 22631 (hybridGM3) R 22629 22631 22631 0 -1 4202496 9510 0 0 0 1435 1 0 0 25 0 1 0 649077583 282873856 9255 33554432000 134512640 135204385 4291540288 18446744073709551615 134537224 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22631/statm: 69061 9255 64 169 0 68891 0
[pid=22632] ppid=22614 vsize=430680 CPUtime=0.03 cores=0-7
/proc/22632/stat : 22632 (python) S 22614 22614 22174 0 -1 4202560 802 0 0 0 1 2 0 0 15 0 1 0 649077586 441016320 17135 33554432000 4194304 5683644 140733469874736 18446744073709551615 264726425158 0 0 16781312 514 15343500047660581124 0 0 17 1 0 0 0
/proc/22632/statm: 107670 17135 371 364 0 43576 0
[pid=22634] ppid=22632 vsize=276240 CPUtime=14.32 cores=0-7
/proc/22634/stat : 22634 (hybridGM3) R 22632 22634 22634 0 -1 4202496 9509 0 0 0 1431 1 0 0 25 0 1 0 649077586 282869760 9252 33554432000 134512640 135204385 4289432112 18446744073709551615 134537403 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/22634/statm: 69060 9252 64 169 0 68890 0

################
# More data... #
################

[pid=22730] ppid=22614 vsize=625544 CPUtime=2.08 cores=0-7
/proc/22730/stat : 22730 (python) S 22614 22614 22174 0 -1 4202560 1203 0 0 0 109 99 0 0 15 0 2 0 649146738 640557056 17491 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738231755 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/22730/statm: 156386 17491 407 364 0 92235 0
[pid=22732] ppid=22730 vsize=276236 CPUtime=587.19 cores=0-7
/proc/22732/stat : 22732 (hybridGM3) T 22730 22732 22732 0 -1 4202496 9537 0 0 0 58718 1 0 0 25 0 1 0 649146739 282865664 9252 33554432000 134512640 135204385 4291883888 18446744073709551615 134537573 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/22732/statm: 69059 9252 64 169 0 68889 0
[pid=22730/tid=22751] ppid=22614 vsize=625544 CPUtime=0 cores=0-7
/proc/22730/task/22751/stat : 22751 (python) S 22614 22614 22174 0 -1 4202560 88 0 0 0 0 0 0 0 15 0 2 0 649167024 640557056 17491 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 514 18446604448765799104 0 0 -1 7 0 0 0
[pid=22734] ppid=22614 vsize=635800 CPUtime=0.66 cores=0-7
/proc/22734/stat : 22734 (python) S 22614 22614 22174 0 -1 4202560 1183 0 0 0 34 32 0 0 17 0 2 0 649147355 651059200 17536 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738231755 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/22734/statm: 158950 17536 400 364 0 94796 0
[pid=22736] ppid=22734 vsize=276240 CPUtime=196.17 cores=0-7
/proc/22736/stat : 22736 (hybridGM3) T 22734 22736 22736 0 -1 4202496 9536 0 0 0 19615 2 0 0 25 0 1 0 649147355 282869760 9249 33554432000 134512640 135204385 4289570592 18446744073709551615 134545851 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/22736/statm: 69060 9249 64 169 0 68890 0
[pid=22734/tid=22752] ppid=22614 vsize=635800 CPUtime=0 cores=0-7
/proc/22734/task/22752/stat : 22752 (python) S 22614 22614 22174 0 -1 4202560 84 0 0 0 0 0 0 0 17 0 2 0 649167043 651059200 17536 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 514 18446604453979477888 0 0 -1 6 0 0 0
[pid=22741] ppid=22614 vsize=646056 CPUtime=0.66 cores=0-7
/proc/22741/stat : 22741 (python) S 22614 22614 22174 0 -1 4202560 1198 0 0 0 36 30 0 0 19 0 2 0 649156266 661561344 17531 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738231755 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/22741/statm: 161514 17531 400 364 0 97357 0
[pid=22743] ppid=22741 vsize=438604 CPUtime=196.48 cores=0-7
/proc/22743/stat : 22743 (adaptg2wsat2009) T 22741 22743 22743 0 -1 4202496 8784 0 0 0 19647 1 0 0 25 0 1 0 649156266 449130496 8499 33554432000 134512640 135107411 4293644272 18446744073709551615 134532277 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/22743/statm: 109651 8499 59 146 0 108527 0
[pid=22741/tid=22762] ppid=22614 vsize=646056 CPUtime=0 cores=0-7
/proc/22741/task/22762/stat : 22762 (python) S 22614 22614 22174 0 -1 4202560 85 0 0 0 0 0 0 0 19 0 2 0 649175929 661561344 17531 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 514 18446604446922948736 0 0 -1 6 0 0 0
[pid=22748] ppid=22614 vsize=656312 CPUtime=0.72 cores=0-7
/proc/22748/stat : 22748 (python) S 22614 22614 22174 0 -1 4202560 1210 0 0 0 37 35 0 0 17 0 2 0 649166951 672063488 17539 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738231755 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/22748/statm: 164078 17539 400 364 0 99918 0
[pid=22750] ppid=22748 vsize=276244 CPUtime=199.46 cores=0-7
/proc/22750/stat : 22750 (hybridGM3) T 22748 22750 22750 0 -1 4202496 9539 0 0 0 19944 2 0 0 25 0 1 0 649166951 282873856 9256 33554432000 134512640 135204385 4292785760 18446744073709551615 134536737 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/22750/statm: 69061 9256 64 169 0 68891 0
[pid=22748/tid=22771] ppid=22614 vsize=656312 CPUtime=0 cores=0-7
/proc/22748/task/22771/stat : 22771 (python) S 22614 22614 22174 0 -1 4202560 88 0 0 0 0 0 0 0 17 0 2 0 649187855 672063488 17539 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 514 18446604445872932864 0 0 -1 7 0 0 0
[pid=22753] ppid=22614 vsize=666568 CPUtime=2.36 cores=0-7
/proc/22753/stat : 22753 (python) S 22614 22614 22174 0 -1 4202560 1223 0 0 0 119 117 0 0 18 0 2 0 649167049 682565632 17600 33554432000 4194304 5683644 140733469874736 18446744073709551615 264726425158 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/22753/statm: 166642 17600 401 364 0 102479 0
[pid=22755] ppid=22753 vsize=438604 CPUtime=670.08 cores=0-7
/proc/22755/stat : 22755 (adaptg2wsat2009) R 22753 22755 22755 0 -1 4202496 8784 0 0 0 67006 2 0 0 25 0 1 0 649167049 449130496 8497 33554432000 134512640 135107411 4293696304 18446744073709551615 134531378 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22755/statm: 109651 8497 59 146 0 108527 0
[pid=22753/tid=22770] ppid=22614 vsize=666568 CPUtime=0 cores=0-7
/proc/22753/task/22770/stat : 22770 (python) S 22614 22614 22174 0 -1 4202560 88 0 0 0 0 0 0 0 18 0 2 0 649186715 682565632 17600 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 514 18446604445872931072 0 0 -1 0 0 0 0
[pid=22759] ppid=22614 vsize=676824 CPUtime=0.35 cores=0-7
/proc/22759/stat : 22759 (python) S 22614 22614 22174 0 -1 4202560 1236 0 0 0 18 17 0 0 19 0 2 0 649175859 693067776 17560 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738231755 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/22759/statm: 169206 17560 406 364 0 105040 0
[pid=22761] ppid=22759 vsize=266472 CPUtime=99.35 cores=0-7
/proc/22761/stat : 22761 (FH) T 22759 22761 22761 0 -1 4202496 8673 0 0 0 9933 2 0 0 25 0 1 0 649175860 272867328 8380 33554432000 134512640 134936988 4293893904 18446744073709551615 134554998 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/22761/statm: 66618 8380 41 104 0 66122 0
[pid=22759/tid=22765] ppid=22614 vsize=676824 CPUtime=0 cores=0-7
/proc/22759/task/22765/stat : 22765 (python) S 22614 22614 22174 0 -1 4202560 84 0 0 0 0 0 0 0 19 0 2 0 649185800 693067776 17560 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 514 18446604449318678848 0 0 -1 4 0 0 0
[pid=22766] ppid=22614 vsize=687080 CPUtime=1.66 cores=0-7
/proc/22766/stat : 22766 (python) S 22614 22614 22174 0 -1 4202560 1257 0 0 0 82 84 0 0 18 0 2 0 649185806 703569920 17594 33554432000 4194304 5683644 140733469874736 18446744073709551615 264726425158 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/22766/statm: 171770 17594 406 364 0 107601 0
[pid=22768] ppid=22766 vsize=471116 CPUtime=469.79 cores=0-7
/proc/22768/stat : 22768 (TNM) R 22766 22768 22768 0 -1 4202496 8461 0 0 0 46978 1 0 0 25 0 1 0 649185806 482422784 8172 33554432000 134512640 135008748 4288687488 18446744073709551615 134527693 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/22768/statm: 117779 8172 46 122 0 116679 0
[pid=22766/tid=22777] ppid=22614 vsize=687080 CPUtime=0 cores=0-7
/proc/22766/task/22777/stat : 22777 (python) S 22614 22614 22174 0 -1 4202560 89 0 0 0 0 0 0 0 19 0 2 0 649195841 703569920 17594 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 514 18446604449318680640 0 0 -1 0 0 0 0
[pid=22772] ppid=22614 vsize=697336 CPUtime=0.35 cores=0-7
/proc/22772/stat : 22772 (python) S 22614 22614 22174 0 -1 4202560 1260 0 0 0 18 17 0 0 19 0 2 0 649187861 714072064 17605 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738231755 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/22772/statm: 174334 17605 407 364 0 110162 0
[pid=22774] ppid=22772 vsize=471116 CPUtime=99.08 cores=0-7
/proc/22774/stat : 22774 (TNM) T 22772 22774 22774 0 -1 4202496 8457 0 0 0 9906 2 0 0 25 0 1 0 649187862 482422784 8171 33554432000 134512640 135008748 4290098832 18446744073709551615 134529367 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/22774/statm: 117779 8171 46 122 0 116679 0
[pid=22772/tid=22784] ppid=22614 vsize=697336 CPUtime=0 cores=0-7
/proc/22772/task/22784/stat : 22784 (python) S 22614 22614 22174 0 -1 4202560 90 0 0 0 0 0 0 0 19 0 2 0 649197896 714072064 17605 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 514 18446604448793892992 0 0 -1 1 0 0 0
[pid=22778] ppid=22614 vsize=707592 CPUtime=0.69 cores=0-7
/proc/22778/stat : 22778 (python) S 22614 22614 22174 0 -1 4202560 1266 0 0 0 32 37 0 0 19 0 2 0 649195847 724574208 17702 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738231755 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/22778/statm: 176898 17702 406 364 0 112723 0
[pid=22780] ppid=22778 vsize=438604 CPUtime=194.61 cores=0-7
/proc/22780/stat : 22780 (adaptg2wsat2009) T 22778 22780 22780 0 -1 4202496 8794 0 0 0 19459 2 0 0 25 0 1 0 649195848 449130496 8498 33554432000 134512640 135107411 4287182272 18446744073709551615 134531627 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/22780/statm: 109651 8498 59 146 0 108527 0
[pid=22778/tid=22791] ppid=22614 vsize=707592 CPUtime=0 cores=0-7
/proc/22778/task/22791/stat : 22791 (python) S 22614 22614 22174 0 -1 4202560 86 0 0 0 0 0 0 0 19 0 2 0 649215320 724574208 17702 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 514 18446604448793891200 0 0 -1 4 0 0 0
[pid=22781] ppid=22614 vsize=717848 CPUtime=0.35 cores=0-7
/proc/22781/stat : 22781 (python) S 22614 22614 22174 0 -1 4202560 1273 0 0 0 16 19 0 0 19 0 2 0 649196454 735076352 17685 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738231755 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/22781/statm: 179462 17685 406 364 0 115284 0
[pid=22783] ppid=22781 vsize=471116 CPUtime=99.33 cores=0-7
/proc/22783/stat : 22783 (TNM) T 22781 22783 22783 0 -1 4202496 8459 0 0 0 9932 1 0 0 25 0 1 0 649196455 482422784 8169 33554432000 134512640 135008748 4292570592 18446744073709551615 134529123 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/22783/statm: 117779 8169 46 122 0 116679 0
[pid=22781/tid=22788] ppid=22614 vsize=717848 CPUtime=0 cores=0-7
/proc/22781/task/22788/stat : 22788 (python) S 22614 22614 22174 0 -1 4202560 92 0 0 0 0 0 0 0 19 0 2 0 649206489 735076352 17685 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 514 18446604449754884288 0 0 -1 1 0 0 0
[pid=22797] ppid=22614 vsize=728104 CPUtime=0 cores=0-7
/proc/22797/stat : 22797 (python) S 22614 22614 22174 0 -1 4202560 924 0 0 0 0 0 0 0 19 0 1 0 649234850 745578496 17572 33554432000 4194304 5683644 140733469874736 18446744073709551615 264726425158 0 0 16781312 514 2829914510649720831 0 0 17 6 0 0 0
/proc/22797/statm: 182026 17572 346 364 0 117845 0
[pid=22799] ppid=22797 vsize=415656 CPUtime=0.11 cores=0-7
/proc/22799/stat : 22799 (NCVWr) R 22797 22799 22799 0 -1 4202496 1788 0 0 0 11 0 0 0 21 0 1 0 649234851 425631744 1500 33554432000 134512640 135113558 4292299792 18446744073709551615 134640574 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/22799/statm: 103914 1500 55 147 0 102789 0
[pid=22614/tid=22621] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22621/stat : 22621 (python) S 22612 22614 22174 0 -1 4202560 42 935 0 0 0 0 0 0 18 0 35 0 649077573 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 4 0 0 0
[pid=22614/tid=22624] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22624/stat : 22624 (python) S 22612 22614 22174 0 -1 4202560 34 935 0 0 0 0 0 0 18 0 35 0 649077576 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 3 0 0 0
[pid=22614/tid=22627] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22627/stat : 22627 (python) S 22612 22614 22174 0 -1 4202560 98 935 0 0 0 0 0 0 15 0 35 0 649077580 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 0 0 0 0
[pid=22614/tid=22630] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22630/stat : 22630 (python) S 22612 22614 22174 0 -1 4202560 45 935 0 0 0 0 0 0 18 0 35 0 649077583 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 5 0 0 0
[pid=22614/tid=22633] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22633/stat : 22633 (python) S 22612 22614 22174 0 -1 4202560 53 935 0 0 0 0 0 0 17 0 35 0 649077586 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 6 0 0 0
[pid=22614/tid=22636] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22636/stat : 22636 (python) S 22612 22614 22174 0 -1 4202560 42 935 0 0 0 0 0 0 18 0 35 0 649077589 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 7 0 0 0
[pid=22614/tid=22639] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22639/stat : 22639 (python) S 22612 22614 22174 0 -1 4202560 51 935 0 0 0 0 0 0 18 0 35 0 649077592 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 1 0 0 0
[pid=22614/tid=22642] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22642/stat : 22642 (python) S 22612 22614 22174 0 -1 4202560 52 935 0 0 0 0 0 0 15 0 35 0 649077596 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 3 0 0 0
[pid=22614/tid=22659] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22659/stat : 22659 (python) S 22612 22614 22174 0 -1 4202560 43 935 0 0 0 0 0 0 18 0 35 0 649097157 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 3 0 0 0
[pid=22614/tid=22668] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22668/stat : 22668 (python) S 22612 22614 22174 0 -1 4202560 50 935 0 0 0 0 0 0 18 0 35 0 649107284 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 1 0 0 0
[pid=22614/tid=22671] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22671/stat : 22671 (python) S 22612 22614 22174 0 -1 4202560 55 935 0 0 0 0 0 0 18 0 35 0 649107288 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 2 0 0 0
[pid=22614/tid=22674] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22674/stat : 22674 (python) S 22612 22614 22174 0 -1 4202560 36 935 0 0 0 0 0 0 18 0 35 0 649107295 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 2 0 0 0
[pid=22614/tid=22680] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22680/stat : 22680 (python) S 22612 22614 22174 0 -1 4202560 40 935 0 0 0 0 0 0 20 0 35 0 649116725 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 3 0 0 0
[pid=22614/tid=22683] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22683/stat : 22683 (python) S 22612 22614 22174 0 -1 4202560 33 935 0 0 0 0 0 0 20 0 35 0 649116832 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 2 0 0 0
[pid=22614/tid=22686] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22686/stat : 22686 (python) S 22612 22614 22174 0 -1 4202560 33 935 0 0 0 0 0 0 21 0 35 0 649116848 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 2 0 0 0
[pid=22614/tid=22690] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22690/stat : 22690 (python) S 22612 22614 22174 0 -1 4202560 34 935 0 0 0 0 0 0 20 0 35 0 649117743 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 2 0 0 0
[pid=22614/tid=22697] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22697/stat : 22697 (python) S 22612 22614 22174 0 -1 4202560 37 935 0 0 0 0 0 0 20 0 35 0 649126761 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 1 0 0 0
[pid=22614/tid=22701] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22701/stat : 22701 (python) S 22612 22614 22174 0 -1 4202560 32 935 0 0 0 0 0 0 19 0 35 0 649126880 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 2 0 0 0
[pid=22614/tid=22708] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22708/stat : 22708 (python) S 22612 22614 22174 0 -1 4202560 58 935 0 0 0 0 0 0 18 0 35 0 649127683 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 2 0 0 0
[pid=22614/tid=22713] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22713/stat : 22713 (python) S 22612 22614 22174 0 -1 4202560 43 935 0 0 0 0 0 0 18 0 35 0 649136515 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 1 0 0 0
[pid=22614/tid=22717] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22717/stat : 22717 (python) S 22612 22614 22174 0 -1 4202560 38 935 0 0 0 0 0 0 20 0 35 0 649136699 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 2 0 0 0
[pid=22614/tid=22720] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22720/stat : 22720 (python) S 22612 22614 22174 0 -1 4202560 34 935 0 0 0 0 0 0 21 0 35 0 649136712 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 2 0 0 0
[pid=22614/tid=22724] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22724/stat : 22724 (python) S 22612 22614 22174 0 -1 4202560 40 935 0 0 0 0 0 0 18 0 35 0 649137723 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 6 0 0 0
[pid=22614/tid=22731] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22731/stat : 22731 (python) S 22612 22614 22174 0 -1 4202560 54 935 0 0 0 0 0 0 18 0 35 0 649146738 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 1 0 0 0
[pid=22614/tid=22735] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22735/stat : 22735 (python) S 22612 22614 22174 0 -1 4202560 32 935 0 0 0 0 0 0 20 0 35 0 649147355 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 6 0 0 0
[pid=22614/tid=22742] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22742/stat : 22742 (python) S 22612 22614 22174 0 -1 4202560 34 935 0 0 0 0 0 0 20 0 35 0 649156266 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 1 0 0 0
[pid=22614/tid=22749] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22749/stat : 22749 (python) S 22612 22614 22174 0 -1 4202560 37 935 0 0 0 0 0 0 20 0 35 0 649166951 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 1 0 0 0
[pid=22614/tid=22754] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22754/stat : 22754 (python) S 22612 22614 22174 0 -1 4202560 44 935 0 0 0 0 0 0 18 0 35 0 649167049 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 6 0 0 0
[pid=22614/tid=22760] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22760/stat : 22760 (python) S 22612 22614 22174 0 -1 4202560 34 935 0 0 0 0 0 0 20 0 35 0 649175859 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 1 0 0 0
[pid=22614/tid=22767] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22767/stat : 22767 (python) S 22612 22614 22174 0 -1 4202560 42 935 0 0 0 0 0 0 18 0 35 0 649185806 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 1 0 0 0
[pid=22614/tid=22773] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22773/stat : 22773 (python) S 22612 22614 22174 0 -1 4202560 33 935 0 0 0 0 0 0 20 0 35 0 649187862 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 6 0 0 0
[pid=22614/tid=22779] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22779/stat : 22779 (python) S 22612 22614 22174 0 -1 4202560 35 935 0 0 0 0 0 0 20 0 35 0 649195848 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 1 0 0 0
[pid=22614/tid=22782] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22782/stat : 22782 (python) S 22612 22614 22174 0 -1 4202560 34 935 0 0 0 0 0 0 19 0 35 0 649196454 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 6 0 0 0
[pid=22614/tid=22798] ppid=22612 vsize=738348 CPUtime=0 cores=0-7
/proc/22614/task/22798/stat : 22798 (python) S 22612 22614 22174 0 -1 4202560 37 935 0 0 0 0 0 0 20 0 35 0 649234851 756068352 19172 33554432000 4194304 5683644 140733469874736 18446744073709551615 264738228657 0 0 16781312 2 18446604453947385920 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 12528.2
Current children cumulated vsize (KiB) 32324088

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 22614 and gives
#  childrusage.ru_utime.tv_sec=3338
#  childrusage.ru_utime.tv_usec=975398
#  childrusage.ru_stime.tv_sec=22
#  childrusage.ru_stime.tv_usec=308608
# CPU time returned by wait4() is 3361.28
# while last known CPU time is 12528.2
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1584.13
CPU time (s): 12528.2
CPU user time (s): 12505.7
CPU system time (s): 22.43
CPU usage (%): 790.856
Max. virtual memory (cumulated for all children) (KiB): 32324088

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 3338.98
system time used= 22.3086
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 159986
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 18527
involuntary context switches= 13755

runsolver used 17.0624 second user time and 46.103 second system time

The end

Launcher Data 

Begin job on node141 at 2011-05-16 19:23:28
IDJOB=3385436
IDBENCH=82701
IDSOLVER=1870
FILE ID=node141/3385436-1305566608
RUNJOBID= node141-1305564961-22188
PBS_JOBID= 13350109
Free space on /tmp= 73280 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v50000-c210000-S964643921-035-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3385436-1305566608/watcher-3385436-1305566608 -o /tmp/evaluation-result-3385436-1305566608/solver-3385436-1305566608 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385436-1305566608.cnf -seed 1321995845 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= d8337de8a71a630ccf2dd02943cd7d7f
RANDOM SEED=1321995845

node141.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5362.96
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      19233924 kB
Buffers:        291176 kB
Cached:        5669588 kB
SwapCached:      15716 kB
Active:        8059668 kB
Inactive:      5387464 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      19233924 kB
SwapTotal:    67111528 kB
SwapFree:     63083184 kB
Dirty:           36440 kB
Writeback:           0 kB
AnonPages:     7478728 kB
Mapped:          17292 kB
Slab:           148032 kB
PageTables:      52820 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 104320484 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73248 MiB
End job on node141 at 2011-05-16 19:49:55