Trace number 3385343

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (MO) 7175.13 908.72

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v40000-c168000-S1184456903-078-UNKNOWN.cnf
MD5SUM33dd7efc1f63d9d3511c7a69c68ea88b
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark108.505
Satisfiable
(Un)Satisfiability was proved
Number of variables40000
Number of clauses168000
Sum of the clauses size504000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3168000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.35/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.35/0.67	c INFO: solving HOME/instance-3385343-1305565332.cnf
0.35/0.67	c INFO: computing task features
0.93/1.24	c INFO: computed variable-clause graph statistics
0.93/1.27	c INFO: computed clause balance statistics
3.82/4.13	c INFO: computed variable balance statistics
6.12/6.44	c INFO: computed small-clause counts
6.12/6.45	c INFO: computed Horn-clause counts
8.12/8.46	c INFO: computed variable graph statistics
8.12/8.46	c INFO: collected features for HOME/instance-3385343-1305565332.cnf in 7.76 s
8.12/8.49	c INFO: running hybridGM3@0 for 97 with 4992 remaining (b = 0.35)
8.22/8.52	c INFO: running hybridGM3@0 for 97 with 4992 remaining (b = 0.21)
8.22/8.55	c INFO: running hybridGM3@0 for 97 with 4992 remaining (b = 0.14)
8.22/8.58	c INFO: running hybridGM3@0 for 97 with 4992 remaining (b = 0.10)
8.33/8.61	c INFO: running hybridGM3@0 for 97 with 4992 remaining (b = 0.07)
8.33/8.65	c INFO: running hybridGM3@0 for 194 with 4992 remaining (b = 0.10)
8.33/8.68	c INFO: running hybridGM3@0 for 194 with 4992 remaining (b = 0.08)
9.20/8.71	c INFO: running hybridGM3@0 for 194 with 4991 remaining (b = 0.06)
802.75/108.28	c INFO: running hybridGM3@0 for 97 with 4892 remaining (b = 0.03)
810.64/109.21	c INFO: running hybridGM3@0 for 97 with 4891 remaining (b = 0.03)
810.64/109.25	c INFO: running hybridGM3@0 for 97 with 4891 remaining (b = 0.03)
810.64/109.30	c INFO: running hybridGM3@0 for 97 with 4891 remaining (b = 0.03)
811.43/109.34	c INFO: running hybridGM3@0 for 97 with 4891 remaining (b = 0.03)
1426.46/186.46	c INFO: running TNM@0 for 97 with 4814 remaining (b = 0.02)
1562.94/203.58	c INFO: running hybridGM3@100 for 97 with 4797 remaining (b = 0.03)
1570.03/204.51	c INFO: running hybridGM3@100 for 97 with 4796 remaining (b = 0.02)
1570.91/204.60	c INFO: running hybridGM3@100 for 97 with 4796 remaining (b = 0.02)
1571.70/204.64	c INFO: running hybridGM3@76 for 97 with 4796 remaining (b = 0.02)
1579.69/205.67	c INFO: running hybridGM3@0 for 97 with 4795 remaining (b = 0.02)
1579.69/205.71	c INFO: running hybridGM3@0 for 97 with 4794 remaining (b = 0.02)
1628.27/211.74	c INFO: running hybridGM3@99 for 97 with 4788 remaining (b = 0.02)
2226.48/286.79	c INFO: running hybridGM3@0 for 194 with 4713 remaining (b = 0.04)
2371.61/304.92	c INFO: running hybridGM3@0 for 194 with 4695 remaining (b = 0.03)
2378.86/305.84	c INFO: running hybridGM3@0 for 194 with 4694 remaining (b = 0.03)
2379.50/305.92	c INFO: running TNM@0 for 97 with 4694 remaining (b = 0.01)
2379.50/305.97	c INFO: running TNM@0 for 97 with 4694 remaining (b = 0.01)
2388.30/307.00	c INFO: running TNM@0 for 97 with 4693 remaining (b = 0.01)
2388.30/307.05	c INFO: running hybridGM3@0 for 194 with 4693 remaining (b = 0.02)
2388.90/307.10	c INFO: running TNM@0 for 97 with 4693 remaining (b = 0.01)
3178.44/406.28	c INFO: running TNM@0 for 97 with 4594 remaining (b = 0.01)
3179.92/406.41	c INFO: running TNM@0 for 97 with 4594 remaining (b = 0.01)
3187.14/407.38	c INFO: running TNM@0 for 97 with 4593 remaining (b = 0.01)
3188.62/407.50	c INFO: running TNM@0 for 97 with 4593 remaining (b = 0.01)
3797.44/483.93	c INFO: running hybridGM3@101 for 97 with 4516 remaining (b = 0.01)
3941.50/502.09	c INFO: running hybridGM3@100 for 97 with 4498 remaining (b = 0.01)
3948.64/502.93	c INFO: running hybridGM3@101 for 97 with 4497 remaining (b = 0.01)
3958.20/504.17	c INFO: running hybridGM3@94 for 97 with 4496 remaining (b = 0.01)
3978.91/506.80	c INFO: running hybridGM3@0 for 194 with 4493 remaining (b = 0.02)
3986.87/507.75	c INFO: running TNM@0 for 97 with 4492 remaining (b = 0.01)
3987.62/507.88	c INFO: running TNM@196 for 1843 with 4492 remaining (b = 0.15)
4025.97/512.69	c INFO: running hybridGM3@0 for 194 with 4488 remaining (b = 0.02)
4555.66/579.29	c INFO: running hybridGM3@298 for 97 with 4421 remaining (b = 0.01)
4700.43/597.47	c INFO: running hybridGM3@297 for 97 with 4403 remaining (b = 0.01)
4706.80/598.29	c INFO: running hybridGM3@296 for 194 with 4402 remaining (b = 0.02)
4764.89/605.55	c INFO: running adaptg2wsat2009++@0 for 97 with 4395 remaining (b = 0.01)
4785.55/608.14	c INFO: running adaptg2wsat2009++@0 for 194 with 4392 remaining (b = 0.01)
5362.37/680.67	c INFO: running hybridGM3@393 for 97 with 4320 remaining (b = 0.01)
5507.16/698.87	c INFO: running hybridGM3@392 for 97 with 4301 remaining (b = 0.01)
5547.85/703.99	c INFO: running hybridGM3@98 for 97 with 4296 remaining (b = 0.01)
5555.70/704.95	c INFO: running hybridGM3@0 for 194 with 4295 remaining (b = 0.01)
5595.42/709.90	c INFO: running hybridGM3@94 for 97 with 4290 remaining (b = 0.01)
5605.76/711.26	c INFO: running adaptg2wsat2009++@100 for 97 with 4289 remaining (b = 0.01)
5797.49/735.35	c INFO: running TNM@0 for 97 with 4265 remaining (b = 0.01)
6121.34/776.03	c INFO: running TNM@0 for 97 with 4224 remaining (b = 0.01)
6266.21/794.24	c INFO: running hybridGM3@0 for 194 with 4206 remaining (b = 0.01)
6350.43/804.86	c INFO: running adaptg2wsat2009++@101 for 97 with 4195 remaining (b = 0.01)
6354.39/805.37	c INFO: running TNM@0 for 97 with 4195 remaining (b = 0.01)
6372.68/807.66	c INFO: running adaptg2wsat2009++@123 for 97 with 4193 remaining (b = 0.01)
6596.22/835.78	c INFO: running TNM@0 for 97 with 4164 remaining (b = 0.01)
6919.90/876.43	c INFO: running adaptg2wsat2009++@0 for 194 with 4124 remaining (b = 0.01)
7109.02/900.25	c INFO: running hybridGM3@195 for 291 with 4100 remaining (b = 0.02)
7124.07/902.13	c INFO: running TNM@0 for 97 with 4098 remaining (b = 0.01)
7152.66/905.78	c INFO: running TNM@0 for 97 with 4094 remaining (b = 0.01)
7170.95/908.08	c INFO: running TNM@0 for 97 with 4092 remaining (b = 0.00)
7175.13/908.62	c INFO: running hybridGM3@195 for 291 with 4092 remaining (b = 0.01)
7175.13/908.62	Traceback (most recent call last):
7175.13/908.62	  File "HOME/solve", line 33, in <module>
7175.13/908.62	    plac.call(main)
7175.13/908.62	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
7175.13/908.62	    cmd, result = parser_from(obj).consume(arglist)
7175.13/908.62	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
7175.13/908.62	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
7175.13/908.62	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
7175.13/908.62	    (_, answer, _) = solver(input_path, budget / cores, cores)
7175.13/908.62	  File "HOME/borg/build/lib/borg/bilevel.py", line 202, in __call__
7175.13/908.62	    solver.go(max_cost * borg.defaults.machine_speed)
7175.13/908.62	  File "HOME/borg/build/lib/borg/solvers.py", line 179, in go
7175.13/908.62	    self._process.start()
7175.13/908.62	  File "/home/evaluation/evaluation/tools64/lib/python2.7/multiprocessing/process.py", line 94, in start
7175.13/908.62	    assert self._popen is None, 'cannot start a process twice'
7175.13/908.62	AssertionError: cannot start a process twice

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3385343-1305565332/watcher-3385343-1305565332 -o /tmp/evaluation-result-3385343-1305565332/solver-3385343-1305565332 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385343-1305565332.cnf -seed 465104676 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.13 5.55 2.52 2/392 29852
/proc/meminfo: memFree=11986660/32951124 swapFree=56316216/67111528
[pid=29852] ppid=29850 vsize=15772 CPUtime=0 cores=0-7
/proc/29852/stat : 29852 (python) S 29850 29852 29579 0 -1 4202496 934 0 0 0 0 0 0 0 20 0 1 0 668135622 16150528 813 33554432000 4194304 5683644 140736284179984 18446744073709551615 212436015104 0 2147483391 16781312 2 18446744071699207358 0 0 17 6 0 0 0
/proc/29852/statm: 3943 813 356 364 0 430 0

[startup+0.0395879 s]
/proc/loadavg: 8.13 5.55 2.52 2/392 29852
/proc/meminfo: memFree=11986660/32951124 swapFree=56316216/67111528
[pid=29852] ppid=29850 vsize=18880 CPUtime=0.01 cores=0-7
/proc/29852/stat : 29852 (python) R 29850 29852 29579 0 -1 4202496 1271 0 0 0 1 0 0 0 20 0 1 0 668135622 19333120 1124 33554432000 4194304 5683644 140736284179984 18446744073709551615 4540219 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/29852/statm: 4720 1124 421 364 0 689 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 18880

[startup+0.100554 s]
/proc/loadavg: 8.13 5.55 2.52 2/392 29852
/proc/meminfo: memFree=11986660/32951124 swapFree=56316216/67111528
[pid=29852] ppid=29850 vsize=87700 CPUtime=0.05 cores=0-7
/proc/29852/stat : 29852 (python) S 29850 29852 29579 0 -1 4202496 2637 0 0 0 4 1 0 0 20 0 1 0 668135622 89804800 2353 33554432000 4194304 5683644 140736284179984 18446744073709551615 212431093383 0 2147483391 16781312 2 18446744071699207358 0 0 17 6 0 0 0
/proc/29852/statm: 21925 2353 717 364 0 1621 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 87700

[startup+0.300525 s]
/proc/loadavg: 8.13 5.55 2.52 2/392 29852
/proc/meminfo: memFree=11986660/32951124 swapFree=56316216/67111528
[pid=29852] ppid=29850 vsize=135748 CPUtime=0.18 cores=0-7
/proc/29852/stat : 29852 (python) R 29850 29852 29579 0 -1 4202496 5963 935 1 0 13 5 0 0 20 0 1 0 668135622 139005952 4150 33554432000 4194304 5683644 140736284179984 18446744073709551615 212435822363 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/29852/statm: 33937 4150 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.18
Current children cumulated vsize (KiB) 135748

[startup+0.700437 s]
/proc/loadavg: 8.13 5.55 2.52 2/392 29852
/proc/meminfo: memFree=11986660/32951124 swapFree=56316216/67111528
[pid=29852] ppid=29850 vsize=283272 CPUtime=0.43 cores=0-7
/proc/29852/stat : 29852 (python) R 29850 29852 29579 0 -1 4202496 10459 935 1 0 32 11 0 0 19 0 1 0 668135622 290070528 8142 33554432000 4194304 5683644 140736284179984 18446744073709551615 212435703648 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/29852/statm: 70818 8143 1761 364 0 6742 0
Current children cumulated CPU time (s) 0.43
Current children cumulated vsize (KiB) 283272

[startup+1.5003 s]
/proc/loadavg: 8.13 5.55 2.52 2/393 29857
/proc/meminfo: memFree=11940908/32951124 swapFree=56316216/67111528
[pid=29852] ppid=29850 vsize=317984 CPUtime=1.22 cores=0-7
/proc/29852/stat : 29852 (python) R 29850 29852 29579 0 -1 4202496 23796 935 1 0 109 13 0 0 20 0 1 0 668135622 325615616 16881 33554432000 4194304 5683644 140736284179984 18446744073709551615 212435703562 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/29852/statm: 79496 16881 1826 364 0 15420 0
Current children cumulated CPU time (s) 1.22
Current children cumulated vsize (KiB) 317984

[startup+3.10098 s]
/proc/loadavg: 7.56 5.47 2.51 2/393 29858
/proc/meminfo: memFree=11924664/32951124 swapFree=56316216/67111528
[pid=29852] ppid=29850 vsize=317984 CPUtime=2.82 cores=0-7
/proc/29852/stat : 29852 (python) R 29850 29852 29579 0 -1 4202496 23796 935 1 0 269 13 0 0 25 0 1 0 668135622 325615616 16881 33554432000 4194304 5683644 140736284179984 18446744073709551615 4539735 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/29852/statm: 79496 16881 1826 364 0 15420 0
Current children cumulated CPU time (s) 2.82
Current children cumulated vsize (KiB) 317984

[startup+6.30039 s]
/proc/loadavg: 7.56 5.47 2.51 2/393 29858
/proc/meminfo: memFree=11925660/32951124 swapFree=56316216/67111528
[pid=29852] ppid=29850 vsize=317984 CPUtime=6.02 cores=0-7
/proc/29852/stat : 29852 (python) R 29850 29852 29579 0 -1 4202496 23797 935 1 0 589 13 0 0 25 0 1 0 668135622 325615616 16882 33554432000 4194304 5683644 140736284179984 18446744073709551615 4602388 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/29852/statm: 79496 16882 1827 364 0 15420 0
Current children cumulated CPU time (s) 6.02
Current children cumulated vsize (KiB) 317984

[startup+12.7011 s]
/proc/loadavg: 7.11 5.44 2.53 9/417 29882
/proc/meminfo: memFree=11634332/32951124 swapFree=56316216/67111528
[pid=29852] ppid=29850 vsize=462452 CPUtime=8.44 cores=0-7
/proc/29852/stat : 29852 (python) S 29850 29852 29579 0 -1 4202496 29608 935 1 0 828 16 0 0 24 0 9 0 668135622 473550848 16392 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447843787 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/29852/statm: 115613 16392 1889 364 0 51510 0
[pid=29859] ppid=29852 vsize=314880 CPUtime=0 cores=0-7
/proc/29859/stat : 29859 (python) S 29852 29852 29579 0 -1 4202560 737 0 0 0 0 0 0 0 15 0 1 0 668136472 322437120 14626 33554432000 4194304 5683644 140736284179984 18446744073709551615 212436037135 0 0 16781312 514 14479952415593300228 0 0 17 1 0 0 0
/proc/29859/statm: 78720 14626 353 364 0 14638 0
[pid=29861] ppid=29859 vsize=276236 CPUtime=4.2 cores=0-7
/proc/29861/stat : 29861 (hybridGM3) R 29859 29861 29861 0 -1 4202496 8563 0 0 0 419 1 0 0 25 0 1 0 668136472 282865664 8299 33554432000 134512640 135204385 4292885808 18446744073709551615 134536744 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/29861/statm: 69059 8299 64 169 0 68889 0
[pid=29862] ppid=29852 vsize=325136 CPUtime=0 cores=0-7
/proc/29862/stat : 29862 (python) S 29852 29852 29579 0 -1 4202560 740 0 0 0 0 0 0 0 15 0 1 0 668136475 332939264 14624 33554432000 4194304 5683644 140736284179984 18446744073709551615 212436037190 0 0 16781312 514 12397934789422711042 0 0 17 1 0 0 0
/proc/29862/statm: 81284 14624 347 364 0 17199 0
[pid=29864] ppid=29862 vsize=276236 CPUtime=4.16 cores=0-7
/proc/29864/stat : 29864 (hybridGM3) R 29862 29864 29864 0 -1 4202496 8563 0 0 0 415 1 0 0 25 0 1 0 668136475 282865664 8300 33554432000 134512640 135204385 4292992752 18446744073709551615 134537565 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/29864/statm: 69059 8300 64 169 0 68889 0
[pid=29865] ppid=29852 vsize=400928 CPUtime=0 cores=0-7
/proc/29865/stat : 29865 (python) S 29852 29852 29579 0 -1 4202560 709 0 0 0 0 0 0 0 15 0 1 0 668136478 410550272 14802 33554432000 4194304 5683644 140736284179984 18446744073709551615 212436037190 0 0 16781312 514 1980343591221821698 0 0 17 7 0 0 0
/proc/29865/statm: 100232 14802 348 364 0 36144 0
[pid=29867] ppid=29865 vsize=276240 CPUtime=4.13 cores=0-7
/proc/29867/stat : 29867 (hybridGM3) R 29865 29867 29867 0 -1 4202496 8552 0 0 0 412 1 0 0 25 0 1 0 668136478 282869760 8301 33554432000 134512640 135204385 4287149440 18446744073709551615 134537292 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/29867/statm: 69060 8301 64 169 0 68890 0
[pid=29868] ppid=29852 vsize=411184 CPUtime=0 cores=0-7
/proc/29868/stat : 29868 (python) S 29852 29852 29579 0 -1 4202560 720 0 0 0 0 0 0 0 15 0 1 0 668136481 421052416 14763 33554432000 4194304 5683644 140736284179984 18446744073709551615 212436037190 0 0 16781312 514 4435495881440985347 0 0 17 6 0 0 0
/proc/29868/statm: 102796 14763 348 364 0 38705 0
[pid=29870] ppid=29868 vsize=276244 CPUtime=4.1 cores=0-7
/proc/29870/stat : 29870 (hybridGM3) R 29868 29870 29870 0 -1 4202496 8556 0 0 0 409 1 0 0 25 0 1 0 668136481 282873856 8304 33554432000 134512640 135204385 4290520528 18446744073709551615 134537292 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/29870/statm: 69061 8304 64 169 0 68891 0
[pid=29871] ppid=29852 vsize=421440 CPUtime=0 cores=0-7
/proc/29871/stat : 29871 (python) S 29852 29852 29579 0 -1 4202560 728 0 0 0 0 0 0 0 15 0 1 0 668136484 431554560 14785 33554432000 4194304 5683644 140736284179984 18446744073709551615 212436037190 0 0 16781312 514 30087040477397252 0 0 17 1 0 0 0
/proc/29871/statm: 105360 14785 348 364 0 41266 0
[pid=29873] ppid=29871 vsize=276244 CPUtime=4.07 cores=0-7
/proc/29873/stat : 29873 (hybridGM3) R 29871 29873 29873 0 -1 4202496 8554 0 0 0 406 1 0 0 25 0 1 0 668136485 282873856 8300 33554432000 134512640 135204385 4294342320 18446744073709551615 134536641 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/29873/statm: 69061 8300 64 169 0 68891 0
[pid=29874] ppid=29852 vsize=431696 CPUtime=0 cores=0-7
/proc/29874/stat : 29874 (python) S 29852 29852 29579 0 -1 4202560 735 0 0 0 0 0 0 0 15 0 1 0 668136487 442056704 14806 33554432000 4194304 5683644 140736284179984 18446744073709551615 212436037190 0 0 16781312 514 10904050330996015362 0 0 17 1 0 0 0
/proc/29874/statm: 107924 14806 348 364 0 43827 0
[pid=29876] ppid=29874 vsize=276240 CPUtime=3.97 cores=0-7
/proc/29876/stat : 29876 (hybridGM3) R 29874 29876 29876 0 -1 4202496 8559 0 0 0 396 1 0 0 25 0 1 0 668136488 282869760 8298 33554432000 134512640 135204385 4291486720 18446744073709551615 134547802 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/29876/statm: 69060 8298 64 169 0 68890 0
[pid=29877] ppid=29852 vsize=441952 CPUtime=0 cores=0-7
/proc/29877/stat : 29877 (python) S 29852 29852 29579 0 -1 4202560 744 0 0 0 0 0 0 0 15 0 1 0 668136491 452558848 14827 33554432000 4194304 5683644 140736284179984 18446744073709551615 212436037190 0 0 16781312 514 17031435118545174786 0 0 17 1 0 0 0
/proc/29877/statm: 110488 14827 348 364 0 46388 0
[pid=29879] ppid=29877 vsize=276240 CPUtime=3.97 cores=0-7
/proc/29879/stat : 29879 (hybridGM3) R 29877 29879 29879 0 -1 4202496 8556 0 0 0 396 1 0 0 25 0 1 0 668136493 282869760 8294 33554432000 134512640 135204385 4286725280 18446744073709551615 134660134 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/29879/statm: 69060 8294 64 169 0 68890 0
[pid=29880] ppid=29852 vsize=452208 CPUtime=0 cores=0-7
/proc/29880/stat : 29880 (python) S 29852 29852 29579 0 -1 4202560 782 0 0 0 0 0 0 0 15 0 1 0 668136494 463060992 14853 33554432000 4194304 5683644 140736284179984 18446744073709551615 212436037190 0 0 16781312 514 15964710927509455105 0 0 17 7 0 0 0
/proc/29880/statm: 113052 14853 354 364 0 48949 0
[pid=29882] ppid=29880 vsize=276240 CPUtime=3.97 cores=0-7
/proc/29882/stat : 29882 (hybridGM3) R 29880 29882 29882 0 -1 4202496 8570 0 0 0 396 1 0 0 25 0 1 0 668136494 282869760 8304 33554432000 134512640 135204385 4290409488 18446744073709551615 134537272 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/29882/statm: 69060 8304 64 169 0 68890 0
[pid=29852/tid=29860] ppid=29850 vsize=462452 CPUtime=0 cores=0-7
/proc/29852/task/29860/stat : 29860 (python) S 29850 29852 29579 0 -1 4202560 26 935 0 0 0 0 0 0 25 0 9 0 668136472 473550848 16392 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 0 0 0 0
[pid=29852/tid=29863] ppid=29850 vsize=462452 CPUtime=0 cores=0-7
/proc/29852/task/29863/stat : 29863 (python) S 29850 29852 29579 0 -1 4202560 21 935 0 0 0 0 0 0 18 0 9 0 668136475 473550848 16392 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 3 0 0 0
[pid=29852/tid=29866] ppid=29850 vsize=462452 CPUtime=0 cores=0-7
/proc/29852/task/29866/stat : 29866 (python) S 29850 29852 29579 0 -1 4202560 80 935 0 0 0 0 0 0 19 0 9 0 668136478 473550848 16392 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 4 0 0 0
[pid=29852/tid=29869] ppid=29850 vsize=462452 CPUtime=0 cores=0-7
/proc/29852/task/29869/stat : 29869 (python) S 29850 29852 29579 0 -1 4202560 33 935 0 0 0 0 0 0 20 0 9 0 668136481 473550848 16392 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 4 0 0 0
[pid=29852/tid=29872] ppid=29850 vsize=462452 CPUtime=0 cores=0-7
/proc/29852/task/29872/stat : 29872 (python) S 29850 29852 29579 0 -1 4202560 33 935 0 0 0 0 0 0 21 0 9 0 668136484 473550848 16392 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 4 0 0 0
[pid=29852/tid=29875] ppid=29850 vsize=462452 CPUtime=0 cores=0-7
/proc/29852/task/29875/stat : 29875 (python) S 29850 29852 29579 0 -1 4202560 33 935 0 0 0 0 0 0 18 0 9 0 668136488 473550848 16392 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 7 0 0 0
[pid=29852/tid=29878] ppid=29850 vsize=462452 CPUtime=0 cores=0-7
/proc/29852/task/29878/stat : 29878 (python) S 29850 29852 29579 0 -1 4202560 34 935 0 0 0 0 0 0 23 0 9 0 668136491 473550848 16392 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 0 0 0 0
[pid=29852/tid=29881] ppid=29850 vsize=462452 CPUtime=0 cores=0-7
/proc/29852/task/29881/stat : 29881 (python) S 29850 29852 29579 0 -1 4202560 33 935 0 0 0 0 0 0 24 0 9 0 668136494 473550848 16392 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 41.01
Current children cumulated vsize (KiB) 5871796

[startup+25.5017 s]
/proc/loadavg: 7.25 5.53 2.59 11/417 29882
/proc/meminfo: memFree=11635324/32951124 swapFree=56316216/67111528
[pid=29852] ppid=29850 vsize=462452 CPUtime=8.44 cores=0-7
/proc/29852/stat : 29852 (python) S 29850 29852 29579 0 -1 4202496 29608 935 1 0 828 16 0 0 24 0 9 0 668135622 473550848 16392 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447843787 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/29852/statm: 115613 16392 1889 364 0 51510 0
[pid=29859] ppid=29852 vsize=314880 CPUtime=0.05 cores=0-7
/proc/29859/stat : 29859 (python) S 29852 29852 29579 0 -1 4202560 906 0 0 0 2 3 0 0 15 0 1 0 668136472 322437120 14649 33554432000 4194304 5683644 140736284179984 18446744073709551615 212436037135 0 0 16781312 514 13413197433937035524 0 0 17 1 0 0 0
/proc/29859/statm: 78720 14649 376 364 0 14638 0
[pid=29861] ppid=29859 vsize=276236 CPUtime=17 cores=0-7
/proc/29861/stat : 29861 (hybridGM3) R 29859 29861 29861 0 -1 4202496 8563 0 0 0 1699 1 0 0 25 0 1 0 668136472 282865664 8299 33554432000 134512640 135204385 4292885808 18446744073709551615 134537383 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/29861/statm: 69059 8299 64 169 0 68889 0
[pid=29862] ppid=29852 vsize=325136 CPUtime=0.05 cores=0-7
/proc/29862/stat : 29862 (python) S 29852 29852 29579 0 -1 4202560 923 0 0 0 2 3 0 0 15 0 1 0 668136475 332939264 14647 33554432000 4194304 5683644 140736284179984 18446744073709551615 212436037190 0 0 16781312 514 13413197433937035522 0 0 17 1 0 0 0
/proc/29862/statm: 81284 14647 370 364 0 17199 0
[pid=29864] ppid=29862 vsize=276236 CPUtime=16.96 cores=0-7
/proc/29864/stat : 29864 (hybridGM3) R 29862 29864 29864 0 -1 4202496 8563 0 0 0 1695 1 0 0 25 0 1 0 668136475 282865664 8300 33554432000 134512640 135204385 4292992752 18446744073709551615 134536641 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/29864/statm: 69059 8300 64 169 0 68889 0
[pid=29865] ppid=29852 vsize=400928 CPUtime=0.05 cores=0-7
/proc/29865/stat : 29865 (python) S 29852 29852 29579 0 -1 4202560 786 0 0 0 3 2 0 0 15 0 1 0 668136478 410550272 14825 33554432000 4194304 5683644 140736284179984 18446744073709551615 212436037190 0 0 16781312 514 13413197433937035522 0 0 17 7 0 0 0
/proc/29865/statm: 100232 14825 371 364 0 36144 0
[pid=29867] ppid=29865 vsize=276240 CPUtime=16.93 cores=0-7
/proc/29867/stat : 29867 (hybridGM3) R 29865 29867 29867 0 -1 4202496 8552 0 0 0 1692 1 0 0 25 0 1 0 668136478 282869760 8301 33554432000 134512640 135204385 4287149440 18446744073709551615 134536663 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/29867/statm: 69060 8301 64 169 0 68890 0
[pid=29868] ppid=29852 vsize=411184 CPUtime=0.05 cores=0-7
/proc/29868/stat : 29868 (python) S 29852 29852 29579 0 -1 4202560 800 0 0 0 2 3 0 0 15 0 1 0 668136481 421052416 14786 33554432000 4194304 5683644 140736284179984 18446744073709551615 212436037190 0 0 16781312 514 13413197433937035523 0 0 17 6 0 0 0
/proc/29868/statm: 102796 14786 371 364 0 38705 0
[pid=29870] ppid=29868 vsize=276244 CPUtime=16.85 cores=0-7
/proc/29870/stat : 29870 (hybridGM3) R 29868 29870 29870 0 -1 4202496 8556 0 0 0 1684 1 0 0 25 0 1 0 668136481 282873856 8304 33554432000 134512640 135204385 4290520528 18446744073709551615 134537123 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/29870/statm: 69061 8304 64 169 0 68891 0
[pid=29871] ppid=29852 vsize=421440 CPUtime=0.05 cores=0-7
/proc/29871/stat : 29871 (python) S 29852 29852 29579 0 -1 4202560 808 0 0 0 3 2 0 0 15 0 1 0 668136484 431554560 14808 33554432000 4194304 5683644 140736284179984 18446744073709551615 212436037190 0 0 16781312 514 13413197433937035524 0 0 17 1 0 0 0
/proc/29871/statm: 105360 14808 371 364 0 41266 0
[pid=29873] ppid=29871 vsize=276244 CPUtime=16.87 cores=0-7
/proc/29873/stat : 29873 (hybridGM3) R 29871 29873 29873 0 -1 4202496 8554 0 0 0 1686 1 0 0 25 0 1 0 668136485 282873856 8300 33554432000 134512640 135204385 4294342320 18446744073709551615 134536676 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/29873/statm: 69061 8300 64 169 0 68891 0

################
# More data... #
################

/proc/29971/statm: 156640 15198 400 364 0 92486 0
[pid=29973] ppid=29971 vsize=434080 CPUtime=295.1 cores=0-7
/proc/29973/stat : 29973 (adaptg2wsat2009) T 29971 29973 29973 0 -1 4202496 7104 0 0 0 29509 1 0 0 25 0 1 0 668196178 444497920 6818 33554432000 134512640 135107411 4287859152 18446744073709551615 134531246 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/29973/statm: 108520 6818 59 146 0 107396 0
[pid=29971/tid=29981] ppid=29852 vsize=626560 CPUtime=0 cores=0-7
/proc/29971/task/29981/stat : 29981 (python) S 29852 29852 29579 0 -1 4202560 93 0 0 0 0 0 0 0 18 0 2 0 668206112 641597440 15198 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 514 18446604451657835712 0 0 -1 1 0 0 0
[pid=29975] ppid=29852 vsize=636816 CPUtime=0.98 cores=0-7
/proc/29975/stat : 29975 (python) S 29852 29852 29579 0 -1 4202560 1185 0 0 0 50 48 0 0 18 0 2 0 668196437 652099584 15157 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/29975/statm: 159204 15157 400 364 0 95047 0
[pid=29977] ppid=29975 vsize=434080 CPUtime=291.94 cores=0-7
/proc/29977/stat : 29977 (adaptg2wsat2009) T 29975 29977 29977 0 -1 4202496 7102 0 0 0 29193 1 0 0 25 0 1 0 668196438 444497920 6818 33554432000 134512640 135107411 4288846208 18446744073709551615 134532277 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/29977/statm: 108520 6818 59 146 0 107396 0
[pid=29975/tid=29998] ppid=29852 vsize=636816 CPUtime=0 cores=0-7
/proc/29975/task/29998/stat : 29998 (python) S 29852 29852 29579 0 -1 4202560 85 0 0 0 0 0 0 0 18 0 2 0 668216102 652099584 15157 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 514 18446604445538634496 0 0 -1 3 0 0 0
[pid=29982] ppid=29852 vsize=647072 CPUtime=0.65 cores=0-7
/proc/29982/stat : 29982 (python) S 29852 29852 29579 0 -1 4202560 1197 0 0 0 36 29 0 0 16 0 2 0 668206118 662601728 15182 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/29982/statm: 161768 15182 400 364 0 97608 0
[pid=29984] ppid=29982 vsize=276240 CPUtime=197.01 cores=0-7
/proc/29984/stat : 29984 (hybridGM3) T 29982 29984 29984 0 -1 4202496 8582 0 0 0 19700 1 0 0 25 0 1 0 668206119 282869760 8298 33554432000 134512640 135204385 4294675552 18446744073709551615 134536663 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/29984/statm: 69060 8298 64 169 0 68890 0
[pid=29982/tid=30013] ppid=29852 vsize=647072 CPUtime=0 cores=0-7
/proc/29982/task/30013/stat : 30013 (python) S 29852 29852 29579 0 -1 4202560 87 0 0 0 0 0 0 0 17 0 2 0 668225829 662601728 15182 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 514 18446604445538632704 0 0 -1 5 0 0 0
[pid=29987] ppid=29852 vsize=657328 CPUtime=0.33 cores=0-7
/proc/29987/stat : 29987 (python) S 29852 29852 29579 0 -1 4202560 1201 0 0 0 15 18 0 0 18 0 2 0 668209158 673103872 15166 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/29987/statm: 164332 15166 401 364 0 100169 0
[pid=29989] ppid=29987 vsize=467024 CPUtime=99.76 cores=0-7
/proc/29989/stat : 29989 (TNM) T 29987 29989 29989 0 -1 4202496 6846 0 0 0 9975 1 0 0 25 0 1 0 668209160 478232576 6558 33554432000 134512640 135008748 4292204992 18446744073709551615 134524720 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/29989/statm: 116756 6558 46 122 0 115656 0
[pid=29987/tid=30003] ppid=29852 vsize=657328 CPUtime=0 cores=0-7
/proc/29987/task/30003/stat : 30003 (python) S 29852 29852 29579 0 -1 4202560 91 0 0 0 0 0 0 0 19 0 2 0 668219193 673103872 15166 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 514 18446604441456880640 0 0 -1 4 0 0 0
[pid=29991] ppid=29852 vsize=667584 CPUtime=0.34 cores=0-7
/proc/29991/stat : 29991 (python) S 29852 29852 29579 0 -1 4202560 1242 0 0 0 16 18 0 0 19 0 2 0 668213226 683606016 15277 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/29991/statm: 166896 15277 406 364 0 102730 0
[pid=29993] ppid=29991 vsize=467024 CPUtime=100.02 cores=0-7
/proc/29993/stat : 29993 (TNM) T 29991 29993 29993 0 -1 4202496 6851 0 0 0 10001 1 0 0 25 0 1 0 668213227 478232576 6559 33554432000 134512640 135008748 4292130368 18446744073709551615 134529183 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/29993/statm: 116756 6559 46 122 0 115656 0
[pid=29991/tid=30008] ppid=29852 vsize=667584 CPUtime=0 cores=0-7
/proc/29991/task/30008/stat : 30008 (python) S 29852 29852 29579 0 -1 4202560 90 0 0 0 0 0 0 0 19 0 2 0 668223260 683606016 15277 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 514 18446604445493491712 0 0 -1 6 0 0 0
[pid=29995] ppid=29852 vsize=677840 CPUtime=0.37 cores=0-7
/proc/29995/stat : 29995 (python) S 29852 29852 29579 0 -1 4202560 1046 0 0 0 17 20 0 0 15 0 1 0 668215047 694108160 15305 33554432000 4194304 5683644 140736284179984 18446744073709551615 212436037190 0 0 16781312 514 13413197433937035522 0 0 17 6 0 0 0
/proc/29995/statm: 169460 15305 379 364 0 105291 0
[pid=29997] ppid=29995 vsize=276240 CPUtime=113.98 cores=0-7
/proc/29997/stat : 29997 (hybridGM3) R 29995 29997 29997 0 -1 4202496 8588 0 0 0 11397 1 0 0 25 0 1 0 668215047 282869760 8299 33554432000 134512640 135204385 4293536496 18446744073709551615 134659313 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/29997/statm: 69060 8299 64 169 0 68890 0
[pid=29999] ppid=29852 vsize=688096 CPUtime=0.33 cores=0-7
/proc/29999/stat : 29999 (python) S 29852 29852 29579 0 -1 4202560 1249 0 0 0 17 16 0 0 19 0 2 0 668216160 704610304 15190 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/29999/statm: 172024 15190 407 364 0 107852 0
[pid=30001] ppid=29999 vsize=467024 CPUtime=99.51 cores=0-7
/proc/30001/stat : 30001 (TNM) T 29999 30001 30001 0 -1 4202496 6847 0 0 0 9949 2 0 0 25 0 1 0 668216161 478232576 6559 33554432000 134512640 135008748 4289029472 18446744073709551615 134528456 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/30001/statm: 116756 6559 46 122 0 115656 0
[pid=29999/tid=30017] ppid=29852 vsize=688096 CPUtime=0 cores=0-7
/proc/29999/task/30017/stat : 30017 (python) S 29852 29852 29579 0 -1 4202560 94 0 0 0 0 0 0 0 19 0 2 0 668226194 704610304 15190 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 514 18446604446438994112 0 0 -1 2 0 0 0
[pid=30004] ppid=29852 vsize=698352 CPUtime=0.24 cores=0-7
/proc/30004/stat : 30004 (python) S 29852 29852 29579 0 -1 4202560 1048 0 0 0 11 13 0 0 18 0 1 0 668219201 715112448 15226 33554432000 4194304 5683644 140736284179984 18446744073709551615 212436037190 0 0 16781312 514 13413197433937035523 0 0 17 1 0 0 0
/proc/30004/statm: 174588 15226 378 364 0 110413 0
[pid=30006] ppid=30004 vsize=467024 CPUtime=71.78 cores=0-7
/proc/30006/stat : 30006 (TNM) R 30004 30006 30006 0 -1 4202496 6848 0 0 0 7178 0 0 0 25 0 1 0 668219201 478232576 6558 33554432000 134512640 135008748 4289632016 18446744073709551615 134529448 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/30006/statm: 116756 6558 46 122 0 115656 0
[pid=30009] ppid=29852 vsize=708608 CPUtime=0.1 cores=0-7
/proc/30009/stat : 30009 (python) S 29852 29852 29579 0 -1 4202560 1067 0 0 0 4 6 0 0 18 0 1 0 668223266 725614592 15346 33554432000 4194304 5683644 140736284179984 18446744073709551615 212436037190 0 0 16781312 514 13413197433937035522 0 0 17 1 0 0 0
/proc/30009/statm: 177152 15346 378 364 0 112974 0
[pid=30011] ppid=30009 vsize=434080 CPUtime=31.87 cores=0-7
/proc/30011/stat : 30011 (adaptg2wsat2009) R 30009 30011 30011 0 -1 4202496 7115 0 0 0 3187 0 0 0 25 0 1 0 668223267 444497920 6819 33554432000 134512640 135107411 4289578368 18446744073709551615 134531535 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/30011/statm: 108520 6819 59 146 0 107396 0
[pid=30014] ppid=29852 vsize=718864 CPUtime=0.02 cores=0-7
/proc/30014/stat : 30014 (python) S 29852 29852 29579 0 -1 4202560 1049 0 0 0 1 1 0 0 18 0 1 0 668225836 736116736 15322 33554432000 4194304 5683644 140736284179984 18446744073709551615 212436037190 0 0 16781312 514 3652027601075011841 0 0 17 6 0 0 0
/proc/30014/statm: 179716 15322 373 364 0 115535 0
[pid=30016] ppid=30014 vsize=467024 CPUtime=6.34 cores=0-7
/proc/30016/stat : 30016 (TNM) R 30014 30016 30016 0 -1 4202496 6847 0 0 0 634 0 0 0 25 0 1 0 668225837 478232576 6557 33554432000 134512640 135008748 4287553312 18446744073709551615 134527477 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/30016/statm: 116756 6557 46 122 0 115656 0
[pid=30018] ppid=29852 vsize=729120 CPUtime=0 cores=0-7
/proc/30018/stat : 30018 (python) S 29852 29852 29579 0 -1 4202560 942 0 0 0 0 0 0 0 18 0 1 0 668226201 746618880 15363 33554432000 4194304 5683644 140736284179984 18446744073709551615 212436037190 0 0 16781312 514 3652023752784314626 0 0 17 6 0 0 0
/proc/30018/statm: 182280 15363 350 364 0 118096 0
[pid=30020] ppid=30018 vsize=467024 CPUtime=2.67 cores=0-7
/proc/30020/stat : 30020 (TNM) R 30018 30020 30020 0 -1 4202496 6843 0 0 0 266 1 0 0 25 0 1 0 668226201 478232576 6556 33554432000 134512640 135008748 4288030384 18446744073709551615 134527475 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/30020/statm: 116756 6556 46 122 0 115656 0
[pid=30021] ppid=29852 vsize=739376 CPUtime=0 cores=0-7
/proc/30021/stat : 30021 (python) S 29852 29852 29579 0 -1 4202560 950 0 0 0 0 0 0 0 19 0 1 0 668226431 757121024 15349 33554432000 4194304 5683644 140736284179984 18446744073709551615 212436037190 0 0 16781312 514 3652016056202952703 0 0 17 6 0 0 0
/proc/30021/statm: 184844 15349 352 364 0 120657 0
[pid=30023] ppid=30021 vsize=462800 CPUtime=0.43 cores=0-7
/proc/30023/stat : 30023 (TNM) R 30021 30023 30023 0 -1 4202496 4199 0 0 0 43 0 0 0 21 0 1 0 668226432 473907200 3906 33554432000 134512640 135008748 4291882752 18446744073709551615 134523324 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/30023/statm: 115700 3906 45 122 0 114600 0
[pid=29852/tid=29860] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29860/stat : 29860 (python) S 29850 29852 29579 0 -1 4202560 36 30416 0 0 0 0 80407 153 18 0 37 0 668136472 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 0 0 0 0
[pid=29852/tid=29863] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29863/stat : 29863 (python) S 29850 29852 29579 0 -1 4202560 27 30416 0 0 0 0 80407 153 18 0 37 0 668136475 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 3 0 0 0
[pid=29852/tid=29866] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29866/stat : 29866 (python) S 29850 29852 29579 0 -1 4202560 94 30416 0 0 0 0 80407 153 18 0 37 0 668136478 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 4 0 0 0
[pid=29852/tid=29869] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29869/stat : 29869 (python) S 29850 29852 29579 0 -1 4202560 46 30416 0 0 0 0 80407 153 18 0 37 0 668136481 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 4 0 0 0
[pid=29852/tid=29872] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29872/stat : 29872 (python) S 29850 29852 29579 0 -1 4202560 45 30416 0 0 0 0 80407 153 18 0 37 0 668136484 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 4 0 0 0
[pid=29852/tid=29875] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29875/stat : 29875 (python) S 29850 29852 29579 0 -1 4202560 39 30416 0 0 0 0 80407 153 18 0 37 0 668136488 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 0 0 0 0
[pid=29852/tid=29878] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29878/stat : 29878 (python) S 29850 29852 29579 0 -1 4202560 39 30416 0 0 0 0 80407 153 18 0 37 0 668136491 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 0 0 0 0
[pid=29852/tid=29881] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29881/stat : 29881 (python) S 29850 29852 29579 0 -1 4202560 37 30416 0 0 0 0 80407 153 18 0 37 0 668136494 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 7 0 0 0
[pid=29852/tid=29896] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29896/stat : 29896 (python) S 29850 29852 29579 0 -1 4202560 39 30416 0 0 0 0 80407 153 20 0 37 0 668154269 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 3 0 0 0
[pid=29852/tid=29906] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29906/stat : 29906 (python) S 29850 29852 29579 0 -1 4202560 68 30416 0 0 0 0 80407 153 18 0 37 0 668164302 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 1 0 0 0
[pid=29852/tid=29910] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29910/stat : 29910 (python) S 29850 29852 29579 0 -1 4202560 45 30416 0 0 0 0 80407 153 18 0 37 0 668166115 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 0 0 0 0
[pid=29852/tid=29913] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29913/stat : 29913 (python) S 29850 29852 29579 0 -1 4202560 45 30416 0 0 0 0 80407 153 18 0 37 0 668166207 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 0 0 0 0
[pid=29852/tid=29916] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29916/stat : 29916 (python) S 29850 29852 29579 0 -1 4202560 34 30416 0 0 0 0 80407 153 21 0 37 0 668166216 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 0 0 0 0
[pid=29852/tid=29919] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29919/stat : 29919 (python) S 29850 29852 29579 0 -1 4202560 36 30416 0 0 0 0 80407 153 23 0 37 0 668166221 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 0 0 0 0
[pid=29852/tid=29922] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29922/stat : 29922 (python) S 29850 29852 29579 0 -1 4202560 32 30416 0 0 0 0 80407 153 20 0 37 0 668166324 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 6 0 0 0
[pid=29852/tid=29925] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29925/stat : 29925 (python) S 29850 29852 29579 0 -1 4202560 41 30416 0 0 0 0 80407 153 18 0 37 0 668166329 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 0 0 0 0
[pid=29852/tid=29928] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29928/stat : 29928 (python) S 29850 29852 29579 0 -1 4202560 31 30416 0 0 0 0 80407 153 25 0 37 0 668166334 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 6 0 0 0
[pid=29852/tid=29934] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29934/stat : 29934 (python) S 29850 29852 29579 0 -1 4202560 32 30416 0 0 0 0 80407 153 20 0 37 0 668176252 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 6 0 0 0
[pid=29852/tid=29938] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29938/stat : 29938 (python) S 29850 29852 29579 0 -1 4202560 33 30416 0 0 0 0 80407 153 21 0 37 0 668176265 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 6 0 0 0
[pid=29852/tid=29942] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29942/stat : 29942 (python) S 29850 29852 29579 0 -1 4202560 35 30416 0 0 0 0 80407 153 20 0 37 0 668176361 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 6 0 0 0
[pid=29852/tid=29946] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29946/stat : 29946 (python) S 29850 29852 29579 0 -1 4202560 39 30416 0 0 0 0 80407 153 18 0 37 0 668176374 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 6 0 0 0
[pid=29852/tid=29957] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29957/stat : 29957 (python) S 29850 29852 29579 0 -1 4202560 49 30416 0 0 0 0 80407 153 18 0 37 0 668186303 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 1 0 0 0
[pid=29852/tid=29961] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29961/stat : 29961 (python) S 29850 29852 29579 0 -1 4202560 33 30416 0 0 0 0 80407 153 19 0 37 0 668186398 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 6 0 0 0
[pid=29852/tid=29966] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29966/stat : 29966 (python) S 29850 29852 29579 0 -1 4202560 40 30416 0 0 0 0 80407 153 18 0 37 0 668186893 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 6 0 0 0
[pid=29852/tid=29972] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29972/stat : 29972 (python) S 29850 29852 29579 0 -1 4202560 46 30416 0 0 0 0 80407 153 18 0 37 0 668196178 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 1 0 0 0
[pid=29852/tid=29976] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29976/stat : 29976 (python) S 29850 29852 29579 0 -1 4202560 41 30416 0 0 0 0 80407 153 18 0 37 0 668196438 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 6 0 0 0
[pid=29852/tid=29983] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29983/stat : 29983 (python) S 29850 29852 29579 0 -1 4202560 33 30416 0 0 0 0 80407 153 20 0 37 0 668206119 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 1 0 0 0
[pid=29852/tid=29988] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29988/stat : 29988 (python) S 29850 29852 29579 0 -1 4202560 35 30416 0 0 0 0 80407 153 19 0 37 0 668209158 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 6 0 0 0
[pid=29852/tid=29992] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29992/stat : 29992 (python) S 29850 29852 29579 0 -1 4202560 32 30416 0 0 0 0 80407 153 20 0 37 0 668213227 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 1 0 0 0
[pid=29852/tid=29996] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/29996/stat : 29996 (python) S 29850 29852 29579 0 -1 4202560 33 30416 0 0 0 0 80407 153 20 0 37 0 668215047 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 6 0 0 0
[pid=29852/tid=30000] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/30000/stat : 30000 (python) S 29850 29852 29579 0 -1 4202560 34 30416 0 0 0 0 80407 153 20 0 37 0 668216161 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 6 0 0 0
[pid=29852/tid=30005] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/30005/stat : 30005 (python) S 29850 29852 29579 0 -1 4202560 32 30416 0 0 0 0 80407 153 19 0 37 0 668219201 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 6 0 0 0
[pid=29852/tid=30010] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/30010/stat : 30010 (python) S 29850 29852 29579 0 -1 4202560 32 30416 0 0 0 0 80407 153 19 0 37 0 668223267 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 6 0 0 0
[pid=29852/tid=30015] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/30015/stat : 30015 (python) S 29850 29852 29579 0 -1 4202560 37 30416 0 0 0 0 80407 153 20 0 37 0 668225837 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 6 0 0 0
[pid=29852/tid=30019] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/30019/stat : 30019 (python) S 29850 29852 29579 0 -1 4202560 32 30416 0 0 0 0 80407 153 20 0 37 0 668226201 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 6 0 0 0
[pid=29852/tid=30022] ppid=29850 vsize=749620 CPUtime=805.6 cores=0-7
/proc/29852/task/30022/stat : 30022 (python) S 29850 29852 29579 0 -1 4202560 35 30416 0 0 0 0 80407 153 20 0 37 0 668226432 767610880 16947 33554432000 4194304 5683644 140736284179984 18446744073709551615 212447840689 0 0 16781312 2 18446604452489698176 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 7175.13
Current children cumulated vsize (KiB) 32178920

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 29852 and gives
#  childrusage.ru_utime.tv_sec=1358
#  childrusage.ru_utime.tv_usec=367496
#  childrusage.ru_stime.tv_sec=12
#  childrusage.ru_stime.tv_usec=31170
# CPU time returned by wait4() is 1370.4
# while last known CPU time is 7175.13
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 908.72
CPU time (s): 7175.13
CPU user time (s): 7162.95
CPU system time (s): 12.18
CPU usage (%): 789.586
Max. virtual memory (cumulated for all children) (KiB): 32178920

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1358.37
system time used= 12.0312
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 192274
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 12971
involuntary context switches= 7332

runsolver used 8.26174 second user time and 22.5466 second system time

The end

Launcher Data 

Begin job on node102 at 2011-05-16 19:02:13
IDJOB=3385343
IDBENCH=82608
IDSOLVER=1870
FILE ID=node102/3385343-1305565332
RUNJOBID= node102-1305564967-29593
PBS_JOBID= 13350148
Free space on /tmp= 73612 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v40000-c168000-S1184456903-078-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3385343-1305565332/watcher-3385343-1305565332 -o /tmp/evaluation-result-3385343-1305565332/solver-3385343-1305565332 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385343-1305565332.cnf -seed 465104676 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 33dd7efc1f63d9d3511c7a69c68ea88b
RANDOM SEED=465104676

node102.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      11987064 kB
Buffers:        120296 kB
Cached:        1022152 kB
SwapCached:      33496 kB
Active:       12122856 kB
Inactive:      8607252 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      11987064 kB
SwapTotal:    67111528 kB
SwapFree:     56316216 kB
Dirty:           34952 kB
Writeback:           0 kB
AnonPages:    19565692 kB
Mapped:          20328 kB
Slab:            77668 kB
PageTables:      86360 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 102328232 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73484 MiB
End job on node102 at 2011-05-16 19:17:23