Trace number 3378229

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (MO) 1994.56 502.733

General information on the benchmark

Namerandom/large/
unif-k7-r85-v200-c17000-S970203146-047-UNKNOWN.cnf
MD5SUM635ea520fe16a78d41337039f98997e5
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark2027.46
Satisfiable
(Un)Satisfiability was proved
Number of variables200
Number of clauses17000
Sum of the clauses size119000
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 517000

Solver Data 

0.32/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.32/0.67	c INFO: solving HOME/instance-3378229-1305463934.cnf
0.32/0.67	c INFO: computing task features
0.39/0.78	c INFO: computed variable-clause graph statistics
0.39/0.79	c INFO: computed clause balance statistics
1.09/1.40	c INFO: computed variable balance statistics
1.29/1.65	c INFO: computed small-clause counts
1.29/1.65	c INFO: computed Horn-clause counts
1.29/1.67	c INFO: computed variable graph statistics
1.29/1.67	c INFO: collected features for HOME/instance-3378229-1305463934.cnf in 1.00 s
1.29/1.68	c INFO: running gnovelty+2@0 for 97 with 1198 remaining (b = 0.35)
1.29/1.69	c INFO: running gnovelty+2@0 for 97 with 1198 remaining (b = 0.12)
1.39/1.70	c INFO: running adaptg2wsat2009++@0 for 97 with 1198 remaining (b = 0.08)
1.39/1.71	c INFO: running adaptg2wsat2009++@0 for 97 with 1198 remaining (b = 0.06)
397.24/101.07	c INFO: running adaptg2wsat2009++@0 for 97 with 1099 remaining (b = 0.05)
401.21/102.04	c INFO: running gnovelty+2@0 for 97 with 1098 remaining (b = 0.04)
401.21/102.06	c INFO: running FH@0 for 97 with 1098 remaining (b = 0.03)
401.21/102.07	c INFO: running adaptg2wsat2009++@0 for 97 with 1098 remaining (b = 0.03)
796.87/201.38	c INFO: running adaptg2wsat2009++@0 for 97 with 999 remaining (b = 0.02)
797.20/201.40	c INFO: running FH@0 for 97 with 999 remaining (b = 0.02)
800.82/202.40	c INFO: running hybridGM3@0 for 194 with 998 remaining (b = 0.04)
800.82/202.41	c INFO: running gnovelty+2@0 for 97 with 998 remaining (b = 0.02)
1196.04/301.71	c INFO: running hybridGM3@0 for 194 with 898 remaining (b = 0.03)
1196.33/301.82	c INFO: running adaptg2wsat2009++@0 for 97 with 898 remaining (b = 0.02)
1199.98/302.76	c INFO: running hybridGM3@0 for 194 with 897 remaining (b = 0.03)
1582.85/399.06	c INFO: running hybridGM3@99 for 97 with 801 remaining (b = 0.01)
1595.15/402.16	c INFO: running FH@0 for 97 with 798 remaining (b = 0.01)
1961.56/494.37	c INFO: running hybridGM3@0 for 194 with 706 remaining (b = 0.02)
1973.45/497.39	c INFO: running hybridGM3@99 for 97 with 703 remaining (b = 0.01)
1994.10/502.51	c INFO: running TNM@0 for 698 with 698 remaining (b = 0.08)
1994.56/502.71	c INFO: running adaptg2wsat2009++@0 for 97 with 697 remaining (b = 0.01)
1994.56/502.71	Traceback (most recent call last):
1994.56/502.71	  File "HOME/solve", line 33, in <module>
1994.56/502.71	    plac.call(main)
1994.56/502.71	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
1994.56/502.71	    cmd, result = parser_from(obj).consume(arglist)
1994.56/502.71	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
1994.56/502.71	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
1994.56/502.71	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
1994.56/502.71	    (_, answer, _) = solver(input_path, budget / cores, cores)
1994.56/502.71	  File "HOME/borg/build/lib/borg/bilevel.py", line 202, in __call__
1994.56/502.71	    solver.go(max_cost * borg.defaults.machine_speed)
1994.56/502.71	  File "HOME/borg/build/lib/borg/solvers.py", line 179, in go
1994.56/502.71	    self._process.start()
1994.56/502.71	  File "/home/evaluation/evaluation/tools64/lib/python2.7/multiprocessing/process.py", line 94, in start
1994.56/502.71	    assert self._popen is None, 'cannot start a process twice'
1994.56/502.71	AssertionError: cannot start a process twice

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378229-1305463934/watcher-3378229-1305463934 -o /tmp/evaluation-result-3378229-1305463934/solver-3378229-1305463934 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378229-1305463934.cnf -seed 1942632377 -budget 4800 -cores 4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.20 7.62 5.88 6/416 14756
/proc/meminfo: memFree=16562072/32951124 swapFree=67056804/67111528
[pid=14756] ppid=14754 vsize=15752 CPUtime=0 cores=0,2,4,6
/proc/14756/stat : 14756 (python) R 14754 14756 14447 0 -1 4202496 839 0 0 0 0 0 0 0 25 0 1 0 657994345 16130048 723 33554432000 4194304 5683644 140733754910800 18446744073709551615 270613108736 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/14756/statm: 3938 723 334 364 0 425 0

[startup+0.0922419 s]
/proc/loadavg: 8.20 7.62 5.88 6/416 14756
/proc/meminfo: memFree=16562072/32951124 swapFree=67056804/67111528
[pid=14756] ppid=14754 vsize=84472 CPUtime=0.05 cores=0,2,4,6
/proc/14756/stat : 14756 (python) R 14754 14756 14447 0 -1 4202496 2396 0 0 0 4 1 0 0 18 0 1 0 657994345 86499328 2125 33554432000 4194304 5683644 140733754910800 18446744073709551615 270612771525 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/14756/statm: 21118 2127 707 364 0 1335 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 84472

[startup+0.10024 s]
/proc/loadavg: 8.20 7.62 5.88 6/416 14756
/proc/meminfo: memFree=16562072/32951124 swapFree=67056804/67111528
[pid=14756] ppid=14754 vsize=87708 CPUtime=0.05 cores=0,2,4,6
/proc/14756/stat : 14756 (python) R 14754 14756 14447 0 -1 4202496 2638 0 0 0 4 1 0 0 17 0 1 0 657994345 89812992 2355 33554432000 4194304 5683644 140733754910800 18446744073709551615 270613110528 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/14756/statm: 21926 2354 717 364 0 1622 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 87708

[startup+0.30019 s]
/proc/loadavg: 8.20 7.62 5.88 6/416 14756
/proc/meminfo: memFree=16562072/32951124 swapFree=67056804/67111528
[pid=14756] ppid=14754 vsize=135748 CPUtime=0.17 cores=0,2,4,6
/proc/14756/stat : 14756 (python) S 14754 14756 14447 0 -1 4202496 5963 933 1 0 12 5 0 0 18 0 1 0 657994345 139005952 4151 33554432000 4194304 5683644 140733754910800 18446744073709551615 270613106693 0 2147483391 16781312 2 18446744071699199166 0 0 17 2 0 0 0
/proc/14756/statm: 33937 4151 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.17
Current children cumulated vsize (KiB) 135748

[startup+0.700134 s]
/proc/loadavg: 8.20 7.62 5.88 6/416 14756
/proc/meminfo: memFree=16562072/32951124 swapFree=67056804/67111528
[pid=14756] ppid=14754 vsize=279368 CPUtime=0.39 cores=0,2,4,6
/proc/14756/stat : 14756 (python) R 14754 14756 14447 0 -1 4202496 9491 933 1 0 29 10 0 0 17 0 1 0 657994345 286072832 7172 33554432000 4194304 5683644 140733754910800 18446744073709551615 4445115 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/14756/statm: 69842 7172 1761 364 0 5766 0
Current children cumulated CPU time (s) 0.39
Current children cumulated vsize (KiB) 279368

[startup+1.50102 s]
/proc/loadavg: 8.20 7.62 5.88 6/417 14761
/proc/meminfo: memFree=16534784/32951124 swapFree=67056804/67111528
[pid=14756] ppid=14754 vsize=282584 CPUtime=1.19 cores=0,2,4,6
/proc/14756/stat : 14756 (python) R 14754 14756 14447 0 -1 4202496 11330 933 1 0 109 10 0 0 18 0 1 0 657994345 289366016 8059 33554432000 4194304 5683644 140733754910800 18446744073709551615 4941968 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/14756/statm: 70646 8059 1827 364 0 6570 0
Current children cumulated CPU time (s) 1.19
Current children cumulated vsize (KiB) 282584

[startup+3.10067 s]
/proc/loadavg: 7.94 7.58 5.87 9/429 14773
/proc/meminfo: memFree=16514476/32951124 swapFree=67056804/67111528
[pid=14756] ppid=14754 vsize=320376 CPUtime=1.4 cores=0,2,4,6
/proc/14756/stat : 14756 (python) S 14754 14756 14447 0 -1 4202496 14093 933 1 0 129 11 0 0 25 0 5 0 657994345 328065024 7309 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624937419 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/14756/statm: 80094 7309 1883 364 0 16003 0
[pid=14762] ppid=14756 vsize=279228 CPUtime=0 cores=0,2,4,6
/proc/14762/stat : 14762 (python) S 14756 14756 14447 0 -1 4202560 724 0 0 0 0 0 0 0 18 0 1 0 657994514 285929472 5714 33554432000 4194304 5683644 140733754910800 18446744073709551615 270613130767 0 0 16781312 514 14329457036419367170 0 0 17 2 0 0 0
/proc/14762/statm: 69807 5714 354 364 0 5725 0
[pid=14764] ppid=14762 vsize=1018616 CPUtime=1.4 cores=0,2,4,6
/proc/14764/stat : 14764 (gnovelty+2) R 14762 14764 14764 0 -1 4202496 763 0 0 0 140 0 0 0 25 0 1 0 657994514 1043062784 501 33554432000 134512640 135663704 4291763840 18446744073709551615 134524918 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/14764/statm: 254654 501 113 282 0 254370 0
[pid=14765] ppid=14756 vsize=289620 CPUtime=0 cores=0,2,4,6
/proc/14765/stat : 14765 (python) S 14756 14756 14447 0 -1 4202560 737 0 0 0 0 0 0 0 18 0 1 0 657994515 296570880 5762 33554432000 4194304 5683644 140733754910800 18446744073709551615 270613130822 0 0 16781312 514 7935337050163609857 0 0 17 2 0 0 0
/proc/14765/statm: 72405 5762 348 364 0 8320 0
[pid=14767] ppid=14765 vsize=1018616 CPUtime=1.4 cores=0,2,4,6
/proc/14767/stat : 14767 (gnovelty+2) R 14765 14767 14767 0 -1 4202496 762 0 0 0 140 0 0 0 25 0 1 0 657994515 1043062784 501 33554432000 134512640 135663704 4287256256 18446744073709551615 134524503 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/14767/statm: 254654 501 113 282 0 254370 0
[pid=14768] ppid=14756 vsize=299876 CPUtime=0 cores=0,2,4,6
/proc/14768/stat : 14768 (python) S 14756 14756 14447 0 -1 4202560 741 0 0 0 0 0 0 0 18 0 1 0 657994516 307073024 5766 33554432000 4194304 5683644 140733754910800 18446744073709551615 270613130822 0 0 16781312 514 6619627076628512767 0 0 17 6 0 0 0
/proc/14768/statm: 74969 5766 348 364 0 10881 0
[pid=14770] ppid=14768 vsize=418500 CPUtime=1.37 cores=0,2,4,6
/proc/14770/stat : 14770 (adaptg2wsat2009) R 14768 14770 14770 0 -1 4202496 1074 0 0 0 137 0 0 0 25 0 1 0 657994516 428544000 808 33554432000 134512640 135107411 4293818528 18446744073709551615 134523312 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/14770/statm: 104625 808 49 146 0 103501 0
[pid=14771] ppid=14756 vsize=310132 CPUtime=0 cores=0,2,4,6
/proc/14771/stat : 14771 (python) S 14756 14756 14447 0 -1 4202560 740 0 0 0 0 0 0 0 18 0 1 0 657994517 317575168 5770 33554432000 4194304 5683644 140733754910800 18446744073709551615 270613130822 0 0 16781312 514 7935325505291550719 0 0 17 2 0 0 0
/proc/14771/statm: 77533 5770 348 364 0 13442 0
[pid=14773] ppid=14771 vsize=418500 CPUtime=1.36 cores=0,2,4,6
/proc/14773/stat : 14773 (adaptg2wsat2009) R 14771 14773 14773 0 -1 4202496 1072 0 0 0 136 0 0 0 25 0 1 0 657994517 428544000 808 33554432000 134512640 135107411 4294879312 18446744073709551615 134523310 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/14773/statm: 104625 808 49 146 0 103501 0
[pid=14756/tid=14763] ppid=14754 vsize=320376 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14763/stat : 14763 (python) S 14754 14756 14447 0 -1 4202560 24 933 0 0 0 0 0 0 20 0 5 0 657994514 328065024 7309 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 2 0 0 0
[pid=14756/tid=14766] ppid=14754 vsize=320376 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14766/stat : 14766 (python) S 14754 14756 14447 0 -1 4202560 25 933 0 0 0 0 0 0 22 0 5 0 657994515 328065024 7309 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 2 0 0 0
[pid=14756/tid=14769] ppid=14754 vsize=320376 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14769/stat : 14769 (python) S 14754 14756 14447 0 -1 4202560 25 933 0 0 0 0 0 0 24 0 5 0 657994516 328065024 7309 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 6 0 0 0
[pid=14756/tid=14772] ppid=14754 vsize=320376 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14772/stat : 14772 (python) S 14754 14756 14447 0 -1 4202560 29 933 0 0 0 0 0 0 25 0 5 0 657994517 328065024 7309 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 6.93
Current children cumulated vsize (KiB) 4373464

[startup+6.3011 s]
/proc/loadavg: 7.94 7.58 5.87 9/429 14773
/proc/meminfo: memFree=16513368/32951124 swapFree=67056804/67111528
[pid=14756] ppid=14754 vsize=320376 CPUtime=1.4 cores=0,2,4,6
/proc/14756/stat : 14756 (python) S 14754 14756 14447 0 -1 4202496 14093 933 1 0 129 11 0 0 25 0 5 0 657994345 328065024 7309 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624937419 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/14756/statm: 80094 7309 1883 364 0 16003 0
[pid=14762] ppid=14756 vsize=279228 CPUtime=0 cores=0,2,4,6
/proc/14762/stat : 14762 (python) S 14756 14756 14447 0 -1 4202560 885 0 0 0 0 0 0 0 18 0 1 0 657994514 285929472 5737 33554432000 4194304 5683644 140733754910800 18446744073709551615 270613130767 0 0 16781312 514 14329457036419367170 0 0 17 2 0 0 0
/proc/14762/statm: 69807 5737 377 364 0 5725 0
[pid=14764] ppid=14762 vsize=1018616 CPUtime=4.6 cores=0,2,4,6
/proc/14764/stat : 14764 (gnovelty+2) R 14762 14764 14764 0 -1 4202496 763 0 0 0 460 0 0 0 25 0 1 0 657994514 1043062784 501 33554432000 134512640 135663704 4291763840 18446744073709551615 134525691 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/14764/statm: 254654 501 113 282 0 254370 0
[pid=14765] ppid=14756 vsize=289620 CPUtime=0.01 cores=0,2,4,6
/proc/14765/stat : 14765 (python) S 14756 14756 14447 0 -1 4202560 897 0 0 0 0 1 0 0 18 0 1 0 657994515 296570880 5785 33554432000 4194304 5683644 140733754910800 18446744073709551615 270613130822 0 0 16781312 514 7935337050163609857 0 0 17 6 0 0 0
/proc/14765/statm: 72405 5785 371 364 0 8320 0
[pid=14767] ppid=14765 vsize=1018616 CPUtime=4.6 cores=0,2,4,6
/proc/14767/stat : 14767 (gnovelty+2) R 14765 14767 14767 0 -1 4202496 762 0 0 0 460 0 0 0 25 0 1 0 657994515 1043062784 501 33554432000 134512640 135663704 4287256256 18446744073709551615 134524553 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/14767/statm: 254654 501 113 282 0 254370 0
[pid=14768] ppid=14756 vsize=299876 CPUtime=0 cores=0,2,4,6
/proc/14768/stat : 14768 (python) S 14756 14756 14447 0 -1 4202560 905 0 0 0 0 0 0 0 18 0 1 0 657994516 307073024 5789 33554432000 4194304 5683644 140733754910800 18446744073709551615 270613130822 0 0 16781312 514 6619627076628480260 0 0 17 6 0 0 0
/proc/14768/statm: 74969 5789 371 364 0 10881 0
[pid=14770] ppid=14768 vsize=418636 CPUtime=4.49 cores=0,2,4,6
/proc/14770/stat : 14770 (adaptg2wsat2009) R 14768 14770 14770 0 -1 4202496 1221 0 0 0 449 0 0 0 25 0 1 0 657994516 428683264 955 33554432000 134512640 135107411 4293818528 18446744073709551615 134531781 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/14770/statm: 104659 955 59 146 0 103535 0
[pid=14771] ppid=14756 vsize=310132 CPUtime=0.01 cores=0,2,4,6
/proc/14771/stat : 14771 (python) S 14756 14756 14447 0 -1 4202560 901 0 0 0 0 1 0 0 18 0 1 0 657994517 317575168 5793 33554432000 4194304 5683644 140733754910800 18446744073709551615 270613130822 0 0 16781312 514 7935325505291518210 0 0 17 2 0 0 0
/proc/14771/statm: 77533 5793 371 364 0 13442 0
[pid=14773] ppid=14771 vsize=418636 CPUtime=4.53 cores=0,2,4,6
/proc/14773/stat : 14773 (adaptg2wsat2009) R 14771 14773 14773 0 -1 4202496 1220 0 0 0 453 0 0 0 25 0 1 0 657994517 428683264 956 33554432000 134512640 135107411 4294879312 18446744073709551615 134531582 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/14773/statm: 104659 956 59 146 0 103535 0
[pid=14756/tid=14763] ppid=14754 vsize=320376 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14763/stat : 14763 (python) S 14754 14756 14447 0 -1 4202560 24 933 0 0 0 0 0 0 20 0 5 0 657994514 328065024 7309 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 2 0 0 0
[pid=14756/tid=14766] ppid=14754 vsize=320376 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14766/stat : 14766 (python) S 14754 14756 14447 0 -1 4202560 25 933 0 0 0 0 0 0 22 0 5 0 657994515 328065024 7309 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 2 0 0 0
[pid=14756/tid=14769] ppid=14754 vsize=320376 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14769/stat : 14769 (python) S 14754 14756 14447 0 -1 4202560 25 933 0 0 0 0 0 0 24 0 5 0 657994516 328065024 7309 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 6 0 0 0
[pid=14756/tid=14772] ppid=14754 vsize=320376 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14772/stat : 14772 (python) S 14754 14756 14447 0 -1 4202560 29 933 0 0 0 0 0 0 25 0 5 0 657994517 328065024 7309 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 19.64
Current children cumulated vsize (KiB) 4373736

[startup+12.701 s]
/proc/loadavg: 8.02 7.61 5.90 9/429 14773
/proc/meminfo: memFree=16511332/32951124 swapFree=67056804/67111528
[pid=14756] ppid=14754 vsize=320376 CPUtime=1.4 cores=0,2,4,6
/proc/14756/stat : 14756 (python) S 14754 14756 14447 0 -1 4202496 14093 933 1 0 129 11 0 0 25 0 5 0 657994345 328065024 7309 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624937419 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/14756/statm: 80094 7309 1883 364 0 16003 0
[pid=14762] ppid=14756 vsize=279228 CPUtime=0.03 cores=0,2,4,6
/proc/14762/stat : 14762 (python) S 14756 14756 14447 0 -1 4202560 888 0 0 0 1 2 0 0 18 0 1 0 657994514 285929472 5737 33554432000 4194304 5683644 140733754910800 18446744073709551615 270613130767 0 0 16781312 514 13761381980128116994 0 0 17 2 0 0 0
/proc/14762/statm: 69807 5737 377 364 0 5725 0
[pid=14764] ppid=14762 vsize=1018616 CPUtime=11 cores=0,2,4,6
/proc/14764/stat : 14764 (gnovelty+2) R 14762 14764 14764 0 -1 4202496 763 0 0 0 1100 0 0 0 25 0 1 0 657994514 1043062784 501 33554432000 134512640 135663704 4291763840 18446744073709551615 134525278 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/14764/statm: 254654 501 113 282 0 254370 0
[pid=14765] ppid=14756 vsize=289620 CPUtime=0.03 cores=0,2,4,6
/proc/14765/stat : 14765 (python) S 14756 14756 14447 0 -1 4202560 902 0 0 0 1 2 0 0 18 0 1 0 657994515 296570880 5785 33554432000 4194304 5683644 140733754910800 18446744073709551615 270613130822 0 0 16781312 514 13761381980128116993 0 0 17 6 0 0 0
/proc/14765/statm: 72405 5785 371 364 0 8320 0
[pid=14767] ppid=14765 vsize=1018616 CPUtime=11 cores=0,2,4,6
/proc/14767/stat : 14767 (gnovelty+2) R 14765 14767 14767 0 -1 4202496 762 0 0 0 1100 0 0 0 25 0 1 0 657994515 1043062784 501 33554432000 134512640 135663704 4287256256 18446744073709551615 134544196 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/14767/statm: 254654 501 113 282 0 254370 0
[pid=14768] ppid=14756 vsize=299876 CPUtime=0.02 cores=0,2,4,6
/proc/14768/stat : 14768 (python) S 14756 14756 14447 0 -1 4202560 907 0 0 0 1 1 0 0 18 0 1 0 657994516 307073024 5789 33554432000 4194304 5683644 140733754910800 18446744073709551615 270613130822 0 0 16781312 514 13761381980128116996 0 0 17 6 0 0 0
/proc/14768/statm: 74969 5789 371 364 0 10881 0
[pid=14770] ppid=14768 vsize=418636 CPUtime=10.77 cores=0,2,4,6
/proc/14770/stat : 14770 (adaptg2wsat2009) R 14768 14770 14770 0 -1 4202496 1221 0 0 0 1077 0 0 0 25 0 1 0 657994516 428683264 955 33554432000 134512640 135107411 4293818528 18446744073709551615 134531781 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/14770/statm: 104659 955 59 146 0 103535 0
[pid=14771] ppid=14756 vsize=310132 CPUtime=0.02 cores=0,2,4,6
/proc/14771/stat : 14771 (python) S 14756 14756 14447 0 -1 4202560 904 0 0 0 1 1 0 0 18 0 1 0 657994517 317575168 5793 33554432000 4194304 5683644 140733754910800 18446744073709551615 270613130822 0 0 16781312 514 13761381980128116994 0 0 17 2 0 0 0
/proc/14771/statm: 77533 5793 371 364 0 13442 0
[pid=14773] ppid=14771 vsize=418636 CPUtime=10.89 cores=0,2,4,6
/proc/14773/stat : 14773 (adaptg2wsat2009) R 14771 14773 14773 0 -1 4202496 1220 0 0 0 1089 0 0 0 25 0 1 0 657994517 428683264 956 33554432000 134512640 135107411 4294879312 18446744073709551615 134531411 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/14773/statm: 104659 956 59 146 0 103535 0
[pid=14756/tid=14763] ppid=14754 vsize=320376 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14763/stat : 14763 (python) S 14754 14756 14447 0 -1 4202560 24 933 0 0 0 0 0 0 20 0 5 0 657994514 328065024 7309 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 2 0 0 0
[pid=14756/tid=14766] ppid=14754 vsize=320376 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14766/stat : 14766 (python) S 14754 14756 14447 0 -1 4202560 25 933 0 0 0 0 0 0 22 0 5 0 657994515 328065024 7309 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 2 0 0 0
[pid=14756/tid=14769] ppid=14754 vsize=320376 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14769/stat : 14769 (python) S 14754 14756 14447 0 -1 4202560 25 933 0 0 0 0 0 0 24 0 5 0 657994516 328065024 7309 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 6 0 0 0
[pid=14756/tid=14772] ppid=14754 vsize=320376 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14772/stat : 14772 (python) S 14754 14756 14447 0 -1 4202560 29 933 0 0 0 0 0 0 25 0 5 0 657994517 328065024 7309 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 45.16
Current children cumulated vsize (KiB) 4373736

[startup+25.5007 s]
/proc/loadavg: 8.10 7.63 5.93 9/429 14773
/proc/meminfo: memFree=16511548/32951124 swapFree=67056804/67111528
[pid=14756] ppid=14754 vsize=320376 CPUtime=1.4 cores=0,2,4,6

################
# More data... #
################

/proc/14771/stat : 14771 (python) S 14756 14756 14447 0 -1 4202560 1086 0 0 0 15 14 0 0 19 0 2 0 657994517 317575168 5823 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624937419 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/14771/statm: 77533 5823 400 364 0 13442 0
[pid=14773] ppid=14771 vsize=418636 CPUtime=99.97 cores=0,2,4,6
/proc/14773/stat : 14773 (adaptg2wsat2009) T 14771 14773 14773 0 -1 4202496 1220 0 0 0 9996 1 0 0 25 0 1 0 657994517 428683264 956 33554432000 134512640 135107411 4294879312 18446744073709551615 134532128 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/14773/statm: 104659 956 59 146 0 103535 0
[pid=14771/tid=14793] ppid=14756 vsize=310132 CPUtime=0 cores=0,2,4,6
/proc/14771/task/14793/stat : 14793 (python) S 14756 14756 14447 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 658004552 317575168 5823 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 514 18446604446170345600 0 0 -1 0 0 0 0
[pid=14784] ppid=14756 vsize=320520 CPUtime=0.3 cores=0,2,4,6
/proc/14784/stat : 14784 (python) S 14756 14756 14447 0 -1 4202560 1088 0 0 0 15 15 0 0 19 0 2 0 658004453 328212480 5844 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624937419 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/14784/statm: 80130 5844 400 364 0 16036 0
[pid=14786] ppid=14784 vsize=418636 CPUtime=98.77 cores=0,2,4,6
/proc/14786/stat : 14786 (adaptg2wsat2009) T 14784 14786 14786 0 -1 4202496 1216 0 0 0 9877 0 0 0 25 0 1 0 658004453 428683264 954 33554432000 134512640 135107411 4289891520 18446744073709551615 134531942 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/14786/statm: 104659 954 59 146 0 103535 0
[pid=14784/tid=14803] ppid=14756 vsize=320520 CPUtime=0 cores=0,2,4,6
/proc/14784/task/14803/stat : 14803 (python) S 14756 14756 14447 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 658014483 328212480 5844 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 514 18446604445403402688 0 0 -1 2 0 0 0
[pid=14789] ppid=14756 vsize=331316 CPUtime=0.3 cores=0,2,4,6
/proc/14789/stat : 14789 (python) S 14756 14756 14447 0 -1 4202560 1083 0 0 0 16 14 0 0 19 0 2 0 658004550 339267584 5983 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624937419 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/14789/statm: 82829 5983 399 364 0 18732 0
[pid=14795] ppid=14789 vsize=1018616 CPUtime=98.72 cores=0,2,4,6
/proc/14795/stat : 14795 (gnovelty+2) T 14789 14795 14795 0 -1 4202496 761 0 0 0 9872 0 0 0 25 0 1 0 658004552 1043062784 501 33554432000 134512640 135663704 4294305744 18446744073709551615 134545635 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/14795/statm: 254654 501 113 282 0 254370 0
[pid=14789/tid=14807] ppid=14756 vsize=331316 CPUtime=0 cores=0,2,4,6
/proc/14789/task/14807/stat : 14807 (python) S 14756 14756 14447 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 658014485 339267584 5983 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 514 18446604442567266752 0 0 -1 6 0 0 0
[pid=14792] ppid=14756 vsize=341572 CPUtime=0.3 cores=0,2,4,6
/proc/14792/stat : 14792 (python) S 14756 14756 14447 0 -1 4202560 1095 0 0 0 16 14 0 0 19 0 2 0 658004552 349769728 5987 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624937419 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/14792/statm: 85393 5987 399 364 0 21293 0
[pid=14796] ppid=14792 vsize=248648 CPUtime=99.98 cores=0,2,4,6
/proc/14796/stat : 14796 (FH) T 14792 14796 14796 0 -1 4202496 1117 0 0 0 9998 0 0 0 25 0 1 0 658004552 254615552 856 33554432000 134512640 134936988 4288450320 18446744073709551615 134545134 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/14796/statm: 62162 856 41 104 0 61666 0
[pid=14792/tid=14812] ppid=14756 vsize=341572 CPUtime=0 cores=0,2,4,6
/proc/14792/task/14812/stat : 14812 (python) S 14756 14756 14447 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 658014584 349769728 5987 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 514 18446604443110125952 0 0 -1 4 0 0 0
[pid=14797] ppid=14756 vsize=351828 CPUtime=0.3 cores=0,2,4,6
/proc/14797/stat : 14797 (python) S 14756 14756 14447 0 -1 4202560 1146 0 0 0 14 16 0 0 18 0 2 0 658004553 360271872 5997 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624937419 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/14797/statm: 87957 5997 406 364 0 23854 0
[pid=14799] ppid=14797 vsize=418636 CPUtime=99.98 cores=0,2,4,6
/proc/14799/stat : 14799 (adaptg2wsat2009) T 14797 14799 14799 0 -1 4202496 1224 0 0 0 9997 1 0 0 25 0 1 0 658004554 428683264 955 33554432000 134512640 135107411 4291406816 18446744073709551615 134531380 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/14799/statm: 104659 955 59 146 0 103535 0
[pid=14797/tid=14811] ppid=14756 vsize=351828 CPUtime=0 cores=0,2,4,6
/proc/14797/task/14811/stat : 14811 (python) S 14756 14756 14447 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 658014584 360271872 5997 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 514 18446604448911902656 0 0 -1 4 0 0 0
[pid=14804] ppid=14756 vsize=362084 CPUtime=0.32 cores=0,2,4,6
/proc/14804/stat : 14804 (python) S 14756 14756 14447 0 -1 4202560 1137 0 0 0 18 14 0 0 19 0 2 0 658014484 370774016 6001 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624937419 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/14804/statm: 90521 6001 406 364 0 26415 0
[pid=14806] ppid=14804 vsize=418636 CPUtime=98.02 cores=0,2,4,6
/proc/14806/stat : 14806 (adaptg2wsat2009) T 14804 14806 14806 0 -1 4202496 1224 0 0 0 9802 0 0 0 25 0 1 0 658014484 428683264 956 33554432000 134512640 135107411 4287705920 18446744073709551615 134531795 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/14806/statm: 104659 956 59 146 0 103535 0
[pid=14804/tid=14827] ppid=14756 vsize=362084 CPUtime=0 cores=0,2,4,6
/proc/14804/task/14827/stat : 14827 (python) S 14756 14756 14447 0 -1 4202560 51 0 0 0 0 0 0 0 19 0 2 0 658024515 370774016 6001 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 514 18446604453925556480 0 0 -1 6 0 0 0
[pid=14808] ppid=14756 vsize=372340 CPUtime=0.32 cores=0,2,4,6
/proc/14808/stat : 14808 (python) S 14756 14756 14447 0 -1 4202560 1119 0 0 0 16 16 0 0 19 0 2 0 658014486 381276160 5999 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624937419 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/14808/statm: 93085 5999 400 364 0 28976 0
[pid=14810] ppid=14808 vsize=248652 CPUtime=99.75 cores=0,2,4,6
/proc/14810/stat : 14810 (FH) T 14808 14810 14810 0 -1 4202496 1124 0 0 0 9972 3 0 0 25 0 1 0 658014486 254619648 858 33554432000 134512640 134936988 4291732256 18446744073709551615 134552382 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/14810/statm: 62163 858 41 104 0 61667 0
[pid=14808/tid=14831] ppid=14756 vsize=372340 CPUtime=0 cores=0,2,4,6
/proc/14808/task/14831/stat : 14831 (python) S 14756 14756 14447 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 658024527 381276160 5999 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 514 18446604441410381760 0 0 -1 0 0 0 0
[pid=14813] ppid=14756 vsize=382596 CPUtime=0.94 cores=0,2,4,6
/proc/14813/stat : 14813 (python) S 14756 14756 14447 0 -1 4202560 1173 0 0 0 51 43 0 0 18 0 2 0 658014586 391778304 6003 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624937419 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/14813/statm: 95649 6003 400 364 0 31537 0
[pid=14815] ppid=14813 vsize=268428 CPUtime=291.59 cores=0,2,4,6
/proc/14815/stat : 14815 (hybridGM3) T 14813 14815 14815 0 -1 4202496 5163 0 0 0 29158 1 0 0 25 0 1 0 658014586 274870272 4897 33554432000 134512640 135204385 4292699712 18446744073709551615 134538395 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/14815/statm: 67107 4897 63 169 0 66937 0
[pid=14813/tid=14854] ppid=14756 vsize=382596 CPUtime=0 cores=0,2,4,6
/proc/14813/task/14854/stat : 14854 (python) S 14756 14756 14447 0 -1 4202560 48 0 0 0 0 0 0 0 18 0 2 0 658034251 391778304 6003 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 514 18446604450591200576 0 0 -1 0 0 0 0
[pid=14816] ppid=14756 vsize=392852 CPUtime=0.31 cores=0,2,4,6
/proc/14816/stat : 14816 (python) S 14756 14756 14447 0 -1 4202560 1130 0 0 0 15 16 0 0 19 0 2 0 658014587 402280448 6007 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624937419 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/14816/statm: 98213 6007 400 364 0 34098 0
[pid=14818] ppid=14816 vsize=1018616 CPUtime=99.65 cores=0,2,4,6
/proc/14818/stat : 14818 (gnovelty+2) T 14816 14818 14818 0 -1 4202496 767 0 0 0 9965 0 0 0 25 0 1 0 658014587 1043062784 501 33554432000 134512640 135663704 4289590464 18446744073709551615 134525286 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/14818/statm: 254654 501 113 282 0 254370 0
[pid=14816/tid=14835] ppid=14756 vsize=392852 CPUtime=0 cores=0,2,4,6
/proc/14816/task/14835/stat : 14835 (python) S 14756 14756 14447 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 658024621 402280448 6007 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 514 18446604441410382656 0 0 -1 0 0 0 0
[pid=14828] ppid=14756 vsize=403108 CPUtime=0.66 cores=0,2,4,6
/proc/14828/stat : 14828 (python) S 14756 14756 14447 0 -1 4202560 1163 0 0 0 36 30 0 0 18 0 2 0 658024517 412782592 6014 33554432000 4194304 5683644 140733754910800 18446744073709551615 270613130822 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/14828/statm: 100777 6014 400 364 0 36659 0
[pid=14830] ppid=14828 vsize=268428 CPUtime=199.84 cores=0,2,4,6
/proc/14830/stat : 14830 (hybridGM3) R 14828 14830 14830 0 -1 4202496 5168 0 0 0 19984 0 0 0 25 0 1 0 658024517 274870272 4897 33554432000 134512640 135204385 4288850288 18446744073709551615 134537792 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/14830/statm: 67107 4897 63 169 0 66937 0
[pid=14828/tid=14919] ppid=14756 vsize=403108 CPUtime=0 cores=0,2,4,6
/proc/14828/task/14919/stat : 14919 (python) S 14756 14756 14447 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 658044084 412782592 6014 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 514 18446604445699664960 0 0 -1 4 0 0 0
[pid=14832] ppid=14756 vsize=413364 CPUtime=0.33 cores=0,2,4,6
/proc/14832/stat : 14832 (python) S 14756 14756 14447 0 -1 4202560 1157 0 0 0 17 16 0 0 19 0 2 0 658024528 423284736 6019 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624937419 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/14832/statm: 103341 6019 400 364 0 39220 0
[pid=14834] ppid=14832 vsize=418636 CPUtime=99.96 cores=0,2,4,6
/proc/14834/stat : 14834 (adaptg2wsat2009) T 14832 14834 14834 0 -1 4202496 1231 0 0 0 9996 0 0 0 25 0 1 0 658024529 428683264 957 33554432000 134512640 135107411 4288594448 18446744073709551615 134531696 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/14834/statm: 104659 957 59 146 0 103535 0
[pid=14832/tid=14856] ppid=14756 vsize=413364 CPUtime=0 cores=0,2,4,6
/proc/14832/task/14856/stat : 14856 (python) S 14756 14756 14447 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 658034560 423284736 6019 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 514 18446604448259460160 0 0 -1 0 0 0 0
[pid=14836] ppid=14756 vsize=423980 CPUtime=0.63 cores=0,2,4,6
/proc/14836/stat : 14836 (python) S 14756 14756 14447 0 -1 4202560 1062 0 0 0 35 28 0 0 18 0 1 0 658024622 434155520 6085 33554432000 4194304 5683644 140733754910800 18446744073709551615 270613130822 0 0 16781312 514 12788622052802134273 0 0 17 4 0 0 0
/proc/14836/statm: 105995 6085 377 364 0 41871 0
[pid=14838] ppid=14836 vsize=268424 CPUtime=196.88 cores=0,2,4,6
/proc/14838/stat : 14838 (hybridGM3) R 14836 14838 14838 0 -1 4202496 5172 0 0 0 19687 1 0 0 25 0 1 0 658024623 274866176 4895 33554432000 134512640 135204385 4290563328 18446744073709551615 134538541 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/14838/statm: 67106 4895 63 169 0 66936 0
[pid=14857] ppid=14756 vsize=434236 CPUtime=0.32 cores=0,2,4,6
/proc/14857/stat : 14857 (python) S 14756 14756 14447 0 -1 4202560 1186 0 0 0 16 16 0 0 19 0 2 0 658034562 444657664 6121 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624937419 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/14857/statm: 108559 6121 407 364 0 44432 0
[pid=14859] ppid=14857 vsize=248648 CPUtime=97.39 cores=0,2,4,6
/proc/14859/stat : 14859 (FH) T 14857 14859 14859 0 -1 4202496 1131 0 0 0 9736 3 0 0 25 0 1 0 658034562 254615552 856 33554432000 134512640 134936988 4291055104 18446744073709551615 134552375 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/14859/statm: 62162 856 41 104 0 61666 0
[pid=14857/tid=14920] ppid=14756 vsize=434236 CPUtime=0 cores=0,2,4,6
/proc/14857/task/14920/stat : 14920 (python) S 14756 14756 14447 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 658044595 444657664 6121 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 514 18446604447659094464 0 0 -1 6 0 0 0
[pid=14916] ppid=14756 vsize=444492 CPUtime=0.01 cores=0,2,4,6
/proc/14916/stat : 14916 (python) S 14756 14756 14447 0 -1 4202560 1006 0 0 0 0 1 0 0 18 0 1 0 658043783 455159808 6095 33554432000 4194304 5683644 140733754910800 18446744073709551615 270613130822 0 0 16781312 514 16101820302371029249 0 0 17 4 0 0 0
/proc/14916/statm: 111123 6095 379 364 0 46993 0
[pid=14918] ppid=14916 vsize=268428 CPUtime=8.21 cores=0,2,4,6
/proc/14918/stat : 14918 (hybridGM3) R 14916 14918 14918 0 -1 4202496 5178 0 0 0 821 0 0 0 25 0 1 0 658043784 274870272 4897 33554432000 134512640 135204385 4290893424 18446744073709551615 134538264 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/14918/statm: 67107 4897 63 169 0 66937 0
[pid=14921] ppid=14756 vsize=454748 CPUtime=0 cores=0,2,4,6
/proc/14921/stat : 14921 (python) S 14756 14756 14447 0 -1 4202560 813 0 0 0 0 0 0 0 19 0 1 0 658044597 465661952 6066 33554432000 4194304 5683644 140733754910800 18446744073709551615 270613130822 0 0 16781312 514 16101824150661758975 0 0 17 4 0 0 0
/proc/14921/statm: 113687 6066 346 364 0 49554 0
[pid=14923] ppid=14921 vsize=453032 CPUtime=0.11 cores=0,2,4,6
/proc/14923/stat : 14923 (TNM) R 14921 14923 14923 0 -1 4202496 956 0 0 0 11 0 0 0 21 0 1 0 658044597 463904768 685 33554432000 134512640 135008748 4293130192 18446744073709551615 134523493 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/14923/statm: 113258 685 45 122 0 112158 0
[pid=14756/tid=14763] ppid=14754 vsize=464992 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14763/stat : 14763 (python) S 14754 14756 14447 0 -1 4202560 24 933 0 0 0 0 0 0 20 0 19 0 657994514 476151808 7636 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 2 0 0 0
[pid=14756/tid=14766] ppid=14754 vsize=464992 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14766/stat : 14766 (python) S 14754 14756 14447 0 -1 4202560 25 933 0 0 0 0 0 0 22 0 19 0 657994515 476151808 7636 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 2 0 0 0
[pid=14756/tid=14769] ppid=14754 vsize=464992 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14769/stat : 14769 (python) S 14754 14756 14447 0 -1 4202560 25 933 0 0 0 0 0 0 24 0 19 0 657994516 476151808 7636 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 6 0 0 0
[pid=14756/tid=14772] ppid=14754 vsize=464992 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14772/stat : 14772 (python) S 14754 14756 14447 0 -1 4202560 29 933 0 0 0 0 0 0 25 0 19 0 657994517 476151808 7636 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 2 0 0 0
[pid=14756/tid=14785] ppid=14754 vsize=464992 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14785/stat : 14785 (python) S 14754 14756 14447 0 -1 4202560 28 933 0 0 0 0 0 0 20 0 19 0 658004453 476151808 7636 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 2 0 0 0
[pid=14756/tid=14790] ppid=14754 vsize=464992 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14790/stat : 14790 (python) S 14754 14756 14447 0 -1 4202560 23 933 0 0 0 0 0 0 20 0 19 0 658004550 476151808 7636 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 4 0 0 0
[pid=14756/tid=14794] ppid=14754 vsize=464992 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14794/stat : 14794 (python) S 14754 14756 14447 0 -1 4202560 26 933 0 0 0 0 0 0 21 0 19 0 658004552 476151808 7636 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 0 0 0 0
[pid=14756/tid=14798] ppid=14754 vsize=464992 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14798/stat : 14798 (python) S 14754 14756 14447 0 -1 4202560 28 933 0 0 0 0 0 0 23 0 19 0 658004554 476151808 7636 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 0 0 0 0
[pid=14756/tid=14805] ppid=14754 vsize=464992 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14805/stat : 14805 (python) S 14754 14756 14447 0 -1 4202560 23 933 0 0 0 0 0 0 20 0 19 0 658014484 476151808 7636 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 0 0 0 0
[pid=14756/tid=14809] ppid=14754 vsize=464992 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14809/stat : 14809 (python) S 14754 14756 14447 0 -1 4202560 26 933 0 0 0 0 0 0 21 0 19 0 658014486 476151808 7636 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 0 0 0 0
[pid=14756/tid=14814] ppid=14754 vsize=464992 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14814/stat : 14814 (python) S 14754 14756 14447 0 -1 4202560 26 933 0 0 0 0 0 0 18 0 19 0 658014586 476151808 7636 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 0 0 0 0
[pid=14756/tid=14817] ppid=14754 vsize=464992 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14817/stat : 14817 (python) S 14754 14756 14447 0 -1 4202560 27 933 0 0 0 0 0 0 22 0 19 0 658014587 476151808 7636 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 4 0 0 0
[pid=14756/tid=14829] ppid=14754 vsize=464992 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14829/stat : 14829 (python) S 14754 14756 14447 0 -1 4202560 31 933 0 0 0 0 0 0 18 0 19 0 658024517 476151808 7636 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 4 0 0 0
[pid=14756/tid=14833] ppid=14754 vsize=464992 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14833/stat : 14833 (python) S 14754 14756 14447 0 -1 4202560 28 933 0 0 0 0 0 0 20 0 19 0 658024529 476151808 7636 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 4 0 0 0
[pid=14756/tid=14837] ppid=14754 vsize=464992 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14837/stat : 14837 (python) S 14754 14756 14447 0 -1 4202560 24 933 0 0 0 0 0 0 20 0 19 0 658024623 476151808 7636 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 4 0 0 0
[pid=14756/tid=14858] ppid=14754 vsize=464992 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14858/stat : 14858 (python) S 14754 14756 14447 0 -1 4202560 27 933 0 0 0 0 0 0 20 0 19 0 658034562 476151808 7636 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 4 0 0 0
[pid=14756/tid=14917] ppid=14754 vsize=464992 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14917/stat : 14917 (python) S 14754 14756 14447 0 -1 4202560 27 933 0 0 0 0 0 0 20 0 19 0 658043783 476151808 7636 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 4 0 0 0
[pid=14756/tid=14922] ppid=14754 vsize=464992 CPUtime=0 cores=0,2,4,6
/proc/14756/task/14922/stat : 14922 (python) S 14754 14756 14447 0 -1 4202560 28 933 0 0 0 0 0 0 20 0 19 0 658044597 476151808 7636 33554432000 4194304 5683644 140733754910800 18446744073709551615 270624934321 0 0 16781312 2 18446604443954554176 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 1994.56
Current children cumulated vsize (KiB) 15942096

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 14756 and gives
#  childrusage.ru_utime.tv_sec=410
#  childrusage.ru_utime.tv_usec=198640
#  childrusage.ru_stime.tv_sec=3
#  childrusage.ru_stime.tv_usec=254505
# CPU time returned by wait4() is 413.453
# while last known CPU time is 1994.56
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 502.733
CPU time (s): 1994.56
CPU user time (s): 1991.34
CPU system time (s): 3.22
CPU usage (%): 396.743
Max. virtual memory (cumulated for all children) (KiB): 15942096

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 410.199
system time used= 3.2545
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 67214
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 7693
involuntary context switches= 2946

runsolver used 3.24551 second user time and 6.88195 second system time

The end

Launcher Data 

Begin job on node108 at 2011-05-15 14:52:14
IDJOB=3378229
IDBENCH=82638
IDSOLVER=1859
FILE ID=node108/3378229-1305463934
RUNJOBID= node108-1305463282-14465
PBS_JOBID= 13324989
Free space on /tmp= 73512 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/large/unif-k7-r85-v200-c17000-S970203146-047-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378229-1305463934/watcher-3378229-1305463934 -o /tmp/evaluation-result-3378229-1305463934/solver-3378229-1305463934 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378229-1305463934.cnf -seed 1942632377 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 635ea520fe16a78d41337039f98997e5
RANDOM SEED=1942632377

node108.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.64
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      16562368 kB
Buffers:        308476 kB
Cached:        5123452 kB
SwapCached:       8700 kB
Active:       11784620 kB
Inactive:      4271732 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      16562368 kB
SwapTotal:    67111528 kB
SwapFree:     67056804 kB
Dirty:           32168 kB
Writeback:           0 kB
AnonPages:    10621628 kB
Mapped:          37152 kB
Slab:           213624 kB
PageTables:      52560 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 80413404 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73480 MiB
End job on node108 at 2011-05-15 15:00:39