Trace number 3378288

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (MO) 1994.45 503.837

General information on the benchmark

Namerandom/large/
unif-k7-r85-v300-c25500-S415002759-031-UNKNOWN.cnf
MD5SUM6d4469ea6d617a494594ad07b4996367
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables300
Number of clauses25500
Sum of the clauses size178500
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 525500

Solver Data 

0.33/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.33/0.67	c INFO: solving HOME/instance-3378288-1305464512.cnf
0.33/0.67	c INFO: computing task features
0.49/0.85	c INFO: computed variable-clause graph statistics
0.49/0.86	c INFO: computed clause balance statistics
1.49/1.82	c INFO: computed variable balance statistics
1.91/2.21	c INFO: computed small-clause counts
1.91/2.21	c INFO: computed Horn-clause counts
1.91/2.25	c INFO: computed variable graph statistics
1.91/2.25	c INFO: collected features for HOME/instance-3378288-1305464512.cnf in 1.57 s
1.91/2.26	c INFO: running gnovelty+2@0 for 97 with 1198 remaining (b = 0.37)
1.91/2.27	c INFO: running gnovelty+2@0 for 97 with 1198 remaining (b = 0.13)
1.91/2.28	c INFO: running adaptg2wsat2009++@0 for 97 with 1198 remaining (b = 0.08)
1.91/2.29	c INFO: running adaptg2wsat2009++@0 for 97 with 1198 remaining (b = 0.07)
397.79/101.62	c INFO: running adaptg2wsat2009++@0 for 97 with 1099 remaining (b = 0.05)
401.77/102.69	c INFO: running gnovelty+2@0 for 97 with 1097 remaining (b = 0.04)
402.18/102.71	c INFO: running FH@0 for 97 with 1097 remaining (b = 0.03)
402.18/102.72	c INFO: running adaptg2wsat2009++@0 for 97 with 1097 remaining (b = 0.03)
791.80/200.98	c INFO: running adaptg2wsat2009++@0 for 97 with 999 remaining (b = 0.02)
800.13/203.05	c INFO: running FH@0 for 97 with 997 remaining (b = 0.02)
800.13/203.07	c INFO: running hybridGM3@0 for 194 with 997 remaining (b = 0.04)
800.49/203.10	c INFO: running gnovelty+2@0 for 97 with 997 remaining (b = 0.02)
1190.89/301.33	c INFO: running hybridGM3@0 for 194 with 899 remaining (b = 0.03)
1199.20/303.42	c INFO: running adaptg2wsat2009++@0 for 97 with 897 remaining (b = 0.01)
1215.47/307.53	c INFO: running hybridGM3@0 for 194 with 893 remaining (b = 0.03)
1582.28/399.82	c INFO: running hybridGM3@99 for 97 with 800 remaining (b = 0.01)
1618.04/408.83	c INFO: running hybridGM3@0 for 194 with 791 remaining (b = 0.02)
1960.26/495.16	c INFO: running TNM@0 for 705 with 705 remaining (b = 0.08)
1971.78/498.07	c INFO: running hybridGM3@99 for 97 with 702 remaining (b = 0.01)
1992.79/503.32	c INFO: running FH@0 for 97 with 697 remaining (b = 0.01)
1994.45/503.81	c INFO: running adaptg2wsat2009++@0 for 97 with 696 remaining (b = 0.01)
1994.45/503.81	Traceback (most recent call last):
1994.45/503.81	  File "HOME/solve", line 33, in <module>
1994.45/503.81	    plac.call(main)
1994.45/503.81	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
1994.45/503.81	    cmd, result = parser_from(obj).consume(arglist)
1994.45/503.81	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
1994.45/503.81	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
1994.45/503.81	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
1994.45/503.81	    (_, answer, _) = solver(input_path, budget / cores, cores)
1994.45/503.81	  File "HOME/borg/build/lib/borg/bilevel.py", line 202, in __call__
1994.45/503.81	    solver.go(max_cost * borg.defaults.machine_speed)
1994.45/503.81	  File "HOME/borg/build/lib/borg/solvers.py", line 179, in go
1994.45/503.81	    self._process.start()
1994.45/503.81	  File "/home/evaluation/evaluation/tools64/lib/python2.7/multiprocessing/process.py", line 94, in start
1994.45/503.81	    assert self._popen is None, 'cannot start a process twice'
1994.45/503.81	AssertionError: cannot start a process twice

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378288-1305464512/watcher-3378288-1305464512 -o /tmp/evaluation-result-3378288-1305464512/solver-3378288-1305464512 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378288-1305464512.cnf -seed 972504358 -budget 4800 -cores 4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.89 8.37 7.03 6/421 15016
/proc/meminfo: memFree=16387648/32951124 swapFree=67056804/67111528
[pid=15016] ppid=15014 vsize=15772 CPUtime=0 cores=0,2,4,6
/proc/15016/stat : 15016 (python) R 15014 15016 14447 0 -1 4202496 892 0 0 0 0 0 0 0 25 0 1 0 658052171 16150528 774 33554432000 4194304 5683644 140734381656512 18446744073709551615 270612771525 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/15016/statm: 3943 775 343 364 0 430 0

[startup+0.049942 s]
/proc/loadavg: 8.89 8.37 7.03 6/421 15016
/proc/meminfo: memFree=16387648/32951124 swapFree=67056804/67111528
[pid=15016] ppid=15014 vsize=18880 CPUtime=0.02 cores=0,2,4,6
/proc/15016/stat : 15016 (python) S 15014 15016 14447 0 -1 4202496 1287 0 0 0 2 0 0 0 25 0 1 0 658052171 19333120 1140 33554432000 4194304 5683644 140734381656512 18446744073709551615 270613108736 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/15016/statm: 4720 1140 421 364 0 689 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 18880

[startup+0.100566 s]
/proc/loadavg: 8.89 8.37 7.03 6/421 15016
/proc/meminfo: memFree=16387648/32951124 swapFree=67056804/67111528
[pid=15016] ppid=15014 vsize=85620 CPUtime=0.05 cores=0,2,4,6
/proc/15016/stat : 15016 (python) S 15014 15016 14447 0 -1 4202496 2630 0 0 0 4 1 0 0 18 0 1 0 658052171 87674880 2348 33554432000 4194304 5683644 140734381656512 18446744073709551615 270608187015 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/15016/statm: 21405 2348 713 364 0 1622 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 85620

[startup+0.300512 s]
/proc/loadavg: 8.89 8.37 7.03 6/421 15016
/proc/meminfo: memFree=16387648/32951124 swapFree=67056804/67111528
[pid=15016] ppid=15014 vsize=135748 CPUtime=0.16 cores=0,2,4,6
/proc/15016/stat : 15016 (python) S 15014 15016 14447 0 -1 4202496 5964 937 1 0 11 5 0 0 18 0 1 0 658052171 139005952 4151 33554432000 4194304 5683644 140734381656512 18446744073709551615 270613106693 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/15016/statm: 33937 4151 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.16
Current children cumulated vsize (KiB) 135748

[startup+0.706477 s]
/proc/loadavg: 8.89 8.37 7.03 6/421 15016
/proc/meminfo: memFree=16387648/32951124 swapFree=67056804/67111528
[pid=15016] ppid=15014 vsize=279708 CPUtime=0.4 cores=0,2,4,6
/proc/15016/stat : 15016 (python) R 15014 15016 14447 0 -1 4202496 9594 937 1 0 30 10 0 0 17 0 1 0 658052171 286420992 7275 33554432000 4194304 5683644 140734381656512 18446744073709551615 4445115 0 0 16781312 2 0 0 0 17 0 0 0 0
/proc/15016/statm: 69927 7275 1761 364 0 5851 0
Current children cumulated CPU time (s) 0.4
Current children cumulated vsize (KiB) 279708

[startup+1.50034 s]
/proc/loadavg: 8.89 8.37 7.03 7/422 15021
/proc/meminfo: memFree=16358500/32951124 swapFree=67056804/67111528
[pid=15016] ppid=15014 vsize=285632 CPUtime=1.19 cores=0,2,4,6
/proc/15016/stat : 15016 (python) R 15014 15016 14447 0 -1 4202496 12502 937 1 0 109 10 0 0 18 0 1 0 658052171 292487168 8793 33554432000 4194304 5683644 140734381656512 18446744073709551615 270612797150 0 0 16781312 2 0 0 0 17 0 0 0 0
/proc/15016/statm: 71408 8793 1826 364 0 7332 0
Current children cumulated CPU time (s) 1.19
Current children cumulated vsize (KiB) 285632

[startup+3.10101 s]
/proc/loadavg: 8.89 8.37 7.03 7/422 15021
/proc/meminfo: memFree=16357632/32951124 swapFree=67056804/67111528
[pid=15016] ppid=15014 vsize=321284 CPUtime=1.98 cores=0,2,4,6
/proc/15016/stat : 15016 (python) S 15014 15016 14447 0 -1 4202496 15275 937 1 0 187 11 0 0 25 0 5 0 658052171 328994816 7529 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624937419 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/15016/statm: 80321 7529 1875 364 0 16230 0
Current children cumulated CPU time (s) 1.98
Current children cumulated vsize (KiB) 321284
heavy processes:

[startup+6.30145 s]
/proc/loadavg: 8.90 8.38 7.04 10/434 15033
/proc/meminfo: memFree=16333080/32951124 swapFree=67056804/67111528
[pid=15016] ppid=15014 vsize=321284 CPUtime=1.98 cores=0,2,4,6
/proc/15016/stat : 15016 (python) S 15014 15016 14447 0 -1 4202496 15275 937 1 0 187 11 0 0 25 0 5 0 658052171 328994816 7529 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624937419 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/15016/statm: 80321 7529 1875 364 0 16230 0
[pid=15022] ppid=15016 vsize=280756 CPUtime=0 cores=0,2,4,6
/proc/15022/stat : 15022 (python) S 15016 15016 14447 0 -1 4202560 720 0 0 0 0 0 0 0 18 0 1 0 658052397 287494144 5977 33554432000 4194304 5683644 140734381656512 18446744073709551615 270613130767 0 0 16781312 514 14329453188128669954 0 0 17 4 0 0 0
/proc/15022/statm: 70189 5977 355 364 0 6107 0
[pid=15024] ppid=15022 vsize=1019708 CPUtime=4.02 cores=0,2,4,6
/proc/15024/stat : 15024 (gnovelty+2) R 15022 15024 15024 0 -1 4202496 954 0 0 0 401 1 0 0 25 0 1 0 658052397 1044180992 690 33554432000 134512640 135663704 4292002560 18446744073709551615 135007681 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/15024/statm: 254927 690 113 282 0 254643 0
[pid=15025] ppid=15016 vsize=291080 CPUtime=0 cores=0,2,4,6
/proc/15025/stat : 15025 (python) S 15016 15016 14447 0 -1 4202560 733 0 0 0 0 0 0 0 18 0 1 0 658052398 298065920 5975 33554432000 4194304 5683644 140734381656512 18446744073709551615 270613130822 0 0 16781312 514 11351676632948506883 0 0 17 6 0 0 0
/proc/15025/statm: 72770 5975 349 364 0 8685 0
[pid=15027] ppid=15025 vsize=1019708 CPUtime=4.02 cores=0,2,4,6
/proc/15027/stat : 15027 (gnovelty+2) R 15025 15027 15027 0 -1 4202496 953 0 0 0 402 0 0 0 25 0 1 0 658052398 1044180992 689 33554432000 134512640 135663704 4294127952 18446744073709551615 134525302 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/15027/statm: 254927 689 113 282 0 254643 0
[pid=15028] ppid=15016 vsize=301268 CPUtime=0 cores=0,2,4,6
/proc/15028/stat : 15028 (python) R 15016 15016 14447 0 -1 4202560 887 0 0 0 0 0 0 0 18 0 1 0 658052399 308498432 6013 33554432000 4194304 5683644 140734381656512 18446744073709551615 270613106612 0 0 16781312 514 0 0 0 17 6 0 0 0
/proc/15028/statm: 75317 6013 366 364 0 11229 0
[pid=15030] ppid=15028 vsize=420220 CPUtime=4.01 cores=0,2,4,6
/proc/15030/stat : 15030 (adaptg2wsat2009) R 15028 15030 15030 0 -1 4202496 1667 0 0 0 401 0 0 0 25 0 1 0 658052400 430305280 1401 33554432000 134512640 135107411 4288201376 18446744073709551615 134531792 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/15030/statm: 105055 1401 59 146 0 103931 0
[pid=15031] ppid=15016 vsize=311592 CPUtime=0 cores=0,2,4,6
/proc/15031/stat : 15031 (python) R 15016 15016 14447 0 -1 4202560 740 0 0 0 0 0 0 0 18 0 1 0 658052400 319070208 6000 33554432000 4194304 5683644 140734381656512 18446744073709551615 270613130822 0 0 16781312 514 0 0 0 17 6 0 0 0
/proc/15031/statm: 77898 6000 349 364 0 13807 0
[pid=15033] ppid=15031 vsize=419820 CPUtime=3.93 cores=0,2,4,6
/proc/15033/stat : 15033 (adaptg2wsat2009) R 15031 15033 15033 0 -1 4202496 1434 0 0 0 393 0 0 0 25 0 1 0 658052401 429895680 1172 33554432000 134512640 135107411 4290234208 18446744073709551615 134523217 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/15033/statm: 104955 1172 49 146 0 103831 0
[pid=15016/tid=15023] ppid=15014 vsize=321284 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15023/stat : 15023 (python) S 15014 15016 14447 0 -1 4202560 24 937 0 0 0 0 0 0 21 0 5 0 658052397 328994816 7529 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 6 0 0 0
[pid=15016/tid=15026] ppid=15014 vsize=321284 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15026/stat : 15026 (python) S 15014 15016 14447 0 -1 4202560 26 937 0 0 0 0 0 0 23 0 5 0 658052398 328994816 7529 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 6 0 0 0
[pid=15016/tid=15029] ppid=15014 vsize=321284 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15029/stat : 15029 (python) S 15014 15016 14447 0 -1 4202560 24 937 0 0 0 0 0 0 25 0 5 0 658052399 328994816 7529 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 6 0 0 0
[pid=15016/tid=15032] ppid=15014 vsize=321284 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15032/stat : 15032 (python) S 15014 15016 14447 0 -1 4202560 24 937 0 0 0 0 0 0 25 0 5 0 658052400 328994816 7529 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 17.96
Current children cumulated vsize (KiB) 4385436

[startup+12.7013 s]
/proc/loadavg: 8.91 8.39 7.05 10/434 15034
/proc/meminfo: memFree=16329180/32951124 swapFree=67056804/67111528
[pid=15016] ppid=15014 vsize=321284 CPUtime=1.98 cores=0,2,4,6
/proc/15016/stat : 15016 (python) S 15014 15016 14447 0 -1 4202496 15275 937 1 0 187 11 0 0 25 0 5 0 658052171 328994816 7529 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624937419 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/15016/statm: 80321 7529 1875 364 0 16230 0
[pid=15022] ppid=15016 vsize=280756 CPUtime=0.03 cores=0,2,4,6
/proc/15022/stat : 15022 (python) S 15016 15016 14447 0 -1 4202560 879 0 0 0 1 2 0 0 18 0 1 0 658052397 287494144 6000 33554432000 4194304 5683644 140734381656512 18446744073709551615 270613130767 0 0 16781312 514 11366610470459900162 0 0 17 4 0 0 0
/proc/15022/statm: 70189 6000 378 364 0 6107 0
[pid=15024] ppid=15022 vsize=1019708 CPUtime=10.41 cores=0,2,4,6
/proc/15024/stat : 15024 (gnovelty+2) R 15022 15024 15024 0 -1 4202496 954 0 0 0 1040 1 0 0 25 0 1 0 658052397 1044180992 690 33554432000 134512640 135663704 4292002560 18446744073709551615 134524868 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/15024/statm: 254927 690 113 282 0 254643 0
[pid=15025] ppid=15016 vsize=291080 CPUtime=0.03 cores=0,2,4,6
/proc/15025/stat : 15025 (python) S 15016 15016 14447 0 -1 4202560 900 0 0 0 2 1 0 0 18 0 1 0 658052398 298065920 5998 33554432000 4194304 5683644 140734381656512 18446744073709551615 270613130822 0 0 16781312 514 11366610470459900163 0 0 17 6 0 0 0
/proc/15025/statm: 72770 5998 372 364 0 8685 0
[pid=15027] ppid=15025 vsize=1019708 CPUtime=10.38 cores=0,2,4,6
/proc/15027/stat : 15027 (gnovelty+2) R 15025 15027 15027 0 -1 4202496 953 0 0 0 1038 0 0 0 25 0 1 0 658052398 1044180992 689 33554432000 134512640 135663704 4294127952 18446744073709551615 134545682 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/15027/statm: 254927 689 113 282 0 254643 0
[pid=15028] ppid=15016 vsize=301268 CPUtime=0.03 cores=0,2,4,6
/proc/15028/stat : 15028 (python) S 15016 15016 14447 0 -1 4202560 909 0 0 0 2 1 0 0 18 0 1 0 658052399 308498432 6019 33554432000 4194304 5683644 140734381656512 18446744073709551615 270613130822 0 0 16781312 514 11366610470459900162 0 0 17 6 0 0 0
/proc/15028/statm: 75317 6019 372 364 0 11229 0
[pid=15030] ppid=15028 vsize=420220 CPUtime=10.41 cores=0,2,4,6
/proc/15030/stat : 15030 (adaptg2wsat2009) R 15028 15030 15030 0 -1 4202496 1667 0 0 0 1041 0 0 0 25 0 1 0 658052400 430305280 1401 33554432000 134512640 135107411 4288201376 18446744073709551615 134532128 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/15030/statm: 105055 1401 59 146 0 103931 0
[pid=15031] ppid=15016 vsize=311592 CPUtime=0.03 cores=0,2,4,6
/proc/15031/stat : 15031 (python) S 15016 15016 14447 0 -1 4202560 913 0 0 0 2 1 0 0 18 0 1 0 658052400 319070208 6023 33554432000 4194304 5683644 140734381656512 18446744073709551615 270613130822 0 0 16781312 514 11366610470459900161 0 0 17 6 0 0 0
/proc/15031/statm: 77898 6023 372 364 0 13807 0
[pid=15033] ppid=15031 vsize=420220 CPUtime=10.14 cores=0,2,4,6
/proc/15033/stat : 15033 (adaptg2wsat2009) R 15031 15033 15033 0 -1 4202496 1662 0 0 0 1014 0 0 0 25 0 1 0 658052401 430305280 1400 33554432000 134512640 135107411 4290234208 18446744073709551615 134531928 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/15033/statm: 105055 1400 59 146 0 103931 0
[pid=15016/tid=15023] ppid=15014 vsize=321284 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15023/stat : 15023 (python) S 15014 15016 14447 0 -1 4202560 24 937 0 0 0 0 0 0 21 0 5 0 658052397 328994816 7529 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 6 0 0 0
[pid=15016/tid=15026] ppid=15014 vsize=321284 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15026/stat : 15026 (python) S 15014 15016 14447 0 -1 4202560 26 937 0 0 0 0 0 0 23 0 5 0 658052398 328994816 7529 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 6 0 0 0
[pid=15016/tid=15029] ppid=15014 vsize=321284 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15029/stat : 15029 (python) S 15014 15016 14447 0 -1 4202560 24 937 0 0 0 0 0 0 25 0 5 0 658052399 328994816 7529 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 6 0 0 0
[pid=15016/tid=15032] ppid=15014 vsize=321284 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15032/stat : 15032 (python) S 15014 15016 14447 0 -1 4202560 24 937 0 0 0 0 0 0 25 0 5 0 658052400 328994816 7529 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 43.44
Current children cumulated vsize (KiB) 4385836

[startup+25.5011 s]
/proc/loadavg: 8.93 8.41 7.08 10/438 15038
/proc/meminfo: memFree=16318368/32951124 swapFree=67056804/67111528
[pid=15016] ppid=15014 vsize=321284 CPUtime=1.98 cores=0,2,4,6
/proc/15016/stat : 15016 (python) S 15014 15016 14447 0 -1 4202496 15275 937 1 0 187 11 0 0 25 0 5 0 658052171 328994816 7529 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624937419 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/15016/statm: 80321 7529 1875 364 0 16230 0
[pid=15022] ppid=15016 vsize=280756 CPUtime=0.06 cores=0,2,4,6
/proc/15022/stat : 15022 (python) S 15016 15016 14447 0 -1 4202560 888 0 0 0 3 3 0 0 18 0 1 0 658052397 287494144 6000 33554432000 4194304 5683644 140734381656512 18446744073709551615 270613130767 0 0 16781312 514 11366610470459900162 0 0 17 4 0 0 0
/proc/15022/statm: 70189 6000 378 364 0 6107 0
[pid=15024] ppid=15022 vsize=1019708 CPUtime=23.21 cores=0,2,4,6
/proc/15024/stat : 15024 (gnovelty+2) R 15022 15024 15024 0 -1 4202496 954 0 0 0 2320 1 0 0 25 0 1 0 658052397 1044180992 690 33554432000 134512640 135663704 4292002560 18446744073709551615 134525224 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/15024/statm: 254927 690 113 282 0 254643 0
[pid=15025] ppid=15016 vsize=291080 CPUtime=0.06 cores=0,2,4,6
/proc/15025/stat : 15025 (python) S 15016 15016 14447 0 -1 4202560 906 0 0 0 4 2 0 0 18 0 1 0 658052398 298065920 5998 33554432000 4194304 5683644 140734381656512 18446744073709551615 270613130822 0 0 16781312 514 11366610470459900163 0 0 17 4 0 0 0
/proc/15025/statm: 72770 5998 372 364 0 8685 0
[pid=15027] ppid=15025 vsize=1019708 CPUtime=23.14 cores=0,2,4,6
/proc/15027/stat : 15027 (gnovelty+2) R 15025 15027 15027 0 -1 4202496 953 0 0 0 2314 0 0 0 25 0 1 0 658052398 1044180992 689 33554432000 134512640 135663704 4294127952 18446744073709551615 134525678 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/15027/statm: 254927 689 113 282 0 254643 0
[pid=15028] ppid=15016 vsize=301268 CPUtime=0.07 cores=0,2,4,6
/proc/15028/stat : 15028 (python) S 15016 15016 14447 0 -1 4202560 914 0 0 0 4 3 0 0 18 0 1 0 658052399 308498432 6019 33554432000 4194304 5683644 140734381656512 18446744073709551615 270613130822 0 0 16781312 514 11366610470459900162 0 0 17 6 0 0 0
/proc/15028/statm: 75317 6019 372 364 0 11229 0
[pid=15030] ppid=15028 vsize=420220 CPUtime=23.21 cores=0,2,4,6
/proc/15030/stat : 15030 (adaptg2wsat2009) R 15028 15030 15030 0 -1 4202496 1667 0 0 0 2321 0 0 0 25 0 1 0 658052400 430305280 1401 33554432000 134512640 135107411 4288201376 18446744073709551615 134532107 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/15030/statm: 105055 1401 59 146 0 103931 0
[pid=15031] ppid=15016 vsize=311592 CPUtime=0.07 cores=0,2,4,6
/proc/15031/stat : 15031 (python) S 15016 15016 14447 0 -1 4202560 917 0 0 0 4 3 0 0 18 0 1 0 658052400 319070208 6023 33554432000 4194304 5683644 140734381656512 18446744073709551615 270613130822 0 0 16781312 514 11366610470459900161 0 0 17 6 0 0 0
/proc/15031/statm: 77898 6023 372 364 0 13807 0
[pid=15033] ppid=15031 vsize=420220 CPUtime=22.65 cores=0,2,4,6
/proc/15033/stat : 15033 (adaptg2wsat2009) R 15031 15033 15033 0 -1 4202496 1662 0 0 0 2265 0 0 0 25 0 1 0 658052401 430305280 1400 33554432000 134512640 135107411 4290234208 18446744073709551615 134532352 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/15033/statm: 105055 1400 59 146 0 103931 0
[pid=15016/tid=15023] ppid=15014 vsize=321284 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15023/stat : 15023 (python) S 15014 15016 14447 0 -1 4202560 24 937 0 0 0 0 0 0 21 0 5 0 658052397 328994816 7529 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 6 0 0 0
[pid=15016/tid=15026] ppid=15014 vsize=321284 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15026/stat : 15026 (python) S 15014 15016 14447 0 -1 4202560 26 937 0 0 0 0 0 0 23 0 5 0 658052398 328994816 7529 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 6 0 0 0
[pid=15016/tid=15029] ppid=15014 vsize=321284 CPUtime=0 cores=0,2,4,6

################
# More data... #
################

/proc/15030/stat : 15030 (adaptg2wsat2009) T 15028 15030 15030 0 -1 4202496 1667 0 0 0 9918 1 0 0 25 0 1 0 658052400 430305280 1401 33554432000 134512640 135107411 4288201376 18446744073709551615 134531781 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/15030/statm: 105055 1401 59 146 0 103931 0
[pid=15028/tid=15053] ppid=15016 vsize=301268 CPUtime=0 cores=0,2,4,6
/proc/15028/task/15053/stat : 15053 (python) S 15016 15016 14447 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 658062333 308498432 6048 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 514 18446604450606913408 0 0 -1 0 0 0 0
[pid=15031] ppid=15016 vsize=311592 CPUtime=0.31 cores=0,2,4,6
/proc/15031/stat : 15031 (python) S 15016 15016 14447 0 -1 4202560 1091 0 0 0 17 14 0 0 19 0 2 0 658052400 319070208 6052 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624937419 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/15031/statm: 77898 6052 400 364 0 13807 0
[pid=15033] ppid=15031 vsize=420220 CPUtime=98.32 cores=0,2,4,6
/proc/15033/stat : 15033 (adaptg2wsat2009) T 15031 15033 15033 0 -1 4202496 1662 0 0 0 9832 0 0 0 25 0 1 0 658052401 430305280 1400 33554432000 134512640 135107411 4290234208 18446744073709551615 134532242 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/15033/statm: 105055 1400 59 146 0 103931 0
[pid=15031/tid=15061] ppid=15016 vsize=311592 CPUtime=0 cores=0,2,4,6
/proc/15031/task/15061/stat : 15061 (python) S 15016 15016 14447 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 658062442 319070208 6052 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 514 18446604450606911616 0 0 -1 4 0 0 0
[pid=15054] ppid=15016 vsize=321924 CPUtime=0.32 cores=0,2,4,6
/proc/15054/stat : 15054 (python) S 15016 15016 14447 0 -1 4202560 1085 0 0 0 15 17 0 0 19 0 2 0 658062333 329650176 6056 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624937419 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/15054/statm: 80481 6056 400 364 0 16387 0
[pid=15056] ppid=15054 vsize=420220 CPUtime=98.24 cores=0,2,4,6
/proc/15056/stat : 15056 (adaptg2wsat2009) T 15054 15056 15056 0 -1 4202496 1663 0 0 0 9821 3 0 0 25 0 1 0 658062334 430305280 1400 33554432000 134512640 135107411 4289890016 18446744073709551615 134531508 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/15056/statm: 105055 1400 59 146 0 103931 0
[pid=15054/tid=15087] ppid=15016 vsize=321924 CPUtime=0 cores=0,2,4,6
/proc/15054/task/15087/stat : 15087 (python) S 15016 15016 14447 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 658072269 329650176 6056 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 514 18446604448815303488 0 0 -1 6 0 0 0
[pid=15058] ppid=15016 vsize=332260 CPUtime=0.33 cores=0,2,4,6
/proc/15058/stat : 15058 (python) S 15016 15016 14447 0 -1 4202560 1096 0 0 0 19 14 0 0 19 0 2 0 658062441 340234240 6079 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624937419 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/15058/statm: 83065 6079 400 364 0 18968 0
[pid=15062] ppid=15058 vsize=1019708 CPUtime=99.73 cores=0,2,4,6
/proc/15062/stat : 15062 (gnovelty+2) T 15058 15062 15062 0 -1 4202496 948 0 0 0 9973 0 0 0 25 0 1 0 658062442 1044180992 690 33554432000 134512640 135663704 4288340208 18446744073709551615 134545583 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/15062/statm: 254927 690 113 282 0 254643 0
[pid=15058/tid=15099] ppid=15016 vsize=332260 CPUtime=0 cores=0,2,4,6
/proc/15058/task/15099/stat : 15099 (python) S 15016 15016 14447 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 658072480 340234240 6079 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 514 18446604453382058304 0 0 -1 0 0 0 0
[pid=15063] ppid=15016 vsize=342464 CPUtime=0.32 cores=0,2,4,6
/proc/15063/stat : 15063 (python) S 15016 15016 14447 0 -1 4202560 1100 0 0 0 18 14 0 0 19 0 2 0 658062442 350683136 6103 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624937419 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/15063/statm: 85616 6103 400 364 0 21516 0
[pid=15065] ppid=15063 vsize=249868 CPUtime=97.66 cores=0,2,4,6
/proc/15065/stat : 15065 (FH) T 15063 15065 15065 0 -1 4202496 1528 0 0 0 9766 0 0 0 25 0 1 0 658062442 255864832 1263 33554432000 134512640 134936988 4292079616 18446744073709551615 134554594 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/15065/statm: 62467 1263 41 104 0 61971 0
[pid=15063/tid=15091] ppid=15016 vsize=342464 CPUtime=0 cores=0,2,4,6
/proc/15063/task/15091/stat : 15091 (python) S 15016 15016 14447 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 658072475 350683136 6103 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 514 18446604447996609664 0 0 -1 6 0 0 0
[pid=15066] ppid=15016 vsize=352820 CPUtime=0.33 cores=0,2,4,6
/proc/15066/stat : 15066 (python) S 15016 15016 14447 0 -1 4202560 1140 0 0 0 19 14 0 0 19 0 2 0 658062444 361287680 6113 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624937419 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/15066/statm: 88205 6113 406 364 0 24102 0
[pid=15068] ppid=15066 vsize=420220 CPUtime=100.3 cores=0,2,4,6
/proc/15068/stat : 15068 (adaptg2wsat2009) T 15066 15068 15068 0 -1 4202496 1670 0 0 0 10030 0 0 0 25 0 1 0 658062444 430305280 1400 33554432000 134512640 135107411 4289859248 18446744073709551615 134532277 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/15068/statm: 105055 1400 59 146 0 103931 0
[pid=15066/tid=15095] ppid=15016 vsize=352820 CPUtime=0 cores=0,2,4,6
/proc/15066/task/15095/stat : 15095 (python) S 15016 15016 14447 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 658072477 361287680 6113 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 514 18446604445403402688 0 0 -1 0 0 0 0
[pid=15088] ppid=15016 vsize=363180 CPUtime=0.33 cores=0,2,4,6
/proc/15088/stat : 15088 (python) S 15016 15016 14447 0 -1 4202560 1139 0 0 0 17 16 0 0 19 0 2 0 658072270 371896320 6129 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624937419 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/15088/statm: 90795 6129 406 364 0 26689 0
[pid=15090] ppid=15088 vsize=420220 CPUtime=100.27 cores=0,2,4,6
/proc/15090/stat : 15090 (adaptg2wsat2009) T 15088 15090 15090 0 -1 4202496 1672 0 0 0 10027 0 0 0 25 0 1 0 658072270 430305280 1400 33554432000 134512640 135107411 4293457728 18446744073709551615 134539273 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/15090/statm: 105055 1400 59 146 0 103931 0
[pid=15088/tid=15109] ppid=15016 vsize=363180 CPUtime=0 cores=0,2,4,6
/proc/15088/task/15109/stat : 15109 (python) S 15016 15016 14447 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 658082303 371896320 6129 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 514 18446604443175335040 0 0 -1 0 0 0 0
[pid=15092] ppid=15016 vsize=373356 CPUtime=0.34 cores=0,2,4,6
/proc/15092/stat : 15092 (python) S 15016 15016 14447 0 -1 4202560 1112 0 0 0 17 17 0 0 19 0 2 0 658072477 382316544 6153 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624937419 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/15092/statm: 93339 6153 400 364 0 29230 0
[pid=15094] ppid=15092 vsize=249872 CPUtime=99.68 cores=0,2,4,6
/proc/15094/stat : 15094 (FH) T 15092 15094 15094 0 -1 4202496 1529 0 0 0 9966 2 0 0 25 0 1 0 658072477 255868928 1264 33554432000 134512640 134936988 4289429568 18446744073709551615 134553428 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/15094/statm: 62468 1264 41 104 0 61972 0
[pid=15092/tid=15113] ppid=15016 vsize=373356 CPUtime=0 cores=0,2,4,6
/proc/15092/task/15113/stat : 15113 (python) S 15016 15016 14447 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 658082512 382316544 6153 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 514 18446604450606914304 0 0 -1 0 0 0 0
[pid=15096] ppid=15016 vsize=383852 CPUtime=0.99 cores=0,2,4,6
/proc/15096/stat : 15096 (python) S 15016 15016 14447 0 -1 4202560 1173 0 0 0 50 49 0 0 18 0 2 0 658072479 393064448 6157 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624937419 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/15096/statm: 95963 6157 400 364 0 31851 0
[pid=15098] ppid=15096 vsize=268420 CPUtime=291.66 cores=0,2,4,6
/proc/15098/stat : 15098 (hybridGM3) T 15096 15098 15098 0 -1 4202496 5393 0 0 0 29166 0 0 0 25 0 1 0 658072479 274862080 5125 33554432000 134512640 135204385 4289580112 18446744073709551615 134537704 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/15098/statm: 67105 5125 63 169 0 66935 0
[pid=15096/tid=15133] ppid=15016 vsize=383852 CPUtime=0 cores=0,2,4,6
/proc/15096/task/15133/stat : 15133 (python) S 15016 15016 14447 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 658092153 393064448 6157 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 514 18446604446335890368 0 0 -1 0 0 0 0
[pid=15100] ppid=15016 vsize=394236 CPUtime=0.35 cores=0,2,4,6
/proc/15100/stat : 15100 (python) S 15016 15016 14447 0 -1 4202560 1129 0 0 0 18 17 0 0 19 0 2 0 658072482 403697664 6201 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624937419 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/15100/statm: 98559 6201 400 364 0 34444 0
[pid=15102] ppid=15100 vsize=1019708 CPUtime=101.22 cores=0,2,4,6
/proc/15102/stat : 15102 (gnovelty+2) T 15100 15102 15102 0 -1 4202496 957 0 0 0 10122 0 0 0 25 0 1 0 658072482 1044180992 690 33554432000 134512640 135663704 4286736192 18446744073709551615 134544196 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/15102/statm: 254927 690 113 282 0 254643 0
[pid=15100/tid=15117] ppid=15016 vsize=394236 CPUtime=0 cores=0,2,4,6
/proc/15100/task/15117/stat : 15117 (python) S 15016 15016 14447 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 658082923 403697664 6201 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 514 18446604441410380864 0 0 -1 0 0 0 0
[pid=15110] ppid=15016 vsize=404628 CPUtime=0.7 cores=0,2,4,6
/proc/15110/stat : 15110 (python) S 15016 15016 14447 0 -1 4202560 1174 0 0 0 36 34 0 0 18 0 2 0 658082304 414339072 6237 33554432000 4194304 5683644 140734381656512 18446744073709551615 270613130822 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/15110/statm: 101157 6237 400 364 0 37039 0
[pid=15112] ppid=15110 vsize=268424 CPUtime=201.4 cores=0,2,4,6
/proc/15112/stat : 15112 (hybridGM3) R 15110 15112 15112 0 -1 4202496 5395 0 0 0 20140 0 0 0 25 0 1 0 658082305 274866176 5127 33554432000 134512640 135204385 4287969936 18446744073709551615 134538264 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/15112/statm: 67106 5127 63 169 0 66936 0
[pid=15110/tid=15144] ppid=15016 vsize=404628 CPUtime=0 cores=0,2,4,6
/proc/15110/task/15144/stat : 15144 (python) S 15016 15016 14447 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 658101977 414339072 6237 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 514 18446604445403405376 0 0 -1 4 0 0 0
[pid=15114] ppid=15016 vsize=415024 CPUtime=0.37 cores=0,2,4,6
/proc/15114/stat : 15114 (python) S 15016 15016 14447 0 -1 4202560 1156 0 0 0 19 18 0 0 19 0 2 0 658082513 424984576 6275 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624937419 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/15114/statm: 103756 6275 400 364 0 39635 0
[pid=15116] ppid=15114 vsize=420220 CPUtime=102.26 cores=0,2,4,6
/proc/15116/stat : 15116 (adaptg2wsat2009) T 15114 15116 15116 0 -1 4202496 1672 0 0 0 10226 0 0 0 25 0 1 0 658082514 430305280 1400 33554432000 134512640 135107411 4292588000 18446744073709551615 134531709 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/15116/statm: 105055 1400 59 146 0 103931 0
[pid=15114/tid=15135] ppid=15016 vsize=415024 CPUtime=0 cores=0,2,4,6
/proc/15114/task/15135/stat : 15135 (python) S 15016 15016 14447 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 658093053 424984576 6275 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 514 18446604443175334144 0 0 -1 2 0 0 0
[pid=15118] ppid=15016 vsize=425184 CPUtime=0.69 cores=0,2,4,6
/proc/15118/stat : 15118 (python) S 15016 15016 14447 0 -1 4202560 1236 0 0 0 36 33 0 0 19 0 2 0 658082924 435388416 6322 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624937419 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/15118/statm: 106296 6322 408 364 0 42172 0
[pid=15120] ppid=15118 vsize=268424 CPUtime=195.1 cores=0,2,4,6
/proc/15120/stat : 15120 (hybridGM3) T 15118 15120 15120 0 -1 4202496 5401 0 0 0 19508 2 0 0 25 0 1 0 658082924 274866176 5127 33554432000 134512640 135204385 4291586736 18446744073709551615 134538264 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/15120/statm: 67106 5127 63 169 0 66936 0
[pid=15118/tid=15146] ppid=15016 vsize=425184 CPUtime=0 cores=0,2,4,6
/proc/15118/task/15146/stat : 15146 (python) S 15016 15016 14447 0 -1 4202560 44 0 0 0 0 0 0 0 19 0 2 0 658102502 435388416 6322 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 514 18446604443175335936 0 0 -1 0 0 0 0
[pid=15136] ppid=15016 vsize=435340 CPUtime=0.33 cores=0,2,4,6
/proc/15136/stat : 15136 (python) S 15016 15016 14447 0 -1 4202560 1039 0 0 0 15 18 0 0 18 0 1 0 658093055 445788160 6312 33554432000 4194304 5683644 140734381656512 18446744073709551615 270613130822 0 0 16781312 514 11366610470459900161 0 0 17 6 0 0 0
/proc/15136/statm: 108835 6312 380 364 0 44708 0
[pid=15138] ppid=15136 vsize=268424 CPUtime=91.37 cores=0,2,4,6
/proc/15138/stat : 15138 (hybridGM3) R 15136 15138 15138 0 -1 4202496 5405 0 0 0 9137 0 0 0 25 0 1 0 658093055 274866176 5127 33554432000 134512640 135204385 4294920112 18446744073709551615 134537717 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/15138/statm: 67106 5127 63 169 0 66936 0
[pid=15141] ppid=15016 vsize=446016 CPUtime=0.01 cores=0,2,4,6
/proc/15141/stat : 15141 (python) S 15016 15016 14447 0 -1 4202560 1017 0 0 0 0 1 0 0 18 0 1 0 658101687 456720384 6316 33554432000 4194304 5683644 140734381656512 18446744073709551615 270613130822 0 0 16781312 514 848577514967105793 0 0 17 6 0 0 0
/proc/15141/statm: 111504 6316 380 364 0 47374 0
[pid=15143] ppid=15141 vsize=458352 CPUtime=8.41 cores=0,2,4,6
/proc/15143/stat : 15143 (TNM) R 15141 15143 15143 0 -1 4202496 1517 0 0 0 841 0 0 0 25 0 1 0 658101688 469352448 1239 33554432000 134512640 135008748 4289698128 18446744073709551615 134527424 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/15143/statm: 114588 1239 46 122 0 113488 0
[pid=15147] ppid=15016 vsize=456056 CPUtime=0 cores=0,2,4,6
/proc/15147/stat : 15147 (python) S 15016 15016 14447 0 -1 4202560 823 0 0 0 0 0 0 0 19 0 1 0 658102503 467001344 6307 33554432000 4194304 5683644 140734381656512 18446744073709551615 270613130822 0 0 16781312 514 7935325505291550719 0 0 17 6 0 0 0
/proc/15147/statm: 114014 6307 346 364 0 49881 0
[pid=15149] ppid=15147 vsize=247956 CPUtime=0.4 cores=0,2,4,6
/proc/15149/stat : 15149 (FH) R 15147 15149 15149 0 -1 4202496 1258 0 0 0 40 0 0 0 21 0 1 0 658102503 253906944 987 33554432000 134512640 134936988 4289463872 18446744073709551615 134523472 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/15149/statm: 61989 987 37 104 0 61493 0
[pid=15016/tid=15023] ppid=15014 vsize=465952 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15023/stat : 15023 (python) S 15014 15016 14447 0 -1 4202560 24 937 0 0 0 0 0 0 21 0 19 0 658052397 477134848 7868 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 6 0 0 0
[pid=15016/tid=15026] ppid=15014 vsize=465952 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15026/stat : 15026 (python) S 15014 15016 14447 0 -1 4202560 26 937 0 0 0 0 0 0 23 0 19 0 658052398 477134848 7868 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 6 0 0 0
[pid=15016/tid=15029] ppid=15014 vsize=465952 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15029/stat : 15029 (python) S 15014 15016 14447 0 -1 4202560 24 937 0 0 0 0 0 0 25 0 19 0 658052399 477134848 7868 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 6 0 0 0
[pid=15016/tid=15032] ppid=15014 vsize=465952 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15032/stat : 15032 (python) S 15014 15016 14447 0 -1 4202560 24 937 0 0 0 0 0 0 25 0 19 0 658052400 477134848 7868 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 6 0 0 0
[pid=15016/tid=15055] ppid=15014 vsize=465952 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15055/stat : 15055 (python) S 15014 15016 14447 0 -1 4202560 28 937 0 0 0 0 0 0 20 0 19 0 658062334 477134848 7868 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 6 0 0 0
[pid=15016/tid=15060] ppid=15014 vsize=465952 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15060/stat : 15060 (python) S 15014 15016 14447 0 -1 4202560 26 937 0 0 0 0 0 0 20 0 19 0 658062441 477134848 7868 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 0 0 0 0
[pid=15016/tid=15064] ppid=15014 vsize=465952 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15064/stat : 15064 (python) S 15014 15016 14447 0 -1 4202560 32 937 0 0 0 0 0 0 20 0 19 0 658062442 477134848 7868 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 0 0 0 0
[pid=15016/tid=15067] ppid=15014 vsize=465952 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15067/stat : 15067 (python) S 15014 15016 14447 0 -1 4202560 28 937 0 0 0 0 0 0 21 0 19 0 658062444 477134848 7868 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 0 0 0 0
[pid=15016/tid=15089] ppid=15014 vsize=465952 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15089/stat : 15089 (python) S 15014 15016 14447 0 -1 4202560 21 937 0 0 0 0 0 0 20 0 19 0 658072270 477134848 7868 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 0 0 0 0
[pid=15016/tid=15093] ppid=15014 vsize=465952 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15093/stat : 15093 (python) S 15014 15016 14447 0 -1 4202560 24 937 0 0 0 0 0 0 20 0 19 0 658072477 477134848 7868 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 6 0 0 0
[pid=15016/tid=15097] ppid=15014 vsize=465952 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15097/stat : 15097 (python) S 15014 15016 14447 0 -1 4202560 29 937 0 0 0 0 0 0 18 0 19 0 658072479 477134848 7868 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 6 0 0 0
[pid=15016/tid=15101] ppid=15014 vsize=465952 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15101/stat : 15101 (python) S 15014 15016 14447 0 -1 4202560 27 937 0 0 0 0 0 0 22 0 19 0 658072482 477134848 7868 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 6 0 0 0
[pid=15016/tid=15111] ppid=15014 vsize=465952 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15111/stat : 15111 (python) S 15014 15016 14447 0 -1 4202560 30 937 0 0 0 0 0 0 18 0 19 0 658082304 477134848 7868 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 6 0 0 0
[pid=15016/tid=15115] ppid=15014 vsize=465952 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15115/stat : 15115 (python) S 15014 15016 14447 0 -1 4202560 27 937 0 0 0 0 0 0 20 0 19 0 658082514 477134848 7868 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 6 0 0 0
[pid=15016/tid=15119] ppid=15014 vsize=465952 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15119/stat : 15119 (python) S 15014 15016 14447 0 -1 4202560 24 937 0 0 0 0 0 0 20 0 19 0 658082924 477134848 7868 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 6 0 0 0
[pid=15016/tid=15137] ppid=15014 vsize=465952 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15137/stat : 15137 (python) S 15014 15016 14447 0 -1 4202560 27 937 0 0 0 0 0 0 20 0 19 0 658093055 477134848 7868 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 6 0 0 0
[pid=15016/tid=15142] ppid=15014 vsize=465952 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15142/stat : 15142 (python) S 15014 15016 14447 0 -1 4202560 27 937 0 0 0 0 0 0 20 0 19 0 658101687 477134848 7868 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 6 0 0 0
[pid=15016/tid=15148] ppid=15014 vsize=465952 CPUtime=0 cores=0,2,4,6
/proc/15016/task/15148/stat : 15148 (python) S 15014 15016 14447 0 -1 4202560 26 937 0 0 0 0 0 0 20 0 19 0 658102503 477134848 7868 33554432000 4194304 5683644 140734381656512 18446744073709551615 270624934321 0 0 16781312 2 18446604446965777728 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 1994.45
Current children cumulated vsize (KiB) 15987124

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 15016 and gives
#  childrusage.ru_utime.tv_sec=216
#  childrusage.ru_utime.tv_usec=99147
#  childrusage.ru_stime.tv_sec=3
#  childrusage.ru_stime.tv_usec=479471
# CPU time returned by wait4() is 219.579
# while last known CPU time is 1994.45
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 503.837
CPU time (s): 1994.45
CPU user time (s): 1991.01
CPU system time (s): 3.44
CPU usage (%): 395.853
Max. virtual memory (cumulated for all children) (KiB): 15987124

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 216.099
system time used= 3.47947
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 60191
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 6278
involuntary context switches= 1167

runsolver used 4.00839 second user time and 8.11876 second system time

The end

Launcher Data 

Begin job on node108 at 2011-05-15 15:01:52
IDJOB=3378288
IDBENCH=82697
IDSOLVER=1859
FILE ID=node108/3378288-1305464512
RUNJOBID= node108-1305463282-14465
PBS_JOBID= 13324989
Free space on /tmp= 73500 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/large/unif-k7-r85-v300-c25500-S415002759-031-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378288-1305464512/watcher-3378288-1305464512 -o /tmp/evaluation-result-3378288-1305464512/solver-3378288-1305464512 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378288-1305464512.cnf -seed 972504358 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 6d4469ea6d617a494594ad07b4996367
RANDOM SEED=972504358

node108.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.64
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      16387928 kB
Buffers:        309488 kB
Cached:        5136568 kB
SwapCached:       8700 kB
Active:       11959400 kB
Inactive:      4271580 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      16387928 kB
SwapTotal:    67111528 kB
SwapFree:     67056804 kB
Dirty:           32404 kB
Writeback:           0 kB
AnonPages:    10782104 kB
Mapped:          38780 kB
Slab:           213776 kB
PageTables:      51892 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 93095568 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73468 MiB
End job on node108 at 2011-05-15 15:10:18