Trace number 3378226

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (MO) 2026.3 511.53

General information on the benchmark

Namerandom/large/
unif-k5-r20-v2000-c40000-S61914820-087-UNKNOWN.cnf
MD5SUM1b9b43753c02a2c226adda648a956e92
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2000
Number of clauses40000
Sum of the clauses size200000
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 540000
Number of clauses of size over 50

Solver Data 

0.34/0.68	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.34/0.68	c INFO: solving HOME/instance-3378226-1305463917.cnf
0.34/0.68	c INFO: computing task features
0.52/0.88	c INFO: computed variable-clause graph statistics
0.52/0.89	c INFO: computed clause balance statistics
1.63/1.94	c INFO: computed variable balance statistics
2.23/2.50	c INFO: computed small-clause counts
2.23/2.50	c INFO: computed Horn-clause counts
2.23/2.55	c INFO: computed variable graph statistics
2.23/2.55	c INFO: collected features for HOME/instance-3378226-1305463917.cnf in 1.86 s
2.23/2.56	c INFO: running gnovelty+2@0 for 97 with 1198 remaining (b = 0.38)
2.23/2.57	c INFO: running gnovelty+2@0 for 97 with 1198 remaining (b = 0.13)
2.23/2.58	c INFO: running adaptg2wsat2009++@0 for 97 with 1198 remaining (b = 0.08)
2.23/2.59	c INFO: running adaptg2wsat2009++@0 for 97 with 1198 remaining (b = 0.07)
402.03/102.84	c INFO: running adaptg2wsat2009++@0 for 97 with 1097 remaining (b = 0.05)
402.03/102.86	c INFO: running gnovelty+2@0 for 97 with 1097 remaining (b = 0.05)
402.03/102.88	c INFO: running FH@0 for 97 with 1097 remaining (b = 0.03)
402.03/102.89	c INFO: running adaptg2wsat2009++@0 for 97 with 1097 remaining (b = 0.03)
797.04/202.18	c INFO: running adaptg2wsat2009++@0 for 97 with 998 remaining (b = 0.02)
801.01/203.11	c INFO: running FH@0 for 97 with 997 remaining (b = 0.02)
801.01/203.13	c INFO: running hybridGM3@0 for 194 with 997 remaining (b = 0.04)
820.85/208.18	c INFO: running gnovelty+2@0 for 97 with 992 remaining (b = 0.02)
1188.09/300.48	c INFO: running hybridGM3@0 for 194 with 900 remaining (b = 0.03)
1200.01/303.40	c INFO: running adaptg2wsat2009++@0 for 97 with 897 remaining (b = 0.01)
1232.19/311.56	c INFO: running hybridGM3@0 for 194 with 889 remaining (b = 0.03)
1590.85/401.80	c INFO: running hybridGM3@102 for 97 with 798 remaining (b = 0.01)
1594.48/402.69	c INFO: running hybridGM3@0 for 194 with 797 remaining (b = 0.02)
1977.43/499.19	c INFO: running hybridGM3@100 for 97 with 701 remaining (b = 0.01)
1992.92/503.07	c INFO: running TNM@0 for 697 with 697 remaining (b = 0.08)
2025.44/511.25	c INFO: running FH@0 for 97 with 689 remaining (b = 0.01)
2026.30/511.51	c INFO: running adaptg2wsat2009++@0 for 97 with 689 remaining (b = 0.01)
2026.30/511.51	Traceback (most recent call last):
2026.30/511.51	  File "HOME/solve", line 33, in <module>
2026.30/511.51	    plac.call(main)
2026.30/511.51	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
2026.30/511.51	    cmd, result = parser_from(obj).consume(arglist)
2026.30/511.51	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
2026.30/511.51	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
2026.30/511.51	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
2026.30/511.51	    (_, answer, _) = solver(input_path, budget / cores, cores)
2026.30/511.51	  File "HOME/borg/build/lib/borg/bilevel.py", line 202, in __call__
2026.30/511.51	    solver.go(max_cost * borg.defaults.machine_speed)
2026.30/511.51	  File "HOME/borg/build/lib/borg/solvers.py", line 179, in go
2026.30/511.51	    self._process.start()
2026.30/511.51	  File "/home/evaluation/evaluation/tools64/lib/python2.7/multiprocessing/process.py", line 94, in start
2026.30/511.51	    assert self._popen is None, 'cannot start a process twice'
2026.30/511.51	AssertionError: cannot start a process twice

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378226-1305463917/watcher-3378226-1305463917 -o /tmp/evaluation-result-3378226-1305463917/solver-3378226-1305463917 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378226-1305463917.cnf -seed 2037844210 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.04 7.65 5.94 6/348 18366
/proc/meminfo: memFree=20040996/32951124 swapFree=66457188/67111528
[pid=18366] ppid=18364 vsize=15776 CPUtime=0 cores=1,3,5,7
/proc/18366/stat : 18366 (python) R 18364 18366 18093 0 -1 4202496 886 0 0 0 0 0 0 0 25 0 1 0 657989916 16154624 769 33554432000 4194304 5683644 140733495900752 18446744073709551615 232335553221 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/18366/statm: 3944 770 343 364 0 431 0

[startup+0.0803159 s]
/proc/loadavg: 8.04 7.65 5.94 6/348 18366
/proc/meminfo: memFree=20040996/32951124 swapFree=66457188/67111528
[pid=18366] ppid=18364 vsize=50016 CPUtime=0.04 cores=1,3,5,7
/proc/18366/stat : 18366 (python) R 18364 18366 18093 0 -1 4202496 1947 0 0 0 3 1 0 0 25 0 1 0 657989916 51216384 1713 33554432000 4194304 5683644 140733495900752 18446744073709551615 232335888308 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/18366/statm: 12505 1717 476 364 0 1205 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 50016

[startup+0.100336 s]
/proc/loadavg: 8.04 7.65 5.94 6/348 18366
/proc/meminfo: memFree=20040996/32951124 swapFree=66457188/67111528
[pid=18366] ppid=18364 vsize=87708 CPUtime=0.05 cores=1,3,5,7
/proc/18366/stat : 18366 (python) S 18364 18366 18093 0 -1 4202496 2636 0 0 0 4 1 0 0 17 0 1 0 657989916 89812992 2354 33554432000 4194304 5683644 140733495900752 18446744073709551615 232335890432 0 2147483391 16781312 2 18446744071699453118 0 0 17 1 0 0 0
/proc/18366/statm: 21927 2354 717 364 0 1623 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 87708

[startup+0.300303 s]
/proc/loadavg: 8.04 7.65 5.94 6/348 18366
/proc/meminfo: memFree=20040996/32951124 swapFree=66457188/67111528
[pid=18366] ppid=18364 vsize=135752 CPUtime=0.18 cores=1,3,5,7
/proc/18366/stat : 18366 (python) R 18364 18366 18093 0 -1 4202496 5960 933 1 0 13 5 0 0 17 0 1 0 657989916 139010048 4151 33554432000 4194304 5683644 140733495900752 18446744073709551615 232335888389 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/18366/statm: 33938 4151 1032 364 0 3124 0
Current children cumulated CPU time (s) 0.18
Current children cumulated vsize (KiB) 135752

[startup+0.700213 s]
/proc/loadavg: 8.04 7.65 5.94 6/348 18366
/proc/meminfo: memFree=20040996/32951124 swapFree=66457188/67111528
[pid=18366] ppid=18364 vsize=280064 CPUtime=0.42 cores=1,3,5,7
/proc/18366/stat : 18366 (python) R 18364 18366 18093 0 -1 4202496 9658 933 1 0 30 12 0 0 15 0 1 0 657989916 286785536 7342 33554432000 4194304 5683644 140733495900752 18446744073709551615 4602358 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/18366/statm: 70016 7342 1761 364 0 5940 0
Current children cumulated CPU time (s) 0.42
Current children cumulated vsize (KiB) 280064

[startup+1.50108 s]
/proc/loadavg: 8.04 7.65 5.94 6/349 18371
/proc/meminfo: memFree=19988536/32951124 swapFree=66457188/67111528
[pid=18366] ppid=18364 vsize=290804 CPUtime=1.23 cores=1,3,5,7
/proc/18366/stat : 18366 (python) R 18364 18366 18093 0 -1 4202496 14007 933 1 0 110 13 0 0 16 0 1 0 657989916 297783296 10085 33554432000 4194304 5683644 140733495900752 18446744073709551615 4525717 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/18366/statm: 72701 10085 1826 364 0 8625 0
Current children cumulated CPU time (s) 1.23
Current children cumulated vsize (KiB) 290804

[startup+3.10075 s]
/proc/loadavg: 8.04 7.65 5.94 6/349 18371
/proc/meminfo: memFree=19988288/32951124 swapFree=66457188/67111528
[pid=18366] ppid=18364 vsize=325512 CPUtime=2.31 cores=1,3,5,7
/proc/18366/stat : 18366 (python) S 18364 18366 18093 0 -1 4202496 16830 933 1 0 217 14 0 0 24 0 5 0 657989916 333324288 8568 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347719115 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/18366/statm: 81378 8568 1875 364 0 17287 0
Current children cumulated CPU time (s) 2.31
Current children cumulated vsize (KiB) 325512
heavy processes:

[startup+6.30119 s]
/proc/loadavg: 8.03 7.66 5.95 9/361 18383
/proc/meminfo: memFree=19952844/32951124 swapFree=66457188/67111528
[pid=18366] ppid=18364 vsize=325512 CPUtime=2.31 cores=1,3,5,7
/proc/18366/stat : 18366 (python) S 18364 18366 18093 0 -1 4202496 16830 933 1 0 217 14 0 0 24 0 5 0 657989916 333324288 8568 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347719115 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/18366/statm: 81378 8568 1875 364 0 17287 0
[pid=18372] ppid=18366 vsize=284984 CPUtime=0 cores=1,3,5,7
/proc/18372/stat : 18372 (python) S 18366 18366 18093 0 -1 4202560 724 0 0 0 0 0 0 0 18 0 1 0 657990172 291823616 6898 33554432000 4194304 5683644 140733495900752 18446744073709551615 232335912463 0 0 16781312 514 4450525380758962436 0 0 17 5 0 0 0
/proc/18372/statm: 71246 6898 355 364 0 7164 0
[pid=18374] ppid=18372 vsize=1020308 CPUtime=3.68 cores=1,3,5,7
/proc/18374/stat : 18374 (gnovelty+2) R 18372 18374 18374 0 -1 4202496 1156 0 0 0 368 0 0 0 25 0 1 0 657990172 1044795392 891 33554432000 134512640 135663704 4288408080 18446744073709551615 134545640 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/18374/statm: 255077 891 113 282 0 254793 0
[pid=18375] ppid=18366 vsize=294900 CPUtime=0 cores=1,3,5,7
/proc/18375/stat : 18375 (python) S 18366 18366 18093 0 -1 4202560 733 0 0 0 0 0 0 0 18 0 1 0 657990173 301977600 7031 33554432000 4194304 5683644 140733495900752 18446744073709551615 232335912518 0 0 16781312 514 3556765363888685317 0 0 17 1 0 0 0
/proc/18375/statm: 73725 7031 349 364 0 9640 0
[pid=18377] ppid=18375 vsize=1020308 CPUtime=3.72 cores=1,3,5,7
/proc/18377/stat : 18377 (gnovelty+2) R 18375 18377 18377 0 -1 4202496 1157 0 0 0 372 0 0 0 25 0 1 0 657990173 1044795392 891 33554432000 134512640 135663704 4287872128 18446744073709551615 134524503 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/18377/statm: 255077 891 113 282 0 254793 0
[pid=18378] ppid=18366 vsize=305156 CPUtime=0 cores=1,3,5,7
/proc/18378/stat : 18378 (python) S 18366 18366 18093 0 -1 4202560 734 0 0 0 0 0 0 0 18 0 1 0 657990174 312479744 7035 33554432000 4194304 5683644 140733495900752 18446744073709551615 232335912518 0 0 16781312 514 6427446737744396552 0 0 17 5 0 0 0
/proc/18378/statm: 76289 7035 349 364 0 12201 0
[pid=18380] ppid=18378 vsize=423400 CPUtime=3.7 cores=1,3,5,7
/proc/18380/stat : 18380 (adaptg2wsat2009) R 18378 18380 18380 0 -1 4202496 2622 0 0 0 370 0 0 0 25 0 1 0 657990174 433561600 2356 33554432000 134512640 135107411 4290947504 18446744073709551615 134531731 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/18380/statm: 105850 2356 59 146 0 104726 0
[pid=18381] ppid=18366 vsize=316020 CPUtime=0 cores=1,3,5,7
/proc/18381/stat : 18381 (python) S 18366 18366 18093 0 -1 4202560 741 0 0 0 0 0 0 0 18 0 1 0 657990175 323604480 7039 33554432000 4194304 5683644 140733495900752 18446744073709551615 232335912518 0 0 16781312 514 17031528302155628807 0 0 17 1 0 0 0
/proc/18381/statm: 79005 7039 349 364 0 14914 0
[pid=18383] ppid=18381 vsize=423400 CPUtime=3.7 cores=1,3,5,7
/proc/18383/stat : 18383 (adaptg2wsat2009) R 18381 18383 18383 0 -1 4202496 2618 0 0 0 370 0 0 0 25 0 1 0 657990175 433561600 2357 33554432000 134512640 135107411 4293822416 18446744073709551615 134532076 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/18383/statm: 105850 2357 59 146 0 104726 0
[pid=18366/tid=18373] ppid=18364 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18373/stat : 18373 (python) S 18364 18366 18093 0 -1 4202560 24 933 0 0 0 0 0 0 20 0 5 0 657990172 333324288 8568 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 5 0 0 0
[pid=18366/tid=18376] ppid=18364 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18376/stat : 18376 (python) S 18364 18366 18093 0 -1 4202560 26 933 0 0 0 0 0 0 20 0 5 0 657990173 333324288 8568 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 1 0 0 0
[pid=18366/tid=18379] ppid=18364 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18379/stat : 18379 (python) S 18364 18366 18093 0 -1 4202560 22 933 0 0 0 0 0 0 22 0 5 0 657990174 333324288 8568 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 5 0 0 0
[pid=18366/tid=18382] ppid=18364 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18382/stat : 18382 (python) S 18364 18366 18093 0 -1 4202560 26 933 0 0 0 0 0 0 24 0 5 0 657990175 333324288 8568 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 17.11
Current children cumulated vsize (KiB) 4413988

[startup+12.701 s]
/proc/loadavg: 8.03 7.66 5.96 9/361 18383
/proc/meminfo: memFree=19950796/32951124 swapFree=66457188/67111528
[pid=18366] ppid=18364 vsize=325512 CPUtime=2.31 cores=1,3,5,7
/proc/18366/stat : 18366 (python) S 18364 18366 18093 0 -1 4202496 16830 933 1 0 217 14 0 0 24 0 5 0 657989916 333324288 8568 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347719115 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/18366/statm: 81378 8568 1875 364 0 17287 0
[pid=18372] ppid=18366 vsize=284984 CPUtime=0.02 cores=1,3,5,7
/proc/18372/stat : 18372 (python) S 18366 18366 18093 0 -1 4202560 883 0 0 0 1 1 0 0 18 0 1 0 657990172 291823616 6921 33554432000 4194304 5683644 140733495900752 18446744073709551615 232335912463 0 0 16781312 514 11425491517150560516 0 0 17 5 0 0 0
/proc/18372/statm: 71246 6921 378 364 0 7164 0
[pid=18374] ppid=18372 vsize=1020308 CPUtime=10.01 cores=1,3,5,7
/proc/18374/stat : 18374 (gnovelty+2) R 18372 18374 18374 0 -1 4202496 1156 0 0 0 1001 0 0 0 25 0 1 0 657990172 1044795392 891 33554432000 134512640 135663704 4288408080 18446744073709551615 134544196 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/18374/statm: 255077 891 113 282 0 254793 0
[pid=18375] ppid=18366 vsize=294900 CPUtime=0.02 cores=1,3,5,7
/proc/18375/stat : 18375 (python) S 18366 18366 18093 0 -1 4202560 907 0 0 0 1 1 0 0 18 0 1 0 657990173 301977600 7054 33554432000 4194304 5683644 140733495900752 18446744073709551615 232335912518 0 0 16781312 514 11425491517150560517 0 0 17 1 0 0 0
/proc/18375/statm: 73725 7054 372 364 0 9640 0
[pid=18377] ppid=18375 vsize=1020308 CPUtime=10.12 cores=1,3,5,7
/proc/18377/stat : 18377 (gnovelty+2) R 18375 18377 18377 0 -1 4202496 1157 0 0 0 1012 0 0 0 25 0 1 0 657990173 1044795392 891 33554432000 134512640 135663704 4287872128 18446744073709551615 134525213 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/18377/statm: 255077 891 113 282 0 254793 0
[pid=18378] ppid=18366 vsize=305156 CPUtime=0.02 cores=1,3,5,7
/proc/18378/stat : 18378 (python) S 18366 18366 18093 0 -1 4202560 892 0 0 0 1 1 0 0 18 0 1 0 657990174 312479744 7058 33554432000 4194304 5683644 140733495900752 18446744073709551615 232335912518 0 0 16781312 514 11425491517150560520 0 0 17 5 0 0 0
/proc/18378/statm: 76289 7058 372 364 0 12201 0
[pid=18380] ppid=18378 vsize=423400 CPUtime=10.05 cores=1,3,5,7
/proc/18380/stat : 18380 (adaptg2wsat2009) R 18378 18380 18380 0 -1 4202496 2622 0 0 0 1005 0 0 0 25 0 1 0 657990174 433561600 2356 33554432000 134512640 135107411 4290947504 18446744073709551615 134531935 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/18380/statm: 105850 2356 59 146 0 104726 0
[pid=18381] ppid=18366 vsize=316020 CPUtime=0.02 cores=1,3,5,7
/proc/18381/stat : 18381 (python) S 18366 18366 18093 0 -1 4202560 911 0 0 0 1 1 0 0 18 0 1 0 657990175 323604480 7062 33554432000 4194304 5683644 140733495900752 18446744073709551615 232335912518 0 0 16781312 514 11425491517150560519 0 0 17 1 0 0 0
/proc/18381/statm: 79005 7062 372 364 0 14914 0
[pid=18383] ppid=18381 vsize=423400 CPUtime=10.04 cores=1,3,5,7
/proc/18383/stat : 18383 (adaptg2wsat2009) R 18381 18383 18383 0 -1 4202496 2618 0 0 0 1004 0 0 0 25 0 1 0 657990175 433561600 2357 33554432000 134512640 135107411 4293822416 18446744073709551615 134531561 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/18383/statm: 105850 2357 59 146 0 104726 0
[pid=18366/tid=18373] ppid=18364 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18373/stat : 18373 (python) S 18364 18366 18093 0 -1 4202560 24 933 0 0 0 0 0 0 20 0 5 0 657990172 333324288 8568 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 5 0 0 0
[pid=18366/tid=18376] ppid=18364 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18376/stat : 18376 (python) S 18364 18366 18093 0 -1 4202560 26 933 0 0 0 0 0 0 20 0 5 0 657990173 333324288 8568 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 1 0 0 0
[pid=18366/tid=18379] ppid=18364 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18379/stat : 18379 (python) S 18364 18366 18093 0 -1 4202560 22 933 0 0 0 0 0 0 22 0 5 0 657990174 333324288 8568 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 5 0 0 0
[pid=18366/tid=18382] ppid=18364 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18382/stat : 18382 (python) S 18364 18366 18093 0 -1 4202560 26 933 0 0 0 0 0 0 24 0 5 0 657990175 333324288 8568 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 42.61
Current children cumulated vsize (KiB) 4413988

[startup+25.5008 s]
/proc/loadavg: 8.02 7.68 5.99 9/361 18384
/proc/meminfo: memFree=19951112/32951124 swapFree=66457188/67111528
[pid=18366] ppid=18364 vsize=325512 CPUtime=2.31 cores=1,3,5,7
/proc/18366/stat : 18366 (python) S 18364 18366 18093 0 -1 4202496 16830 933 1 0 217 14 0 0 24 0 5 0 657989916 333324288 8568 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347719115 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/18366/statm: 81378 8568 1875 364 0 17287 0
[pid=18372] ppid=18366 vsize=284984 CPUtime=0.05 cores=1,3,5,7
/proc/18372/stat : 18372 (python) S 18366 18366 18093 0 -1 4202560 889 0 0 0 3 2 0 0 18 0 1 0 657990172 291823616 6921 33554432000 4194304 5683644 140733495900752 18446744073709551615 232335912463 0 0 16781312 514 11425491517150560516 0 0 17 5 0 0 0
/proc/18372/statm: 71246 6921 378 364 0 7164 0
[pid=18374] ppid=18372 vsize=1020308 CPUtime=22.66 cores=1,3,5,7
/proc/18374/stat : 18374 (gnovelty+2) R 18372 18374 18374 0 -1 4202496 1156 0 0 0 2266 0 0 0 25 0 1 0 657990172 1044795392 891 33554432000 134512640 135663704 4288408080 18446744073709551615 134545607 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/18374/statm: 255077 891 113 282 0 254793 0
[pid=18375] ppid=18366 vsize=294900 CPUtime=0.05 cores=1,3,5,7
/proc/18375/stat : 18375 (python) S 18366 18366 18093 0 -1 4202560 913 0 0 0 2 3 0 0 18 0 1 0 657990173 301977600 7054 33554432000 4194304 5683644 140733495900752 18446744073709551615 232335912518 0 0 16781312 514 11425491517150560517 0 0 17 1 0 0 0
/proc/18375/statm: 73725 7054 372 364 0 9640 0
[pid=18377] ppid=18375 vsize=1020308 CPUtime=22.92 cores=1,3,5,7
/proc/18377/stat : 18377 (gnovelty+2) R 18375 18377 18377 0 -1 4202496 1157 0 0 0 2292 0 0 0 25 0 1 0 657990173 1044795392 891 33554432000 134512640 135663704 4287872128 18446744073709551615 134525678 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/18377/statm: 255077 891 113 282 0 254793 0
[pid=18378] ppid=18366 vsize=305156 CPUtime=0.05 cores=1,3,5,7
/proc/18378/stat : 18378 (python) S 18366 18366 18093 0 -1 4202560 898 0 0 0 2 3 0 0 18 0 1 0 657990174 312479744 7058 33554432000 4194304 5683644 140733495900752 18446744073709551615 232335912518 0 0 16781312 514 11425491517150560520 0 0 17 5 0 0 0
/proc/18378/statm: 76289 7058 372 364 0 12201 0
[pid=18380] ppid=18378 vsize=423400 CPUtime=22.79 cores=1,3,5,7
/proc/18380/stat : 18380 (adaptg2wsat2009) R 18378 18380 18380 0 -1 4202496 2622 0 0 0 2279 0 0 0 25 0 1 0 657990174 433561600 2356 33554432000 134512640 135107411 4290947504 18446744073709551615 134532349 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/18380/statm: 105850 2356 59 146 0 104726 0
[pid=18381] ppid=18366 vsize=316020 CPUtime=0.05 cores=1,3,5,7
/proc/18381/stat : 18381 (python) S 18366 18366 18093 0 -1 4202560 915 0 0 0 2 3 0 0 18 0 1 0 657990175 323604480 7062 33554432000 4194304 5683644 140733495900752 18446744073709551615 232335912518 0 0 16781312 514 11425491517150560519 0 0 17 1 0 0 0
/proc/18381/statm: 79005 7062 372 364 0 14914 0
[pid=18383] ppid=18381 vsize=423400 CPUtime=22.78 cores=1,3,5,7
/proc/18383/stat : 18383 (adaptg2wsat2009) R 18381 18383 18383 0 -1 4202496 2618 0 0 0 2278 0 0 0 25 0 1 0 657990175 433561600 2357 33554432000 134512640 135107411 4293822416 18446744073709551615 134531582 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/18383/statm: 105850 2357 59 146 0 104726 0
[pid=18366/tid=18373] ppid=18364 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18373/stat : 18373 (python) S 18364 18366 18093 0 -1 4202560 24 933 0 0 0 0 0 0 20 0 5 0 657990172 333324288 8568 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 5 0 0 0
[pid=18366/tid=18376] ppid=18364 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18376/stat : 18376 (python) S 18364 18366 18093 0 -1 4202560 26 933 0 0 0 0 0 0 20 0 5 0 657990173 333324288 8568 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 1 0 0 0
[pid=18366/tid=18379] ppid=18364 vsize=325512 CPUtime=0 cores=1,3,5,7

################
# More data... #
################

/proc/18381/stat : 18381 (python) S 18366 18366 18093 0 -1 4202560 1091 0 0 0 12 12 0 0 18 0 2 0 657990175 323604480 7091 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347719115 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/18381/statm: 79005 7091 400 364 0 14914 0
[pid=18383] ppid=18381 vsize=423400 CPUtime=99.75 cores=1,3,5,7
/proc/18383/stat : 18383 (adaptg2wsat2009) T 18381 18383 18383 0 -1 4202496 2618 0 0 0 9975 0 0 0 25 0 1 0 657990175 433561600 2357 33554432000 134512640 135107411 4293822416 18446744073709551615 134531935 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/18383/statm: 105850 2357 59 146 0 104726 0
[pid=18381/tid=18391] ppid=18366 vsize=316020 CPUtime=0 cores=1,3,5,7
/proc/18381/task/18391/stat : 18391 (python) S 18366 18366 18093 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 658000199 323604480 7091 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 514 18446604469763056832 0 0 -1 5 0 0 0
[pid=18392] ppid=18366 vsize=326184 CPUtime=0.25 cores=1,3,5,7
/proc/18392/stat : 18392 (python) S 18366 18366 18093 0 -1 4202560 1096 0 0 0 12 13 0 0 19 0 2 0 658000200 334012416 7095 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347719115 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/18392/statm: 81546 7095 400 364 0 17452 0
[pid=18395] ppid=18392 vsize=423400 CPUtime=99.99 cores=1,3,5,7
/proc/18395/stat : 18395 (adaptg2wsat2009) T 18392 18395 18395 0 -1 4202496 2623 0 0 0 9999 0 0 0 25 0 1 0 658000202 433561600 2356 33554432000 134512640 135107411 4290293120 18446744073709551615 134531582 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/18395/statm: 105850 2356 59 146 0 104726 0
[pid=18392/tid=18418] ppid=18366 vsize=326184 CPUtime=0 cores=1,3,5,7
/proc/18392/task/18418/stat : 18418 (python) S 18366 18366 18093 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 658010226 334012416 7095 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 514 18446604466625717312 0 0 -1 5 0 0 0
[pid=18397] ppid=18366 vsize=336100 CPUtime=0.25 cores=1,3,5,7
/proc/18397/stat : 18397 (python) S 18366 18366 18093 0 -1 4202560 1099 0 0 0 12 13 0 0 19 0 2 0 658000203 344166400 7143 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347719115 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/18397/statm: 84025 7143 400 364 0 19928 0
[pid=18399] ppid=18397 vsize=1020308 CPUtime=99.29 cores=1,3,5,7
/proc/18399/stat : 18399 (gnovelty+2) T 18397 18399 18399 0 -1 4202496 1152 0 0 0 9929 0 0 0 25 0 1 0 658000203 1044795392 891 33554432000 134512640 135663704 4289121376 18446744073709551615 134525151 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/18399/statm: 255077 891 113 282 0 254793 0
[pid=18397/tid=18414] ppid=18366 vsize=336100 CPUtime=0 cores=1,3,5,7
/proc/18397/task/18414/stat : 18414 (python) S 18366 18366 18093 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 658010133 344166400 7143 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 514 18446604454395489024 0 0 -1 7 0 0 0
[pid=18400] ppid=18366 vsize=346444 CPUtime=0.25 cores=1,3,5,7
/proc/18400/stat : 18400 (python) S 18366 18366 18093 0 -1 4202560 1100 0 0 0 13 12 0 0 19 0 2 0 658000204 354758656 7147 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347719115 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/18400/statm: 86611 7147 400 364 0 22511 0
[pid=18402] ppid=18400 vsize=253036 CPUtime=99.98 cores=1,3,5,7
/proc/18402/stat : 18402 (FH) T 18400 18402 18402 0 -1 4202496 2496 0 0 0 9998 0 0 0 25 0 1 0 658000204 259108864 2233 33554432000 134512640 134936988 4287491376 18446744073709551615 134554229 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/18402/statm: 63259 2233 41 104 0 62763 0
[pid=18400/tid=18422] ppid=18366 vsize=346444 CPUtime=0 cores=1,3,5,7
/proc/18400/task/18422/stat : 18422 (python) S 18366 18366 18093 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 658010228 354758656 7147 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 514 18446604468615351104 0 0 -1 3 0 0 0
[pid=18403] ppid=18366 vsize=356796 CPUtime=0.27 cores=1,3,5,7
/proc/18403/stat : 18403 (python) S 18366 18366 18093 0 -1 4202560 1137 0 0 0 15 12 0 0 19 0 2 0 658000205 365359104 7157 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347719115 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/18403/statm: 89199 7157 406 364 0 25096 0
[pid=18405] ppid=18403 vsize=423400 CPUtime=102.5 cores=1,3,5,7
/proc/18405/stat : 18405 (adaptg2wsat2009) T 18403 18405 18405 0 -1 4202496 2623 0 0 0 10250 0 0 0 25 0 1 0 658000205 433561600 2357 33554432000 134512640 135107411 4291467136 18446744073709551615 134531709 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/18405/statm: 105850 2357 59 146 0 104726 0
[pid=18403/tid=18427] ppid=18366 vsize=356796 CPUtime=0 cores=1,3,5,7
/proc/18403/task/18427/stat : 18427 (python) S 18366 18366 18093 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 658010733 365359104 7157 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 514 18446604454395487232 0 0 -1 7 0 0 0
[pid=18415] ppid=18366 vsize=367020 CPUtime=0.26 cores=1,3,5,7
/proc/18415/stat : 18415 (python) S 18366 18366 18093 0 -1 4202560 1143 0 0 0 14 12 0 0 19 0 2 0 658010134 375828480 7161 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347719115 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/18415/statm: 91755 7161 406 364 0 27649 0
[pid=18417] ppid=18415 vsize=423400 CPUtime=98.24 cores=1,3,5,7
/proc/18417/stat : 18417 (adaptg2wsat2009) T 18415 18417 18417 0 -1 4202496 2628 0 0 0 9824 0 0 0 25 0 1 0 658010135 433561600 2356 33554432000 134512640 135107411 4286582576 18446744073709551615 134532277 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/18417/statm: 105850 2356 59 146 0 104726 0
[pid=18415/tid=18437] ppid=18366 vsize=367020 CPUtime=0 cores=1,3,5,7
/proc/18415/task/18437/stat : 18437 (python) S 18366 18366 18093 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 658019963 375828480 7161 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 514 18446604454395486336 0 0 -1 5 0 0 0
[pid=18419] ppid=18366 vsize=376980 CPUtime=0.27 cores=1,3,5,7
/proc/18419/stat : 18419 (python) S 18366 18366 18093 0 -1 4202560 1116 0 0 0 12 15 0 0 19 0 2 0 658010227 386027520 7158 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347719115 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/18419/statm: 94245 7158 399 364 0 30136 0
[pid=18421] ppid=18419 vsize=253036 CPUtime=100.26 cores=1,3,5,7
/proc/18421/stat : 18421 (FH) T 18419 18421 18421 0 -1 4202496 2498 0 0 0 10026 0 0 0 25 0 1 0 658010228 259108864 2232 33554432000 134512640 134936988 4288731696 18446744073709551615 134552472 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/18421/statm: 63259 2232 41 104 0 62763 0
[pid=18419/tid=18441] ppid=18366 vsize=376980 CPUtime=0 cores=1,3,5,7
/proc/18419/task/18441/stat : 18441 (python) S 18366 18366 18093 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 658020255 386027520 7158 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 514 18446604468615348416 0 0 -1 3 0 0 0
[pid=18423] ppid=18366 vsize=387984 CPUtime=0.81 cores=1,3,5,7
/proc/18423/stat : 18423 (python) S 18366 18366 18093 0 -1 4202560 1178 0 0 0 41 40 0 0 18 0 2 0 658010229 397295616 7163 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347719115 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/18423/statm: 96996 7163 400 364 0 32884 0
[pid=18425] ppid=18423 vsize=268424 CPUtime=293.53 cores=1,3,5,7
/proc/18425/stat : 18425 (hybridGM3) T 18423 18425 18425 0 -1 4202496 6187 0 0 0 29353 0 0 0 25 0 1 0 658010230 274866176 5920 33554432000 134512640 135204385 4293035520 18446744073709551615 134538395 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/18425/statm: 67106 5920 63 169 0 66936 0
[pid=18423/tid=18456] ppid=18366 vsize=387984 CPUtime=0 cores=1,3,5,7
/proc/18423/task/18456/stat : 18456 (python) S 18366 18366 18093 0 -1 4202560 48 0 0 0 0 0 0 0 18 0 2 0 658030095 397295616 7163 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 514 18446604468615350208 0 0 -1 5 0 0 0
[pid=18428] ppid=18366 vsize=397756 CPUtime=0.28 cores=1,3,5,7
/proc/18428/stat : 18428 (python) S 18366 18366 18093 0 -1 4202560 1140 0 0 0 14 14 0 0 19 0 2 0 658010734 407302144 7232 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347719115 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/18428/statm: 99439 7232 399 364 0 35324 0
[pid=18430] ppid=18428 vsize=1020308 CPUtime=100.43 cores=1,3,5,7
/proc/18430/stat : 18430 (gnovelty+2) T 18428 18430 18430 0 -1 4202496 1159 0 0 0 10043 0 0 0 25 0 1 0 658010735 1044795392 892 33554432000 134512640 135663704 4291724496 18446744073709551615 134525258 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/18430/statm: 255077 892 113 282 0 254793 0
[pid=18428/tid=18445] ppid=18366 vsize=397756 CPUtime=0 cores=1,3,5,7
/proc/18428/task/18445/stat : 18445 (python) S 18366 18366 18093 0 -1 4202560 51 0 0 0 0 0 0 0 19 0 2 0 658021070 407302144 7232 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 514 18446604465468462272 0 0 -1 7 0 0 0
[pid=18438] ppid=18366 vsize=408276 CPUtime=0.58 cores=1,3,5,7
/proc/18438/stat : 18438 (python) S 18366 18366 18093 0 -1 4202560 1184 0 0 0 28 30 0 0 18 0 2 0 658019965 418074624 7237 33554432000 4194304 5683644 140733495900752 18446744073709551615 232335912518 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/18438/statm: 102069 7237 400 364 0 37951 0
[pid=18440] ppid=18438 vsize=268428 CPUtime=210.37 cores=1,3,5,7
/proc/18440/stat : 18440 (hybridGM3) R 18438 18440 18440 0 -1 4202496 6190 0 0 0 21037 0 0 0 25 0 1 0 658019968 274870272 5919 33554432000 134512640 135204385 4287530720 18446744073709551615 134538475 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/18440/statm: 67107 5919 63 169 0 66937 0
[pid=18438/tid=18466] ppid=18366 vsize=408276 CPUtime=0 cores=1,3,5,7
/proc/18438/task/18466/stat : 18466 (python) S 18366 18366 18093 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 658039834 418074624 7237 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 514 18446604466530742400 0 0 -1 3 0 0 0
[pid=18442] ppid=18366 vsize=418468 CPUtime=0.26 cores=1,3,5,7
/proc/18442/stat : 18442 (python) S 18366 18366 18093 0 -1 4202560 1154 0 0 0 13 13 0 0 19 0 2 0 658020256 428511232 7241 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347719115 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/18442/statm: 104617 7241 400 364 0 40496 0
[pid=18444] ppid=18442 vsize=423400 CPUtime=98.67 cores=1,3,5,7
/proc/18444/stat : 18444 (adaptg2wsat2009) T 18442 18444 18444 0 -1 4202496 2627 0 0 0 9867 0 0 0 25 0 1 0 658020257 433561600 2356 33554432000 134512640 135107411 4289039920 18446744073709551615 134531797 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/18444/statm: 105850 2356 59 146 0 104726 0
[pid=18442/tid=18457] ppid=18366 vsize=418468 CPUtime=0 cores=1,3,5,7
/proc/18442/task/18457/stat : 18457 (python) S 18366 18366 18093 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 658030183 428511232 7241 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 514 18446604464863562496 0 0 -1 7 0 0 0
[pid=18446] ppid=18366 vsize=428876 CPUtime=0.55 cores=1,3,5,7
/proc/18446/stat : 18446 (python) S 18366 18366 18093 0 -1 4202560 1226 0 0 0 28 27 0 0 19 0 2 0 658021072 439169024 7252 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347719115 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/18446/statm: 107219 7252 407 364 0 43095 0
[pid=18448] ppid=18446 vsize=268424 CPUtime=199.66 cores=1,3,5,7
/proc/18448/stat : 18448 (hybridGM3) T 18446 18448 18448 0 -1 4202496 6193 0 0 0 19966 0 0 0 25 0 1 0 658021072 274866176 5920 33554432000 134512640 135204385 4288509008 18446744073709551615 134545276 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/18448/statm: 67106 5920 63 169 0 66936 0
[pid=18446/tid=18471] ppid=18366 vsize=428876 CPUtime=0 cores=1,3,5,7
/proc/18446/task/18471/stat : 18471 (python) S 18366 18366 18093 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 658041039 439169024 7252 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 514 18446604463105320960 0 0 -1 7 0 0 0
[pid=18458] ppid=18366 vsize=439144 CPUtime=0.28 cores=1,3,5,7
/proc/18458/stat : 18458 (python) S 18366 18366 18093 0 -1 4202560 1046 0 0 0 13 15 0 0 18 0 1 0 658030185 449683456 7228 33554432000 4194304 5683644 140733495900752 18446744073709551615 232335912518 0 0 16781312 514 11425491517150560519 0 0 17 5 0 0 0
/proc/18458/statm: 109786 7228 380 364 0 45659 0
[pid=18460] ppid=18458 vsize=268424 CPUtime=108.15 cores=1,3,5,7
/proc/18460/stat : 18460 (hybridGM3) R 18458 18460 18460 0 -1 4202496 6193 0 0 0 10814 1 0 0 25 0 1 0 658030185 274866176 5918 33554432000 134512640 135204385 4293536576 18446744073709551615 134537704 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/18460/statm: 67106 5918 63 169 0 66936 0
[pid=18468] ppid=18366 vsize=449816 CPUtime=0.01 cores=1,3,5,7
/proc/18468/stat : 18468 (python) S 18366 18366 18093 0 -1 4202560 1022 0 0 0 0 1 0 0 18 0 1 0 658040223 460611584 7244 33554432000 4194304 5683644 140733495900752 18446744073709551615 232335912518 0 0 16781312 514 4530973348028449029 0 0 17 3 0 0 0
/proc/18468/statm: 112454 7244 380 364 0 48324 0
[pid=18470] ppid=18468 vsize=461504 CPUtime=8.29 cores=1,3,5,7
/proc/18470/stat : 18470 (TNM) R 18468 18470 18470 0 -1 4202496 2471 0 0 0 829 0 0 0 25 0 1 0 658040224 472580096 2194 33554432000 134512640 135008748 4288222224 18446744073709551615 134529451 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/18470/statm: 115376 2194 46 122 0 114276 0
[pid=18472] ppid=18366 vsize=460248 CPUtime=0 cores=1,3,5,7
/proc/18472/stat : 18472 (python) S 18366 18366 18093 0 -1 4202560 819 0 0 0 0 0 0 0 20 0 1 0 658041041 471293952 7215 33554432000 4194304 5683644 140733495900752 18446744073709551615 232335912518 0 0 16781312 514 13001278326147383295 0 0 17 3 0 0 0
/proc/18472/statm: 115062 7215 347 364 0 50929 0
[pid=18474] ppid=18472 vsize=248616 CPUtime=0.17 cores=1,3,5,7
/proc/18474/stat : 18474 (FH) R 18472 18474 18474 0 -1 4202496 1486 0 0 0 17 0 0 0 21 0 1 0 658041041 254582784 1213 33554432000 134512640 134936988 4289301200 18446744073709551615 134523472 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/18474/statm: 62154 1213 37 104 0 61658 0
[pid=18366/tid=18373] ppid=18364 vsize=469536 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18373/stat : 18373 (python) S 18364 18366 18093 0 -1 4202560 24 933 0 0 0 0 0 0 20 0 19 0 657990172 480804864 8775 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 5 0 0 0
[pid=18366/tid=18376] ppid=18364 vsize=469536 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18376/stat : 18376 (python) S 18364 18366 18093 0 -1 4202560 26 933 0 0 0 0 0 0 20 0 19 0 657990173 480804864 8775 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 1 0 0 0
[pid=18366/tid=18379] ppid=18364 vsize=469536 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18379/stat : 18379 (python) S 18364 18366 18093 0 -1 4202560 22 933 0 0 0 0 0 0 22 0 19 0 657990174 480804864 8775 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 5 0 0 0
[pid=18366/tid=18382] ppid=18364 vsize=469536 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18382/stat : 18382 (python) S 18364 18366 18093 0 -1 4202560 26 933 0 0 0 0 0 0 24 0 19 0 657990175 480804864 8775 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 1 0 0 0
[pid=18366/tid=18393] ppid=18364 vsize=469536 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18393/stat : 18393 (python) S 18364 18366 18093 0 -1 4202560 24 933 0 0 0 0 0 0 20 0 19 0 658000200 480804864 8775 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 5 0 0 0
[pid=18366/tid=18398] ppid=18364 vsize=469536 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18398/stat : 18398 (python) S 18364 18366 18093 0 -1 4202560 24 933 0 0 0 0 0 0 20 0 19 0 658000203 480804864 8775 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 1 0 0 0
[pid=18366/tid=18401] ppid=18364 vsize=469536 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18401/stat : 18401 (python) S 18364 18366 18093 0 -1 4202560 24 933 0 0 0 0 0 0 22 0 19 0 658000204 480804864 8775 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 3 0 0 0
[pid=18366/tid=18404] ppid=18364 vsize=469536 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18404/stat : 18404 (python) S 18364 18366 18093 0 -1 4202560 28 933 0 0 0 0 0 0 20 0 19 0 658000205 480804864 8775 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 7 0 0 0
[pid=18366/tid=18416] ppid=18364 vsize=469536 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18416/stat : 18416 (python) S 18364 18366 18093 0 -1 4202560 23 933 0 0 0 0 0 0 20 0 19 0 658010134 480804864 8775 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 7 0 0 0
[pid=18366/tid=18420] ppid=18364 vsize=469536 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18420/stat : 18420 (python) S 18364 18366 18093 0 -1 4202560 28 933 0 0 0 0 0 0 20 0 19 0 658010228 480804864 8775 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 7 0 0 0
[pid=18366/tid=18424] ppid=18364 vsize=469536 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18424/stat : 18424 (python) S 18364 18366 18093 0 -1 4202560 30 933 0 0 0 0 0 0 18 0 19 0 658010230 480804864 8775 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 5 0 0 0
[pid=18366/tid=18429] ppid=18364 vsize=469536 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18429/stat : 18429 (python) S 18364 18366 18093 0 -1 4202560 28 933 0 0 0 0 0 0 20 0 19 0 658010734 480804864 8775 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 7 0 0 0
[pid=18366/tid=18439] ppid=18364 vsize=469536 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18439/stat : 18439 (python) S 18364 18366 18093 0 -1 4202560 33 933 0 0 0 0 0 0 18 0 19 0 658019965 480804864 8775 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 7 0 0 0
[pid=18366/tid=18443] ppid=18364 vsize=469536 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18443/stat : 18443 (python) S 18364 18366 18093 0 -1 4202560 31 933 0 0 0 0 0 0 20 0 19 0 658020257 480804864 8775 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 7 0 0 0
[pid=18366/tid=18447] ppid=18364 vsize=469536 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18447/stat : 18447 (python) S 18364 18366 18093 0 -1 4202560 26 933 0 0 0 0 0 0 20 0 19 0 658021072 480804864 8775 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 5 0 0 0
[pid=18366/tid=18459] ppid=18364 vsize=469536 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18459/stat : 18459 (python) S 18364 18366 18093 0 -1 4202560 29 933 0 0 0 0 0 0 20 0 19 0 658030185 480804864 8775 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 5 0 0 0
[pid=18366/tid=18469] ppid=18364 vsize=469536 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18469/stat : 18469 (python) S 18364 18366 18093 0 -1 4202560 29 933 0 0 0 0 0 0 20 0 19 0 658040224 480804864 8775 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 3 0 0 0
[pid=18366/tid=18473] ppid=18364 vsize=469536 CPUtime=0 cores=1,3,5,7
/proc/18366/task/18473/stat : 18473 (python) S 18364 18366 18093 0 -1 4202560 26 933 0 0 0 0 0 0 20 0 19 0 658041041 480804864 8775 33554432000 4194304 5683644 140733495900752 18446744073709551615 232347716017 0 0 16781312 2 18446604469763057728 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 2026.3
Current children cumulated vsize (KiB) 16092456

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 18366 and gives
#  childrusage.ru_utime.tv_sec=332
#  childrusage.ru_utime.tv_usec=397467
#  childrusage.ru_stime.tv_sec=2
#  childrusage.ru_stime.tv_usec=977547
# CPU time returned by wait4() is 335.375
# while last known CPU time is 2026.3
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 511.53
CPU time (s): 2026.3
CPU user time (s): 2023.47
CPU system time (s): 2.83
CPU usage (%): 396.126
Max. virtual memory (cumulated for all children) (KiB): 16092456

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 332.397
system time used= 2.97755
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 70633
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 6971
involuntary context switches= 1415

runsolver used 3.2775 second user time and 7.76682 second system time

The end

Launcher Data 

Begin job on node132 at 2011-05-15 14:51:57
IDJOB=3378226
IDBENCH=82635
IDSOLVER=1859
FILE ID=node132/3378226-1305463917
RUNJOBID= node132-1305463312-18112
PBS_JOBID= 13324991
Free space on /tmp= 73324 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/large/unif-k5-r20-v2000-c40000-S61914820-087-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378226-1305463917/watcher-3378226-1305463917 -o /tmp/evaluation-result-3378226-1305463917/solver-3378226-1305463917 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378226-1305463917.cnf -seed 2037844210 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 1b9b43753c02a2c226adda648a956e92
RANDOM SEED=2037844210

node132.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      20041152 kB
Buffers:        301868 kB
Cached:        5981480 kB
SwapCached:       6904 kB
Active:        6153836 kB
Inactive:      6429764 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      20041152 kB
SwapTotal:    67111528 kB
SwapFree:     66457188 kB
Dirty:           33084 kB
Writeback:           0 kB
AnonPages:     6297996 kB
Mapped:          26696 kB
Slab:           225868 kB
PageTables:      37020 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 47725504 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73292 MiB
End job on node132 at 2011-05-15 15:00:31