Trace number 3378137

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (MO) 1975.1 498.331

General information on the benchmark

Namerandom/large/
unif-k7-r85-v300-c25500-S1286362348-008-UNKNOWN.cnf
MD5SUM63b66954174098a654eacd27832fcf38
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables300
Number of clauses25500
Sum of the clauses size178500
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 525500

Solver Data 

0.38/0.66	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.38/0.66	c INFO: solving HOME/instance-3378137-1305463076.cnf
0.38/0.66	c INFO: computing task features
0.55/0.84	c INFO: computed variable-clause graph statistics
0.55/0.84	c INFO: computed clause balance statistics
1.45/1.74	c INFO: computed variable balance statistics
1.75/2.09	c INFO: computed small-clause counts
1.75/2.09	c INFO: computed Horn-clause counts
1.86/2.12	c INFO: computed variable graph statistics
1.86/2.12	c INFO: collected features for HOME/instance-3378137-1305463076.cnf in 1.45 s
1.86/2.13	c INFO: running gnovelty+2@0 for 97 with 1198 remaining (b = 0.37)
1.86/2.14	c INFO: running gnovelty+2@0 for 97 with 1198 remaining (b = 0.13)
1.86/2.15	c INFO: running adaptg2wsat2009++@0 for 97 with 1198 remaining (b = 0.08)
1.86/2.16	c INFO: running adaptg2wsat2009++@0 for 97 with 1198 remaining (b = 0.07)
397.68/101.50	c INFO: running adaptg2wsat2009++@0 for 97 with 1099 remaining (b = 0.05)
401.66/102.43	c INFO: running gnovelty+2@0 for 97 with 1098 remaining (b = 0.04)
401.66/102.47	c INFO: running FH@0 for 97 with 1098 remaining (b = 0.03)
401.66/102.49	c INFO: running adaptg2wsat2009++@0 for 97 with 1098 remaining (b = 0.03)
792.74/200.79	c INFO: running adaptg2wsat2009++@0 for 97 with 999 remaining (b = 0.02)
800.67/202.75	c INFO: running FH@0 for 97 with 997 remaining (b = 0.02)
800.67/202.77	c INFO: running hybridGM3@0 for 194 with 997 remaining (b = 0.04)
800.67/202.79	c INFO: running gnovelty+2@0 for 97 with 997 remaining (b = 0.02)
1191.66/301.08	c INFO: running hybridGM3@0 for 194 with 899 remaining (b = 0.03)
1196.42/302.21	c INFO: running adaptg2wsat2009++@0 for 97 with 898 remaining (b = 0.01)
1199.60/303.06	c INFO: running hybridGM3@0 for 194 with 897 remaining (b = 0.03)
1582.91/399.42	c INFO: running hybridGM3@99 for 97 with 801 remaining (b = 0.01)
1611.55/406.64	c INFO: running hybridGM3@0 for 194 with 794 remaining (b = 0.02)
1960.75/494.69	c INFO: running TNM@0 for 705 with 705 remaining (b = 0.08)
1973.04/497.72	c INFO: running FH@0 for 97 with 702 remaining (b = 0.01)
1973.04/497.76	c INFO: running hybridGM3@97 for 97 with 702 remaining (b = 0.01)
1975.10/498.30	c INFO: running adaptg2wsat2009++@99 for 97 with 702 remaining (b = 0.01)
1975.10/498.30	Traceback (most recent call last):
1975.10/498.30	  File "HOME/solve", line 33, in <module>
1975.10/498.30	    plac.call(main)
1975.10/498.30	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
1975.10/498.31	    cmd, result = parser_from(obj).consume(arglist)
1975.10/498.31	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
1975.10/498.31	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
1975.10/498.31	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
1975.10/498.31	    (_, answer, _) = solver(input_path, budget / cores, cores)
1975.10/498.31	  File "HOME/borg/build/lib/borg/bilevel.py", line 202, in __call__
1975.10/498.31	    solver.go(max_cost * borg.defaults.machine_speed)
1975.10/498.31	  File "HOME/borg/build/lib/borg/solvers.py", line 179, in go
1975.10/498.31	    self._process.start()
1975.10/498.31	  File "/home/evaluation/evaluation/tools64/lib/python2.7/multiprocessing/process.py", line 94, in start
1975.10/498.31	    assert self._popen is None, 'cannot start a process twice'
1975.10/498.31	AssertionError: cannot start a process twice

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378137-1305463076/watcher-3378137-1305463076 -o /tmp/evaluation-result-3378137-1305463076/solver-3378137-1305463076 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378137-1305463076.cnf -seed 361929384 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.13 5.91 4.17 5/364 21932
/proc/meminfo: memFree=19321912/32951124 swapFree=57331800/67111528
[pid=21932] ppid=21930 vsize=15772 CPUtime=0 cores=1,3,5,7
/proc/21932/stat : 21932 (python) S 21930 21932 21694 0 -1 4202496 901 0 0 0 0 0 0 0 25 0 1 0 657909963 16150528 783 33554432000 4194304 5683644 140733306097424 18446744073709551615 212436015104 0 2147483391 16781312 2 18446744071699207358 0 0 17 1 0 0 0
/proc/21932/statm: 3943 783 346 364 0 430 0

[startup+0.0467391 s]
/proc/loadavg: 8.13 5.91 4.17 5/364 21932
/proc/meminfo: memFree=19321912/32951124 swapFree=57331800/67111528
[pid=21932] ppid=21930 vsize=18880 CPUtime=0.02 cores=1,3,5,7
/proc/21932/stat : 21932 (python) R 21930 21932 21694 0 -1 4202496 1286 0 0 0 2 0 0 0 25 0 1 0 657909963 19333120 1140 33554432000 4194304 5683644 140733306097424 18446744073709551615 4438566 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/21932/statm: 4720 1140 421 364 0 689 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 18880

[startup+0.100722 s]
/proc/loadavg: 8.13 5.91 4.17 5/364 21932
/proc/meminfo: memFree=19321912/32951124 swapFree=57331800/67111528
[pid=21932] ppid=21930 vsize=91824 CPUtime=0.06 cores=1,3,5,7
/proc/21932/stat : 21932 (python) R 21930 21932 21694 0 -1 4202496 2675 0 0 0 4 2 0 0 17 0 1 0 657909963 94027776 2392 33554432000 4194304 5683644 140733306097424 18446744073709551615 4480016 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/21932/statm: 22956 2392 754 364 0 1621 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 91824

[startup+0.30069 s]
/proc/loadavg: 8.13 5.91 4.17 5/364 21932
/proc/meminfo: memFree=19321912/32951124 swapFree=57331800/67111528
[pid=21932] ppid=21930 vsize=135748 CPUtime=0.18 cores=1,3,5,7
/proc/21932/stat : 21932 (python) S 21930 21932 21694 0 -1 4202496 5965 934 1 0 13 5 0 0 18 0 1 0 657909963 139005952 4151 33554432000 4194304 5683644 140733306097424 18446744073709551615 212436013061 0 2147483391 16781312 2 18446744071699207358 0 0 17 1 0 0 0
/proc/21932/statm: 33937 4151 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.18
Current children cumulated vsize (KiB) 135748

[startup+0.700604 s]
/proc/loadavg: 8.13 5.91 4.17 5/364 21932
/proc/meminfo: memFree=19321912/32951124 swapFree=57331800/67111528
[pid=21932] ppid=21930 vsize=280064 CPUtime=0.45 cores=1,3,5,7
/proc/21932/stat : 21932 (python) R 21930 21932 21694 0 -1 4202496 9683 934 1 0 33 12 0 0 16 0 1 0 657909963 286785536 7362 33554432000 4194304 5683644 140733306097424 18446744073709551615 4603222 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/21932/statm: 70016 7362 1761 364 0 5940 0
Current children cumulated CPU time (s) 0.45
Current children cumulated vsize (KiB) 280064

[startup+1.50045 s]
/proc/loadavg: 8.13 5.91 4.17 6/365 21937
/proc/meminfo: memFree=19291704/32951124 swapFree=57331800/67111528
[pid=21932] ppid=21930 vsize=285636 CPUtime=1.25 cores=1,3,5,7
/proc/21932/stat : 21932 (python) R 21930 21932 21694 0 -1 4202496 12505 934 1 0 113 12 0 0 17 0 1 0 657909963 292491264 8794 33554432000 4194304 5683644 140733306097424 18446744073709551615 212439425733 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/21932/statm: 71409 8794 1826 364 0 7333 0
Current children cumulated CPU time (s) 1.25
Current children cumulated vsize (KiB) 285636

[startup+3.10011 s]
/proc/loadavg: 8.13 5.91 4.17 6/365 21937
/proc/meminfo: memFree=19290836/32951124 swapFree=57331800/67111528
[pid=21932] ppid=21930 vsize=321288 CPUtime=1.91 cores=1,3,5,7
/proc/21932/stat : 21932 (python) S 21930 21932 21694 0 -1 4202496 15289 934 1 0 178 13 0 0 20 0 5 0 657909963 328998912 7538 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447843787 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/21932/statm: 80322 7538 1883 364 0 16231 0
Current children cumulated CPU time (s) 1.91
Current children cumulated vsize (KiB) 321288
heavy processes:

[startup+6.30051 s]
/proc/loadavg: 8.12 5.94 4.19 9/377 21949
/proc/meminfo: memFree=19266188/32951124 swapFree=57331800/67111528
[pid=21932] ppid=21930 vsize=321288 CPUtime=1.91 cores=1,3,5,7
/proc/21932/stat : 21932 (python) S 21930 21932 21694 0 -1 4202496 15289 934 1 0 178 13 0 0 20 0 5 0 657909963 328998912 7538 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447843787 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/21932/statm: 80322 7538 1883 364 0 16231 0
[pid=21938] ppid=21932 vsize=280760 CPUtime=0 cores=1,3,5,7
/proc/21938/stat : 21938 (python) S 21932 21932 21694 0 -1 4202560 880 0 0 0 0 0 0 0 18 0 1 0 657910177 287498240 6001 33554432000 4194304 5683644 140733306097424 18446744073709551615 212436037135 0 0 16781312 514 6034299862558671109 0 0 17 1 0 0 0
/proc/21938/statm: 70190 6001 378 364 0 6108 0
[pid=21940] ppid=21938 vsize=1019708 CPUtime=4.12 cores=1,3,5,7
/proc/21940/stat : 21940 (gnovelty+2) R 21938 21940 21940 0 -1 4202496 949 0 0 0 412 0 0 0 25 0 1 0 657910177 1044180992 689 33554432000 134512640 135663704 4294517744 18446744073709551615 134525715 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/21940/statm: 254927 689 113 282 0 254643 0
[pid=21941] ppid=21932 vsize=291084 CPUtime=0.01 cores=1,3,5,7
/proc/21941/stat : 21941 (python) S 21932 21932 21694 0 -1 4202560 895 0 0 0 0 1 0 0 18 0 1 0 657910178 298070016 5999 33554432000 4194304 5683644 140733306097424 18446744073709551615 212436037190 0 0 16781312 514 1031175060259307784 0 0 17 7 0 0 0
/proc/21941/statm: 72771 5999 372 364 0 8686 0
[pid=21943] ppid=21941 vsize=1019708 CPUtime=4.15 cores=1,3,5,7
/proc/21943/stat : 21943 (gnovelty+2) R 21941 21943 21943 0 -1 4202496 950 0 0 0 415 0 0 0 25 0 1 0 657910178 1044180992 689 33554432000 134512640 135663704 4293381408 18446744073709551615 134525276 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/21943/statm: 254927 689 113 282 0 254643 0
[pid=21944] ppid=21932 vsize=301272 CPUtime=0.01 cores=1,3,5,7
/proc/21944/stat : 21944 (python) S 21932 21932 21694 0 -1 4202560 897 0 0 0 1 0 0 0 18 0 1 0 657910178 308502528 6020 33554432000 4194304 5683644 140733306097424 18446744073709551615 212436037190 0 0 16781312 514 16562816390839369991 0 0 17 7 0 0 0
/proc/21944/statm: 75318 6020 372 364 0 11230 0
[pid=21946] ppid=21944 vsize=420220 CPUtime=4.05 cores=1,3,5,7
/proc/21946/stat : 21946 (adaptg2wsat2009) R 21944 21946 21946 0 -1 4202496 1666 0 0 0 405 0 0 0 25 0 1 0 657910179 430305280 1400 33554432000 134512640 135107411 4293236016 18446744073709551615 134532277 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/21946/statm: 105055 1400 59 146 0 103931 0
[pid=21947] ppid=21932 vsize=312136 CPUtime=0.01 cores=1,3,5,7
/proc/21947/stat : 21947 (python) S 21932 21932 21694 0 -1 4202560 897 0 0 0 1 0 0 0 18 0 1 0 657910179 319627264 6024 33554432000 4194304 5683644 140733306097424 18446744073709551615 212436037190 0 0 16781312 514 16562824087420796927 0 0 17 1 0 0 0
/proc/21947/statm: 78034 6024 372 364 0 13943 0
[pid=21949] ppid=21947 vsize=420220 CPUtime=4.13 cores=1,3,5,7
/proc/21949/stat : 21949 (adaptg2wsat2009) R 21947 21949 21949 0 -1 4202496 1658 0 0 0 413 0 0 0 25 0 1 0 657910180 430305280 1399 33554432000 134512640 135107411 4289976624 18446744073709551615 134531600 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/21949/statm: 105055 1399 59 146 0 103931 0
[pid=21932/tid=21939] ppid=21930 vsize=321288 CPUtime=0 cores=1,3,5,7
/proc/21932/task/21939/stat : 21939 (python) S 21930 21932 21694 0 -1 4202560 24 934 0 0 0 0 0 0 20 0 5 0 657910177 328998912 7538 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 1 0 0 0
[pid=21932/tid=21942] ppid=21930 vsize=321288 CPUtime=0 cores=1,3,5,7
/proc/21932/task/21942/stat : 21942 (python) S 21930 21932 21694 0 -1 4202560 22 934 0 0 0 0 0 0 22 0 5 0 657910178 328998912 7538 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 7 0 0 0
[pid=21932/tid=21945] ppid=21930 vsize=321288 CPUtime=0 cores=1,3,5,7
/proc/21932/task/21945/stat : 21945 (python) S 21930 21932 21694 0 -1 4202560 22 934 0 0 0 0 0 0 24 0 5 0 657910179 328998912 7538 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 7 0 0 0
[pid=21932/tid=21948] ppid=21930 vsize=321288 CPUtime=0 cores=1,3,5,7
/proc/21932/task/21948/stat : 21948 (python) S 21930 21932 21694 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 5 0 657910180 328998912 7538 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 18.39
Current children cumulated vsize (KiB) 4386396

[startup+12.7013 s]
/proc/loadavg: 8.11 5.98 4.21 9/377 21949
/proc/meminfo: memFree=19262840/32951124 swapFree=57331800/67111528
[pid=21932] ppid=21930 vsize=321288 CPUtime=1.91 cores=1,3,5,7
/proc/21932/stat : 21932 (python) S 21930 21932 21694 0 -1 4202496 15289 934 1 0 178 13 0 0 20 0 5 0 657909963 328998912 7538 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447843787 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/21932/statm: 80322 7538 1883 364 0 16231 0
[pid=21938] ppid=21932 vsize=280760 CPUtime=0.02 cores=1,3,5,7
/proc/21938/stat : 21938 (python) S 21932 21932 21694 0 -1 4202560 882 0 0 0 1 1 0 0 18 0 1 0 657910177 287498240 6001 33554432000 4194304 5683644 140733306097424 18446744073709551615 212436037135 0 0 16781312 514 7140175734219178245 0 0 17 1 0 0 0
/proc/21938/statm: 70190 6001 378 364 0 6108 0
[pid=21940] ppid=21938 vsize=1019708 CPUtime=10.49 cores=1,3,5,7
/proc/21940/stat : 21940 (gnovelty+2) R 21938 21940 21940 0 -1 4202496 949 0 0 0 1049 0 0 0 25 0 1 0 657910177 1044180992 689 33554432000 134512640 135663704 4294517744 18446744073709551615 134525276 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/21940/statm: 254927 689 113 282 0 254643 0
[pid=21941] ppid=21932 vsize=291084 CPUtime=0.02 cores=1,3,5,7
/proc/21941/stat : 21941 (python) S 21932 21932 21694 0 -1 4202560 900 0 0 0 1 1 0 0 18 0 1 0 657910178 298070016 5999 33554432000 4194304 5683644 140733306097424 18446744073709551615 212436037190 0 0 16781312 514 7140175734219178248 0 0 17 7 0 0 0
/proc/21941/statm: 72771 5999 372 364 0 8686 0
[pid=21943] ppid=21941 vsize=1019708 CPUtime=10.55 cores=1,3,5,7
/proc/21943/stat : 21943 (gnovelty+2) R 21941 21943 21943 0 -1 4202496 950 0 0 0 1055 0 0 0 25 0 1 0 657910178 1044180992 689 33554432000 134512640 135663704 4293381408 18446744073709551615 134524534 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/21943/statm: 254927 689 113 282 0 254643 0
[pid=21944] ppid=21932 vsize=301272 CPUtime=0.02 cores=1,3,5,7
/proc/21944/stat : 21944 (python) S 21932 21932 21694 0 -1 4202560 899 0 0 0 1 1 0 0 18 0 1 0 657910178 308502528 6020 33554432000 4194304 5683644 140733306097424 18446744073709551615 212436037190 0 0 16781312 514 7140175734219178247 0 0 17 7 0 0 0
/proc/21944/statm: 75318 6020 372 364 0 11230 0
[pid=21946] ppid=21944 vsize=420220 CPUtime=10.35 cores=1,3,5,7
/proc/21946/stat : 21946 (adaptg2wsat2009) R 21944 21946 21946 0 -1 4202496 1666 0 0 0 1035 0 0 0 25 0 1 0 657910179 430305280 1400 33554432000 134512640 135107411 4293236016 18446744073709551615 134531731 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/21946/statm: 105055 1400 59 146 0 103931 0
[pid=21947] ppid=21932 vsize=312136 CPUtime=0.03 cores=1,3,5,7
/proc/21947/stat : 21947 (python) S 21932 21932 21694 0 -1 4202560 903 0 0 0 2 1 0 0 18 0 1 0 657910179 319627264 6024 33554432000 4194304 5683644 140733306097424 18446744073709551615 212436037190 0 0 16781312 514 7140175734219178247 0 0 17 1 0 0 0
/proc/21947/statm: 78034 6024 372 364 0 13943 0
[pid=21949] ppid=21947 vsize=420220 CPUtime=10.53 cores=1,3,5,7
/proc/21949/stat : 21949 (adaptg2wsat2009) R 21947 21949 21949 0 -1 4202496 1658 0 0 0 1053 0 0 0 25 0 1 0 657910180 430305280 1399 33554432000 134512640 135107411 4289976624 18446744073709551615 134531248 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/21949/statm: 105055 1399 59 146 0 103931 0
[pid=21932/tid=21939] ppid=21930 vsize=321288 CPUtime=0 cores=1,3,5,7
/proc/21932/task/21939/stat : 21939 (python) S 21930 21932 21694 0 -1 4202560 24 934 0 0 0 0 0 0 20 0 5 0 657910177 328998912 7538 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 1 0 0 0
[pid=21932/tid=21942] ppid=21930 vsize=321288 CPUtime=0 cores=1,3,5,7
/proc/21932/task/21942/stat : 21942 (python) S 21930 21932 21694 0 -1 4202560 22 934 0 0 0 0 0 0 22 0 5 0 657910178 328998912 7538 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 7 0 0 0
[pid=21932/tid=21945] ppid=21930 vsize=321288 CPUtime=0 cores=1,3,5,7
/proc/21932/task/21945/stat : 21945 (python) S 21930 21932 21694 0 -1 4202560 22 934 0 0 0 0 0 0 24 0 5 0 657910179 328998912 7538 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 7 0 0 0
[pid=21932/tid=21948] ppid=21930 vsize=321288 CPUtime=0 cores=1,3,5,7
/proc/21932/task/21948/stat : 21948 (python) S 21930 21932 21694 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 5 0 657910180 328998912 7538 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 43.92
Current children cumulated vsize (KiB) 4386396

[startup+25.5009 s]
/proc/loadavg: 8.08 6.08 4.27 9/377 21949
/proc/meminfo: memFree=19263472/32951124 swapFree=57331800/67111528
[pid=21932] ppid=21930 vsize=321288 CPUtime=1.91 cores=1,3,5,7
/proc/21932/stat : 21932 (python) S 21930 21932 21694 0 -1 4202496 15289 934 1 0 178 13 0 0 20 0 5 0 657909963 328998912 7538 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447843787 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/21932/statm: 80322 7538 1883 364 0 16231 0
[pid=21938] ppid=21932 vsize=280760 CPUtime=0.05 cores=1,3,5,7
/proc/21938/stat : 21938 (python) S 21932 21932 21694 0 -1 4202560 888 0 0 0 3 2 0 0 18 0 1 0 657910177 287498240 6001 33554432000 4194304 5683644 140733306097424 18446744073709551615 212436037135 0 0 16781312 514 7140175734219178245 0 0 17 1 0 0 0
/proc/21938/statm: 70190 6001 378 364 0 6108 0
[pid=21940] ppid=21938 vsize=1019708 CPUtime=23.23 cores=1,3,5,7
/proc/21940/stat : 21940 (gnovelty+2) R 21938 21940 21940 0 -1 4202496 949 0 0 0 2323 0 0 0 25 0 1 0 657910177 1044180992 689 33554432000 134512640 135663704 4294517744 18446744073709551615 134524750 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/21940/statm: 254927 689 113 282 0 254643 0
[pid=21941] ppid=21932 vsize=291084 CPUtime=0.05 cores=1,3,5,7
/proc/21941/stat : 21941 (python) S 21932 21932 21694 0 -1 4202560 905 0 0 0 2 3 0 0 18 0 1 0 657910178 298070016 5999 33554432000 4194304 5683644 140733306097424 18446744073709551615 212436037190 0 0 16781312 514 7140175734219178248 0 0 17 7 0 0 0
/proc/21941/statm: 72771 5999 372 364 0 8686 0
[pid=21943] ppid=21941 vsize=1019708 CPUtime=23.35 cores=1,3,5,7
/proc/21943/stat : 21943 (gnovelty+2) R 21941 21943 21943 0 -1 4202496 950 0 0 0 2335 0 0 0 25 0 1 0 657910178 1044180992 689 33554432000 134512640 135663704 4293381408 18446744073709551615 134524551 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/21943/statm: 254927 689 113 282 0 254643 0
[pid=21944] ppid=21932 vsize=301272 CPUtime=0.06 cores=1,3,5,7
/proc/21944/stat : 21944 (python) S 21932 21932 21694 0 -1 4202560 903 0 0 0 3 3 0 0 18 0 1 0 657910178 308502528 6020 33554432000 4194304 5683644 140733306097424 18446744073709551615 212436037190 0 0 16781312 514 7140175734219178247 0 0 17 3 0 0 0
/proc/21944/statm: 75318 6020 372 364 0 11230 0
[pid=21946] ppid=21944 vsize=420220 CPUtime=22.94 cores=1,3,5,7
/proc/21946/stat : 21946 (adaptg2wsat2009) R 21944 21946 21946 0 -1 4202496 1666 0 0 0 2294 0 0 0 25 0 1 0 657910179 430305280 1400 33554432000 134512640 135107411 4293236016 18446744073709551615 134532277 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/21946/statm: 105055 1400 59 146 0 103931 0
[pid=21947] ppid=21932 vsize=312136 CPUtime=0.05 cores=1,3,5,7
/proc/21947/stat : 21947 (python) S 21932 21932 21694 0 -1 4202560 907 0 0 0 3 2 0 0 18 0 1 0 657910179 319627264 6024 33554432000 4194304 5683644 140733306097424 18446744073709551615 212436037190 0 0 16781312 514 7140175734219178247 0 0 17 1 0 0 0
/proc/21947/statm: 78034 6024 372 364 0 13943 0
[pid=21949] ppid=21947 vsize=420220 CPUtime=23.33 cores=1,3,5,7
/proc/21949/stat : 21949 (adaptg2wsat2009) R 21947 21949 21949 0 -1 4202496 1658 0 0 0 2333 0 0 0 25 0 1 0 657910180 430305280 1399 33554432000 134512640 135107411 4289976624 18446744073709551615 134531246 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/21949/statm: 105055 1399 59 146 0 103931 0
[pid=21932/tid=21939] ppid=21930 vsize=321288 CPUtime=0 cores=1,3,5,7
/proc/21932/task/21939/stat : 21939 (python) S 21930 21932 21694 0 -1 4202560 24 934 0 0 0 0 0 0 20 0 5 0 657910177 328998912 7538 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 1 0 0 0
[pid=21932/tid=21942] ppid=21930 vsize=321288 CPUtime=0 cores=1,3,5,7
/proc/21932/task/21942/stat : 21942 (python) S 21930 21932 21694 0 -1 4202560 22 934 0 0 0 0 0 0 22 0 5 0 657910178 328998912 7538 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 7 0 0 0
[pid=21932/tid=21945] ppid=21930 vsize=321288 CPUtime=0 cores=1,3,5,7

################
# More data... #
################

/proc/21947/stat : 21947 (python) S 21932 21932 21694 0 -1 4202560 1085 0 0 0 14 12 0 0 19 0 2 0 657910179 319627264 6053 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/21947/statm: 78034 6053 400 364 0 13943 0
[pid=21949] ppid=21947 vsize=420220 CPUtime=100.06 cores=1,3,5,7
/proc/21949/stat : 21949 (adaptg2wsat2009) T 21947 21949 21949 0 -1 4202496 1658 0 0 0 10006 0 0 0 25 0 1 0 657910180 430305280 1399 33554432000 134512640 135107411 4289976624 18446744073709551615 134531392 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/21949/statm: 105055 1399 59 146 0 103931 0
[pid=21947/tid=21972] ppid=21932 vsize=312136 CPUtime=0 cores=1,3,5,7
/proc/21947/task/21972/stat : 21972 (python) S 21932 21932 21694 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 657920205 319627264 6053 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 514 18446604456769026240 0 0 -1 1 0 0 0
[pid=21969] ppid=21932 vsize=321860 CPUtime=0.27 cores=1,3,5,7
/proc/21969/stat : 21969 (python) S 21932 21932 21694 0 -1 4202560 1084 0 0 0 14 13 0 0 19 0 2 0 657920113 329584640 6057 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/21969/statm: 80465 6057 400 364 0 16371 0
[pid=21971] ppid=21969 vsize=420220 CPUtime=98.99 cores=1,3,5,7
/proc/21971/stat : 21971 (adaptg2wsat2009) T 21969 21971 21971 0 -1 4202496 1661 0 0 0 9899 0 0 0 25 0 1 0 657920114 430305280 1399 33554432000 134512640 135107411 4293134528 18446744073709551615 134539082 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/21971/statm: 105055 1399 59 146 0 103931 0
[pid=21969/tid=21999] ppid=21932 vsize=321860 CPUtime=0 cores=1,3,5,7
/proc/21969/task/21999/stat : 21999 (python) S 21932 21932 21694 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 657930041 329584640 6057 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 514 18446604467297976768 0 0 -1 1 0 0 0
[pid=21973] ppid=21932 vsize=332280 CPUtime=0.28 cores=1,3,5,7
/proc/21973/stat : 21973 (python) S 21932 21932 21694 0 -1 4202560 1087 0 0 0 15 13 0 0 19 0 2 0 657920206 340254720 6063 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/21973/statm: 83070 6063 400 364 0 18973 0
[pid=21975] ppid=21973 vsize=1019708 CPUtime=98.2 cores=1,3,5,7
/proc/21975/stat : 21975 (gnovelty+2) T 21973 21975 21975 0 -1 4202496 948 0 0 0 9820 0 0 0 25 0 1 0 657920207 1044180992 689 33554432000 134512640 135663704 4288502432 18446744073709551615 134524888 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/21975/statm: 254927 689 113 282 0 254643 0
[pid=21973/tid=22007] ppid=21932 vsize=332280 CPUtime=0 cores=1,3,5,7
/proc/21973/task/22007/stat : 22007 (python) S 21932 21932 21694 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 657930240 340254720 6063 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 514 18446604464860808256 0 0 -1 3 0 0 0
[pid=21978] ppid=21932 vsize=342388 CPUtime=0.27 cores=1,3,5,7
/proc/21978/stat : 21978 (python) S 21932 21932 21694 0 -1 4202560 1089 0 0 0 14 13 0 0 19 0 2 0 657920211 350605312 6108 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/21978/statm: 85597 6108 400 364 0 21497 0
[pid=21980] ppid=21978 vsize=249872 CPUtime=100.26 cores=1,3,5,7
/proc/21980/stat : 21980 (FH) T 21978 21980 21980 0 -1 4202496 1523 0 0 0 10026 0 0 0 25 0 1 0 657920212 255868928 1264 33554432000 134512640 134936988 4289075248 18446744073709551615 134552382 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/21980/statm: 62468 1264 41 104 0 61972 0
[pid=21978/tid=22003] ppid=21932 vsize=342388 CPUtime=0 cores=1,3,5,7
/proc/21978/task/22003/stat : 22003 (python) S 21932 21932 21694 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 657930238 350605312 6108 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 514 18446604458612632256 0 0 -1 1 0 0 0
[pid=21981] ppid=21932 vsize=353248 CPUtime=0.27 cores=1,3,5,7
/proc/21981/stat : 21981 (python) S 21932 21932 21694 0 -1 4202560 1138 0 0 0 14 13 0 0 19 0 2 0 657920213 361725952 6118 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/21981/statm: 88312 6118 406 364 0 24209 0
[pid=21983] ppid=21981 vsize=420220 CPUtime=99.45 cores=1,3,5,7
/proc/21983/stat : 21983 (adaptg2wsat2009) T 21981 21983 21983 0 -1 4202496 1667 0 0 0 9945 0 0 0 25 0 1 0 657920213 430305280 1399 33554432000 134512640 135107411 4293688336 18446744073709551615 134532121 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/21983/statm: 105055 1399 59 146 0 103931 0
[pid=21981/tid=22011] ppid=21932 vsize=353248 CPUtime=0 cores=1,3,5,7
/proc/21981/task/22011/stat : 22011 (python) S 21932 21932 21694 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 657930242 361725952 6118 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 514 18446604468639229056 0 0 -1 5 0 0 0
[pid=22000] ppid=21932 vsize=363020 CPUtime=0.28 cores=1,3,5,7
/proc/22000/stat : 22000 (python) S 21932 21932 21694 0 -1 4202560 1136 0 0 0 14 14 0 0 19 0 2 0 657930042 371732480 6235 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/22000/statm: 90755 6235 405 364 0 26649 0
[pid=22002] ppid=22000 vsize=420220 CPUtime=99.67 cores=1,3,5,7
/proc/22002/stat : 22002 (adaptg2wsat2009) T 22000 22002 22002 0 -1 4202496 1668 0 0 0 9967 0 0 0 25 0 1 0 657930043 430305280 1400 33554432000 134512640 135107411 4287174480 18446744073709551615 134532313 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/22002/statm: 105055 1400 59 146 0 103931 0
[pid=22000/tid=22023] ppid=21932 vsize=363020 CPUtime=0 cores=1,3,5,7
/proc/22000/task/22023/stat : 22023 (python) S 21932 21932 21694 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 657940070 371732480 6235 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 514 18446604454224050944 0 0 -1 1 0 0 0
[pid=22004] ppid=21932 vsize=373468 CPUtime=0.28 cores=1,3,5,7
/proc/22004/stat : 22004 (python) S 21932 21932 21694 0 -1 4202560 1110 0 0 0 14 14 0 0 19 0 2 0 657930239 382431232 6234 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/22004/statm: 93367 6234 400 364 0 29258 0
[pid=22006] ppid=22004 vsize=249876 CPUtime=99.89 cores=1,3,5,7
/proc/22006/stat : 22006 (FH) T 22004 22006 22006 0 -1 4202496 1531 0 0 0 9989 0 0 0 25 0 1 0 657930239 255873024 1264 33554432000 134512640 134936988 4290157952 18446744073709551615 134553554 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/22006/statm: 62469 1264 41 104 0 61973 0
[pid=22004/tid=22031] ppid=21932 vsize=373468 CPUtime=0 cores=1,3,5,7
/proc/22004/task/22031/stat : 22031 (python) S 21932 21932 21694 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 657940269 382431232 6234 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 514 18446604459263480000 0 0 -1 1 0 0 0
[pid=22008] ppid=21932 vsize=383664 CPUtime=0.83 cores=1,3,5,7
/proc/22008/stat : 22008 (python) S 21932 21932 21694 0 -1 4202560 1167 0 0 0 42 41 0 0 18 0 2 0 657930241 392871936 6238 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/22008/statm: 95916 6238 400 364 0 31804 0
[pid=22010] ppid=22008 vsize=268420 CPUtime=291.39 cores=1,3,5,7
/proc/22010/stat : 22010 (hybridGM3) T 22008 22010 22010 0 -1 4202496 5391 0 0 0 29139 0 0 0 25 0 1 0 657930241 274862080 5126 33554432000 134512640 135204385 4291826528 18446744073709551615 134538386 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/22010/statm: 67105 5126 63 169 0 66935 0
[pid=22008/tid=22136] ppid=21932 vsize=383664 CPUtime=0 cores=1,3,5,7
/proc/22008/task/22136/stat : 22136 (python) S 21932 21932 21694 0 -1 4202560 49 0 0 0 0 0 0 0 18 0 2 0 657949904 392871936 6238 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 514 18446604457191543232 0 0 -1 5 0 0 0
[pid=22012] ppid=21932 vsize=394176 CPUtime=0.29 cores=1,3,5,7
/proc/22012/stat : 22012 (python) S 21932 21932 21694 0 -1 4202560 1127 0 0 0 14 15 0 0 19 0 2 0 657930243 403636224 6242 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/22012/statm: 98544 6242 400 364 0 34429 0
[pid=22014] ppid=22012 vsize=1019708 CPUtime=97.04 cores=1,3,5,7
/proc/22014/stat : 22014 (gnovelty+2) T 22012 22014 22014 0 -1 4202496 957 0 0 0 9704 0 0 0 25 0 1 0 657930243 1044180992 689 33554432000 134512640 135663704 4286684032 18446744073709551615 134524897 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/22014/statm: 254927 689 113 282 0 254643 0
[pid=22012/tid=22027] ppid=21932 vsize=394176 CPUtime=0 cores=1,3,5,7
/proc/22012/task/22027/stat : 22027 (python) S 21932 21932 21694 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 657940183 403636224 6242 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 514 18446604457227712576 0 0 -1 5 0 0 0
[pid=22024] ppid=21932 vsize=404044 CPUtime=0.58 cores=1,3,5,7
/proc/22024/stat : 22024 (python) S 21932 21932 21694 0 -1 4202560 1162 0 0 0 29 29 0 0 18 0 2 0 657940071 413741056 6245 33554432000 4194304 5683644 140733306097424 18446744073709551615 212436037190 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/22024/statm: 101011 6245 399 364 0 36893 0
[pid=22026] ppid=22024 vsize=268424 CPUtime=196.81 cores=1,3,5,7
/proc/22026/stat : 22026 (hybridGM3) R 22024 22026 22026 0 -1 4202496 5395 0 0 0 19681 0 0 0 25 0 1 0 657940072 274866176 5127 33554432000 134512640 135204385 4291303040 18446744073709551615 134538355 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/22026/statm: 67106 5127 63 169 0 66936 0
[pid=22024/tid=22154] ppid=21932 vsize=404044 CPUtime=0 cores=1,3,5,7
/proc/22024/task/22154/stat : 22154 (python) S 21932 21932 21694 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 657959738 413741056 6245 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 514 18446604471010591040 0 0 -1 3 0 0 0
[pid=22028] ppid=21932 vsize=415012 CPUtime=0.31 cores=1,3,5,7
/proc/22028/stat : 22028 (python) S 21932 21932 21694 0 -1 4202560 1149 0 0 0 14 17 0 0 19 0 2 0 657940185 424972288 6250 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/22028/statm: 103753 6250 400 364 0 39632 0
[pid=22030] ppid=22028 vsize=420220 CPUtime=101.27 cores=1,3,5,7
/proc/22030/stat : 22030 (adaptg2wsat2009) T 22028 22030 22030 0 -1 4202496 1670 0 0 0 10127 0 0 0 25 0 1 0 657940185 430305280 1400 33554432000 134512640 135107411 4294717536 18446744073709551615 134531239 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/22030/statm: 105055 1400 59 146 0 103931 0
[pid=22028/tid=22137] ppid=21932 vsize=415012 CPUtime=0 cores=1,3,5,7
/proc/22028/task/22137/stat : 22137 (python) S 21932 21932 21694 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 657950627 424972288 6250 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 514 18446604471010591936 0 0 -1 7 0 0 0
[pid=22032] ppid=21932 vsize=424784 CPUtime=0.56 cores=1,3,5,7
/proc/22032/stat : 22032 (python) S 21932 21932 21694 0 -1 4202560 1220 0 0 0 29 27 0 0 19 0 2 0 657940270 434978816 6317 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/22032/statm: 106196 6317 407 364 0 42072 0
[pid=22034] ppid=22032 vsize=268420 CPUtime=194.63 cores=1,3,5,7
/proc/22034/stat : 22034 (hybridGM3) T 22032 22034 22034 0 -1 4202496 5401 0 0 0 19462 1 0 0 25 0 1 0 657940271 274862080 5126 33554432000 134512640 135204385 4287226992 18446744073709551615 134537746 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/22034/statm: 67105 5126 63 169 0 66935 0
[pid=22032/tid=22151] ppid=21932 vsize=424784 CPUtime=0 cores=1,3,5,7
/proc/22032/task/22151/stat : 22151 (python) S 21932 21932 21694 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 657959734 434978816 6317 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 514 18446604471161274560 0 0 -1 1 0 0 0
[pid=22138] ppid=21932 vsize=435328 CPUtime=0.25 cores=1,3,5,7
/proc/22138/stat : 22138 (python) S 21932 21932 21694 0 -1 4202560 1028 0 0 0 12 13 0 0 18 0 1 0 657950628 445775872 6318 33554432000 4194304 5683644 140733306097424 18446744073709551615 212436037190 0 0 16781312 514 7140175734219178245 0 0 17 7 0 0 0
/proc/22138/statm: 108832 6318 380 364 0 44705 0
[pid=22140] ppid=22138 vsize=268424 CPUtime=88.02 cores=1,3,5,7
/proc/22140/stat : 22140 (hybridGM3) R 22138 22140 22140 0 -1 4202496 5401 0 0 0 8802 0 0 0 25 0 1 0 657950628 274866176 5127 33554432000 134512640 135204385 4292835280 18446744073709551615 134538284 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22140/statm: 67106 5127 63 169 0 66936 0
[pid=22148] ppid=21932 vsize=445772 CPUtime=0 cores=1,3,5,7
/proc/22148/stat : 22148 (python) S 21932 21932 21694 0 -1 4202560 853 0 0 0 0 0 0 0 18 0 1 0 657959432 456470528 6299 33554432000 4194304 5683644 140733306097424 18446744073709551615 212436037190 0 0 16781312 514 11795554974639816965 0 0 17 1 0 0 0
/proc/22148/statm: 111443 6299 357 364 0 47313 0
[pid=22150] ppid=22148 vsize=454088 CPUtime=3.37 cores=1,3,5,7
/proc/22150/stat : 22150 (TNM) R 22148 22150 22150 0 -1 4202496 1270 0 0 0 337 0 0 0 25 0 1 0 657959433 464986112 995 33554432000 134512640 135008748 4287961120 18446744073709551615 134523370 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/22150/statm: 113522 995 45 122 0 112422 0
[pid=22152] ppid=21932 vsize=456200 CPUtime=0 cores=1,3,5,7
/proc/22152/stat : 22152 (python) S 21932 21932 21694 0 -1 4202560 817 0 0 0 0 0 0 0 19 0 1 0 657959736 467148800 6293 33554432000 4194304 5683644 140733306097424 18446744073709551615 212436037190 0 0 16781312 514 4607089789240344575 0 0 17 7 0 0 0
/proc/22152/statm: 114050 6293 347 364 0 49917 0
[pid=22155] ppid=22152 vsize=247956 CPUtime=0.47 cores=1,3,5,7
/proc/22155/stat : 22155 (FH) R 22152 22155 22155 0 -1 4202496 1258 0 0 0 47 0 0 0 21 0 1 0 657959738 253906944 987 33554432000 134512640 134936988 4293226768 18446744073709551615 134523462 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/22155/statm: 61989 987 37 104 0 61493 0
[pid=21932/tid=21939] ppid=21930 vsize=465796 CPUtime=0 cores=1,3,5,7
/proc/21932/task/21939/stat : 21939 (python) S 21930 21932 21694 0 -1 4202560 24 934 0 0 0 0 0 0 20 0 19 0 657910177 476975104 7861 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 1 0 0 0
[pid=21932/tid=21942] ppid=21930 vsize=465796 CPUtime=0 cores=1,3,5,7
/proc/21932/task/21942/stat : 21942 (python) S 21930 21932 21694 0 -1 4202560 22 934 0 0 0 0 0 0 22 0 19 0 657910178 476975104 7861 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 7 0 0 0
[pid=21932/tid=21945] ppid=21930 vsize=465796 CPUtime=0 cores=1,3,5,7
/proc/21932/task/21945/stat : 21945 (python) S 21930 21932 21694 0 -1 4202560 22 934 0 0 0 0 0 0 24 0 19 0 657910179 476975104 7861 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 7 0 0 0
[pid=21932/tid=21948] ppid=21930 vsize=465796 CPUtime=0 cores=1,3,5,7
/proc/21932/task/21948/stat : 21948 (python) S 21930 21932 21694 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 19 0 657910180 476975104 7861 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 1 0 0 0
[pid=21932/tid=21970] ppid=21930 vsize=465796 CPUtime=0 cores=1,3,5,7
/proc/21932/task/21970/stat : 21970 (python) S 21930 21932 21694 0 -1 4202560 29 934 0 0 0 0 0 0 20 0 19 0 657920114 476975104 7861 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 1 0 0 0
[pid=21932/tid=21974] ppid=21930 vsize=465796 CPUtime=0 cores=1,3,5,7
/proc/21932/task/21974/stat : 21974 (python) S 21930 21932 21694 0 -1 4202560 23 934 0 0 0 0 0 0 20 0 19 0 657920207 476975104 7861 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 1 0 0 0
[pid=21932/tid=21979] ppid=21930 vsize=465796 CPUtime=0 cores=1,3,5,7
/proc/21932/task/21979/stat : 21979 (python) S 21930 21932 21694 0 -1 4202560 34 934 0 0 0 0 0 0 21 0 19 0 657920211 476975104 7861 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 1 0 0 0
[pid=21932/tid=21982] ppid=21930 vsize=465796 CPUtime=0 cores=1,3,5,7
/proc/21932/task/21982/stat : 21982 (python) S 21930 21932 21694 0 -1 4202560 28 934 0 0 0 0 0 0 23 0 19 0 657920213 476975104 7861 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 1 0 0 0
[pid=21932/tid=22001] ppid=21930 vsize=465796 CPUtime=0 cores=1,3,5,7
/proc/21932/task/22001/stat : 22001 (python) S 21930 21932 21694 0 -1 4202560 21 934 0 0 0 0 0 0 20 0 19 0 657930043 476975104 7861 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 1 0 0 0
[pid=21932/tid=22005] ppid=21930 vsize=465796 CPUtime=0 cores=1,3,5,7
/proc/21932/task/22005/stat : 22005 (python) S 21930 21932 21694 0 -1 4202560 24 934 0 0 0 0 0 0 20 0 19 0 657930239 476975104 7861 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 5 0 0 0
[pid=21932/tid=22009] ppid=21930 vsize=465796 CPUtime=0 cores=1,3,5,7
/proc/21932/task/22009/stat : 22009 (python) S 21930 21932 21694 0 -1 4202560 29 934 0 0 0 0 0 0 18 0 19 0 657930241 476975104 7861 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 7 0 0 0
[pid=21932/tid=22013] ppid=21930 vsize=465796 CPUtime=0 cores=1,3,5,7
/proc/21932/task/22013/stat : 22013 (python) S 21930 21932 21694 0 -1 4202560 25 934 0 0 0 0 0 0 22 0 19 0 657930243 476975104 7861 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 7 0 0 0
[pid=21932/tid=22025] ppid=21930 vsize=465796 CPUtime=0 cores=1,3,5,7
/proc/21932/task/22025/stat : 22025 (python) S 21930 21932 21694 0 -1 4202560 32 934 0 0 0 0 0 0 18 0 19 0 657940072 476975104 7861 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 7 0 0 0
[pid=21932/tid=22029] ppid=21930 vsize=465796 CPUtime=0 cores=1,3,5,7
/proc/21932/task/22029/stat : 22029 (python) S 21930 21932 21694 0 -1 4202560 31 934 0 0 0 0 0 0 20 0 19 0 657940185 476975104 7861 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 7 0 0 0
[pid=21932/tid=22033] ppid=21930 vsize=465796 CPUtime=0 cores=1,3,5,7
/proc/21932/task/22033/stat : 22033 (python) S 21930 21932 21694 0 -1 4202560 28 934 0 0 0 0 0 0 20 0 19 0 657940271 476975104 7861 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 7 0 0 0
[pid=21932/tid=22139] ppid=21930 vsize=465796 CPUtime=0 cores=1,3,5,7
/proc/21932/task/22139/stat : 22139 (python) S 21930 21932 21694 0 -1 4202560 29 934 0 0 0 0 0 0 20 0 19 0 657950628 476975104 7861 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 7 0 0 0
[pid=21932/tid=22149] ppid=21930 vsize=465796 CPUtime=0 cores=1,3,5,7
/proc/21932/task/22149/stat : 22149 (python) S 21930 21932 21694 0 -1 4202560 24 934 0 0 0 0 0 0 20 0 19 0 657959433 476975104 7861 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 7 0 0 0
[pid=21932/tid=22153] ppid=21930 vsize=465796 CPUtime=0 cores=1,3,5,7
/proc/21932/task/22153/stat : 22153 (python) S 21930 21932 21694 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 19 0 657959736 476975104 7861 33554432000 4194304 5683644 140733306097424 18446744073709551615 212447840689 0 0 16781312 2 18446604456769025344 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 1975.1
Current children cumulated vsize (KiB) 15982168

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 21932 and gives
#  childrusage.ru_utime.tv_sec=205
#  childrusage.ru_utime.tv_usec=732723
#  childrusage.ru_stime.tv_sec=3
#  childrusage.ru_stime.tv_usec=29539
# CPU time returned by wait4() is 208.762
# while last known CPU time is 1975.1
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 498.331
CPU time (s): 1975.1
CPU user time (s): 1972.18
CPU system time (s): 2.92
CPU usage (%): 396.343
Max. virtual memory (cumulated for all children) (KiB): 15982168

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 205.733
system time used= 3.02954
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 59711
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 6018
involuntary context switches= 750

runsolver used 3.45247 second user time and 7.37588 second system time

The end

Launcher Data 

Begin job on node102 at 2011-05-15 14:37:56
IDJOB=3378137
IDBENCH=82546
IDSOLVER=1859
FILE ID=node102/3378137-1305463076
RUNJOBID= node102-1305462764-21713
PBS_JOBID= 13324973
Free space on /tmp= 73616 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/large/unif-k7-r85-v300-c25500-S1286362348-008-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378137-1305463076/watcher-3378137-1305463076 -o /tmp/evaluation-result-3378137-1305463076/solver-3378137-1305463076 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378137-1305463076.cnf -seed 361929384 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 63b66954174098a654eacd27832fcf38
RANDOM SEED=361929384

node102.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      19322192 kB
Buffers:         36040 kB
Cached:         730044 kB
SwapCached:      89916 kB
Active:        4151000 kB
Inactive:      9282876 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      19322192 kB
SwapTotal:    67111528 kB
SwapFree:     57331800 kB
Dirty:           32212 kB
Writeback:           0 kB
AnonPages:    12588980 kB
Mapped:          21240 kB
Slab:            62780 kB
PageTables:      68640 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 69002196 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73576 MiB
End job on node102 at 2011-05-15 14:46:16