Trace number 3317131

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (TO) 4800.08 1208.32

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v20000-c84000-S357264423-067-UNKNOWN.cnf
MD5SUM425b4c5e5535f7d6b9d17ca8feb0ac3e
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark78.815
Satisfiable
(Un)Satisfiability was proved
Number of variables20000
Number of clauses84000
Sum of the clauses size252000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 384000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.35/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.35/0.67	c INFO: solving HOME/instance-3317131-1304267441.cnf
0.35/0.67	c INFO: computing task features
0.63/0.95	c INFO: computed variable-clause graph statistics
0.63/0.96	c INFO: computed clause balance statistics
2.14/2.47	c INFO: computed variable balance statistics
3.34/3.67	c INFO: computed small-clause counts
3.34/3.68	c INFO: computed Horn-clause counts
3.94/4.23	c INFO: computed variable graph statistics
3.94/4.23	c INFO: collected features for HOME/instance-3317131-1304267441.cnf in 3.56 s
3.94/4.26	c INFO: running hybridGM3@0 for 97 with 4796 remaining (b = 0.43)
3.94/4.29	c INFO: running hybridGM3@0 for 97 with 4796 remaining (b = 0.26)
4.06/4.32	c INFO: running hybridGM3@0 for 97 with 4796 remaining (b = 0.19)
4.06/4.35	c INFO: running hybridGM3@0 for 194 with 4796 remaining (b = 0.26)
401.36/103.88	c INFO: running hybridGM3@0 for 97 with 4696 remaining (b = 0.12)
405.34/104.82	c INFO: running hybridGM3@0 for 97 with 4695 remaining (b = 0.12)
405.34/104.84	c INFO: running hybridGM3@0 for 97 with 4695 remaining (b = 0.12)
785.16/199.95	c INFO: running hybridGM3@100 for 97 with 4600 remaining (b = 0.12)
785.16/199.98	c INFO: running hybridGM3@94 for 97 with 4600 remaining (b = 0.12)
789.54/201.01	c INFO: running hybridGM3@98 for 97 with 4599 remaining (b = 0.12)
805.52/205.03	c INFO: running hybridGM3@100 for 97 with 4595 remaining (b = 0.12)
1181.32/299.17	c INFO: running TNM@0 for 97 with 4501 remaining (b = 0.14)
1189.28/301.10	c INFO: running hybridGM3@0 for 194 with 4499 remaining (b = 0.20)
1189.28/301.14	c INFO: running hybridGM3@0 for 194 with 4499 remaining (b = 0.15)
1193.23/302.16	c INFO: running hybridGM3@0 for 194 with 4498 remaining (b = 0.12)
1580.51/399.35	c INFO: running hybridGM3@0 for 194 with 4401 remaining (b = 0.09)
1972.89/497.82	c INFO: running hybridGM3@101 for 97 with 4302 remaining (b = 0.04)
1972.89/497.86	c INFO: running hybridGM3@100 for 97 with 4302 remaining (b = 0.04)
1976.87/498.89	c INFO: running hybridGM3@100 for 97 with 4301 remaining (b = 0.04)
2371.63/598.00	c INFO: running hybridGM3@296 for 97 with 4202 remaining (b = 0.03)
2376.03/599.04	c INFO: running hybridGM3@296 for 97 with 4201 remaining (b = 0.04)
2376.03/599.07	c INFO: running hybridGM3@0 for 194 with 4201 remaining (b = 0.07)
2384.38/601.10	c INFO: running TNM@0 for 97 with 4199 remaining (b = 0.04)
2750.87/693.16	c INFO: running TNM@100 for 1455 with 4107 remaining (b = 0.31)
2755.24/694.21	c INFO: running adaptg2wsat2009++@0 for 97 with 4106 remaining (b = 0.03)
2779.52/700.31	c INFO: running hybridGM3@0 for 194 with 4100 remaining (b = 0.05)
3153.95/794.39	c INFO: running hybridGM3@0 for 194 with 4006 remaining (b = 0.04)
3159.93/795.81	c INFO: running adaptg2wsat2009++@97 for 97 with 4004 remaining (b = 0.02)
3558.66/895.98	c INFO: running hybridGM3@397 for 97 with 3904 remaining (b = 0.02)
3586.99/903.06	c INFO: running gnovelty+2@0 for 97 with 3897 remaining (b = 0.03)
3937.49/991.14	c INFO: running hybridGM3@99 for 97 with 3809 remaining (b = 0.02)
3941.46/992.19	c INFO: running hybridGM3@394 for 97 with 3808 remaining (b = 0.02)
3985.97/1003.34	c INFO: running TNM@0 for 97 with 3797 remaining (b = 0.02)
4323.71/1088.34	c INFO: running hybridGM3@193 for 97 with 3712 remaining (b = 0.02)
4327.68/1089.39	c INFO: running TNM@0 for 97 with 3711 remaining (b = 0.02)
4384.08/1103.54	c INFO: running hybridGM3@0 for 194 with 3697 remaining (b = 0.03)
4701.97/1183.54	c INFO: running TNM@0 for 97 with 3617 remaining (b = 0.01)
4726.16/1189.61	c INFO: running TNM@94 for 1455 with 3611 remaining (b = 0.17)
4800.08/1208.28	c INFO: running hybridGM3@0 for 194 with 3592 remaining (b = 0.02)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317131-1304267441/watcher-3317131-1304267441 -o /tmp/evaluation-result-3317131-1304267441/solver-3317131-1304267441 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317131-1304267441.cnf -seed 1247388603 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.53 3.05 1.13 5/209 28598
/proc/meminfo: memFree=29684992/32951124 swapFree=67102756/67111528
[pid=28598] ppid=28596 vsize=15384 CPUtime=0 cores=1,3,5,7
/proc/28598/stat : 28598 (python) R 28596 28598 28463 0 -1 4202496 755 0 0 0 0 0 0 0 23 0 1 0 538342353 15753216 644 33554432000 4194304 5683644 140736723752192 18446744073709551615 246516528109 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/28598/statm: 3846 662 322 364 0 333 0

[startup+0.0250819 s]
/proc/loadavg: 7.53 3.05 1.13 5/209 28598
/proc/meminfo: memFree=29684992/32951124 swapFree=67102756/67111528
[pid=28598] ppid=28596 vsize=16400 CPUtime=0.01 cores=1,3,5,7
/proc/28598/stat : 28598 (python) R 28596 28598 28463 0 -1 4202496 1112 0 0 0 1 0 0 0 23 0 1 0 538342353 16793600 980 33554432000 4194304 5683644 140736723752192 18446744073709551615 246516830053 0 2147483391 16781312 2 0 0 0 17 5 0 0 0
/proc/28598/statm: 4100 980 407 364 0 587 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16400

[startup+0.10106 s]
/proc/loadavg: 7.53 3.05 1.13 5/209 28598
/proc/meminfo: memFree=29684992/32951124 swapFree=67102756/67111528
[pid=28598] ppid=28596 vsize=89768 CPUtime=0.06 cores=1,3,5,7
/proc/28598/stat : 28598 (python) S 28596 28598 28463 0 -1 4202496 2673 0 0 0 4 2 0 0 23 0 1 0 538342353 91922432 2388 33554432000 4194304 5683644 140736723752192 18446744073709551615 246516832256 0 2147483391 16781312 2 0 0 0 17 5 0 0 0
/proc/28598/statm: 22442 2388 750 364 0 1622 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 89768

[startup+0.30103 s]
/proc/loadavg: 7.53 3.05 1.13 5/209 28598
/proc/meminfo: memFree=29684992/32951124 swapFree=67102756/67111528
[pid=28598] ppid=28596 vsize=135748 CPUtime=0.18 cores=1,3,5,7
/proc/28598/stat : 28598 (python) R 28596 28598 28463 0 -1 4202496 5963 935 1 0 12 6 0 0 17 0 1 0 538342353 139005952 4151 33554432000 4194304 5683644 140736723752192 18446744073709551615 4620076 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/28598/statm: 33937 4151 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.18
Current children cumulated vsize (KiB) 135748

[startup+0.700964 s]
/proc/loadavg: 7.53 3.05 1.13 5/209 28598
/proc/meminfo: memFree=29684992/32951124 swapFree=67102756/67111528
[pid=28598] ppid=28596 vsize=281824 CPUtime=0.43 cores=1,3,5,7
/proc/28598/stat : 28598 (python) R 28596 28598 28463 0 -1 4202496 10100 935 1 0 31 12 0 0 15 0 1 0 538342353 288587776 7783 33554432000 4194304 5683644 140736723752192 18446744073709551615 4571152 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/28598/statm: 70456 7784 1761 364 0 6380 0
Current children cumulated CPU time (s) 0.43
Current children cumulated vsize (KiB) 281824

[startup+1.50082 s]
/proc/loadavg: 7.53 3.05 1.13 7/210 28603
/proc/meminfo: memFree=29643192/32951124 swapFree=67102756/67111528
[pid=28598] ppid=28596 vsize=297692 CPUtime=1.24 cores=1,3,5,7
/proc/28598/stat : 28598 (python) R 28596 28598 28463 0 -1 4202496 16354 935 1 0 111 13 0 0 16 0 1 0 538342353 304836608 11808 33554432000 4194304 5683644 140736723752192 18446744073709551615 246516492097 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/28598/statm: 74423 11808 1826 364 0 10347 0
Current children cumulated CPU time (s) 1.24
Current children cumulated vsize (KiB) 297692

[startup+3.10053 s]
/proc/loadavg: 7.32 3.08 1.15 7/210 28603
/proc/meminfo: memFree=29643192/32951124 swapFree=67102756/67111528
[pid=28598] ppid=28596 vsize=297692 CPUtime=2.84 cores=1,3,5,7
/proc/28598/stat : 28598 (python) R 28596 28598 28463 0 -1 4202496 16355 935 1 0 271 13 0 0 18 0 1 0 538342353 304836608 11809 33554432000 4194304 5683644 140736723752192 18446744073709551615 4457125 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/28598/statm: 74423 11809 1827 364 0 10347 0
Current children cumulated CPU time (s) 2.84
Current children cumulated vsize (KiB) 297692

[startup+6.30091 s]
/proc/loadavg: 7.32 3.08 1.15 11/222 28615
/proc/meminfo: memFree=29537820/32951124 swapFree=67102756/67111528
[pid=28598] ppid=28596 vsize=328356 CPUtime=4.09 cores=1,3,5,7
/proc/28598/stat : 28598 (python) S 28596 28598 28463 0 -1 4202496 19234 935 1 0 395 14 0 0 20 0 5 0 538342353 336236544 9285 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528660939 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/28598/statm: 82089 9285 1881 364 0 17998 0
[pid=28604] ppid=28598 vsize=287828 CPUtime=0 cores=1,3,5,7
/proc/28604/stat : 28604 (python) S 28598 28598 28463 0 -1 4202560 743 0 0 0 0 0 0 0 18 0 1 0 538342779 294735872 7610 33554432000 4194304 5683644 140736723752192 18446744073709551615 246516854287 0 0 16781312 514 4470689049622839560 0 0 17 5 0 0 0
/proc/28604/statm: 71957 7610 355 364 0 7875 0
[pid=28606] ppid=28604 vsize=268424 CPUtime=2.02 cores=1,3,5,7
/proc/28606/stat : 28606 (hybridGM3) R 28604 28606 28606 0 -1 4202496 6658 0 0 0 201 1 0 0 25 0 1 0 538342779 274866176 6395 33554432000 134512640 135204385 4291208656 18446744073709551615 134545851 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/28606/statm: 67106 6395 64 169 0 66936 0
[pid=28607] ppid=28598 vsize=297744 CPUtime=0 cores=1,3,5,7
/proc/28607/stat : 28607 (python) S 28598 28598 28463 0 -1 4202560 745 0 0 0 0 0 0 0 18 0 1 0 538342782 304889856 7742 33554432000 4194304 5683644 140736723752192 18446744073709551615 246516854342 0 0 16781312 514 4397716661910601992 0 0 17 5 0 0 0
/proc/28607/statm: 74436 7742 349 364 0 10351 0
[pid=28609] ppid=28607 vsize=268424 CPUtime=1.98 cores=1,3,5,7
/proc/28609/stat : 28609 (hybridGM3) R 28607 28609 28609 0 -1 4202496 6658 0 0 0 198 0 0 0 25 0 1 0 538342782 274866176 6396 33554432000 134512640 135204385 4287939648 18446744073709551615 134536641 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/28609/statm: 67106 6396 64 169 0 66936 0
[pid=28610] ppid=28598 vsize=308000 CPUtime=0 cores=1,3,5,7
/proc/28610/stat : 28610 (python) S 28598 28598 28463 0 -1 4202560 749 0 0 0 0 0 0 0 15 0 1 0 538342785 315392000 7746 33554432000 4194304 5683644 140736723752192 18446744073709551615 246516854342 0 0 16781312 514 4404898946741141768 0 0 17 7 0 0 0
/proc/28610/statm: 77000 7746 349 364 0 12912 0
[pid=28612] ppid=28610 vsize=268424 CPUtime=1.96 cores=1,3,5,7
/proc/28612/stat : 28612 (hybridGM3) R 28610 28612 28612 0 -1 4202496 6658 0 0 0 195 1 0 0 25 0 1 0 538342785 274866176 6396 33554432000 134512640 135204385 4290548880 18446744073709551615 134545823 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/28612/statm: 67106 6396 64 169 0 66936 0
[pid=28613] ppid=28598 vsize=318864 CPUtime=0 cores=1,3,5,7
/proc/28613/stat : 28613 (python) S 28598 28598 28463 0 -1 4202560 756 0 0 0 0 0 0 0 15 0 1 0 538342788 326516736 7750 33554432000 4194304 5683644 140736723752192 18446744073709551615 246516854342 0 0 16781312 514 15103995907978264839 0 0 17 1 0 0 0
/proc/28613/statm: 79716 7750 349 364 0 15625 0
[pid=28615] ppid=28613 vsize=268428 CPUtime=1.94 cores=1,3,5,7
/proc/28615/stat : 28615 (hybridGM3) R 28613 28615 28615 0 -1 4202496 6656 0 0 0 193 1 0 0 25 0 1 0 538342788 274870272 6394 33554432000 134512640 135204385 4290353312 18446744073709551615 134537242 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/28615/statm: 67107 6394 64 169 0 66937 0
[pid=28598/tid=28605] ppid=28596 vsize=328356 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28605/stat : 28605 (python) S 28596 28598 28463 0 -1 4202560 27 935 0 0 0 0 0 0 23 0 5 0 538342779 336236544 9285 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 7 0 0 0
[pid=28598/tid=28608] ppid=28596 vsize=328356 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28608/stat : 28608 (python) S 28596 28598 28463 0 -1 4202560 25 935 0 0 0 0 0 0 25 0 5 0 538342782 336236544 9285 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 3 0 0 0
[pid=28598/tid=28611] ppid=28596 vsize=328356 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28611/stat : 28611 (python) S 28596 28598 28463 0 -1 4202560 23 935 0 0 0 0 0 0 20 0 5 0 538342785 336236544 9285 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 7 0 0 0
[pid=28598/tid=28614] ppid=28596 vsize=328356 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28614/stat : 28614 (python) S 28596 28598 28463 0 -1 4202560 23 935 0 0 0 0 0 0 20 0 5 0 538342788 336236544 9285 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 11.99
Current children cumulated vsize (KiB) 2614492

[startup+12.7007 s]
/proc/loadavg: 7.26 3.21 1.21 6/194 28647
/proc/meminfo: memFree=29597192/32951124 swapFree=67102756/67111528
[pid=28598] ppid=28596 vsize=328356 CPUtime=4.09 cores=1,3,5,7
/proc/28598/stat : 28598 (python) S 28596 28598 28463 0 -1 4202496 19234 935 1 0 395 14 0 0 20 0 5 0 538342353 336236544 9285 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528660939 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/28598/statm: 82089 9285 1881 364 0 17998 0
[pid=28604] ppid=28598 vsize=287828 CPUtime=0 cores=1,3,5,7
/proc/28604/stat : 28604 (python) S 28598 28598 28463 0 -1 4202560 907 0 0 0 0 0 0 0 18 0 1 0 538342779 294735872 7633 33554432000 4194304 5683644 140736723752192 18446744073709551615 246516854287 0 0 16781312 514 4470689049622839560 0 0 17 5 0 0 0
/proc/28604/statm: 71957 7633 378 364 0 7875 0
[pid=28606] ppid=28604 vsize=268424 CPUtime=8.4 cores=1,3,5,7
/proc/28606/stat : 28606 (hybridGM3) R 28604 28606 28606 0 -1 4202496 6658 0 0 0 839 1 0 0 25 0 1 0 538342779 274866176 6395 33554432000 134512640 135204385 4291208656 18446744073709551615 134537272 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/28606/statm: 67106 6395 64 169 0 66936 0
[pid=28607] ppid=28598 vsize=297744 CPUtime=0 cores=1,3,5,7
/proc/28607/stat : 28607 (python) S 28598 28598 28463 0 -1 4202560 913 0 0 0 0 0 0 0 18 0 1 0 538342782 304889856 7765 33554432000 4194304 5683644 140736723752192 18446744073709551615 246516854342 0 0 16781312 514 4397716661910601992 0 0 17 5 0 0 0
/proc/28607/statm: 74436 7765 372 364 0 10351 0
[pid=28609] ppid=28607 vsize=268424 CPUtime=8.38 cores=1,3,5,7
/proc/28609/stat : 28609 (hybridGM3) R 28607 28609 28609 0 -1 4202496 6658 0 0 0 837 1 0 0 25 0 1 0 538342782 274866176 6396 33554432000 134512640 135204385 4287939648 18446744073709551615 134537272 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/28609/statm: 67106 6396 64 169 0 66936 0
[pid=28610] ppid=28598 vsize=308000 CPUtime=0 cores=1,3,5,7
/proc/28610/stat : 28610 (python) S 28598 28598 28463 0 -1 4202560 931 0 0 0 0 0 0 0 15 0 1 0 538342785 315392000 7769 33554432000 4194304 5683644 140736723752192 18446744073709551615 246516854342 0 0 16781312 514 4404898946741141768 0 0 17 7 0 0 0
/proc/28610/statm: 77000 7769 372 364 0 12912 0
[pid=28612] ppid=28610 vsize=268424 CPUtime=8.31 cores=1,3,5,7
/proc/28612/stat : 28612 (hybridGM3) R 28610 28612 28612 0 -1 4202496 6658 0 0 0 830 1 0 0 25 0 1 0 538342785 274866176 6396 33554432000 134512640 135204385 4290548880 18446744073709551615 134537403 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/28612/statm: 67106 6396 64 169 0 66936 0
[pid=28613] ppid=28598 vsize=318864 CPUtime=0 cores=1,3,5,7
/proc/28613/stat : 28613 (python) S 28598 28598 28463 0 -1 4202560 922 0 0 0 0 0 0 0 15 0 1 0 538342788 326516736 7773 33554432000 4194304 5683644 140736723752192 18446744073709551615 246516854342 0 0 16781312 514 15103995907978264839 0 0 17 1 0 0 0
/proc/28613/statm: 79716 7773 372 364 0 15625 0
[pid=28615] ppid=28613 vsize=268428 CPUtime=8.34 cores=1,3,5,7
/proc/28615/stat : 28615 (hybridGM3) R 28613 28615 28615 0 -1 4202496 6656 0 0 0 833 1 0 0 25 0 1 0 538342788 274870272 6394 33554432000 134512640 135204385 4290353312 18446744073709551615 134537272 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/28615/statm: 67107 6394 64 169 0 66937 0
[pid=28598/tid=28605] ppid=28596 vsize=328356 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28605/stat : 28605 (python) S 28596 28598 28463 0 -1 4202560 27 935 0 0 0 0 0 0 23 0 5 0 538342779 336236544 9285 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 7 0 0 0
[pid=28598/tid=28608] ppid=28596 vsize=328356 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28608/stat : 28608 (python) S 28596 28598 28463 0 -1 4202560 25 935 0 0 0 0 0 0 25 0 5 0 538342782 336236544 9285 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 3 0 0 0
[pid=28598/tid=28611] ppid=28596 vsize=328356 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28611/stat : 28611 (python) S 28596 28598 28463 0 -1 4202560 23 935 0 0 0 0 0 0 20 0 5 0 538342785 336236544 9285 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 7 0 0 0
[pid=28598/tid=28614] ppid=28596 vsize=328356 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28614/stat : 28614 (python) S 28596 28598 28463 0 -1 4202560 23 935 0 0 0 0 0 0 20 0 5 0 538342788 336236544 9285 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 37.52
Current children cumulated vsize (KiB) 2614492

[startup+25.5014 s]
/proc/loadavg: 7.45 3.38 1.29 9/206 28659
/proc/meminfo: memFree=29493948/32951124 swapFree=67102756/67111528
[pid=28598] ppid=28596 vsize=328356 CPUtime=4.09 cores=1,3,5,7
/proc/28598/stat : 28598 (python) S 28596 28598 28463 0 -1 4202496 19234 935 1 0 395 14 0 0 20 0 5 0 538342353 336236544 9285 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528660939 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/28598/statm: 82089 9285 1881 364 0 17998 0
[pid=28604] ppid=28598 vsize=287828 CPUtime=0.02 cores=1,3,5,7
/proc/28604/stat : 28604 (python) S 28598 28598 28463 0 -1 4202560 913 0 0 0 1 1 0 0 18 0 1 0 538342779 294735872 7633 33554432000 4194304 5683644 140736723752192 18446744073709551615 246516854287 0 0 16781312 514 16170596674538799368 0 0 17 5 0 0 0
/proc/28604/statm: 71957 7633 378 364 0 7875 0
[pid=28606] ppid=28604 vsize=268424 CPUtime=21.17 cores=1,3,5,7
/proc/28606/stat : 28606 (hybridGM3) R 28604 28606 28606 0 -1 4202496 6660 0 0 0 2116 1 0 0 25 0 1 0 538342779 274866176 6397 33554432000 134512640 135204385 4291208656 18446744073709551615 134547471 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/28606/statm: 67106 6397 64 169 0 66936 0
[pid=28607] ppid=28598 vsize=297744 CPUtime=0.02 cores=1,3,5,7
/proc/28607/stat : 28607 (python) S 28598 28598 28463 0 -1 4202560 921 0 0 0 1 1 0 0 18 0 1 0 538342782 304889856 7765 33554432000 4194304 5683644 140736723752192 18446744073709551615 246516854342 0 0 16781312 514 16170596674538799368 0 0 17 5 0 0 0
/proc/28607/statm: 74436 7765 372 364 0 10351 0
[pid=28609] ppid=28607 vsize=268424 CPUtime=21.15 cores=1,3,5,7
/proc/28609/stat : 28609 (hybridGM3) R 28607 28609 28609 0 -1 4202496 6658 0 0 0 2114 1 0 0 25 0 1 0 538342782 274866176 6396 33554432000 134512640 135204385 4287939648 18446744073709551615 134536777 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/28609/statm: 67106 6396 64 169 0 66936 0
[pid=28610] ppid=28598 vsize=308000 CPUtime=0.01 cores=1,3,5,7
/proc/28610/stat : 28610 (python) S 28598 28598 28463 0 -1 4202560 933 0 0 0 0 1 0 0 15 0 1 0 538342785 315392000 7769 33554432000 4194304 5683644 140736723752192 18446744073709551615 246516854342 0 0 16781312 514 16170596674538799368 0 0 17 7 0 0 0
/proc/28610/statm: 77000 7769 372 364 0 12912 0
[pid=28612] ppid=28610 vsize=268424 CPUtime=21.01 cores=1,3,5,7
/proc/28612/stat : 28612 (hybridGM3) R 28610 28612 28612 0 -1 4202496 6658 0 0 0 2100 1 0 0 25 0 1 0 538342785 274866176 6396 33554432000 134512640 135204385 4290548880 18446744073709551615 134549101 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/28612/statm: 67106 6396 64 169 0 66936 0
[pid=28613] ppid=28598 vsize=318864 CPUtime=0.02 cores=1,3,5,7
/proc/28613/stat : 28613 (python) S 28598 28598 28463 0 -1 4202560 928 0 0 0 1 1 0 0 15 0 1 0 538342788 326516736 7773 33554432000 4194304 5683644 140736723752192 18446744073709551615 246516854342 0 0 16781312 514 16170596674538799367 0 0 17 1 0 0 0
/proc/28613/statm: 79716 7773 372 364 0 15625 0
[pid=28615] ppid=28613 vsize=268428 CPUtime=21.14 cores=1,3,5,7
/proc/28615/stat : 28615 (hybridGM3) R 28613 28615 28615 0 -1 4202496 6659 0 0 0 2113 1 0 0 25 0 1 0 538342788 274870272 6397 33554432000 134512640 135204385 4290353312 18446744073709551615 134537502 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/28615/statm: 67107 6397 64 169 0 66937 0
[pid=28598/tid=28605] ppid=28596 vsize=328356 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28605/stat : 28605 (python) S 28596 28598 28463 0 -1 4202560 27 935 0 0 0 0 0 0 23 0 5 0 538342779 336236544 9285 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 7 0 0 0
[pid=28598/tid=28608] ppid=28596 vsize=328356 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28608/stat : 28608 (python) S 28596 28598 28463 0 -1 4202560 25 935 0 0 0 0 0 0 25 0 5 0 538342782 336236544 9285 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 3 0 0 0
[pid=28598/tid=28611] ppid=28596 vsize=328356 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28611/stat : 28611 (python) S 28596 28598 28463 0 -1 4202560 23 935 0 0 0 0 0 0 20 0 5 0 538342785 336236544 9285 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 7 0 0 0

################
# More data... #
################

/proc/28721/statm: 114743 3328 46 122 0 113643 0
[pid=28719/tid=28737] ppid=28598 vsize=328536 CPUtime=0 cores=1,3,5,7
/proc/28719/task/28737/stat : 28737 (python) S 28598 28598 28463 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 538382285 336420864 7848 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 514 18446604471122969856 0 0 -1 1 0 0 0
[pid=28723] ppid=28598 vsize=339020 CPUtime=0.76 cores=1,3,5,7
/proc/28723/stat : 28723 (python) S 28598 28598 28463 0 -1 4202560 1182 0 0 0 39 37 0 0 16 0 2 0 538372463 347156480 7853 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528660939 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/28723/statm: 84755 7853 400 364 0 20658 0
[pid=28725] ppid=28723 vsize=277424 CPUtime=485.74 cores=1,3,5,7
/proc/28725/stat : 28725 (hybridGM3) T 28723 28725 28725 0 -1 4202496 24306 0 0 0 48568 6 0 0 25 0 1 0 538372464 284082176 9170 33554432000 134512640 135204385 4289738896 18446744073709551615 134545975 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/28725/statm: 69356 9170 85 169 0 69186 0
[pid=28723/tid=28757] ppid=28598 vsize=339020 CPUtime=0 cores=1,3,5,7
/proc/28723/task/28757/stat : 28757 (python) S 28598 28598 28463 0 -1 4202560 49 0 0 0 0 0 0 0 15 0 2 0 538392131 347156480 7853 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 514 18446604454224048320 0 0 -1 1 0 0 0
[pid=28726] ppid=28598 vsize=349368 CPUtime=0.76 cores=1,3,5,7
/proc/28726/stat : 28726 (python) S 28598 28598 28463 0 -1 4202560 1235 0 0 0 42 34 0 0 18 0 2 0 538372467 357752832 7863 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528660939 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/28726/statm: 87342 7863 406 364 0 23242 0
[pid=28728] ppid=28726 vsize=268424 CPUtime=487.79 cores=1,3,5,7
/proc/28728/stat : 28728 (hybridGM3) T 28726 28728 28728 0 -1 4202496 6670 0 0 0 48778 1 0 0 25 0 1 0 538372467 274866176 6402 33554432000 134512640 135204385 4289946064 18446744073709551615 134537292 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/28728/statm: 67106 6402 64 169 0 66936 0
[pid=28726/tid=28758] ppid=28598 vsize=349368 CPUtime=0 cores=1,3,5,7
/proc/28726/task/28758/stat : 28758 (python) S 28598 28598 28463 0 -1 4202560 46 0 0 0 0 0 0 0 18 0 2 0 538392135 357752832 7863 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 514 18446604471161850688 0 0 -1 1 0 0 0
[pid=28729] ppid=28598 vsize=359720 CPUtime=0.44 cores=1,3,5,7
/proc/28729/stat : 28729 (python) S 28598 28598 28463 0 -1 4202560 1211 0 0 0 22 22 0 0 15 0 2 0 538372569 368353280 7867 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528660939 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/28729/statm: 89930 7867 406 364 0 25827 0
[pid=28731] ppid=28729 vsize=268420 CPUtime=294.01 cores=1,3,5,7
/proc/28731/stat : 28731 (hybridGM3) T 28729 28731 28731 0 -1 4202496 6667 0 0 0 29400 1 0 0 25 0 1 0 538372570 274862080 6399 33554432000 134512640 135204385 4287712944 18446744073709551615 134537294 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/28731/statm: 67105 6399 64 169 0 66935 0
[pid=28729/tid=28759] ppid=28598 vsize=359720 CPUtime=0 cores=1,3,5,7
/proc/28729/task/28759/stat : 28759 (python) S 28598 28598 28463 0 -1 4202560 49 0 0 0 0 0 0 0 15 0 2 0 538392238 368353280 7867 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 514 18446604462119996160 0 0 -1 7 0 0 0
[pid=28738] ppid=28598 vsize=370080 CPUtime=0.3 cores=1,3,5,7
/proc/28738/stat : 28738 (python) S 28598 28598 28463 0 -1 4202560 1195 0 0 0 15 15 0 0 16 0 2 0 538382288 378961920 7865 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528660939 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/28738/statm: 92520 7865 400 364 0 28414 0
[pid=28740] ppid=28738 vsize=268420 CPUtime=199.82 cores=1,3,5,7
/proc/28740/stat : 28740 (hybridGM3) T 28738 28740 28740 0 -1 4202496 6670 0 0 0 19981 1 0 0 25 0 1 0 538382289 274862080 6400 33554432000 134512640 135204385 4290777072 18446744073709551615 134536993 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/28740/statm: 67105 6400 64 169 0 66935 0
[pid=28738/tid=28770] ppid=28598 vsize=370080 CPUtime=0 cores=1,3,5,7
/proc/28738/task/28770/stat : 28770 (python) S 28598 28598 28463 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 538402460 378961920 7865 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 514 18446604471116486528 0 0 -1 7 0 0 0
[pid=28766] ppid=28598 vsize=380268 CPUtime=0.29 cores=1,3,5,7
/proc/28766/stat : 28766 (python) S 28598 28598 28463 0 -1 4202560 1195 0 0 0 15 14 0 0 16 0 2 0 538402260 389394432 7888 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528660939 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/28766/statm: 95067 7888 400 364 0 30958 0
[pid=28768] ppid=28766 vsize=268424 CPUtime=196.67 cores=1,3,5,7
/proc/28768/stat : 28768 (hybridGM3) T 28766 28768 28768 0 -1 4202496 6670 0 0 0 19666 1 0 0 25 0 1 0 538402261 274866176 6398 33554432000 134512640 135204385 4287118048 18446744073709551615 134547366 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/28768/statm: 67106 6398 64 169 0 66936 0
[pid=28766/tid=28852] ppid=28598 vsize=380268 CPUtime=0 cores=1,3,5,7
/proc/28766/task/28852/stat : 28852 (python) S 28598 28598 28463 0 -1 4202560 45 0 0 0 0 0 0 0 16 0 2 0 538421930 389394432 7888 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 514 18446604471110855552 0 0 -1 5 0 0 0
[pid=28771] ppid=28598 vsize=390644 CPUtime=0.18 cores=1,3,5,7
/proc/28771/stat : 28771 (python) S 28598 28598 28463 0 -1 4202560 1176 0 0 0 8 10 0 0 18 0 2 0 538402463 400019456 7892 33554432000 4194304 5683644 140736723752192 18446744073709551615 246516854342 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/28771/statm: 97661 7892 400 364 0 33549 0
[pid=28773] ppid=28771 vsize=458972 CPUtime=115.48 cores=1,3,5,7
/proc/28773/stat : 28773 (TNM) R 28771 28773 28773 0 -1 4202496 3598 0 0 0 11548 0 0 0 25 0 1 0 538402464 469987328 3329 33554432000 134512640 135008748 4293985712 18446744073709551615 134527847 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/28773/statm: 114743 3329 46 122 0 113643 0
[pid=28771/tid=28825] ppid=28598 vsize=390644 CPUtime=0 cores=1,3,5,7
/proc/28771/task/28825/stat : 28825 (python) S 28598 28598 28463 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538412380 400019456 7892 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 514 18446604471110853760 0 0 -1 7 0 0 0
[pid=28822] ppid=28598 vsize=400812 CPUtime=0.3 cores=1,3,5,7
/proc/28822/stat : 28822 (python) S 28598 28598 28463 0 -1 4202560 1183 0 0 0 12 18 0 0 18 0 2 0 538411774 410431488 7899 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528660939 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/28822/statm: 100203 7899 400 364 0 36088 0
[pid=28824] ppid=28822 vsize=425004 CPUtime=199.35 cores=1,3,5,7
/proc/28824/stat : 28824 (adaptg2wsat2009) T 28822 28824 28824 0 -1 4202496 3739 0 0 0 19935 0 0 0 25 0 1 0 538411774 435204096 3466 33554432000 134512640 135107411 4290420800 18446744073709551615 134532371 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/28824/statm: 106251 3466 59 146 0 105127 0
[pid=28822/tid=28848] ppid=28598 vsize=400812 CPUtime=0 cores=1,3,5,7
/proc/28822/task/28848/stat : 28848 (python) S 28598 28598 28463 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 538421788 410431488 7899 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 514 18446604465019299136 0 0 -1 3 0 0 0
[pid=28826] ppid=28598 vsize=411336 CPUtime=0.65 cores=1,3,5,7
/proc/28826/stat : 28826 (python) S 28598 28598 28463 0 -1 4202560 1200 0 0 0 37 28 0 0 15 0 2 0 538412384 421208064 7903 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528660939 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/28826/statm: 102834 7903 400 364 0 38716 0
[pid=28828] ppid=28826 vsize=268424 CPUtime=388.94 cores=1,3,5,7
/proc/28828/stat : 28828 (hybridGM3) T 28826 28828 28828 0 -1 4202496 6674 0 0 0 38893 1 0 0 25 0 1 0 538412384 274866176 6402 33554432000 134512640 135204385 4294642432 18446744073709551615 134536737 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/28828/statm: 67106 6402 64 169 0 66936 0
[pid=28826/tid=28872] ppid=28598 vsize=411336 CPUtime=0 cores=1,3,5,7
/proc/28826/task/28872/stat : 28872 (python) S 28598 28598 28463 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 538432655 421208064 7903 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 514 18446604465019300032 0 0 -1 5 0 0 0
[pid=28849] ppid=28598 vsize=421324 CPUtime=0.3 cores=1,3,5,7
/proc/28849/stat : 28849 (python) S 28598 28598 28463 0 -1 4202560 1192 0 0 0 14 16 0 0 16 0 2 0 538421792 431435776 7952 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528660939 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/28849/statm: 105331 7952 400 364 0 41210 0
[pid=28851] ppid=28849 vsize=268424 CPUtime=196.68 cores=1,3,5,7
/proc/28851/stat : 28851 (hybridGM3) T 28849 28851 28851 0 -1 4202496 6670 0 0 0 19667 1 0 0 25 0 1 0 538421793 274866176 6398 33554432000 134512640 135204385 4287518096 18446744073709551615 134536646 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/28851/statm: 67106 6398 64 169 0 66936 0
[pid=28849/tid=28890] ppid=28598 vsize=421324 CPUtime=0 cores=1,3,5,7
/proc/28849/task/28890/stat : 28890 (python) S 28598 28598 28463 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 538441463 431435776 7952 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 514 18446604463709226048 0 0 -1 5 0 0 0
[pid=28873] ppid=28598 vsize=431580 CPUtime=0.16 cores=1,3,5,7
/proc/28873/stat : 28873 (python) S 28598 28598 28463 0 -1 4202560 1217 0 0 0 8 8 0 0 19 0 2 0 538432659 441937920 7964 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528660939 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/28873/statm: 107895 7964 408 364 0 43771 0
[pid=28875] ppid=28873 vsize=1023176 CPUtime=99.94 cores=1,3,5,7
/proc/28875/stat : 28875 (gnovelty+2) T 28873 28875 28875 0 -1 4202496 2198 0 0 0 9994 0 0 0 25 0 1 0 538432660 1047732224 1889 33554432000 134512640 135663704 4292625008 18446744073709551615 134545529 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/28875/statm: 255794 1889 113 282 0 255510 0
[pid=28873/tid=28891] ppid=28598 vsize=431580 CPUtime=0 cores=1,3,5,7
/proc/28873/task/28891/stat : 28891 (python) S 28598 28598 28463 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538442683 441937920 7964 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 514 18446604471127571456 0 0 -1 5 0 0 0
[pid=28892] ppid=28598 vsize=441836 CPUtime=0.16 cores=1,3,5,7
/proc/28892/stat : 28892 (python) S 28598 28598 28463 0 -1 4202560 1215 0 0 0 8 8 0 0 19 0 2 0 538442687 452440064 7980 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528660939 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/28892/statm: 110459 7980 408 364 0 46332 0
[pid=28894] ppid=28892 vsize=458972 CPUtime=99.45 cores=1,3,5,7
/proc/28894/stat : 28894 (TNM) T 28892 28894 28894 0 -1 4202496 3604 0 0 0 9945 0 0 0 25 0 1 0 538442687 469987328 3328 33554432000 134512640 135008748 4290225152 18446744073709551615 134529409 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/28894/statm: 114743 3328 46 122 0 113643 0
[pid=28892/tid=28902] ppid=28598 vsize=441836 CPUtime=0 cores=1,3,5,7
/proc/28892/task/28902/stat : 28902 (python) S 28598 28598 28463 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 538452703 452440064 7980 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 514 18446604462119995264 0 0 -1 7 0 0 0
[pid=28899] ppid=28598 vsize=452524 CPUtime=0.16 cores=1,3,5,7
/proc/28899/stat : 28899 (python) S 28598 28598 28463 0 -1 4202560 1200 0 0 0 10 6 0 0 19 0 2 0 538451292 463384576 7985 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528660939 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/28899/statm: 113131 7985 402 364 0 49001 0
[pid=28901] ppid=28899 vsize=458972 CPUtime=99.66 cores=1,3,5,7
/proc/28901/stat : 28901 (TNM) T 28899 28901 28901 0 -1 4202496 3604 0 0 0 9966 0 0 0 25 0 1 0 538451292 469987328 3328 33554432000 134512640 135008748 4292355360 18446744073709551615 134528967 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/28901/statm: 114743 3328 46 122 0 113643 0
[pid=28899/tid=28915] ppid=28598 vsize=452524 CPUtime=0 cores=1,3,5,7
/proc/28899/task/28915/stat : 28915 (python) S 28598 28598 28463 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538461309 463384576 7985 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 514 18446604471118158784 0 0 -1 1 0 0 0
[pid=28903] ppid=28598 vsize=462952 CPUtime=0.17 cores=1,3,5,7
/proc/28903/stat : 28903 (python) S 28598 28598 28463 0 -1 4202560 1046 0 0 0 9 8 0 0 15 0 1 0 538452707 474062848 7959 33554432000 4194304 5683644 140736723752192 18446744073709551615 246516854342 0 0 16781312 514 16170596674538799365 0 0 17 5 0 0 0
/proc/28903/statm: 115738 7959 373 364 0 51605 0
[pid=28905] ppid=28903 vsize=268428 CPUtime=102.27 cores=1,3,5,7
/proc/28905/stat : 28905 (hybridGM3) R 28903 28905 28905 0 -1 4202496 6670 0 0 0 10227 0 0 0 25 0 1 0 538452708 274870272 6399 33554432000 134512640 135204385 4292540320 18446744073709551615 134547775 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/28905/statm: 67107 6399 64 169 0 66937 0
[pid=28911] ppid=28598 vsize=473388 CPUtime=0.04 cores=1,3,5,7
/proc/28911/stat : 28911 (python) S 28598 28598 28463 0 -1 4202560 1039 0 0 0 2 2 0 0 18 0 1 0 538460707 484749312 7964 33554432000 4194304 5683644 140736723752192 18446744073709551615 246516854342 0 0 16781312 514 16170596674538799368 0 0 17 5 0 0 0
/proc/28911/statm: 118347 7964 373 364 0 54211 0
[pid=28913] ppid=28911 vsize=458972 CPUtime=24.58 cores=1,3,5,7
/proc/28913/stat : 28913 (TNM) R 28911 28913 28913 0 -1 4202496 3605 0 0 0 2458 0 0 0 25 0 1 0 538460708 469987328 3328 33554432000 134512640 135008748 4290780464 18446744073709551615 134529483 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/28913/statm: 114743 3328 46 122 0 113643 0
[pid=28598/tid=28605] ppid=28596 vsize=483148 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28605/stat : 28605 (python) S 28596 28598 28463 0 -1 4202560 31 935 0 0 0 0 0 0 18 0 20 0 538342779 494743552 9615 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 7 0 0 0
[pid=28598/tid=28608] ppid=28596 vsize=483148 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28608/stat : 28608 (python) S 28596 28598 28463 0 -1 4202560 28 935 0 0 0 0 0 0 18 0 20 0 538342782 494743552 9615 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 3 0 0 0
[pid=28598/tid=28611] ppid=28596 vsize=483148 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28611/stat : 28611 (python) S 28596 28598 28463 0 -1 4202560 27 935 0 0 0 0 0 0 18 0 20 0 538342785 494743552 9615 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 7 0 0 0
[pid=28598/tid=28614] ppid=28596 vsize=483148 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28614/stat : 28614 (python) S 28596 28598 28463 0 -1 4202560 23 935 0 0 0 0 0 0 18 0 20 0 538342788 494743552 9615 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 1 0 0 0
[pid=28598/tid=28720] ppid=28596 vsize=483148 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28720/stat : 28720 (python) S 28596 28598 28463 0 -1 4202560 34 935 0 0 0 0 0 0 18 0 20 0 538372271 494743552 9615 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 1 0 0 0
[pid=28598/tid=28724] ppid=28596 vsize=483148 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28724/stat : 28724 (python) S 28596 28598 28463 0 -1 4202560 34 935 0 0 0 0 0 0 18 0 20 0 538372463 494743552 9615 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 7 0 0 0
[pid=28598/tid=28727] ppid=28596 vsize=483148 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28727/stat : 28727 (python) S 28596 28598 28463 0 -1 4202560 38 935 0 0 0 0 0 0 15 0 20 0 538372467 494743552 9615 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 1 0 0 0
[pid=28598/tid=28730] ppid=28596 vsize=483148 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28730/stat : 28730 (python) S 28596 28598 28463 0 -1 4202560 84 935 0 0 0 0 0 0 18 0 20 0 538372570 494743552 9615 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 7 0 0 0
[pid=28598/tid=28739] ppid=28596 vsize=483148 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28739/stat : 28739 (python) S 28596 28598 28463 0 -1 4202560 31 935 0 0 0 0 0 0 20 0 20 0 538382288 494743552 9615 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 7 0 0 0
[pid=28598/tid=28767] ppid=28596 vsize=483148 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28767/stat : 28767 (python) S 28596 28598 28463 0 -1 4202560 28 935 0 0 0 0 0 0 20 0 20 0 538402260 494743552 9615 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 5 0 0 0
[pid=28598/tid=28772] ppid=28596 vsize=483148 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28772/stat : 28772 (python) S 28596 28598 28463 0 -1 4202560 35 935 0 0 0 0 0 0 18 0 20 0 538402464 494743552 9615 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 5 0 0 0
[pid=28598/tid=28823] ppid=28596 vsize=483148 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28823/stat : 28823 (python) S 28596 28598 28463 0 -1 4202560 33 935 0 0 0 0 0 0 18 0 20 0 538411774 494743552 9615 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 5 0 0 0
[pid=28598/tid=28827] ppid=28596 vsize=483148 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28827/stat : 28827 (python) S 28596 28598 28463 0 -1 4202560 38 935 0 0 0 0 0 0 18 0 20 0 538412384 494743552 9615 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 5 0 0 0
[pid=28598/tid=28850] ppid=28596 vsize=483148 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28850/stat : 28850 (python) S 28596 28598 28463 0 -1 4202560 32 935 0 0 0 0 0 0 20 0 20 0 538421793 494743552 9615 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 5 0 0 0
[pid=28598/tid=28874] ppid=28596 vsize=483148 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28874/stat : 28874 (python) S 28596 28598 28463 0 -1 4202560 30 935 0 0 0 0 0 0 20 0 20 0 538432660 494743552 9615 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 5 0 0 0
[pid=28598/tid=28893] ppid=28596 vsize=483148 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28893/stat : 28893 (python) S 28596 28598 28463 0 -1 4202560 29 935 0 0 0 0 0 0 20 0 20 0 538442687 494743552 9615 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 5 0 0 0
[pid=28598/tid=28900] ppid=28596 vsize=483148 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28900/stat : 28900 (python) S 28596 28598 28463 0 -1 4202560 28 935 0 0 0 0 0 0 20 0 20 0 538451292 494743552 9615 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 5 0 0 0
[pid=28598/tid=28904] ppid=28596 vsize=483148 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28904/stat : 28904 (python) S 28596 28598 28463 0 -1 4202560 26 935 0 0 0 0 0 0 20 0 20 0 538452708 494743552 9615 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 5 0 0 0
[pid=28598/tid=28912] ppid=28596 vsize=483148 CPUtime=0 cores=1,3,5,7
/proc/28598/task/28912/stat : 28912 (python) S 28596 28598 28463 0 -1 4202560 28 935 0 0 0 0 0 0 20 0 20 0 538460707 494743552 9615 33554432000 4194304 5683644 140736723752192 18446744073709551615 246528657841 0 0 16781312 2 18446604466799784000 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.08
Current children cumulated vsize (KiB) 14692344

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 28598 and gives
#  childrusage.ru_utime.tv_sec=622
#  childrusage.ru_utime.tv_usec=795320
#  childrusage.ru_stime.tv_sec=3
#  childrusage.ru_stime.tv_usec=6542
# CPU time returned by wait4() is 625.802
# while last known CPU time is 4800.08
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1208.32
CPU time (s): 4800.08
CPU user time (s): 4796.22
CPU system time (s): 3.86
CPU usage (%): 397.253
Max. virtual memory (cumulated for all children) (KiB): 14692348

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 622.795
system time used= 3.00654
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 54700
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 7195
involuntary context switches= 1947

runsolver used 5.30819 second user time and 13.9199 second system time

The end

Launcher Data 

Begin job on node134 at 2011-05-01 18:30:41
IDJOB=3317131
IDBENCH=82596
IDSOLVER=1705
FILE ID=node134/3317131-1304267441
RUNJOBID= node134-1304267281-28482
PBS_JOBID= 13173945
Free space on /tmp= 72496 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v20000-c84000-S357264423-067-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317131-1304267441/watcher-3317131-1304267441 -o /tmp/evaluation-result-3317131-1304267441/solver-3317131-1304267441 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317131-1304267441.cnf -seed 1247388603 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 425b4c5e5535f7d6b9d17ca8feb0ac3e
RANDOM SEED=1247388603

node134.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 7186.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      29685272 kB
Buffers:       1058744 kB
Cached:        1643920 kB
SwapCached:       4712 kB
Active:         449208 kB
Inactive:      2405976 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      29685272 kB
SwapTotal:    67111528 kB
SwapFree:     67102756 kB
Dirty:           33384 kB
Writeback:           0 kB
AnonPages:      150192 kB
Mapped:          21464 kB
Slab:           341664 kB
PageTables:      10892 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  5819816 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72464 MiB
End job on node134 at 2011-05-01 18:50:52