Trace number 3317042

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (MO) 2756.43 712.817

General information on the benchmark

Namerandom/large/
unif-k7-r85-v200-c17000-S862393334-098-UNKNOWN.cnf
MD5SUM860d0404ebc2282cbe692f8828f81b2c
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark454.473
Satisfiable
(Un)Satisfiability was proved
Number of variables200
Number of clauses17000
Sum of the clauses size119000
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 517000

Solver Data 

0.63/19.98	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.63/19.98	c INFO: solving HOME/instance-3317042-1304267279.cnf
0.63/19.98	c INFO: computing task features
0.64/20.09	c INFO: computed variable-clause graph statistics
0.74/20.10	c INFO: computed clause balance statistics
1.34/20.71	c INFO: computed variable balance statistics
1.55/20.94	c INFO: computed small-clause counts
1.55/20.94	c INFO: computed Horn-clause counts
1.55/20.96	c INFO: computed variable graph statistics
1.55/20.96	c INFO: collected features for HOME/instance-3317042-1304267279.cnf in 0.98 s
1.61/21.16	c INFO: running gnovelty+2@0 for 97 with 4798 remaining (b = 0.36)
1.61/21.19	c INFO: running gnovelty+2@0 for 97 with 4798 remaining (b = 0.12)
1.67/21.22	c INFO: running adaptg2wsat2009++@0 for 97 with 4798 remaining (b = 0.08)
1.67/21.25	c INFO: running adaptg2wsat2009++@0 for 97 with 4798 remaining (b = 0.07)
401.43/121.34	c INFO: running adaptg2wsat2009++@0 for 97 with 4698 remaining (b = 0.05)
401.43/121.37	c INFO: running gnovelty+2@0 for 97 with 4698 remaining (b = 0.04)
401.69/121.42	c INFO: running FH@0 for 97 with 4698 remaining (b = 0.03)
401.69/121.46	c INFO: running adaptg2wsat2009++@0 for 97 with 4698 remaining (b = 0.03)
797.64/220.61	c INFO: running adaptg2wsat2009++@0 for 97 with 4599 remaining (b = 0.02)
797.64/220.65	c INFO: running FH@0 for 97 with 4599 remaining (b = 0.02)
801.18/221.51	c INFO: running hybridGM3@0 for 194 with 4598 remaining (b = 0.04)
801.18/221.57	c INFO: running gnovelty+2@0 for 97 with 4598 remaining (b = 0.02)
1192.74/319.80	c INFO: running hybridGM3@0 for 194 with 4500 remaining (b = 0.03)
1192.74/319.86	c INFO: running hybridGM3@0 for 194 with 4500 remaining (b = 0.03)
1200.27/321.79	c INFO: running TNM@0 for 1552 with 4498 remaining (b = 0.20)
1583.10/417.89	c INFO: running adaptg2wsat2009++@0 for 97 with 4402 remaining (b = 0.01)
1974.92/516.21	c INFO: running hybridGM3@100 for 97 with 4303 remaining (b = 0.01)
1974.92/516.26	c INFO: running hybridGM3@99 for 97 with 4303 remaining (b = 0.01)
1982.13/518.09	c INFO: running hybridGM3@98 for 97 with 4301 remaining (b = 0.01)
2353.83/611.45	c INFO: running FH@0 for 97 with 4208 remaining (b = 0.01)
2357.80/612.48	c INFO: running hybridGM3@295 for 97 with 4207 remaining (b = 0.01)
2380.87/618.30	c INFO: running hybridGM3@193 for 97 with 4201 remaining (b = 0.01)
2752.06/711.66	c INFO: running hybridGM3@0 for 194 with 4108 remaining (b = 0.02)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317042-1304267279/watcher-3317042-1304267279 -o /tmp/evaluation-result-3317042-1304267279/solver-3317042-1304267279 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317042-1304267279.cnf -seed 489172721 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.00 0.00 0.00 2/176 12340
/proc/meminfo: memFree=29928364/32950928 swapFree=67086616/67111528
[pid=12340] ppid=12336 vsize=11800 CPUtime=0 cores=1,3,5,7
/proc/12340/stat : 12340 (runsolver) S 12336 12340 12303 0 -1 4202560 46 0 0 0 0 0 0 0 18 0 1 0 538325043 12083200 54 33554432000 4194304 5434984 140736144689632 18446744073709551615 5055495 0 2147483391 4096 24578 18446744071699207358 0 0 17 5 0 0 0
/proc/12340/statm: 2950 54 34 303 0 2644 0

[startup+0.0185401 s]
/proc/loadavg: 0.00 0.00 0.00 2/176 12340
/proc/meminfo: memFree=29928364/32950928 swapFree=67086616/67111528
[pid=12340] ppid=12336 vsize=1964 CPUtime=0 cores=1,3,5,7
/proc/12340/stat : 12340 (python) D 12336 12340 12303 0 -1 4194304 76 0 1 0 0 0 0 0 17 0 1 0 538325043 2011136 29 33554432000 4194304 5683644 140734942225728 18446744073709551615 239471772688 0 0 4096 0 18446744071562234166 0 0 17 7 0 0 0
/proc/12340/statm: 491 29 21 364 0 37 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 1964

[startup+0.100479 s]
/proc/loadavg: 0.00 0.00 0.00 2/176 12340
/proc/meminfo: memFree=29928364/32950928 swapFree=67086616/67111528
[pid=12340] ppid=12336 vsize=15184 CPUtime=0 cores=1,3,5,7
/proc/12340/stat : 12340 (python) S 12336 12340 12303 0 -1 4202496 719 0 6 0 0 0 0 0 17 0 1 0 538325043 15548416 615 33554432000 4194304 5683644 140734942225728 18446744073709551615 239476692992 0 2147483391 16781312 2 18446744071699207358 0 0 17 7 0 0 2
/proc/12340/statm: 3796 615 321 364 0 283 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15184

[startup+0.30055 s]
/proc/loadavg: 0.00 0.00 0.00 2/176 12340
/proc/meminfo: memFree=29928364/32950928 swapFree=67086616/67111528
[pid=12340] ppid=12336 vsize=16816 CPUtime=0.02 cores=1,3,5,7
/proc/12340/stat : 12340 (python) D 12336 12340 12303 0 -1 4202496 1223 0 6 0 2 0 0 0 16 0 1 0 538325043 17219584 1086 33554432000 4194304 5683644 140734942225728 18446744073709551615 239476690789 0 0 16781312 2 0 0 0 17 7 0 0 4
/proc/12340/statm: 4204 1086 417 364 0 691 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 16816

[startup+0.700417 s]
/proc/loadavg: 0.00 0.00 0.00 2/176 12340
/proc/meminfo: memFree=29928364/32950928 swapFree=67086616/67111528
[pid=12340] ppid=12336 vsize=21108 CPUtime=0.04 cores=1,3,5,7
/proc/12340/stat : 12340 (python) S 12336 12340 12303 0 -1 4202496 1308 0 8 0 3 1 0 0 16 0 1 0 538325043 21614592 1166 33554432000 4194304 5683644 140734942225728 18446744073709551615 239476690789 0 2147483391 16781312 2 18446744071699207358 0 0 17 7 0 0 7
/proc/12340/statm: 5277 1166 428 364 0 724 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 21108

[startup+1.50025 s]
/proc/loadavg: 0.00 0.00 0.00 2/178 12342
/proc/meminfo: memFree=29919116/32950928 swapFree=67086616/67111528
[pid=12340] ppid=12336 vsize=38908 CPUtime=0.06 cores=1,3,5,7
/proc/12340/stat : 12340 (python) D 12336 12340 12303 0 -1 4202496 1616 0 14 0 4 2 0 0 16 0 1 0 538325043 39841792 1458 33554432000 4194304 5683644 140734942225728 18446744073709551615 239476690789 0 0 16781312 2 0 0 0 17 7 0 0 11
/proc/12340/statm: 9727 1458 455 364 0 1027 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 38908

[startup+3.10098 s]
/proc/loadavg: 0.00 0.00 0.00 2/178 12342
/proc/meminfo: memFree=29913076/32950928 swapFree=67086616/67111528
[pid=12340] ppid=12336 vsize=91832 CPUtime=0.11 cores=1,3,5,7
/proc/12340/stat : 12340 (python) S 12336 12340 12303 0 -1 4202496 2657 0 22 0 7 4 0 0 16 0 1 0 538325043 94035968 2394 33554432000 4194304 5683644 140734942225728 18446744073709551615 239476692992 0 2147483391 16781312 2 18446744071699207358 0 0 17 7 0 0 52
/proc/12340/statm: 22958 2394 755 364 0 1623 0
Current children cumulated CPU time (s) 0.11
Current children cumulated vsize (KiB) 91832
heavy processes:

[startup+6.30034 s]
/proc/loadavg: 0.00 0.00 0.00 2/178 12342
/proc/meminfo: memFree=29899544/32950928 swapFree=67086616/67111528
[pid=12340] ppid=12336 vsize=113080 CPUtime=0.18 cores=1,3,5,7
/proc/12340/stat : 12340 (python) S 12336 12340 12303 0 -1 4202496 3488 0 34 0 9 9 0 0 15 0 1 0 538325043 115793920 3169 33554432000 4194304 5683644 140734942225728 18446744073709551615 239476690789 0 2147483391 16781312 2 18446744071699207358 0 0 17 7 0 0 120
/proc/12340/statm: 28270 3169 884 364 0 2292 0
Current children cumulated CPU time (s) 0.18
Current children cumulated vsize (KiB) 113080
heavy processes:

[startup+12.7011 s]
/proc/loadavg: 0.07 0.02 0.00 2/178 12350
/proc/meminfo: memFree=29879404/32950928 swapFree=67086616/67111528
[pid=12340] ppid=12336 vsize=194556 CPUtime=0.44 cores=1,3,5,7
/proc/12340/stat : 12340 (python) D 12336 12340 12303 0 -1 4202496 6842 930 74 8 20 24 0 0 15 0 1 0 538325043 199225344 4965 33554432000 4194304 5683644 140734942225728 18446744073709551615 239476694784 0 0 16781312 2 18446744071562234166 0 0 17 7 0 0 258
/proc/12340/statm: 48639 4965 1431 364 0 3758 0
Current children cumulated CPU time (s) 0.44
Current children cumulated vsize (KiB) 194556
heavy processes:

[startup+25.5006 s]
/proc/loadavg: 0.22 0.05 0.01 10/202 12374
/proc/meminfo: memFree=29714128/32950928 swapFree=67086616/67111528
[pid=12340] ppid=12336 vsize=320512 CPUtime=1.72 cores=1,3,5,7
/proc/12340/stat : 12340 (python) S 12336 12340 12303 0 -1 4202496 14064 930 110 8 142 30 0 0 22 0 5 0 538325043 328204288 7326 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488521675 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 417
/proc/12340/statm: 80128 7326 1883 364 0 16037 0
[pid=12351] ppid=12340 vsize=279232 CPUtime=0 cores=1,3,5,7
/proc/12351/stat : 12351 (python) S 12340 12340 12303 0 -1 4202560 881 0 0 0 0 0 0 0 15 0 1 0 538327160 285933568 5737 33554432000 4194304 5683644 140734942225728 18446744073709551615 239476715023 0 0 16781312 514 4473575542523658504 0 0 17 5 0 0 0
/proc/12351/statm: 69808 5737 377 364 0 5726 0
[pid=12353] ppid=12351 vsize=1018616 CPUtime=4.31 cores=1,3,5,7
/proc/12353/stat : 12353 (gnovelty+2) R 12351 12353 12353 0 -1 4202496 764 0 0 0 431 0 0 0 25 0 1 0 538327161 1043062784 500 33554432000 134512640 135663704 4287008368 18446744073709551615 134524919 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/12353/statm: 254654 500 113 282 0 254370 0
[pid=12354] ppid=12340 vsize=289624 CPUtime=0 cores=1,3,5,7
/proc/12354/stat : 12354 (python) S 12340 12340 12303 0 -1 4202560 897 0 0 0 0 0 0 0 18 0 1 0 538327163 296574976 5785 33554432000 4194304 5683644 140734942225728 18446744073709551615 239476715078 0 0 16781312 514 4473583239105052936 0 0 17 7 0 0 0
/proc/12354/statm: 72406 5785 371 364 0 8321 0
[pid=12356] ppid=12354 vsize=1018616 CPUtime=4.3 cores=1,3,5,7
/proc/12356/stat : 12356 (gnovelty+2) R 12354 12356 12356 0 -1 4202496 763 0 0 0 430 0 0 0 25 0 1 0 538327163 1043062784 500 33554432000 134512640 135663704 4287517136 18446744073709551615 134524923 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/12356/statm: 254654 500 113 282 0 254370 0
[pid=12357] ppid=12340 vsize=300012 CPUtime=0 cores=1,3,5,7
/proc/12357/stat : 12357 (python) S 12340 12340 12303 0 -1 4202560 885 0 0 0 0 0 0 0 18 0 1 0 538327166 307212288 5806 33554432000 4194304 5683644 140734942225728 18446744073709551615 239476715078 0 0 16781312 514 5636301942071197960 0 0 17 5 0 0 0
/proc/12357/statm: 75003 5806 371 364 0 10915 0
[pid=12359] ppid=12357 vsize=418636 CPUtime=4.23 cores=1,3,5,7
/proc/12359/stat : 12359 (adaptg2wsat2009) R 12357 12359 12359 0 -1 4202496 1219 0 0 0 423 0 0 0 25 0 1 0 538327166 428683264 954 33554432000 134512640 135107411 4287577824 18446744073709551615 134532121 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/12359/statm: 104659 954 59 146 0 103535 0
[pid=12360] ppid=12340 vsize=310268 CPUtime=0 cores=1,3,5,7
/proc/12360/stat : 12360 (python) S 12340 12340 12303 0 -1 4202560 914 0 0 0 0 0 0 0 18 0 1 0 538327169 317714432 5809 33554432000 4194304 5683644 140734942225728 18446744073709551615 239476715078 0 0 16781312 514 5636294245489803528 0 0 17 5 0 0 0
/proc/12360/statm: 77567 5809 370 364 0 13476 0
[pid=12362] ppid=12360 vsize=418636 CPUtime=4.21 cores=1,3,5,7
/proc/12362/stat : 12362 (adaptg2wsat2009) R 12360 12362 12362 0 -1 4202496 1222 0 0 0 421 0 0 0 25 0 1 0 538327169 428683264 957 33554432000 134512640 135107411 4293537984 18446744073709551615 134531942 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/12362/statm: 104659 957 59 146 0 103535 0
[pid=12340/tid=12352] ppid=12336 vsize=320512 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12352/stat : 12352 (python) S 12336 12340 12303 0 -1 4202560 23 930 0 8 0 0 0 0 17 0 5 0 538327160 328204288 7326 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 1 0 0 0
[pid=12340/tid=12355] ppid=12336 vsize=320512 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12355/stat : 12355 (python) S 12336 12340 12303 0 -1 4202560 25 930 0 8 0 0 0 0 19 0 5 0 538327163 328204288 7326 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 5 0 0 0
[pid=12340/tid=12358] ppid=12336 vsize=320512 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12358/stat : 12358 (python) S 12336 12340 12303 0 -1 4202560 21 930 0 8 0 0 0 0 20 0 5 0 538327166 328204288 7326 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 5 0 0 0
[pid=12340/tid=12361] ppid=12336 vsize=320512 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12361/stat : 12361 (python) S 12336 12340 12303 0 -1 4202560 29 930 0 8 0 0 0 0 22 0 5 0 538327169 328204288 7326 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 18.77
Current children cumulated vsize (KiB) 4374152
heavy processes:

[startup+51.1007 s]
/proc/loadavg: 2.60 0.62 0.20 9/202 12419
/proc/meminfo: memFree=29790152/32950928 swapFree=67086616/67111528
[pid=12340] ppid=12336 vsize=320512 CPUtime=1.72 cores=1,3,5,7
/proc/12340/stat : 12340 (python) S 12336 12340 12303 0 -1 4202496 14064 930 110 8 142 30 0 0 22 0 5 0 538325043 328204288 7326 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488521675 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 417
/proc/12340/statm: 80128 7326 1883 364 0 16037 0
[pid=12351] ppid=12340 vsize=279232 CPUtime=0.03 cores=1,3,5,7
/proc/12351/stat : 12351 (python) S 12340 12340 12303 0 -1 4202560 891 0 0 0 1 2 0 0 15 0 1 0 538327160 285933568 5737 33554432000 4194304 5683644 140734942225728 18446744073709551615 239476715023 0 0 16781312 514 6495123048528576776 0 0 17 5 0 0 0
/proc/12351/statm: 69808 5737 377 364 0 5726 0
[pid=12353] ppid=12351 vsize=1018616 CPUtime=29.9 cores=1,3,5,7
/proc/12353/stat : 12353 (gnovelty+2) R 12351 12353 12353 0 -1 4202496 764 0 0 0 2990 0 0 0 25 0 1 0 538327161 1043062784 500 33554432000 134512640 135663704 4287008368 18446744073709551615 134524984 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/12353/statm: 254654 500 113 282 0 254370 0
[pid=12354] ppid=12340 vsize=289624 CPUtime=0.03 cores=1,3,5,7
/proc/12354/stat : 12354 (python) S 12340 12340 12303 0 -1 4202560 905 0 0 0 2 1 0 0 18 0 1 0 538327163 296574976 5785 33554432000 4194304 5683644 140734942225728 18446744073709551615 239476715078 0 0 16781312 514 6495123048528576776 0 0 17 7 0 0 0
/proc/12354/statm: 72406 5785 371 364 0 8321 0
[pid=12356] ppid=12354 vsize=1018616 CPUtime=29.9 cores=1,3,5,7
/proc/12356/stat : 12356 (gnovelty+2) R 12354 12356 12356 0 -1 4202496 763 0 0 0 2990 0 0 0 25 0 1 0 538327163 1043062784 500 33554432000 134512640 135663704 4287517136 18446744073709551615 134525224 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/12356/statm: 254654 500 113 282 0 254370 0
[pid=12357] ppid=12340 vsize=300012 CPUtime=0.02 cores=1,3,5,7
/proc/12357/stat : 12357 (python) S 12340 12340 12303 0 -1 4202560 899 0 0 0 1 1 0 0 18 0 1 0 538327166 307212288 5806 33554432000 4194304 5683644 140734942225728 18446744073709551615 239476715078 0 0 16781312 514 6495123048528576776 0 0 17 5 0 0 0
/proc/12357/statm: 75003 5806 371 364 0 10915 0
[pid=12359] ppid=12357 vsize=418636 CPUtime=29.66 cores=1,3,5,7
/proc/12359/stat : 12359 (adaptg2wsat2009) R 12357 12359 12359 0 -1 4202496 1219 0 0 0 2964 2 0 0 25 0 1 0 538327166 428683264 954 33554432000 134512640 135107411 4287577824 18446744073709551615 134532128 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/12359/statm: 104659 954 59 146 0 103535 0
[pid=12360] ppid=12340 vsize=310268 CPUtime=0.03 cores=1,3,5,7
/proc/12360/stat : 12360 (python) S 12340 12340 12303 0 -1 4202560 926 0 0 0 2 1 0 0 18 0 1 0 538327169 317714432 5810 33554432000 4194304 5683644 140734942225728 18446744073709551615 239476715078 0 0 16781312 514 6495123048528576776 0 0 17 5 0 0 0
/proc/12360/statm: 77567 5810 371 364 0 13476 0
[pid=12362] ppid=12360 vsize=418636 CPUtime=29.72 cores=1,3,5,7
/proc/12362/stat : 12362 (adaptg2wsat2009) R 12360 12362 12362 0 -1 4202496 1222 0 0 0 2972 0 0 0 25 0 1 0 538327169 428683264 957 33554432000 134512640 135107411 4293537984 18446744073709551615 134531736 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/12362/statm: 104659 957 59 146 0 103535 0
[pid=12340/tid=12352] ppid=12336 vsize=320512 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12352/stat : 12352 (python) S 12336 12340 12303 0 -1 4202560 23 930 0 8 0 0 0 0 17 0 5 0 538327160 328204288 7326 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 1 0 0 0
[pid=12340/tid=12355] ppid=12336 vsize=320512 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12355/stat : 12355 (python) S 12336 12340 12303 0 -1 4202560 25 930 0 8 0 0 0 0 19 0 5 0 538327163 328204288 7326 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 5 0 0 0
[pid=12340/tid=12358] ppid=12336 vsize=320512 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12358/stat : 12358 (python) S 12336 12340 12303 0 -1 4202560 21 930 0 8 0 0 0 0 20 0 5 0 538327166 328204288 7326 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 5 0 0 0
[pid=12340/tid=12361] ppid=12336 vsize=320512 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12361/stat : 12361 (python) S 12336 12340 12303 0 -1 4202560 29 930 0 8 0 0 0 0 22 0 5 0 538327169 328204288 7326 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 121.01
Current children cumulated vsize (KiB) 4374152

[startup+102.307 s]
/proc/loadavg: 5.84 1.86 0.65 9/202 12420
/proc/meminfo: memFree=29789980/32950928 swapFree=67086616/67111528
[pid=12340] ppid=12336 vsize=320512 CPUtime=1.72 cores=1,3,5,7
/proc/12340/stat : 12340 (python) S 12336 12340 12303 0 -1 4202496 14064 930 110 8 142 30 0 0 22 0 5 0 538325043 328204288 7326 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488521675 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 417
/proc/12340/statm: 80128 7326 1883 364 0 16037 0
[pid=12351] ppid=12340 vsize=279232 CPUtime=0.09 cores=1,3,5,7
/proc/12351/stat : 12351 (python) S 12340 12340 12303 0 -1 4202560 907 0 0 0 4 5 0 0 15 0 1 0 538327160 285933568 5737 33554432000 4194304 5683644 140734942225728 18446744073709551615 239476715023 0 0 16781312 514 6495123048528576776 0 0 17 5 0 0 0
/proc/12351/statm: 69808 5737 377 364 0 5726 0
[pid=12353] ppid=12351 vsize=1018616 CPUtime=81.1 cores=1,3,5,7
/proc/12353/stat : 12353 (gnovelty+2) R 12351 12353 12353 0 -1 4202496 764 0 0 0 8110 0 0 0 25 0 1 0 538327161 1043062784 500 33554432000 134512640 135663704 4287008368 18446744073709551615 134544196 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/12353/statm: 254654 500 113 282 0 254370 0
[pid=12354] ppid=12340 vsize=289624 CPUtime=0.09 cores=1,3,5,7
/proc/12354/stat : 12354 (python) S 12340 12340 12303 0 -1 4202560 921 0 0 0 5 4 0 0 18 0 1 0 538327163 296574976 5785 33554432000 4194304 5683644 140734942225728 18446744073709551615 239476715078 0 0 16781312 514 6495123048528576776 0 0 17 7 0 0 0

################
# More data... #
################

[pid=12357/tid=12424] ppid=12340 vsize=300012 CPUtime=0 cores=1,3,5,7
/proc/12357/task/12424/stat : 12424 (python) S 12340 12340 12303 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 538337177 307212288 5836 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 514 18446604454224997056 0 0 -1 3 0 0 0
[pid=12360] ppid=12340 vsize=310268 CPUtime=0.12 cores=1,3,5,7
/proc/12360/stat : 12360 (python) S 12340 12340 12303 0 -1 4202560 1103 0 0 0 8 4 0 0 19 0 2 0 538327169 317714432 5840 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488521675 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/12360/statm: 77567 5840 400 364 0 13476 0
[pid=12362] ppid=12360 vsize=418636 CPUtime=99.77 cores=1,3,5,7
/proc/12362/stat : 12362 (adaptg2wsat2009) T 12360 12362 12362 0 -1 4202496 1222 0 0 0 9977 0 0 0 25 0 1 0 538327169 428683264 957 33554432000 134512640 135107411 4293537984 18446744073709551615 134532364 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/12362/statm: 104659 957 59 146 0 103535 0
[pid=12360/tid=12431] ppid=12340 vsize=310268 CPUtime=0 cores=1,3,5,7
/proc/12360/task/12431/stat : 12431 (python) S 12340 12340 12303 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 538337182 317714432 5840 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 514 18446604454224995264 0 0 -1 5 0 0 0
[pid=12425] ppid=12340 vsize=320524 CPUtime=0.13 cores=1,3,5,7
/proc/12425/stat : 12425 (python) S 12340 12340 12303 0 -1 4202560 1115 0 0 0 6 7 0 0 19 0 2 0 538337178 328216576 5844 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488521675 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/12425/statm: 80131 5844 400 364 0 16037 0
[pid=12427] ppid=12425 vsize=418636 CPUtime=100.12 cores=1,3,5,7
/proc/12427/stat : 12427 (adaptg2wsat2009) T 12425 12427 12427 0 -1 4202496 1224 0 0 0 10012 0 0 0 25 0 1 0 538337178 428683264 957 33554432000 134512640 135107411 4294300032 18446744073709551615 134531928 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/12427/statm: 104659 957 59 146 0 103535 0
[pid=12425/tid=12465] ppid=12340 vsize=320524 CPUtime=0 cores=1,3,5,7
/proc/12425/task/12465/stat : 12465 (python) S 12340 12340 12303 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538347190 328216576 5844 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 514 18446604463076202304 0 0 -1 3 0 0 0
[pid=12428] ppid=12340 vsize=330932 CPUtime=0.13 cores=1,3,5,7
/proc/12428/stat : 12428 (python) S 12340 12340 12303 0 -1 4202560 1118 0 0 0 6 7 0 0 18 0 2 0 538337181 338874368 5886 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488521675 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/12428/statm: 82733 5886 400 364 0 18636 0
[pid=12430] ppid=12428 vsize=1018616 CPUtime=100.15 cores=1,3,5,7
/proc/12430/stat : 12430 (gnovelty+2) T 12428 12430 12430 0 -1 4202496 766 0 0 0 10015 0 0 0 25 0 1 0 538337181 1043062784 500 33554432000 134512640 135663704 4289700480 18446744073709551615 134524918 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/12430/statm: 254654 500 113 282 0 254370 0
[pid=12428/tid=12469] ppid=12340 vsize=330932 CPUtime=0 cores=1,3,5,7
/proc/12428/task/12469/stat : 12469 (python) S 12340 12340 12303 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538347196 338874368 5886 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 514 18446604454685006592 0 0 -1 1 0 0 0
[pid=12432] ppid=12340 vsize=341188 CPUtime=0.13 cores=1,3,5,7
/proc/12432/stat : 12432 (python) S 12340 12340 12303 0 -1 4202560 1170 0 0 0 7 6 0 0 19 0 2 0 538337185 349376512 5917 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488521675 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/12432/statm: 85297 5917 406 364 0 21197 0
[pid=12434] ppid=12432 vsize=248648 CPUtime=98.88 cores=1,3,5,7
/proc/12434/stat : 12434 (FH) T 12432 12434 12434 0 -1 4202496 1123 0 0 0 9888 0 0 0 25 0 1 0 538337186 254615552 856 33554432000 134512640 134936988 4289184864 18446744073709551615 134545104 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/12434/statm: 62162 856 41 104 0 61666 0
[pid=12432/tid=12457] ppid=12340 vsize=341188 CPUtime=0 cores=1,3,5,7
/proc/12432/task/12457/stat : 12457 (python) S 12340 12340 12303 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 538347100 349376512 5917 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 514 18446604471133272768 0 0 -1 7 0 0 0
[pid=12435] ppid=12340 vsize=351444 CPUtime=0.13 cores=1,3,5,7
/proc/12435/stat : 12435 (python) S 12340 12340 12303 0 -1 4202560 1158 0 0 0 6 7 0 0 19 0 2 0 538337189 359878656 5921 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488521675 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/12435/statm: 87861 5921 406 364 0 23758 0
[pid=12437] ppid=12435 vsize=418636 CPUtime=98.01 cores=1,3,5,7
/proc/12437/stat : 12437 (adaptg2wsat2009) T 12435 12437 12437 0 -1 4202496 1223 0 0 0 9801 0 0 0 25 0 1 0 538337189 428683264 955 33554432000 134512640 135107411 4294086032 18446744073709551615 134532277 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/12437/statm: 104659 955 59 146 0 103535 0
[pid=12435/tid=12458] ppid=12340 vsize=351444 CPUtime=0 cores=1,3,5,7
/proc/12435/task/12458/stat : 12458 (python) S 12340 12340 12303 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 538347103 359878656 5921 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 514 18446604471133271872 0 0 -1 5 0 0 0
[pid=12459] ppid=12340 vsize=362248 CPUtime=0.15 cores=1,3,5,7
/proc/12459/stat : 12459 (python) S 12340 12340 12303 0 -1 4202560 1151 0 0 0 8 7 0 0 19 0 2 0 538347105 370941952 6035 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488521675 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/12459/statm: 90562 6035 400 364 0 26456 0
[pid=12461] ppid=12459 vsize=418636 CPUtime=99.12 cores=1,3,5,7
/proc/12461/stat : 12461 (adaptg2wsat2009) T 12459 12461 12461 0 -1 4202496 1227 0 0 0 9912 0 0 0 25 0 1 0 538347105 428683264 955 33554432000 134512640 135107411 4289859056 18446744073709551615 134531537 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/12461/statm: 104659 955 59 146 0 103535 0
[pid=12459/tid=12491] ppid=12340 vsize=362248 CPUtime=0 cores=1,3,5,7
/proc/12459/task/12491/stat : 12491 (python) S 12340 12340 12303 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 538357020 370941952 6035 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 514 18446604471159272448 0 0 -1 7 0 0 0
[pid=12462] ppid=12340 vsize=372504 CPUtime=0.15 cores=1,3,5,7
/proc/12462/stat : 12462 (python) S 12340 12340 12303 0 -1 4202560 1158 0 0 0 9 6 0 0 19 0 2 0 538347109 381444096 6039 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488521675 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/12462/statm: 93126 6039 400 364 0 29017 0
[pid=12464] ppid=12462 vsize=248652 CPUtime=99.14 cores=1,3,5,7
/proc/12464/stat : 12464 (FH) T 12462 12464 12464 0 -1 4202496 1126 0 0 0 9914 0 0 0 25 0 1 0 538347109 254619648 856 33554432000 134512640 134936988 4293179792 18446744073709551615 134552375 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/12464/statm: 62163 856 41 104 0 61667 0
[pid=12462/tid=12494] ppid=12340 vsize=372504 CPUtime=0 cores=1,3,5,7
/proc/12462/task/12494/stat : 12494 (python) S 12340 12340 12303 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538357025 381444096 6039 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 514 18446604454685005696 0 0 -1 5 0 0 0
[pid=12466] ppid=12340 vsize=382760 CPUtime=0.61 cores=1,3,5,7
/proc/12466/stat : 12466 (python) S 12340 12340 12303 0 -1 4202560 1192 0 0 0 34 27 0 0 18 0 2 0 538347194 391946240 6043 33554432000 4194304 5683644 140734942225728 18446744073709551615 239476715078 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/12466/statm: 95690 6043 399 364 0 31578 0
[pid=12468] ppid=12466 vsize=268424 CPUtime=391.51 cores=1,3,5,7
/proc/12468/stat : 12468 (hybridGM3) R 12466 12468 12468 0 -1 4202496 5167 0 0 0 39151 0 0 0 25 0 1 0 538347194 274866176 4896 33554432000 134512640 135204385 4291244736 18446744073709551615 134660160 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/12468/statm: 67106 4896 63 169 0 66936 0
[pid=12466/tid=12518] ppid=12340 vsize=382760 CPUtime=0 cores=1,3,5,7
/proc/12466/task/12518/stat : 12518 (python) S 12340 12340 12303 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 538366827 391946240 6043 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 514 18446604465347174592 0 0 -1 5 0 0 0
[pid=12470] ppid=12340 vsize=393016 CPUtime=0.15 cores=1,3,5,7
/proc/12470/stat : 12470 (python) S 12340 12340 12303 0 -1 4202560 1165 0 0 0 5 10 0 0 19 0 2 0 538347201 402448384 6047 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488521675 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/12470/statm: 98254 6047 400 364 0 34139 0
[pid=12472] ppid=12470 vsize=1018616 CPUtime=98.42 cores=1,3,5,7
/proc/12472/stat : 12472 (gnovelty+2) T 12470 12472 12472 0 -1 4202496 777 0 0 0 9842 0 0 0 25 0 1 0 538347201 1043062784 501 33554432000 134512640 135663704 4290443152 18446744073709551615 134524739 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/12472/statm: 254654 501 113 282 0 254370 0
[pid=12470/tid=12499] ppid=12340 vsize=393016 CPUtime=0 cores=1,3,5,7
/proc/12470/task/12499/stat : 12499 (python) S 12340 12340 12303 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 538357218 402448384 6047 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 514 18446604471151169984 0 0 -1 3 0 0 0
[pid=12492] ppid=12340 vsize=403272 CPUtime=0.48 cores=1,3,5,7
/proc/12492/stat : 12492 (python) S 12340 12340 12303 0 -1 4202560 1191 0 0 0 25 23 0 0 18 0 2 0 538357024 412950528 6051 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488521675 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/12492/statm: 100818 6051 400 364 0 36700 0
[pid=12495] ppid=12492 vsize=268424 CPUtime=291.48 cores=1,3,5,7
/proc/12495/stat : 12495 (hybridGM3) T 12492 12495 12495 0 -1 4202496 5166 0 0 0 29148 0 0 0 25 0 1 0 538357025 274866176 4895 33554432000 134512640 135204385 4289404080 18446744073709551615 134537785 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/12495/statm: 67106 4895 63 169 0 66936 0
[pid=12492/tid=12525] ppid=12340 vsize=403272 CPUtime=0 cores=1,3,5,7
/proc/12492/task/12525/stat : 12525 (python) S 12340 12340 12303 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 538376659 412950528 6051 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 514 18446604471133270080 0 0 -1 3 0 0 0
[pid=12496] ppid=12340 vsize=413528 CPUtime=0.63 cores=1,3,5,7
/proc/12496/stat : 12496 (python) S 12340 12340 12303 0 -1 4202560 1205 0 0 0 35 28 0 0 18 0 2 0 538357030 423452672 6055 33554432000 4194304 5683644 140734942225728 18446744073709551615 239476715078 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/12496/statm: 103382 6055 400 364 0 39261 0
[pid=12498] ppid=12496 vsize=268424 CPUtime=387.65 cores=1,3,5,7
/proc/12498/stat : 12498 (hybridGM3) R 12496 12498 12498 0 -1 4202496 5170 0 0 0 38765 0 0 0 25 0 1 0 538357030 274866176 4896 33554432000 134512640 135204385 4288892768 18446744073709551615 134538016 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/12498/statm: 67106 4896 63 169 0 66936 0
[pid=12496/tid=12526] ppid=12340 vsize=413528 CPUtime=0 cores=1,3,5,7
/proc/12496/task/12526/stat : 12526 (python) S 12340 12340 12303 0 -1 4202560 46 0 0 0 0 0 0 0 18 0 2 0 538376665 423452672 6055 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 514 18446604454223780800 0 0 -1 5 0 0 0
[pid=12500] ppid=12340 vsize=423784 CPUtime=0.63 cores=1,3,5,7
/proc/12500/stat : 12500 (python) S 12340 12340 12303 0 -1 4202560 1076 0 0 0 33 30 0 0 18 0 1 0 538357222 433954816 6037 33554432000 4194304 5683644 140734942225728 18446744073709551615 239476715078 0 0 16781312 514 6495123048528576774 0 0 17 5 0 0 0
/proc/12500/statm: 105946 6037 379 364 0 41822 0
[pid=12502] ppid=12500 vsize=457124 CPUtime=385.07 cores=1,3,5,7
/proc/12502/stat : 12502 (TNM) R 12500 12502 12502 0 -1 4202496 1114 0 0 0 38507 0 0 0 25 0 1 0 538357223 468094976 837 33554432000 134512640 135008748 4292181696 18446744073709551615 134528448 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/12502/statm: 114281 837 46 122 0 113181 0
[pid=12519] ppid=12340 vsize=434180 CPUtime=0.16 cores=1,3,5,7
/proc/12519/stat : 12519 (python) S 12340 12340 12303 0 -1 4202560 1181 0 0 0 8 8 0 0 19 0 2 0 538366832 444600320 6106 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488521675 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/12519/statm: 108545 6106 408 364 0 44418 0
[pid=12521] ppid=12519 vsize=418636 CPUtime=99.66 cores=1,3,5,7
/proc/12521/stat : 12521 (adaptg2wsat2009) T 12519 12521 12521 0 -1 4202496 1235 0 0 0 9966 0 0 0 25 0 1 0 538366832 428683264 955 33554432000 134512640 135107411 4293299504 18446744073709551615 134531942 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/12521/statm: 104659 955 59 146 0 103535 0
[pid=12519/tid=12528] ppid=12340 vsize=434180 CPUtime=0 cores=1,3,5,7
/proc/12519/task/12528/stat : 12528 (python) S 12340 12340 12303 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 538376848 444600320 6106 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 514 18446604464775841856 0 0 -1 5 0 0 0
[pid=12536] ppid=12340 vsize=444596 CPUtime=0.15 cores=1,3,5,7
/proc/12536/stat : 12536 (python) S 12340 12340 12303 0 -1 4202560 1174 0 0 0 7 8 0 0 19 0 2 0 538386189 455266304 6144 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488521675 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/12536/statm: 111149 6144 402 364 0 47019 0
[pid=12538] ppid=12536 vsize=248652 CPUtime=100.13 cores=1,3,5,7
/proc/12538/stat : 12538 (FH) T 12536 12538 12538 0 -1 4202496 1134 0 0 0 10013 0 0 0 25 0 1 0 538386189 254619648 857 33554432000 134512640 134936988 4291439840 18446744073709551615 134545269 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/12538/statm: 62163 857 41 104 0 61667 0
[pid=12536/tid=12640] ppid=12340 vsize=444596 CPUtime=0 cores=1,3,5,7
/proc/12536/task/12640/stat : 12640 (python) S 12340 12340 12303 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 538396205 455266304 6144 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 514 18446604456075932096 0 0 -1 5 0 0 0
[pid=12641] ppid=12340 vsize=454852 CPUtime=0 cores=1,3,5,7
/proc/12641/stat : 12641 (python) S 12340 12340 12303 0 -1 4202560 852 0 0 0 0 0 0 0 19 0 1 0 538396210 465768448 6107 33554432000 4194304 5683644 140734942225728 18446744073709551615 239476715078 0 0 16781312 514 6758443168414302207 0 0 17 5 0 0 0
/proc/12641/statm: 113713 6107 350 364 0 49580 0
[pid=12643] ppid=12641 vsize=268424 CPUtime=1.03 cores=1,3,5,7
/proc/12643/stat : 12643 (hybridGM3) R 12641 12643 12643 0 -1 4202496 5174 0 0 0 103 0 0 0 24 0 1 0 538396210 274866176 4897 33554432000 134512640 135204385 4288885936 18446744073709551615 134538041 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/12643/statm: 67106 4897 63 169 0 66936 0
[pid=12340/tid=12352] ppid=12336 vsize=465096 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12352/stat : 12352 (python) S 12336 12340 12303 0 -1 4202560 23 930 0 8 0 0 0 0 17 0 19 0 538327160 476258304 7673 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 1 0 0 0
[pid=12340/tid=12355] ppid=12336 vsize=465096 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12355/stat : 12355 (python) S 12336 12340 12303 0 -1 4202560 25 930 0 8 0 0 0 0 19 0 19 0 538327163 476258304 7673 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 5 0 0 0
[pid=12340/tid=12358] ppid=12336 vsize=465096 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12358/stat : 12358 (python) S 12336 12340 12303 0 -1 4202560 21 930 0 8 0 0 0 0 20 0 19 0 538327166 476258304 7673 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 5 0 0 0
[pid=12340/tid=12361] ppid=12336 vsize=465096 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12361/stat : 12361 (python) S 12336 12340 12303 0 -1 4202560 29 930 0 8 0 0 0 0 22 0 19 0 538327169 476258304 7673 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 5 0 0 0
[pid=12340/tid=12426] ppid=12336 vsize=465096 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12426/stat : 12426 (python) S 12336 12340 12303 0 -1 4202560 26 930 0 8 0 0 0 0 20 0 19 0 538337178 476258304 7673 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 5 0 0 0
[pid=12340/tid=12429] ppid=12336 vsize=465096 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12429/stat : 12429 (python) S 12336 12340 12303 0 -1 4202560 23 930 0 8 0 0 0 0 22 0 19 0 538337181 476258304 7673 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 7 0 0 0
[pid=12340/tid=12433] ppid=12336 vsize=465096 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12433/stat : 12433 (python) S 12336 12340 12303 0 -1 4202560 27 930 0 8 0 0 0 0 24 0 19 0 538337186 476258304 7673 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 7 0 0 0
[pid=12340/tid=12436] ppid=12336 vsize=465096 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12436/stat : 12436 (python) S 12336 12340 12303 0 -1 4202560 30 930 0 8 0 0 0 0 20 0 19 0 538337189 476258304 7673 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 5 0 0 0
[pid=12340/tid=12460] ppid=12336 vsize=465096 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12460/stat : 12460 (python) S 12336 12340 12303 0 -1 4202560 27 930 0 8 0 0 0 0 20 0 19 0 538347105 476258304 7673 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 7 0 0 0
[pid=12340/tid=12463] ppid=12336 vsize=465096 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12463/stat : 12463 (python) S 12336 12340 12303 0 -1 4202560 29 930 0 8 0 0 0 0 22 0 19 0 538347109 476258304 7673 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 5 0 0 0
[pid=12340/tid=12467] ppid=12336 vsize=465096 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12467/stat : 12467 (python) S 12336 12340 12303 0 -1 4202560 37 930 0 8 0 0 0 0 18 0 19 0 538347194 476258304 7673 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 5 0 0 0
[pid=12340/tid=12471] ppid=12336 vsize=465096 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12471/stat : 12471 (python) S 12336 12340 12303 0 -1 4202560 28 930 0 8 0 0 0 0 21 0 19 0 538347201 476258304 7673 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 5 0 0 0
[pid=12340/tid=12493] ppid=12336 vsize=465096 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12493/stat : 12493 (python) S 12336 12340 12303 0 -1 4202560 24 930 0 8 0 0 0 0 18 0 19 0 538357024 476258304 7673 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 5 0 0 0
[pid=12340/tid=12497] ppid=12336 vsize=465096 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12497/stat : 12497 (python) S 12336 12340 12303 0 -1 4202560 36 930 0 8 0 0 0 0 18 0 19 0 538357030 476258304 7673 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 5 0 0 0
[pid=12340/tid=12501] ppid=12336 vsize=465096 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12501/stat : 12501 (python) S 12336 12340 12303 0 -1 4202560 28 930 0 8 0 0 0 0 20 0 19 0 538357223 476258304 7673 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 5 0 0 0
[pid=12340/tid=12520] ppid=12336 vsize=465096 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12520/stat : 12520 (python) S 12336 12340 12303 0 -1 4202560 24 930 0 8 0 0 0 0 20 0 19 0 538366832 476258304 7673 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 5 0 0 0
[pid=12340/tid=12537] ppid=12336 vsize=465096 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12537/stat : 12537 (python) S 12336 12340 12303 0 -1 4202560 25 930 0 8 0 0 0 0 20 0 19 0 538386189 476258304 7673 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 5 0 0 0
[pid=12340/tid=12642] ppid=12336 vsize=465096 CPUtime=0 cores=1,3,5,7
/proc/12340/task/12642/stat : 12642 (python) S 12336 12340 12303 0 -1 4202560 27 930 0 8 0 0 0 0 20 0 19 0 538396210 476258304 7673 33554432000 4194304 5683644 140734942225728 18446744073709551615 239488518577 0 0 16781312 2 18446604460168895424 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 2756.43
Current children cumulated vsize (KiB) 15946356

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 12340 and gives
#  childrusage.ru_utime.tv_sec=2
#  childrusage.ru_utime.tv_usec=221662
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=348946
# CPU time returned by wait4() is 2.57061
# while last known CPU time is 2756.43
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 712.817
CPU time (s): 2756.43
CPU user time (s): 2754.13
CPU system time (s): 2.3
CPU usage (%): 386.695
Max. virtual memory (cumulated for all children) (KiB): 15946356

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2.22166
system time used= 0.348946
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 29151
page faults= 118
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 17396
involuntary context switches= 137

runsolver used 3.78043 second user time and 7.2539 second system time

The end

Launcher Data 

Begin job on node148 at 2011-05-01 18:27:59
IDJOB=3317042
IDBENCH=82507
IDSOLVER=1705
FILE ID=node148/3317042-1304267279
RUNJOBID= node148-1304267279-12322
PBS_JOBID= 13173931
Free space on /tmp= 73120 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k7-r85-v200-c17000-S862393334-098-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317042-1304267279/watcher-3317042-1304267279 -o /tmp/evaluation-result-3317042-1304267279/solver-3317042-1304267279 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317042-1304267279.cnf -seed 489172721 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 860d0404ebc2282cbe692f8828f81b2c
RANDOM SEED=489172721

node148.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      29929172 kB
Buffers:        796356 kB
Cached:        1810840 kB
SwapCached:       3772 kB
Active:         309084 kB
Inactive:      2339976 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      29929172 kB
SwapTotal:    67111528 kB
SwapFree:     67086616 kB
Dirty:           64804 kB
Writeback:           0 kB
AnonPages:       40532 kB
Mapped:          13704 kB
Slab:           307568 kB
PageTables:       5052 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   185504 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73048 MiB
End job on node148 at 2011-05-01 18:39:55