Trace number 3250900

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587? (TO) 4800.53 1202.14

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m52-q918.75-n50-p30-s52376212.cnf
MD5SUMf34a5dd1cdce70a322dd395bcb5f1303
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2548
Number of clauses582679
Sum of the clauses size1167806
Maximum clause length52
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2582629
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 550

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3250900-1304113432/watcher-3250900-1304113432 -o /tmp/evaluation-result-3250900-1304113432/solver-3250900-1304113432 -C 4800 -W 1300 -M 15500 plingeling -t 4 HOME/instance-3250900-1304113432.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.31 8.03 8.05 3/181 12150
/proc/meminfo: memFree=30110272/32951124 swapFree=67102692/67111528
[pid=12150] ppid=12148 vsize=10004 CPUtime=0 cores=0,2,4,6
/proc/12150/stat : 12150 (plingeling) R 12148 12150 11448 0 -1 4202496 594 1251 0 0 0 0 0 0 22 0 1 0 522941396 10244096 523 33554432000 4194304 4417332 140736233426784 18446744073709551615 246516834048 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/12150/statm: 2501 523 146 55 0 387 0

[startup+0.0544499 s]
/proc/loadavg: 7.31 8.03 8.05 3/181 12150
/proc/meminfo: memFree=30110272/32951124 swapFree=67102692/67111528
[pid=12150] ppid=12148 vsize=18340 CPUtime=0.04 cores=0,2,4,6
/proc/12150/stat : 12150 (plingeling) R 12148 12150 11448 0 -1 4202496 2727 1251 0 0 4 0 0 0 22 0 1 0 522941396 18780160 2608 33554432000 4194304 4417332 140736233426784 18446744073709551615 4231470 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/12150/statm: 4585 2608 147 55 0 2471 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 18340

[startup+0.100443 s]
/proc/loadavg: 7.31 8.03 8.05 3/181 12150
/proc/meminfo: memFree=30110272/32951124 swapFree=67102692/67111528
[pid=12150] ppid=12148 vsize=26532 CPUtime=0.08 cores=0,2,4,6
/proc/12150/stat : 12150 (plingeling) R 12148 12150 11448 0 -1 4202496 4775 1251 0 0 8 0 0 0 22 0 1 0 522941396 27168768 4656 33554432000 4194304 4417332 140736233426784 18446744073709551615 246516834048 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/12150/statm: 6633 4656 147 55 0 4519 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 26532

[startup+0.30041 s]
/proc/loadavg: 7.31 8.03 8.05 3/181 12150
/proc/meminfo: memFree=30110272/32951124 swapFree=67102692/67111528
[pid=12150] ppid=12148 vsize=75684 CPUtime=0.29 cores=0,2,4,6
/proc/12150/stat : 12150 (plingeling) R 12148 12150 11448 0 -1 4202496 17063 1251 0 0 27 2 0 0 23 0 1 0 522941396 77500416 16944 33554432000 4194304 4417332 140736233426784 18446744073709551615 246516834048 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/12150/statm: 18921 16944 147 55 0 16807 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 75684

[startup+0.700292 s]
/proc/loadavg: 7.31 8.03 8.05 3/181 12150
/proc/meminfo: memFree=30110272/32951124 swapFree=67102692/67111528
[pid=12150] ppid=12148 vsize=119020 CPUtime=1.64 cores=0,2,4,6
/proc/12150/stat : 12150 (plingeling) S 12148 12150 11448 0 -1 4202496 19283 1251 0 0 161 3 0 0 24 0 5 0 522941396 121876480 17570 33554432000 4194304 4417332 140736233426784 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/12150/statm: 29755 17570 175 55 0 27641 0
Current children cumulated CPU time (s) 1.64
Current children cumulated vsize (KiB) 119020

[startup+1.50115 s]
/proc/loadavg: 7.31 8.03 8.05 9/190 12164
/proc/meminfo: memFree=30012000/32951124 swapFree=67102692/67111528
[pid=12150] ppid=12148 vsize=199416 CPUtime=4.84 cores=0,2,4,6
/proc/12150/stat : 12150 (plingeling) S 12148 12150 11448 0 -1 4202496 37508 1251 0 0 479 5 0 0 24 0 5 0 522941396 204201984 21290 33554432000 4194304 4417332 140736233426784 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/12150/statm: 49854 21290 179 55 0 47740 0
[pid=12150/tid=12161] ppid=12148 vsize=199416 CPUtime=1.11 cores=0,2,4,6
/proc/12150/task/12161/stat : 12161 (plingeling) R 12148 12150 11448 0 -1 4202560 4779 1251 0 0 111 0 0 0 25 0 5 0 522941434 204201984 21290 33554432000 4194304 4417332 140736233426784 18446744073709551615 4258401 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12150/tid=12162] ppid=12148 vsize=199416 CPUtime=1.1 cores=0,2,4,6
/proc/12150/task/12162/stat : 12162 (plingeling) R 12148 12150 11448 0 -1 4202560 4286 1251 0 0 110 0 0 0 25 0 5 0 522941434 204201984 21290 33554432000 4194304 4417332 140736233426784 18446744073709551615 4235999 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12150/tid=12163] ppid=12148 vsize=199416 CPUtime=1.11 cores=0,2,4,6
/proc/12150/task/12163/stat : 12163 (plingeling) R 12148 12150 11448 0 -1 4202560 4622 1251 0 0 111 0 0 0 25 0 5 0 522941434 204201984 21290 33554432000 4194304 4417332 140736233426784 18446744073709551615 4258449 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12150/tid=12164] ppid=12148 vsize=199416 CPUtime=1.11 cores=0,2,4,6
/proc/12150/task/12164/stat : 12164 (plingeling) R 12148 12150 11448 0 -1 4202560 6746 1251 0 0 110 1 0 0 25 0 5 0 522941434 204201984 21290 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364479 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4.84
Current children cumulated vsize (KiB) 199416

[startup+3.10085 s]
/proc/loadavg: 7.31 8.03 8.05 9/190 12164
/proc/meminfo: memFree=29990296/32951124 swapFree=67102692/67111528
[pid=12150] ppid=12148 vsize=215368 CPUtime=11.23 cores=0,2,4,6
/proc/12150/stat : 12150 (plingeling) S 12148 12150 11448 0 -1 4202496 50132 1251 0 0 1116 7 0 0 24 0 5 0 522941396 220536832 25278 33554432000 4194304 4417332 140736233426784 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/12150/statm: 53842 25278 179 55 0 51728 0
[pid=12150/tid=12161] ppid=12148 vsize=215368 CPUtime=2.71 cores=0,2,4,6
/proc/12150/task/12161/stat : 12161 (plingeling) R 12148 12150 11448 0 -1 4202560 5292 1251 0 0 270 1 0 0 25 0 5 0 522941434 220536832 25278 33554432000 4194304 4417332 140736233426784 18446744073709551615 4236508 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12150/tid=12162] ppid=12148 vsize=215368 CPUtime=2.69 cores=0,2,4,6
/proc/12150/task/12162/stat : 12162 (plingeling) R 12148 12150 11448 0 -1 4202560 5260 1251 0 0 269 0 0 0 25 0 5 0 522941434 220536832 25278 33554432000 4194304 4417332 140736233426784 18446744073709551615 4249143 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12150/tid=12163] ppid=12148 vsize=215368 CPUtime=2.71 cores=0,2,4,6
/proc/12150/task/12163/stat : 12163 (plingeling) R 12148 12150 11448 0 -1 4202560 5597 1251 0 0 271 0 0 0 25 0 5 0 522941434 220536832 25278 33554432000 4194304 4417332 140736233426784 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12150/tid=12164] ppid=12148 vsize=215368 CPUtime=2.71 cores=0,2,4,6
/proc/12150/task/12164/stat : 12164 (plingeling) R 12148 12150 11448 0 -1 4202560 16908 1251 0 0 269 2 0 0 25 0 5 0 522941434 220536832 25278 33554432000 4194304 4417332 140736233426784 18446744073709551615 4365960 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 11.23
Current children cumulated vsize (KiB) 215368

[startup+6.30026 s]
/proc/loadavg: 7.37 8.03 8.05 9/190 12164
/proc/meminfo: memFree=29954400/32951124 swapFree=67102692/67111528
[pid=12150] ppid=12148 vsize=252560 CPUtime=24.01 cores=0,2,4,6
/proc/12150/stat : 12150 (plingeling) S 12148 12150 11448 0 -1 4202496 66983 1251 0 0 2392 9 0 0 24 0 5 0 522941396 258621440 34577 33554432000 4194304 4417332 140736233426784 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/12150/statm: 63140 34577 180 55 0 61026 0
[pid=12150/tid=12161] ppid=12148 vsize=252560 CPUtime=5.91 cores=0,2,4,6
/proc/12150/task/12161/stat : 12161 (plingeling) R 12148 12150 11448 0 -1 4202560 6591 1251 0 0 590 1 0 0 25 0 5 0 522941434 258621440 34577 33554432000 4194304 4417332 140736233426784 18446744073709551615 4236017 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12150/tid=12162] ppid=12148 vsize=252560 CPUtime=5.87 cores=0,2,4,6
/proc/12150/task/12162/stat : 12162 (plingeling) R 12148 12150 11448 0 -1 4202560 6559 1251 0 0 587 0 0 0 25 0 5 0 522941434 258621440 34577 33554432000 4194304 4417332 140736233426784 18446744073709551615 4258460 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12150/tid=12163] ppid=12148 vsize=252560 CPUtime=5.91 cores=0,2,4,6
/proc/12150/task/12163/stat : 12163 (plingeling) R 12148 12150 11448 0 -1 4202560 5598 1251 0 0 591 0 0 0 25 0 5 0 522941434 258621440 34577 33554432000 4194304 4417332 140736233426784 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12150/tid=12164] ppid=12148 vsize=252560 CPUtime=5.91 cores=0,2,4,6
/proc/12150/task/12164/stat : 12164 (plingeling) R 12148 12150 11448 0 -1 4202560 31160 1251 0 0 587 4 0 0 25 0 5 0 522941434 258621440 34577 33554432000 4194304 4417332 140736233426784 18446744073709551615 4199406 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 24.01
Current children cumulated vsize (KiB) 252560

[startup+12.7011 s]
/proc/loadavg: 7.42 8.03 8.05 10/190 12164
/proc/meminfo: memFree=29887092/32951124 swapFree=67102692/67111528
[pid=12150] ppid=12148 vsize=268852 CPUtime=49.58 cores=0,2,4,6
/proc/12150/stat : 12150 (plingeling) S 12148 12150 11448 0 -1 4202496 90058 1251 0 0 4945 13 0 0 24 0 5 0 522941396 275304448 37627 33554432000 4194304 4417332 140736233426784 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/12150/statm: 67213 37627 180 55 0 65099 0
[pid=12150/tid=12161] ppid=12148 vsize=268852 CPUtime=12.31 cores=0,2,4,6
/proc/12150/task/12161/stat : 12161 (plingeling) R 12148 12150 11448 0 -1 4202560 6591 1251 0 0 1230 1 0 0 25 0 5 0 522941434 275304448 37627 33554432000 4194304 4417332 140736233426784 18446744073709551615 4249171 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12150/tid=12162] ppid=12148 vsize=268852 CPUtime=12.24 cores=0,2,4,6
/proc/12150/task/12162/stat : 12162 (plingeling) R 12148 12150 11448 0 -1 4202560 6559 1251 0 0 1224 0 0 0 25 0 5 0 522941434 275304448 37627 33554432000 4194304 4417332 140736233426784 18446744073709551615 4249526 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12150/tid=12163] ppid=12148 vsize=268852 CPUtime=12.31 cores=0,2,4,6
/proc/12150/task/12163/stat : 12163 (plingeling) R 12148 12150 11448 0 -1 4202560 7732 1251 0 0 1230 1 0 0 25 0 5 0 522941434 275304448 37627 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364942 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12150/tid=12164] ppid=12148 vsize=268852 CPUtime=12.31 cores=0,2,4,6
/proc/12150/task/12164/stat : 12164 (plingeling) R 12148 12150 11448 0 -1 4202560 52101 1251 0 0 1224 7 0 0 25 0 5 0 522941434 275304448 37627 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364671 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 49.58
Current children cumulated vsize (KiB) 268852

[startup+25.5008 s]
/proc/loadavg: 7.55 8.03 8.04 9/190 12164
/proc/meminfo: memFree=29732360/32951124 swapFree=67102692/67111528
[pid=12150] ppid=12148 vsize=330312 CPUtime=100.7 cores=0,2,4,6
/proc/12150/stat : 12150 (plingeling) S 12148 12150 11448 0 -1 4202496 138279 1251 0 0 10050 20 0 0 24 0 5 0 522941396 338239488 54016 33554432000 4194304 4417332 140736233426784 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/12150/statm: 82578 54016 181 55 0 80464 0
[pid=12150/tid=12161] ppid=12148 vsize=330312 CPUtime=25.11 cores=0,2,4,6
/proc/12150/task/12161/stat : 12161 (plingeling) R 12148 12150 11448 0 -1 4202560 8583 1251 0 0 2510 1 0 0 25 0 5 0 522941434 338239488 54016 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364599 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12150/tid=12162] ppid=12148 vsize=330312 CPUtime=24.97 cores=0,2,4,6
/proc/12150/task/12162/stat : 12162 (plingeling) R 12148 12150 11448 0 -1 4202560 6637 1251 0 0 2497 0 0 0 25 0 5 0 522941434 338239488 54016 33554432000 4194304 4417332 140736233426784 18446744073709551615 4366384 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12150/tid=12163] ppid=12148 vsize=330312 CPUtime=25.11 cores=0,2,4,6
/proc/12150/task/12163/stat : 12163 (plingeling) R 12148 12150 11448 0 -1 4202560 9280 1251 0 0 2510 1 0 0 25 0 5 0 522941434 338239488 54016 33554432000 4194304 4417332 140736233426784 18446744073709551615 4331683 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12150/tid=12164] ppid=12148 vsize=330312 CPUtime=25.11 cores=0,2,4,6
/proc/12150/task/12164/stat : 12164 (plingeling) R 12148 12150 11448 0 -1 4202560 96704 1251 0 0 2497 14 0 0 25 0 5 0 522941434 338239488 54016 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364470 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 100.7
Current children cumulated vsize (KiB) 330312

[startup+51.1011 s]
/proc/loadavg: 8.09 8.12 8.07 10/190 12165
/proc/meminfo: memFree=29576480/32951124 swapFree=67102692/67111528
[pid=12150] ppid=12148 vsize=410348 CPUtime=202.97 cores=0,2,4,6
/proc/12150/stat : 12150 (plingeling) S 12148 12150 11448 0 -1 4202496 158288 1251 0 0 20275 22 0 0 24 0 5 0 522941396 420196352 74025 33554432000 4194304 4417332 140736233426784 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/12150/statm: 102587 74025 181 55 0 100473 0
[pid=12150/tid=12161] ppid=12148 vsize=410348 CPUtime=50.72 cores=0,2,4,6
/proc/12150/task/12161/stat : 12161 (plingeling) R 12148 12150 11448 0 -1 4202560 14369 1251 0 0 5070 2 0 0 25 0 5 0 522941434 420196352 74025 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364683 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12150/tid=12162] ppid=12148 vsize=410348 CPUtime=50.43 cores=0,2,4,6
/proc/12150/task/12162/stat : 12162 (plingeling) R 12148 12150 11448 0 -1 4202560 6637 1251 0 0 5043 0 0 0 25 0 5 0 522941434 420196352 74025 33554432000 4194304 4417332 140736233426784 18446744073709551615 4261379 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12150/tid=12163] ppid=12148 vsize=410348 CPUtime=50.71 cores=0,2,4,6
/proc/12150/task/12163/stat : 12163 (plingeling) R 12148 12150 11448 0 -1 4202560 13722 1251 0 0 5070 1 0 0 25 0 5 0 522941434 420196352 74025 33554432000 4194304 4417332 140736233426784 18446744073709551615 4249519 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12150/tid=12164] ppid=12148 vsize=410348 CPUtime=50.71 cores=0,2,4,6
/proc/12150/task/12164/stat : 12164 (plingeling) R 12148 12150 11448 0 -1 4202560 106485 1251 0 0 5056 15 0 0 25 0 5 0 522941434 420196352 74025 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 202.97
Current children cumulated vsize (KiB) 410348

[startup+102.307 s]
/proc/loadavg: 8.17 8.13 8.08 9/190 12166
/proc/meminfo: memFree=29406096/32951124 swapFree=67102692/67111528
[pid=12150] ppid=12148 vsize=415848 CPUtime=407.52 cores=0,2,4,6
/proc/12150/stat : 12150 (plingeling) S 12148 12150 11448 0 -1 4202496 210479 1251 0 0 40723 29 0 0 24 0 5 0 522941396 425828352 75400 33554432000 4194304 4417332 140736233426784 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/12150/statm: 103962 75400 181 55 0 101848 0
[pid=12150/tid=12161] ppid=12148 vsize=415848 CPUtime=101.91 cores=0,2,4,6
/proc/12150/task/12161/stat : 12161 (plingeling) R 12148 12150 11448 0 -1 4202560 19490 1251 0 0 10189 2 0 0 25 0 5 0 522941434 425828352 75400 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364742 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12150/tid=12162] ppid=12148 vsize=415848 CPUtime=101.35 cores=0,2,4,6
/proc/12150/task/12162/stat : 12162 (plingeling) R 12148 12150 11448 0 -1 4202560 8440 1251 0 0 10135 0 0 0 25 0 5 0 522941434 425828352 75400 33554432000 4194304 4417332 140736233426784 18446744073709551615 4309982 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12150/tid=12163] ppid=12148 vsize=415848 CPUtime=101.91 cores=0,2,4,6
/proc/12150/task/12163/stat : 12163 (plingeling) R 12148 12150 11448 0 -1 4202560 14636 1251 0 0 10190 1 0 0 25 0 5 0 522941434 425828352 75400 33554432000 4194304 4417332 140736233426784 18446744073709551615 4249584 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12150/tid=12164] ppid=12148 vsize=415848 CPUtime=101.91 cores=0,2,4,6
/proc/12150/task/12164/stat : 12164 (plingeling) R 12148 12150 11448 0 -1 4202560 150838 1251 0 0 10171 20 0 0 25 0 5 0 522941434 425828352 75400 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364336 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 407.52
Current children cumulated vsize (KiB) 415848

[startup+162.301 s]
/proc/loadavg: 8.32 8.17 8.09 9/190 12168
/proc/meminfo: memFree=28918396/32951124 swapFree=67102692/67111528
[pid=12150] ppid=12148 vsize=667412 CPUtime=647.17 cores=0,2,4,6
/proc/12150/stat : 12150 (plingeling) S 12148 12150 11448 0 -1 4202496 277134 1251 0 0 64680 37 0 0 24 0 5 0 522941396 683429888 138291 33554432000 4194304 4417332 140736233426784 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/12150/statm: 166853 138291 181 55 0 164739 0
[pid=12150/tid=12161] ppid=12148 vsize=667412 CPUtime=161.92 cores=0,2,4,6
/proc/12150/task/12161/stat : 12161 (plingeling) R 12148 12150 11448 0 -1 4202560 21832 1251 0 0 16189 3 0 0 25 0 5 0 522941434 683429888 138291 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364494 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12150/tid=12162] ppid=12148 vsize=667412 CPUtime=161.03 cores=0,2,4,6
/proc/12150/task/12162/stat : 12162 (plingeling) R 12148 12150 11448 0 -1 4202560 10385 1251 0 0 16102 1 0 0 25 0 5 0 522941434 683429888 138291 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12150/tid=12163] ppid=12148 vsize=667412 CPUtime=161.91 cores=0,2,4,6
/proc/12150/task/12163/stat : 12163 (plingeling) R 12148 12150 11448 0 -1 4202560 22858 1251 0 0 16189 2 0 0 25 0 5 0 522941434 683429888 138291 33554432000 4194304 4417332 140736233426784 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12150/tid=12164] ppid=12148 vsize=667412 CPUtime=161.92 cores=0,2,4,6
/proc/12150/task/12164/stat : 12164 (plingeling) R 12148 12150 11448 0 -1 4202560 204984 1251 0 0 16164 28 0 0 25 0 5 0 522941434 683429888 138291 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364560 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 647.17
Current children cumulated vsize (KiB) 667412


################
# More data... #
################

/proc/meminfo: memFree=27640160/32951124 swapFree=67102692/67111528
[pid=12150] ppid=12148 vsize=1509504 CPUtime=3043.97 cores=0,2,4,6
/proc/12150/stat : 12150 (plingeling) S 12148 12150 11448 0 -1 4202496 1063115 1251 0 0 304240 157 0 0 24 0 5 0 522941396 1545732096 348814 33554432000 4194304 4417332 140736233426784 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/12150/statm: 377376 348814 181 55 0 375262 0
[pid=12150/tid=12161] ppid=12148 vsize=1509504 CPUtime=761.94 cores=0,2,4,6
/proc/12150/task/12161/stat : 12161 (plingeling) R 12148 12150 11448 0 -1 4202560 123281 1251 0 0 76178 16 0 0 25 0 5 0 522941434 1545732096 348814 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364565 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12150/tid=12162] ppid=12148 vsize=1509504 CPUtime=757.75 cores=0,2,4,6
/proc/12150/task/12162/stat : 12162 (plingeling) R 12148 12150 11448 0 -1 4202560 93565 1251 0 0 75764 11 0 0 25 0 5 0 522941434 1545732096 348814 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364776 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12150/tid=12163] ppid=12148 vsize=1509504 CPUtime=761.94 cores=0,2,4,6
/proc/12150/task/12163/stat : 12163 (plingeling) R 12148 12150 11448 0 -1 4202560 270015 1251 0 0 76158 36 0 0 25 0 5 0 522941434 1545732096 348814 33554432000 4194304 4417332 140736233426784 18446744073709551615 4366384 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12150/tid=12164] ppid=12148 vsize=1509504 CPUtime=761.94 cores=0,2,4,6
/proc/12150/task/12164/stat : 12164 (plingeling) R 12148 12150 11448 0 -1 4202560 559179 1251 0 0 76104 90 0 0 25 0 5 0 522941434 1545732096 348814 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3043.97
Current children cumulated vsize (KiB) 1509504

[startup+822.301 s]
/proc/loadavg: 8.32 8.14 8.10 9/190 12185
/proc/meminfo: memFree=27133472/32951124 swapFree=67102692/67111528
[pid=12150] ppid=12148 vsize=1745832 CPUtime=3283.63 cores=0,2,4,6
/proc/12150/stat : 12150 (plingeling) S 12148 12150 11448 0 -1 4202496 1202976 1251 0 0 328185 178 0 0 24 0 5 0 522941396 1787731968 407896 33554432000 4194304 4417332 140736233426784 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/12150/statm: 436458 407896 181 55 0 434344 0
[pid=12150/tid=12161] ppid=12148 vsize=1745832 CPUtime=821.94 cores=0,2,4,6
/proc/12150/task/12161/stat : 12161 (plingeling) R 12148 12150 11448 0 -1 4202560 155551 1251 0 0 82173 21 0 0 25 0 5 0 522941434 1787731968 407896 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12150/tid=12162] ppid=12148 vsize=1745832 CPUtime=817.42 cores=0,2,4,6
/proc/12150/task/12162/stat : 12162 (plingeling) R 12148 12150 11448 0 -1 4202560 121400 1251 0 0 81728 14 0 0 25 0 5 0 522941434 1787731968 407896 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12150/tid=12163] ppid=12148 vsize=1745832 CPUtime=821.93 cores=0,2,4,6
/proc/12150/task/12163/stat : 12163 (plingeling) R 12148 12150 11448 0 -1 4202560 335551 1251 0 0 82147 46 0 0 25 0 5 0 522941434 1787731968 407896 33554432000 4194304 4417332 140736233426784 18446744073709551615 4354236 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12150/tid=12164] ppid=12148 vsize=1745832 CPUtime=821.94 cores=0,2,4,6
/proc/12150/task/12164/stat : 12164 (plingeling) R 12148 12150 11448 0 -1 4202560 573399 1251 0 0 82101 93 0 0 25 0 5 0 522941434 1787731968 407896 33554432000 4194304 4417332 140736233426784 18446744073709551615 4365963 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3283.63
Current children cumulated vsize (KiB) 1745832

[startup+882.301 s]
/proc/loadavg: 8.18 8.13 8.09 9/190 12187
/proc/meminfo: memFree=27008720/32951124 swapFree=67102692/67111528
[pid=12150] ppid=12148 vsize=1862952 CPUtime=3523.31 cores=0,2,4,6
/proc/12150/stat : 12150 (plingeling) S 12148 12150 11448 0 -1 4202496 1415912 1251 0 0 352120 211 0 0 24 0 5 0 522941396 1907662848 437176 33554432000 4194304 4417332 140736233426784 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/12150/statm: 465738 437176 181 55 0 463624 0
[pid=12150/tid=12161] ppid=12148 vsize=1862952 CPUtime=881.94 cores=0,2,4,6
/proc/12150/task/12161/stat : 12161 (plingeling) R 12148 12150 11448 0 -1 4202560 208240 1251 0 0 88164 30 0 0 25 0 5 0 522941434 1907662848 437176 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12150/tid=12162] ppid=12148 vsize=1862952 CPUtime=877.09 cores=0,2,4,6
/proc/12150/task/12162/stat : 12162 (plingeling) R 12148 12150 11448 0 -1 4202560 195128 1251 0 0 87684 25 0 0 25 0 5 0 522941434 1907662848 437176 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364547 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12150/tid=12163] ppid=12148 vsize=1862952 CPUtime=881.94 cores=0,2,4,6
/proc/12150/task/12163/stat : 12163 (plingeling) R 12148 12150 11448 0 -1 4202560 376511 1251 0 0 88142 52 0 0 25 0 5 0 522941434 1907662848 437176 33554432000 4194304 4417332 140736233426784 18446744073709551615 4237221 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12150/tid=12164] ppid=12148 vsize=1862952 CPUtime=881.93 cores=0,2,4,6
/proc/12150/task/12164/stat : 12164 (plingeling) R 12148 12150 11448 0 -1 4202560 618958 1251 0 0 88093 100 0 0 25 0 5 0 522941434 1907662848 437176 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364502 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3523.31
Current children cumulated vsize (KiB) 1862952

[startup+942.301 s]
/proc/loadavg: 5.81 7.45 7.86 5/180 12206
/proc/meminfo: memFree=28153364/32951124 swapFree=67102692/67111528
[pid=12150] ppid=12148 vsize=2087516 CPUtime=3762.98 cores=0,2,4,6
/proc/12150/stat : 12150 (plingeling) S 12148 12150 11448 0 -1 4202496 1537589 1251 0 0 376070 228 0 0 24 0 5 0 522941396 2137616384 493317 33554432000 4194304 4417332 140736233426784 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/12150/statm: 521879 493317 181 55 0 519765 0
[pid=12150/tid=12161] ppid=12148 vsize=2087516 CPUtime=941.94 cores=0,2,4,6
/proc/12150/task/12161/stat : 12161 (plingeling) R 12148 12150 11448 0 -1 4202560 209328 1251 0 0 94164 30 0 0 25 0 5 0 522941434 2137616384 493317 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364796 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12150/tid=12162] ppid=12148 vsize=2087516 CPUtime=936.76 cores=0,2,4,6
/proc/12150/task/12162/stat : 12162 (plingeling) R 12148 12150 11448 0 -1 4202560 227896 1251 0 0 93648 28 0 0 25 0 5 0 522941434 2137616384 493317 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12150/tid=12163] ppid=12148 vsize=2087516 CPUtime=941.93 cores=0,2,4,6
/proc/12150/task/12163/stat : 12163 (plingeling) R 12148 12150 11448 0 -1 4202560 427948 1251 0 0 94135 58 0 0 25 0 5 0 522941434 2137616384 493317 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12150/tid=12164] ppid=12148 vsize=2087516 CPUtime=941.93 cores=0,2,4,6
/proc/12150/task/12164/stat : 12164 (plingeling) R 12148 12150 11448 0 -1 4202560 655342 1251 0 0 94085 108 0 0 25 0 5 0 522941434 2137616384 493317 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364732 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3762.98
Current children cumulated vsize (KiB) 2087516

[startup+1002.3 s]
/proc/loadavg: 4.66 6.82 7.62 6/180 12207
/proc/meminfo: memFree=27992652/32951124 swapFree=67102692/67111528
[pid=12150] ppid=12148 vsize=2247332 CPUtime=4002.65 cores=0,2,4,6
/proc/12150/stat : 12150 (plingeling) S 12148 12150 11448 0 -1 4202496 1616721 1251 0 0 400027 238 0 0 24 0 5 0 522941396 2301267968 533271 33554432000 4194304 4417332 140736233426784 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/12150/statm: 561833 533271 181 55 0 559719 0
[pid=12150/tid=12161] ppid=12148 vsize=2247332 CPUtime=1001.94 cores=0,2,4,6
/proc/12150/task/12161/stat : 12161 (plingeling) R 12148 12150 11448 0 -1 4202560 213424 1251 0 0 100164 30 0 0 25 0 5 0 522941434 2301267968 533271 33554432000 4194304 4417332 140736233426784 18446744073709551615 4249110 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12150/tid=12162] ppid=12148 vsize=2247332 CPUtime=996.44 cores=0,2,4,6
/proc/12150/task/12162/stat : 12162 (plingeling) R 12148 12150 11448 0 -1 4202560 251289 1251 0 0 99613 31 0 0 25 0 5 0 522941434 2301267968 533271 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364526 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12150/tid=12163] ppid=12148 vsize=2247332 CPUtime=1001.93 cores=0,2,4,6
/proc/12150/task/12163/stat : 12163 (plingeling) R 12148 12150 11448 0 -1 4202560 476376 1251 0 0 100129 64 0 0 25 0 5 0 522941434 2301267968 533271 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12150/tid=12164] ppid=12148 vsize=2247332 CPUtime=1001.93 cores=0,2,4,6
/proc/12150/task/12164/stat : 12164 (plingeling) R 12148 12150 11448 0 -1 4202560 658557 1251 0 0 100084 109 0 0 25 0 5 0 522941434 2301267968 533271 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4002.65
Current children cumulated vsize (KiB) 2247332

[startup+1062.3 s]
/proc/loadavg: 4.38 6.33 7.40 5/180 12218
/proc/meminfo: memFree=27754452/32951124 swapFree=67102692/67111528
[pid=12150] ppid=12148 vsize=2485544 CPUtime=4242.3 cores=0,2,4,6
/proc/12150/stat : 12150 (plingeling) S 12148 12150 11448 0 -1 4202496 1741810 1251 0 0 423975 255 0 0 24 0 5 0 522941396 2545197056 592824 33554432000 4194304 4417332 140736233426784 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/12150/statm: 621386 592824 181 55 0 619272 0
[pid=12150/tid=12161] ppid=12148 vsize=2485544 CPUtime=1061.92 cores=0,2,4,6
/proc/12150/task/12161/stat : 12161 (plingeling) R 12148 12150 11448 0 -1 4202560 308295 1251 0 0 106150 42 0 0 25 0 5 0 522941434 2545197056 592824 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12150/tid=12162] ppid=12148 vsize=2485544 CPUtime=1056.11 cores=0,2,4,6
/proc/12150/task/12162/stat : 12162 (plingeling) R 12148 12150 11448 0 -1 4202560 261011 1251 0 0 105579 32 0 0 25 0 5 0 522941434 2545197056 592824 33554432000 4194304 4417332 140736233426784 18446744073709551615 4309897 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12150/tid=12163] ppid=12148 vsize=2485544 CPUtime=1061.94 cores=0,2,4,6
/proc/12150/task/12163/stat : 12163 (plingeling) R 12148 12150 11448 0 -1 4202560 476376 1251 0 0 106130 64 0 0 25 0 5 0 522941434 2545197056 592824 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364463 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12150/tid=12164] ppid=12148 vsize=2485544 CPUtime=1061.92 cores=0,2,4,6
/proc/12150/task/12164/stat : 12164 (plingeling) R 12148 12150 11448 0 -1 4202560 679053 1251 0 0 106080 112 0 0 25 0 5 0 522941434 2545197056 592824 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4242.3
Current children cumulated vsize (KiB) 2485544

[startup+1122.3 s]
/proc/loadavg: 4.20 5.92 7.19 5/180 12220
/proc/meminfo: memFree=27410104/32951124 swapFree=67102692/67111528
[pid=12150] ppid=12148 vsize=2829388 CPUtime=4481.97 cores=0,2,4,6
/proc/12150/stat : 12150 (plingeling) S 12148 12150 11448 0 -1 4202496 1880162 1251 0 0 447919 278 0 0 24 0 5 0 522941396 2897293312 678785 33554432000 4194304 4417332 140736233426784 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/12150/statm: 707347 678785 181 55 0 705233 0
[pid=12150/tid=12161] ppid=12148 vsize=2829388 CPUtime=1121.92 cores=0,2,4,6
/proc/12150/task/12161/stat : 12161 (plingeling) R 12148 12150 11448 0 -1 4202560 316487 1251 0 0 112148 44 0 0 25 0 5 0 522941434 2897293312 678785 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12150/tid=12162] ppid=12148 vsize=2829388 CPUtime=1115.79 cores=0,2,4,6
/proc/12150/task/12162/stat : 12162 (plingeling) R 12148 12150 11448 0 -1 4202560 296704 1251 0 0 111540 39 0 0 25 0 5 0 522941434 2897293312 678785 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364800 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12150/tid=12163] ppid=12148 vsize=2829388 CPUtime=1121.94 cores=0,2,4,6
/proc/12150/task/12163/stat : 12163 (plingeling) R 12148 12150 11448 0 -1 4202560 536605 1251 0 0 112122 72 0 0 25 0 5 0 522941434 2897293312 678785 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364536 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12150/tid=12164] ppid=12148 vsize=2829388 CPUtime=1121.92 cores=0,2,4,6
/proc/12150/task/12164/stat : 12164 (plingeling) R 12148 12150 11448 0 -1 4202560 713291 1251 0 0 112073 119 0 0 25 0 5 0 522941434 2897293312 678785 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364509 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4481.97
Current children cumulated vsize (KiB) 2829388

[startup+1182.3 s]
/proc/loadavg: 4.29 5.62 7.00 5/180 12221
/proc/meminfo: memFree=27387500/32951124 swapFree=67102692/67111528
[pid=12150] ppid=12148 vsize=2852256 CPUtime=4721.66 cores=0,2,4,6
/proc/12150/stat : 12150 (plingeling) S 12148 12150 11448 0 -1 4202496 1948606 1251 0 0 471877 289 0 0 24 0 5 0 522941396 2920710144 684502 33554432000 4194304 4417332 140736233426784 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/12150/statm: 713064 684502 181 55 0 710950 0
[pid=12150/tid=12161] ppid=12148 vsize=2852256 CPUtime=1181.92 cores=0,2,4,6
/proc/12150/task/12161/stat : 12161 (plingeling) R 12148 12150 11448 0 -1 4202560 349704 1251 0 0 118142 50 0 0 25 0 5 0 522941434 2920710144 684502 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12150/tid=12162] ppid=12148 vsize=2852256 CPUtime=1175.46 cores=0,2,4,6
/proc/12150/task/12162/stat : 12162 (plingeling) R 12148 12150 11448 0 -1 4202560 297322 1251 0 0 117506 40 0 0 25 0 5 0 522941434 2920710144 684502 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364465 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12150/tid=12163] ppid=12148 vsize=2852256 CPUtime=1181.94 cores=0,2,4,6
/proc/12150/task/12163/stat : 12163 (plingeling) R 12148 12150 11448 0 -1 4202560 536605 1251 0 0 118122 72 0 0 25 0 5 0 522941434 2920710144 684502 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12150/tid=12164] ppid=12148 vsize=2852256 CPUtime=1181.92 cores=0,2,4,6
/proc/12150/task/12164/stat : 12164 (plingeling) R 12148 12150 11448 0 -1 4202560 747900 1251 0 0 118069 123 0 0 25 0 5 0 522941434 2920710144 684502 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4721.66
Current children cumulated vsize (KiB) 2852256



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202 s]
/proc/loadavg: 4.21 5.51 6.94 5/180 12222
/proc/meminfo: memFree=27379440/32951124 swapFree=67102692/67111528
[pid=12150] ppid=12148 vsize=2859672 CPUtime=4800.37 cores=0,2,4,6
/proc/12150/stat : 12150 (plingeling) S 12148 12150 11448 0 -1 4202496 1950460 1251 0 0 479748 289 0 0 24 0 5 0 522941396 2928304128 686356 33554432000 4194304 4417332 140736233426784 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/12150/statm: 714918 686356 181 55 0 712804 0
[pid=12150/tid=12161] ppid=12148 vsize=2859672 CPUtime=1201.63 cores=0,2,4,6
/proc/12150/task/12161/stat : 12161 (plingeling) R 12148 12150 11448 0 -1 4202560 349704 1251 0 0 120113 50 0 0 25 0 5 0 522941434 2928304128 686356 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12150/tid=12162] ppid=12148 vsize=2859672 CPUtime=1195.05 cores=0,2,4,6
/proc/12150/task/12162/stat : 12162 (plingeling) R 12148 12150 11448 0 -1 4202560 299176 1251 0 0 119465 40 0 0 25 0 5 0 522941434 2928304128 686356 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364465 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12150/tid=12163] ppid=12148 vsize=2859672 CPUtime=1201.65 cores=0,2,4,6
/proc/12150/task/12163/stat : 12163 (plingeling) R 12148 12150 11448 0 -1 4202560 536605 1251 0 0 120093 72 0 0 25 0 5 0 522941434 2928304128 686356 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364753 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12150/tid=12164] ppid=12148 vsize=2859672 CPUtime=1201.63 cores=0,2,4,6
/proc/12150/task/12164/stat : 12164 (plingeling) R 12148 12150 11448 0 -1 4202560 747900 1251 0 0 120040 123 0 0 25 0 5 0 522941434 2928304128 686356 33554432000 4194304 4417332 140736233426784 18446744073709551615 4364536 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.37
Current children cumulated vsize (KiB) 2859672

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.11 s]
/proc/loadavg: 4.21 5.51 6.94 5/180 12222
/proc/meminfo: memFree=27379440/32951124 swapFree=67102692/67111528
[pid=12150] ppid=12148 vsize=0 CPUtime=4800.49 cores=0,2,4,6
/proc/12150/stat : 12150 (plingeling) Z 12148 12150 11448 0 -1 4203548 1950460 1251 0 0 479750 299 0 0 18 0 2 0 522941396 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12150/statm: 0 0 0 0 0 0 0
[pid=12150/tid=12162] ppid=12148 vsize=0 CPUtime=1195.16 cores=0,2,4,6
/proc/12150/task/12162/stat : 12162 (plingeling) R 12148 12150 11448 0 -1 4203588 299176 1251 0 0 119466 50 0 0 25 0 2 0 522941434 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.49
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.14
CPU time (s): 4800.53
CPU user time (s): 4797.5
CPU system time (s): 3.02954
CPU usage (%): 399.331
Max. virtual memory (cumulated for all children) (KiB): 3033756

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4797.5
system time used= 3.02954
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1951711
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 84
involuntary context switches= 78938

runsolver used 1.9587 second user time and 4.6123 second system time

The end

Launcher Data 

Begin job on node134 at 2011-04-29 23:43:52
IDJOB=3250900
IDBENCH=88204
IDSOLVER=1605
FILE ID=node134/3250900-1304113432
RUNJOBID= node134-1304113432-12125
PBS_JOBID= 13170663
Free space on /tmp= 71576 MiB

SOLVER NAME= Plingeling 587
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m52-q918.75-n50-p30-s52376212.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3250900-1304113432/watcher-3250900-1304113432 -o /tmp/evaluation-result-3250900-1304113432/solver-3250900-1304113432 -C 4800 -W 1300 -M 15500  plingeling -t 4 HOME/instance-3250900-1304113432.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= f34a5dd1cdce70a322dd395bcb5f1303
RANDOM SEED=175682419

node134.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 7186.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      30110428 kB
Buffers:        995256 kB
Cached:        1370000 kB
SwapCached:       4464 kB
Active:         143064 kB
Inactive:      2299812 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      30110428 kB
SwapTotal:    67111528 kB
SwapFree:     67102692 kB
Dirty:            9888 kB
Writeback:           0 kB
AnonPages:       74592 kB
Mapped:          15300 kB
Slab:           334948 kB
PageTables:       4852 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   224628 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71576 MiB
End job on node134 at 2011-04-30 00:03:56