Trace number 3266302

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587? (TO) 4800.34 1201.92

General information on the benchmark

NameSAT02/handmade/pehoushek/plainoldcnf/
5cnf_3900_3900_220.shuffled.cnf
MD5SUMf0b7e0675915478b15cab3695fd0fd62
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.035994
Satisfiable
(Un)Satisfiability was proved
Number of variables220
Number of clauses3432
Sum of the clauses size17000
Maximum clause length5
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32
Number of clauses of size 4156
Number of clauses of size 53274
Number of clauses of size over 50

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3266302-1304108455/watcher-3266302-1304108455 -o /tmp/evaluation-result-3266302-1304108455/solver-3266302-1304108455 -C 4800 -W 1300 -M 15500 plingeling -t 4 HOME/instance-3266302-1304108455.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.07 8.08 8.05 5/179 10123
/proc/meminfo: memFree=30800768/32951132 swapFree=67080208/67111528
[pid=10123] ppid=10121 vsize=9492 CPUtime=0 cores=0,2,4,6
/proc/10123/stat : 10123 (plingeling) R 10121 10123 9622 0 -1 4202496 434 1251 0 0 0 0 0 0 25 0 1 0 522445107 9719808 363 33554432000 4194304 4417332 140737227235632 18446744073709551615 210647146240 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/10123/statm: 2373 363 147 55 0 259 0

[startup+0.121894 s]
/proc/loadavg: 8.07 8.08 8.05 5/179 10123
/proc/meminfo: memFree=30800768/32951132 swapFree=67080208/67111528
[pid=10123] ppid=10121 vsize=54200 CPUtime=0.45 cores=0,2,4,6
/proc/10123/stat : 10123 (plingeling) S 10121 10123 9622 0 -1 4202496 1866 1251 0 0 45 0 0 0 25 0 5 0 522445107 55500800 1355 33554432000 4194304 4417332 140737227235632 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/10123/statm: 13550 1355 178 55 0 11436 0
Current children cumulated CPU time (s) 0.45
Current children cumulated vsize (KiB) 54200

[startup+0.200882 s]
/proc/loadavg: 8.07 8.08 8.05 5/179 10123
/proc/meminfo: memFree=30800768/32951132 swapFree=67080208/67111528
[pid=10123] ppid=10121 vsize=55524 CPUtime=0.76 cores=0,2,4,6
/proc/10123/stat : 10123 (plingeling) S 10121 10123 9622 0 -1 4202496 2720 1251 0 0 76 0 0 0 25 0 5 0 522445107 56856576 1695 33554432000 4194304 4417332 140737227235632 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/10123/statm: 13881 1695 178 55 0 11767 0
Current children cumulated CPU time (s) 0.76
Current children cumulated vsize (KiB) 55524

[startup+0.300865 s]
/proc/loadavg: 8.07 8.08 8.05 5/179 10123
/proc/meminfo: memFree=30800768/32951132 swapFree=67080208/67111528
[pid=10123] ppid=10121 vsize=55792 CPUtime=1.16 cores=0,2,4,6
/proc/10123/stat : 10123 (plingeling) S 10121 10123 9622 0 -1 4202496 3623 1251 0 0 116 0 0 0 25 0 5 0 522445107 57131008 1765 33554432000 4194304 4417332 140737227235632 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/10123/statm: 13948 1765 178 55 0 11834 0
Current children cumulated CPU time (s) 1.16
Current children cumulated vsize (KiB) 55792

[startup+0.700799 s]
/proc/loadavg: 8.07 8.08 8.05 5/179 10123
/proc/meminfo: memFree=30800768/32951132 swapFree=67080208/67111528
[pid=10123] ppid=10121 vsize=58732 CPUtime=2.76 cores=0,2,4,6
/proc/10123/stat : 10123 (plingeling) S 10121 10123 9622 0 -1 4202496 7446 1251 0 0 276 0 0 0 25 0 5 0 522445107 60141568 2500 33554432000 4194304 4417332 140737227235632 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/10123/statm: 14683 2500 178 55 0 12569 0
Current children cumulated CPU time (s) 2.76
Current children cumulated vsize (KiB) 58732

[startup+1.50067 s]
/proc/loadavg: 8.07 8.08 8.05 9/184 10133
/proc/meminfo: memFree=30790692/32951132 swapFree=67080208/67111528
[pid=10123] ppid=10121 vsize=63164 CPUtime=5.96 cores=0,2,4,6
/proc/10123/stat : 10123 (plingeling) S 10121 10123 9622 0 -1 4202496 15075 1251 0 0 595 1 0 0 25 0 5 0 522445107 64679936 3608 33554432000 4194304 4417332 140737227235632 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/10123/statm: 15791 3608 178 55 0 13677 0
[pid=10123/tid=10130] ppid=10121 vsize=63164 CPUtime=1.48 cores=0,2,4,6
/proc/10123/task/10130/stat : 10130 (plingeling) R 10121 10123 9622 0 -1 4202560 3645 1251 0 0 148 0 0 0 25 0 5 0 522445108 64679936 3608 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364942 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=10123/tid=10131] ppid=10121 vsize=63164 CPUtime=1.48 cores=0,2,4,6
/proc/10123/task/10131/stat : 10131 (plingeling) R 10121 10123 9622 0 -1 4202560 3481 1251 0 0 148 0 0 0 25 0 5 0 522445108 64679936 3608 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364556 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10123/tid=10132] ppid=10121 vsize=63164 CPUtime=1.48 cores=0,2,4,6
/proc/10123/task/10132/stat : 10132 (plingeling) R 10121 10123 9622 0 -1 4202560 3626 1251 0 0 148 0 0 0 25 0 5 0 522445108 64679936 3608 33554432000 4194304 4417332 140737227235632 18446744073709551615 4366573 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10123/tid=10133] ppid=10121 vsize=63164 CPUtime=1.48 cores=0,2,4,6
/proc/10123/task/10133/stat : 10133 (plingeling) R 10121 10123 9622 0 -1 4202560 3713 1251 0 0 148 0 0 0 25 0 5 0 522445108 64679936 3608 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 5.96
Current children cumulated vsize (KiB) 63164

[startup+3.1004 s]
/proc/loadavg: 8.07 8.08 8.05 9/184 10133
/proc/meminfo: memFree=30784860/32951132 swapFree=67080208/67111528
[pid=10123] ppid=10121 vsize=63536 CPUtime=12.35 cores=0,2,4,6
/proc/10123/stat : 10123 (plingeling) S 10121 10123 9622 0 -1 4202496 27646 1251 0 0 1232 3 0 0 25 0 5 0 522445107 65060864 3679 33554432000 4194304 4417332 140737227235632 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/10123/statm: 15884 3679 179 55 0 13770 0
[pid=10123/tid=10130] ppid=10121 vsize=63536 CPUtime=3.09 cores=0,2,4,6
/proc/10123/task/10130/stat : 10130 (plingeling) R 10121 10123 9622 0 -1 4202560 6892 1251 0 0 308 1 0 0 25 0 5 0 522445108 65060864 3679 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364732 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=10123/tid=10131] ppid=10121 vsize=63536 CPUtime=3.07 cores=0,2,4,6
/proc/10123/task/10131/stat : 10131 (plingeling) R 10121 10123 9622 0 -1 4202560 6630 1251 0 0 307 0 0 0 25 0 5 0 522445108 65060864 3679 33554432000 4194304 4417332 140737227235632 18446744073709551615 18446744073699065904 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10123/tid=10132] ppid=10121 vsize=63536 CPUtime=3.07 cores=0,2,4,6
/proc/10123/task/10132/stat : 10132 (plingeling) R 10121 10123 9622 0 -1 4202560 7021 1251 0 0 307 0 0 0 25 0 5 0 522445108 65060864 3679 33554432000 4194304 4417332 140737227235632 18446744073709551615 4237349 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10123/tid=10133] ppid=10121 vsize=63536 CPUtime=3.08 cores=0,2,4,6
/proc/10123/task/10133/stat : 10133 (plingeling) R 10121 10123 9622 0 -1 4202560 6493 1251 0 0 308 0 0 0 25 0 5 0 522445108 65060864 3679 33554432000 4194304 4417332 140737227235632 18446744073709551615 4353957 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 12.35
Current children cumulated vsize (KiB) 63536

[startup+6.30086 s]
/proc/loadavg: 8.07 8.08 8.05 9/184 10133
/proc/meminfo: memFree=30774956/32951132 swapFree=67080208/67111528
[pid=10123] ppid=10121 vsize=70120 CPUtime=25.02 cores=0,2,4,6
/proc/10123/stat : 10123 (plingeling) S 10121 10123 9622 0 -1 4202496 47108 1251 0 0 2494 8 0 0 25 0 5 0 522445107 71802880 5348 33554432000 4194304 4417332 140737227235632 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/10123/statm: 17530 5348 179 55 0 15416 0
[pid=10123/tid=10130] ppid=10121 vsize=70120 CPUtime=6.17 cores=0,2,4,6
/proc/10123/task/10130/stat : 10130 (plingeling) R 10121 10123 9622 0 -1 4202560 11698 1251 0 0 614 3 0 0 25 0 5 0 522445108 71802880 5348 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=10123/tid=10131] ppid=10121 vsize=70120 CPUtime=6.28 cores=0,2,4,6
/proc/10123/task/10131/stat : 10131 (plingeling) R 10121 10123 9622 0 -1 4202560 11345 1251 0 0 626 2 0 0 25 0 5 0 522445108 71802880 5348 33554432000 4194304 4417332 140737227235632 18446744073709551615 4366569 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10123/tid=10132] ppid=10121 vsize=70120 CPUtime=6.26 cores=0,2,4,6
/proc/10123/task/10132/stat : 10132 (plingeling) R 10121 10123 9622 0 -1 4202560 11896 1251 0 0 625 1 0 0 25 0 5 0 522445108 71802880 5348 33554432000 4194304 4417332 140737227235632 18446744073709551615 4365003 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10123/tid=10133] ppid=10121 vsize=70120 CPUtime=6.29 cores=0,2,4,6
/proc/10123/task/10133/stat : 10133 (plingeling) R 10121 10123 9622 0 -1 4202560 11559 1251 0 0 627 2 0 0 25 0 5 0 522445108 71802880 5348 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364695 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 25.02
Current children cumulated vsize (KiB) 70120

[startup+12.7008 s]
/proc/loadavg: 8.06 8.08 8.05 9/184 10133
/proc/meminfo: memFree=30765192/32951132 swapFree=67080208/67111528
[pid=10123] ppid=10121 vsize=75268 CPUtime=50.57 cores=0,2,4,6
/proc/10123/stat : 10123 (plingeling) S 10121 10123 9622 0 -1 4202496 84509 1251 0 0 5042 15 0 0 25 0 5 0 522445107 77074432 6603 33554432000 4194304 4417332 140737227235632 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/10123/statm: 18817 6603 179 55 0 16703 0
[pid=10123/tid=10130] ppid=10121 vsize=75268 CPUtime=12.57 cores=0,2,4,6
/proc/10123/task/10130/stat : 10130 (plingeling) R 10121 10123 9622 0 -1 4202560 21925 1251 0 0 1252 5 0 0 25 0 5 0 522445108 77074432 6603 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=10123/tid=10131] ppid=10121 vsize=75268 CPUtime=12.67 cores=0,2,4,6
/proc/10123/task/10131/stat : 10131 (plingeling) R 10121 10123 9622 0 -1 4202560 20832 1251 0 0 1264 3 0 0 25 0 5 0 522445108 77074432 6603 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364942 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10123/tid=10132] ppid=10121 vsize=75268 CPUtime=12.63 cores=0,2,4,6
/proc/10123/task/10132/stat : 10132 (plingeling) R 10121 10123 9622 0 -1 4202560 21297 1251 0 0 1260 3 0 0 25 0 5 0 522445108 77074432 6603 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10123/tid=10133] ppid=10121 vsize=75268 CPUtime=12.68 cores=0,2,4,6
/proc/10123/task/10133/stat : 10133 (plingeling) R 10121 10123 9622 0 -1 4202560 19845 1251 0 0 1265 3 0 0 25 0 5 0 522445108 77074432 6603 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364568 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 50.57
Current children cumulated vsize (KiB) 75268

[startup+25.5006 s]
/proc/loadavg: 8.11 8.09 8.05 9/184 10134
/proc/meminfo: memFree=30746332/32951132 swapFree=67080208/67111528
[pid=10123] ppid=10121 vsize=93452 CPUtime=101.71 cores=0,2,4,6
/proc/10123/stat : 10123 (plingeling) S 10121 10123 9622 0 -1 4202496 140696 1251 0 0 10146 25 0 0 25 0 5 0 522445107 95694848 11160 33554432000 4194304 4417332 140737227235632 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/10123/statm: 23363 11160 179 55 0 21249 0
[pid=10123/tid=10130] ppid=10121 vsize=93452 CPUtime=25.36 cores=0,2,4,6
/proc/10123/task/10130/stat : 10130 (plingeling) R 10121 10123 9622 0 -1 4202560 35682 1251 0 0 2529 7 0 0 25 0 5 0 522445108 95694848 11160 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364736 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=10123/tid=10131] ppid=10121 vsize=93452 CPUtime=25.46 cores=0,2,4,6
/proc/10123/task/10131/stat : 10131 (plingeling) R 10121 10123 9622 0 -1 4202560 35366 1251 0 0 2540 6 0 0 25 0 5 0 522445108 95694848 11160 33554432000 4194304 4417332 140737227235632 18446744073709551615 4251076 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10123/tid=10132] ppid=10121 vsize=93452 CPUtime=25.36 cores=0,2,4,6
/proc/10123/task/10132/stat : 10132 (plingeling) R 10121 10123 9622 0 -1 4202560 36439 1251 0 0 2531 5 0 0 25 0 5 0 522445108 95694848 11160 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364687 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10123/tid=10133] ppid=10121 vsize=93452 CPUtime=25.48 cores=0,2,4,6
/proc/10123/task/10133/stat : 10133 (plingeling) R 10121 10123 9622 0 -1 4202560 32599 1251 0 0 2543 5 0 0 25 0 5 0 522445108 95694848 11160 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364800 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 101.71
Current children cumulated vsize (KiB) 93452

[startup+51.1003 s]
/proc/loadavg: 8.13 8.10 8.05 9/184 10135
/proc/meminfo: memFree=30744200/32951132 swapFree=67080208/67111528
[pid=10123] ppid=10121 vsize=104716 CPUtime=203.93 cores=0,2,4,6
/proc/10123/stat : 10123 (plingeling) S 10121 10123 9622 0 -1 4202496 234784 1251 0 0 20345 48 0 0 25 0 5 0 522445107 107229184 13997 33554432000 4194304 4417332 140737227235632 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/10123/statm: 26179 13997 179 55 0 24065 0
[pid=10123/tid=10130] ppid=10121 vsize=104716 CPUtime=50.92 cores=0,2,4,6
/proc/10123/task/10130/stat : 10130 (plingeling) R 10121 10123 9622 0 -1 4202560 57904 1251 0 0 5074 18 0 0 25 0 5 0 522445108 107229184 13997 33554432000 4194304 4417332 140737227235632 18446744073709551615 4353774 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=10123/tid=10131] ppid=10121 vsize=104716 CPUtime=51.06 cores=0,2,4,6
/proc/10123/task/10131/stat : 10131 (plingeling) R 10121 10123 9622 0 -1 4202560 58164 1251 0 0 5096 10 0 0 25 0 5 0 522445108 107229184 13997 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364652 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10123/tid=10132] ppid=10121 vsize=104716 CPUtime=50.83 cores=0,2,4,6
/proc/10123/task/10132/stat : 10132 (plingeling) R 10121 10123 9622 0 -1 4202560 64672 1251 0 0 5073 10 0 0 25 0 5 0 522445108 107229184 13997 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10123/tid=10133] ppid=10121 vsize=104716 CPUtime=51.08 cores=0,2,4,6
/proc/10123/task/10133/stat : 10133 (plingeling) R 10121 10123 9622 0 -1 4202560 53434 1251 0 0 5099 9 0 0 25 0 5 0 522445108 107229184 13997 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364479 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 203.93
Current children cumulated vsize (KiB) 104716

[startup+102.301 s]
/proc/loadavg: 8.11 8.09 8.05 9/184 10136
/proc/meminfo: memFree=30731528/32951132 swapFree=67080208/67111528
[pid=10123] ppid=10121 vsize=115812 CPUtime=408.47 cores=0,2,4,6
/proc/10123/stat : 10123 (plingeling) S 10121 10123 9622 0 -1 4202496 378782 1251 0 0 40774 73 0 0 25 0 5 0 522445107 118591488 16771 33554432000 4194304 4417332 140737227235632 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/10123/statm: 28953 16771 179 55 0 26839 0
[pid=10123/tid=10130] ppid=10121 vsize=115812 CPUtime=102.1 cores=0,2,4,6
/proc/10123/task/10130/stat : 10130 (plingeling) R 10121 10123 9622 0 -1 4202560 100574 1251 0 0 10184 26 0 0 25 0 5 0 522445108 118591488 16771 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364750 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=10123/tid=10131] ppid=10121 vsize=115812 CPUtime=102.26 cores=0,2,4,6
/proc/10123/task/10131/stat : 10131 (plingeling) R 10121 10123 9622 0 -1 4202560 93603 1251 0 0 10211 15 0 0 25 0 5 0 522445108 118591488 16771 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10123/tid=10132] ppid=10121 vsize=115812 CPUtime=101.79 cores=0,2,4,6
/proc/10123/task/10132/stat : 10132 (plingeling) R 10121 10123 9622 0 -1 4202560 99478 1251 0 0 10163 16 0 0 25 0 5 0 522445108 118591488 16771 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364536 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10123/tid=10133] ppid=10121 vsize=115812 CPUtime=102.28 cores=0,2,4,6
/proc/10123/task/10133/stat : 10133 (plingeling) R 10121 10123 9622 0 -1 4202560 84517 1251 0 0 10214 14 0 0 25 0 5 0 522445108 118591488 16771 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364750 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 408.47
Current children cumulated vsize (KiB) 115812

[startup+162.301 s]
/proc/loadavg: 8.14 8.11 8.06 9/184 10137
/proc/meminfo: memFree=30747752/32951132 swapFree=67080208/67111528
[pid=10123] ppid=10121 vsize=118700 CPUtime=648.11 cores=0,2,4,6
/proc/10123/stat : 10123 (plingeling) S 10121 10123 9622 0 -1 4202496 509074 1251 0 0 64714 97 0 0 25 0 5 0 522445107 121548800 17493 33554432000 4194304 4417332 140737227235632 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/10123/statm: 29675 17493 179 55 0 27561 0
[pid=10123/tid=10130] ppid=10121 vsize=118700 CPUtime=162.07 cores=0,2,4,6
/proc/10123/task/10130/stat : 10130 (plingeling) R 10121 10123 9622 0 -1 4202560 135712 1251 0 0 16174 33 0 0 25 0 5 0 522445108 121548800 17493 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364541 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=10123/tid=10131] ppid=10121 vsize=118700 CPUtime=162.24 cores=0,2,4,6
/proc/10123/task/10131/stat : 10131 (plingeling) R 10121 10123 9622 0 -1 4202560 125583 1251 0 0 16204 20 0 0 25 0 5 0 522445108 121548800 17493 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364482 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10123/tid=10132] ppid=10121 vsize=118700 CPUtime=161.48 cores=0,2,4,6
/proc/10123/task/10132/stat : 10132 (plingeling) R 10121 10123 9622 0 -1 4202560 131440 1251 0 0 16127 21 0 0 25 0 5 0 522445108 121548800 17493 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364750 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10123/tid=10133] ppid=10121 vsize=118700 CPUtime=162.27 cores=0,2,4,6
/proc/10123/task/10133/stat : 10133 (plingeling) R 10121 10123 9622 0 -1 4202560 115729 1251 0 0 16207 20 0 0 25 0 5 0 522445108 121548800 17493 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364494 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 648.11
Current children cumulated vsize (KiB) 118700


################
# More data... #
################

[pid=10123/tid=10131] ppid=10121 vsize=198396 CPUtime=702.13 cores=0,2,4,6
/proc/10123/task/10131/stat : 10131 (plingeling) R 10121 10123 9622 0 -1 4202560 343368 1251 0 0 70156 57 0 0 25 0 5 0 522445108 203157504 37183 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364742 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10123/tid=10132] ppid=10121 vsize=198396 CPUtime=698.68 cores=0,2,4,6
/proc/10123/task/10132/stat : 10132 (plingeling) R 10121 10123 9622 0 -1 4202560 358523 1251 0 0 69807 61 0 0 25 0 5 0 522445108 203157504 37183 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364536 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10123/tid=10133] ppid=10121 vsize=198396 CPUtime=702.23 cores=0,2,4,6
/proc/10123/task/10133/stat : 10133 (plingeling) R 10121 10123 9622 0 -1 4202560 314616 1251 0 0 70169 54 0 0 25 0 5 0 522445108 203157504 37183 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364479 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2804.92
Current children cumulated vsize (KiB) 198396

[startup+762.301 s]
/proc/loadavg: 8.16 8.09 8.06 9/184 10153
/proc/meminfo: memFree=30665144/32951132 swapFree=67080208/67111528
[pid=10123] ppid=10121 vsize=166860 CPUtime=3044.57 cores=0,2,4,6
/proc/10123/stat : 10123 (plingeling) S 10121 10123 9622 0 -1 4202496 1458361 1251 0 0 304188 269 0 0 25 0 5 0 522445107 170864640 29500 33554432000 4194304 4417332 140737227235632 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/10123/statm: 41715 29500 179 55 0 39601 0
[pid=10123/tid=10130] ppid=10121 vsize=166860 CPUtime=761.82 cores=0,2,4,6
/proc/10123/task/10130/stat : 10130 (plingeling) R 10121 10123 9622 0 -1 4202560 382582 1251 0 0 76096 86 0 0 25 0 5 0 522445108 170864640 29500 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364549 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=10123/tid=10131] ppid=10121 vsize=166860 CPUtime=762.11 cores=0,2,4,6
/proc/10123/task/10131/stat : 10131 (plingeling) R 10121 10123 9622 0 -1 4202560 367587 1251 0 0 76151 60 0 0 25 0 5 0 522445108 170864640 29500 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364796 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10123/tid=10132] ppid=10121 vsize=166860 CPUtime=758.37 cores=0,2,4,6
/proc/10123/task/10132/stat : 10132 (plingeling) R 10121 10123 9622 0 -1 4202560 376621 1251 0 0 75772 65 0 0 25 0 5 0 522445108 170864640 29500 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10123/tid=10133] ppid=10121 vsize=166860 CPUtime=762.22 cores=0,2,4,6
/proc/10123/task/10133/stat : 10133 (plingeling) R 10121 10123 9622 0 -1 4202560 330961 1251 0 0 76166 56 0 0 25 0 5 0 522445108 170864640 29500 33554432000 4194304 4417332 140737227235632 18446744073709551615 4366384 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3044.57
Current children cumulated vsize (KiB) 166860

[startup+822.301 s]
/proc/loadavg: 8.17 8.10 8.06 9/184 10192
/proc/meminfo: memFree=30802848/32951132 swapFree=67080208/67111528
[pid=10123] ppid=10121 vsize=205064 CPUtime=3284.03 cores=0,2,4,6
/proc/10123/stat : 10123 (plingeling) S 10121 10123 9622 0 -1 4202496 1520138 1251 0 0 328117 286 0 0 25 0 5 0 522445107 209985536 39084 33554432000 4194304 4417332 140737227235632 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/10123/statm: 51266 39084 179 55 0 49152 0
[pid=10123/tid=10130] ppid=10121 vsize=205064 CPUtime=821.76 cores=0,2,4,6
/proc/10123/task/10130/stat : 10130 (plingeling) R 10121 10123 9622 0 -1 4202560 398381 1251 0 0 82086 90 0 0 25 0 5 0 522445108 209985536 39084 33554432000 4194304 4417332 140737227235632 18446744073709551615 4354410 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=10123/tid=10131] ppid=10121 vsize=205064 CPUtime=822.08 cores=0,2,4,6
/proc/10123/task/10131/stat : 10131 (plingeling) R 10121 10123 9622 0 -1 4202560 383886 1251 0 0 82145 63 0 0 25 0 5 0 522445108 209985536 39084 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364656 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10123/tid=10132] ppid=10121 vsize=205064 CPUtime=818.08 cores=0,2,4,6
/proc/10123/task/10132/stat : 10132 (plingeling) R 10121 10123 9622 0 -1 4202560 393589 1251 0 0 81740 68 0 0 25 0 5 0 522445108 209985536 39084 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10123/tid=10133] ppid=10121 vsize=205064 CPUtime=822.07 cores=0,2,4,6
/proc/10123/task/10133/stat : 10133 (plingeling) R 10121 10123 9622 0 -1 4202560 343672 1251 0 0 82144 63 0 0 25 0 5 0 522445108 209985536 39084 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3284.03
Current children cumulated vsize (KiB) 205064

[startup+882.307 s]
/proc/loadavg: 8.14 8.11 8.07 9/184 10194
/proc/meminfo: memFree=30801200/32951132 swapFree=67080208/67111528
[pid=10123] ppid=10121 vsize=188176 CPUtime=3523.71 cores=0,2,4,6
/proc/10123/stat : 10123 (plingeling) S 10121 10123 9622 0 -1 4202496 1590201 1251 0 0 352070 301 0 0 25 0 5 0 522445107 192692224 34862 33554432000 4194304 4417332 140737227235632 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/10123/statm: 47044 34862 179 55 0 44930 0
[pid=10123/tid=10130] ppid=10121 vsize=188176 CPUtime=881.77 cores=0,2,4,6
/proc/10123/task/10130/stat : 10130 (plingeling) R 10121 10123 9622 0 -1 4202560 411490 1251 0 0 88084 93 0 0 25 0 5 0 522445108 192692224 34862 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364753 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=10123/tid=10131] ppid=10121 vsize=188176 CPUtime=882.07 cores=0,2,4,6
/proc/10123/task/10131/stat : 10131 (plingeling) R 10121 10123 9622 0 -1 4202560 405240 1251 0 0 88141 66 0 0 25 0 5 0 522445108 192692224 34862 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10123/tid=10132] ppid=10121 vsize=188176 CPUtime=877.79 cores=0,2,4,6
/proc/10123/task/10132/stat : 10132 (plingeling) R 10121 10123 9622 0 -1 4202560 411430 1251 0 0 87708 71 0 0 25 0 5 0 522445108 192692224 34862 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364448 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10123/tid=10133] ppid=10121 vsize=188176 CPUtime=882.04 cores=0,2,4,6
/proc/10123/task/10133/stat : 10133 (plingeling) R 10121 10123 9622 0 -1 4202560 361431 1251 0 0 88134 70 0 0 25 0 5 0 522445108 192692224 34862 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364470 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3523.71
Current children cumulated vsize (KiB) 188176

[startup+942.301 s]
/proc/loadavg: 8.06 8.09 8.06 9/184 10195
/proc/meminfo: memFree=30771292/32951132 swapFree=67080208/67111528
[pid=10123] ppid=10121 vsize=202800 CPUtime=3763.36 cores=0,2,4,6
/proc/10123/stat : 10123 (plingeling) S 10121 10123 9622 0 -1 4202496 1651431 1251 0 0 376023 313 0 0 25 0 5 0 522445107 207667200 38518 33554432000 4194304 4417332 140737227235632 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/10123/statm: 50700 38518 179 55 0 48586 0
[pid=10123/tid=10130] ppid=10121 vsize=202800 CPUtime=941.75 cores=0,2,4,6
/proc/10123/task/10130/stat : 10130 (plingeling) R 10121 10123 9622 0 -1 4202560 427102 1251 0 0 94080 95 0 0 25 0 5 0 522445108 207667200 38518 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=10123/tid=10131] ppid=10121 vsize=202800 CPUtime=942.06 cores=0,2,4,6
/proc/10123/task/10131/stat : 10131 (plingeling) R 10121 10123 9622 0 -1 4202560 425223 1251 0 0 94136 70 0 0 25 0 5 0 522445108 207667200 38518 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364479 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10123/tid=10132] ppid=10121 vsize=202800 CPUtime=937.5 cores=0,2,4,6
/proc/10123/task/10132/stat : 10132 (plingeling) R 10121 10123 9622 0 -1 4202560 424244 1251 0 0 93677 73 0 0 25 0 5 0 522445108 207667200 38518 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10123/tid=10133] ppid=10121 vsize=202800 CPUtime=942.01 cores=0,2,4,6
/proc/10123/task/10133/stat : 10133 (plingeling) R 10121 10123 9622 0 -1 4202560 374252 1251 0 0 94128 73 0 0 25 0 5 0 522445108 207667200 38518 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3763.36
Current children cumulated vsize (KiB) 202800

[startup+1002.3 s]
/proc/loadavg: 8.02 8.07 8.06 10/184 10197
/proc/meminfo: memFree=30760484/32951132 swapFree=67080208/67111528
[pid=10123] ppid=10121 vsize=210348 CPUtime=4003.02 cores=0,2,4,6
/proc/10123/stat : 10123 (plingeling) S 10121 10123 9622 0 -1 4202496 1725526 1251 0 0 399975 327 0 0 25 0 5 0 522445107 215396352 40138 33554432000 4194304 4417332 140737227235632 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/10123/statm: 52587 40138 179 55 0 50473 0
[pid=10123/tid=10130] ppid=10121 vsize=210348 CPUtime=1001.74 cores=0,2,4,6
/proc/10123/task/10130/stat : 10130 (plingeling) R 10121 10123 9622 0 -1 4202560 439724 1251 0 0 100077 97 0 0 25 0 5 0 522445108 215396352 40138 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364990 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=10123/tid=10131] ppid=10121 vsize=210348 CPUtime=1002.05 cores=0,2,4,6
/proc/10123/task/10131/stat : 10131 (plingeling) R 10121 10123 9622 0 -1 4202560 447416 1251 0 0 100131 74 0 0 25 0 5 0 522445108 215396352 40138 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364683 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10123/tid=10132] ppid=10121 vsize=210348 CPUtime=997.2 cores=0,2,4,6
/proc/10123/task/10132/stat : 10132 (plingeling) R 10121 10123 9622 0 -1 4202560 441486 1251 0 0 99644 76 0 0 25 0 5 0 522445108 215396352 40138 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364652 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10123/tid=10133] ppid=10121 vsize=210348 CPUtime=1001.98 cores=0,2,4,6
/proc/10123/task/10133/stat : 10133 (plingeling) R 10121 10123 9622 0 -1 4202560 396290 1251 0 0 100120 78 0 0 25 0 5 0 522445108 215396352 40138 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4003.02
Current children cumulated vsize (KiB) 210348

[startup+1062.3 s]
/proc/loadavg: 8.00 8.05 8.05 9/184 10198
/proc/meminfo: memFree=30728968/32951132 swapFree=67080208/67111528
[pid=10123] ppid=10121 vsize=229264 CPUtime=4242.68 cores=0,2,4,6
/proc/10123/stat : 10123 (plingeling) S 10121 10123 9622 0 -1 4202496 1787510 1251 0 0 423929 339 0 0 25 0 5 0 522445107 234766336 45134 33554432000 4194304 4417332 140737227235632 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/10123/statm: 57316 45134 179 55 0 55202 0
[pid=10123/tid=10130] ppid=10121 vsize=229264 CPUtime=1061.75 cores=0,2,4,6
/proc/10123/task/10130/stat : 10130 (plingeling) R 10121 10123 9622 0 -1 4202560 456331 1251 0 0 106074 101 0 0 25 0 5 0 522445108 234766336 45134 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=10123/tid=10131] ppid=10121 vsize=229264 CPUtime=1062.03 cores=0,2,4,6
/proc/10123/task/10131/stat : 10131 (plingeling) R 10121 10123 9622 0 -1 4202560 465722 1251 0 0 106126 77 0 0 25 0 5 0 522445108 234766336 45134 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10123/tid=10132] ppid=10121 vsize=229264 CPUtime=1056.92 cores=0,2,4,6
/proc/10123/task/10132/stat : 10132 (plingeling) R 10121 10123 9622 0 -1 4202560 455807 1251 0 0 105614 78 0 0 25 0 5 0 522445108 234766336 45134 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10123/tid=10133] ppid=10121 vsize=229264 CPUtime=1061.96 cores=0,2,4,6
/proc/10123/task/10133/stat : 10133 (plingeling) R 10121 10123 9622 0 -1 4202560 409040 1251 0 0 106114 82 0 0 25 0 5 0 522445108 234766336 45134 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364942 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4242.68
Current children cumulated vsize (KiB) 229264

[startup+1122.3 s]
/proc/loadavg: 8.00 8.04 8.05 9/184 10200
/proc/meminfo: memFree=30725836/32951132 swapFree=67080208/67111528
[pid=10123] ppid=10121 vsize=214948 CPUtime=4482.33 cores=0,2,4,6
/proc/10123/stat : 10123 (plingeling) S 10121 10123 9622 0 -1 4202496 1847829 1251 0 0 447882 351 0 0 25 0 5 0 522445107 220106752 41555 33554432000 4194304 4417332 140737227235632 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/10123/statm: 53737 41555 179 55 0 51623 0
[pid=10123/tid=10130] ppid=10121 vsize=214948 CPUtime=1121.74 cores=0,2,4,6
/proc/10123/task/10130/stat : 10130 (plingeling) R 10121 10123 9622 0 -1 4202560 472248 1251 0 0 112070 104 0 0 25 0 5 0 522445108 220106752 41555 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=10123/tid=10131] ppid=10121 vsize=214948 CPUtime=1122.02 cores=0,2,4,6
/proc/10123/task/10131/stat : 10131 (plingeling) R 10121 10123 9622 0 -1 4202560 484851 1251 0 0 112122 80 0 0 25 0 5 0 522445108 220106752 41555 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10123/tid=10132] ppid=10121 vsize=214948 CPUtime=1116.61 cores=0,2,4,6
/proc/10123/task/10132/stat : 10132 (plingeling) R 10121 10123 9622 0 -1 4202560 467893 1251 0 0 111581 80 0 0 25 0 5 0 522445108 220106752 41555 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10123/tid=10133] ppid=10121 vsize=214948 CPUtime=1121.93 cores=0,2,4,6
/proc/10123/task/10133/stat : 10133 (plingeling) R 10121 10123 9622 0 -1 4202560 422227 1251 0 0 112108 85 0 0 25 0 5 0 522445108 220106752 41555 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364473 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4482.33
Current children cumulated vsize (KiB) 214948

[startup+1182.3 s]
/proc/loadavg: 8.04 8.05 8.05 9/184 10202
/proc/meminfo: memFree=30743424/32951132 swapFree=67080208/67111528
[pid=10123] ppid=10121 vsize=186836 CPUtime=4722 cores=0,2,4,6
/proc/10123/stat : 10123 (plingeling) S 10121 10123 9622 0 -1 4202496 1916741 1251 0 0 471837 363 0 0 25 0 5 0 522445107 191320064 34527 33554432000 4194304 4417332 140737227235632 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/10123/statm: 46709 34527 179 55 0 44595 0
[pid=10123/tid=10130] ppid=10121 vsize=186836 CPUtime=1181.74 cores=0,2,4,6
/proc/10123/task/10130/stat : 10130 (plingeling) R 10121 10123 9622 0 -1 4202560 487390 1251 0 0 118067 107 0 0 25 0 5 0 522445108 191320064 34527 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364473 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=10123/tid=10131] ppid=10121 vsize=186836 CPUtime=1182 cores=0,2,4,6
/proc/10123/task/10131/stat : 10131 (plingeling) R 10121 10123 9622 0 -1 4202560 505335 1251 0 0 118117 83 0 0 25 0 5 0 522445108 191320064 34527 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364671 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10123/tid=10132] ppid=10121 vsize=186836 CPUtime=1176.32 cores=0,2,4,6
/proc/10123/task/10132/stat : 10132 (plingeling) R 10121 10123 9622 0 -1 4202560 484214 1251 0 0 117549 83 0 0 25 0 5 0 522445108 191320064 34527 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364750 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10123/tid=10133] ppid=10121 vsize=186836 CPUtime=1181.92 cores=0,2,4,6
/proc/10123/task/10133/stat : 10133 (plingeling) R 10121 10123 9622 0 -1 4202560 439192 1251 0 0 118103 89 0 0 25 0 5 0 522445108 191320064 34527 33554432000 4194304 4417332 140737227235632 18446744073709551615 4366303 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4722
Current children cumulated vsize (KiB) 186836



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1201.9 s]
/proc/loadavg: 8.02 8.04 8.04 9/184 10202
/proc/meminfo: memFree=30737704/32951132 swapFree=67080208/67111528
[pid=10123] ppid=10121 vsize=210748 CPUtime=4800.29 cores=0,2,4,6
/proc/10123/stat : 10123 (plingeling) S 10121 10123 9622 0 -1 4202496 1935395 1251 0 0 479662 367 0 0 25 0 5 0 522445107 215805952 40505 33554432000 4194304 4417332 140737227235632 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/10123/statm: 52687 40505 179 55 0 50573 0
[pid=10123/tid=10130] ppid=10121 vsize=210748 CPUtime=1201.34 cores=0,2,4,6
/proc/10123/task/10130/stat : 10130 (plingeling) R 10121 10123 9622 0 -1 4202560 496754 1251 0 0 120025 109 0 0 25 0 5 0 522445108 215805952 40505 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364470 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=10123/tid=10131] ppid=10121 vsize=210748 CPUtime=1201.6 cores=0,2,4,6
/proc/10123/task/10131/stat : 10131 (plingeling) R 10121 10123 9622 0 -1 4202560 508821 1251 0 0 120076 84 0 0 25 0 5 0 522445108 215805952 40505 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364536 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10123/tid=10132] ppid=10121 vsize=210748 CPUtime=1195.82 cores=0,2,4,6
/proc/10123/task/10132/stat : 10132 (plingeling) R 10121 10123 9622 0 -1 4202560 487028 1251 0 0 119498 84 0 0 25 0 5 0 522445108 215805952 40505 33554432000 4194304 4417332 140737227235632 18446744073709551615 4365960 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10123/tid=10133] ppid=10121 vsize=210748 CPUtime=1201.52 cores=0,2,4,6
/proc/10123/task/10133/stat : 10133 (plingeling) R 10121 10123 9622 0 -1 4202560 442182 1251 0 0 120062 90 0 0 25 0 5 0 522445108 215805952 40505 33554432000 4194304 4417332 140737227235632 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.29
Current children cumulated vsize (KiB) 210748

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1201.92
CPU time (s): 4800.34
CPU user time (s): 4796.65
CPU system time (s): 3.68944
CPU usage (%): 399.389
Max. virtual memory (cumulated for all children) (KiB): 239172

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4796.65
system time used= 3.68944
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1936646
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 132
involuntary context switches= 21896

runsolver used 1.76873 second user time and 4.32834 second system time

The end

Launcher Data 

Begin job on node129 at 2011-04-29 22:20:55
IDJOB=3266302
IDBENCH=87453
IDSOLVER=1605
FILE ID=node129/3266302-1304108455
RUNJOBID= node129-1304107926-9936
PBS_JOBID= 13170649
Free space on /tmp= 66368 MiB

SOLVER NAME= Plingeling 587
BENCH NAME= SAT02/handmade/pehoushek/plainoldcnf/5cnf_3900_3900_220.shuffled.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3266302-1304108455/watcher-3266302-1304108455 -o /tmp/evaluation-result-3266302-1304108455/solver-3266302-1304108455 -C 4800 -W 1300 -M 15500  plingeling -t 4 HOME/instance-3266302-1304108455.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= f0b7e0675915478b15cab3695fd0fd62
RANDOM SEED=545329693

node129.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      30800916 kB
Buffers:        363184 kB
Cached:        1166276 kB
SwapCached:       5960 kB
Active:         303124 kB
Inactive:      1460252 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      30800916 kB
SwapTotal:    67111528 kB
SwapFree:     67080208 kB
Dirty:             420 kB
Writeback:           0 kB
AnonPages:      230464 kB
Mapped:          13012 kB
Slab:           323632 kB
PageTables:       5232 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS:   443716 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 66368 MiB
End job on node129 at 2011-04-29 22:40:59