Trace number 3371290

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 4800.51 1212.47

General information on the benchmark

NameSAT-Race-2010-CNF/hardware-verification/
velev/1dlx_c_iq60_a.cnf
MD5SUM1e60a63f1ba68b9b3c423ca413b4fa2b
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark802.03
Satisfiable
(Un)Satisfiability was proved
Number of variables651875
Number of clauses9927770
Sum of the clauses size28821360
Maximum clause length195
Minimum clause length1
Number of clauses of size 11
Number of clauses of size 28783867
Number of clauses of size 3839934
Number of clauses of size 410777
Number of clauses of size 570916
Number of clauses of size over 5222275

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3371290-1305432733/watcher-3371290-1305432733 -o /tmp/evaluation-result-3371290-1305432733/solver-3371290-1305432733 -C 4800 -W 1300 -M 15500 plingeling -t 4 HOME/instance-3371290-1305432733.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.66 5.92 3.32 5/368 24431
/proc/meminfo: memFree=21575132/32950928 swapFree=65730864/67111528
[pid=24431] ppid=24429 vsize=16648 CPUtime=0 cores=1,3,5,7
/proc/24431/stat : 24431 (plingeling) R 24429 24431 23830 0 -1 4202496 2230 1254 0 0 0 0 0 0 23 0 1 0 733504239 17047552 2159 33554432000 4194304 4417452 140736389637568 18446744073709551615 260905090827 0 0 4096 17442 0 0 0 17 1 0 0 0
/proc/24431/statm: 4162 2159 148 55 0 2048 0

[startup+0.054112 s]
/proc/loadavg: 7.66 5.92 3.32 5/368 24431
/proc/meminfo: memFree=21575132/32950928 swapFree=65730864/67111528
[pid=24431] ppid=24429 vsize=24304 CPUtime=0.04 cores=1,3,5,7
/proc/24431/stat : 24431 (plingeling) R 24429 24431 23830 0 -1 4202496 4171 1254 0 0 4 0 0 0 23 0 1 0 733504239 24887296 4100 33554432000 4194304 4417452 140736389637568 18446744073709551615 4235968 0 0 4096 17442 0 0 0 17 1 0 0 0
/proc/24431/statm: 6076 4100 149 55 0 3962 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 24304

[startup+0.101102 s]
/proc/loadavg: 7.66 5.92 3.32 5/368 24431
/proc/meminfo: memFree=21575132/32950928 swapFree=65730864/67111528
[pid=24431] ppid=24429 vsize=31604 CPUtime=0.09 cores=1,3,5,7
/proc/24431/stat : 24431 (plingeling) R 24429 24431 23830 0 -1 4202496 5996 1254 0 0 9 0 0 0 23 0 1 0 733504239 32362496 5925 33554432000 4194304 4417452 140736389637568 18446744073709551615 4231544 0 0 4096 17442 0 0 0 17 1 0 0 0
/proc/24431/statm: 7901 5925 149 55 0 5787 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 31604

[startup+0.301065 s]
/proc/loadavg: 7.66 5.92 3.32 5/368 24431
/proc/meminfo: memFree=21575132/32950928 swapFree=65730864/67111528
[pid=24431] ppid=24429 vsize=50824 CPUtime=0.29 cores=1,3,5,7
/proc/24431/stat : 24431 (plingeling) R 24429 24431 23830 0 -1 4202496 10801 1254 0 0 28 1 0 0 24 0 1 0 733504239 52043776 10730 33554432000 4194304 4417452 140736389637568 18446744073709551615 4231636 0 0 4096 17442 0 0 0 17 1 0 0 0
/proc/24431/statm: 12706 10730 149 55 0 10592 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 50824

[startup+0.700993 s]
/proc/loadavg: 7.66 5.92 3.32 5/368 24431
/proc/meminfo: memFree=21575132/32950928 swapFree=65730864/67111528
[pid=24431] ppid=24429 vsize=117652 CPUtime=0.69 cores=1,3,5,7
/proc/24431/stat : 24431 (plingeling) R 24429 24431 23830 0 -1 4202496 27508 1254 0 0 66 3 0 0 25 0 1 0 733504239 120475648 27437 33554432000 4194304 4417452 140736389637568 18446744073709551615 4209599 0 0 4096 17442 0 0 0 17 1 0 0 0
/proc/24431/statm: 29413 27437 149 55 0 27299 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 117652

[startup+1.50085 s]
/proc/loadavg: 7.45 5.90 3.33 6/369 24437
/proc/meminfo: memFree=21446296/32950928 swapFree=65730864/67111528
[pid=24431] ppid=24429 vsize=218264 CPUtime=1.49 cores=1,3,5,7
/proc/24431/stat : 24431 (plingeling) R 24429 24431 23830 0 -1 4202496 52661 1254 0 0 142 7 0 0 25 0 1 0 733504239 223502336 52590 33554432000 4194304 4417452 140736389637568 18446744073709551615 4231313 0 0 4096 17442 0 0 0 17 1 0 0 0
/proc/24431/statm: 54566 52590 149 55 0 52452 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 218264

[startup+3.10055 s]
/proc/loadavg: 7.45 5.90 3.33 6/369 24437
/proc/meminfo: memFree=21260172/32950928 swapFree=65730864/67111528
[pid=24431] ppid=24429 vsize=419400 CPUtime=3.08 cores=1,3,5,7
/proc/24431/stat : 24431 (plingeling) R 24429 24431 23830 0 -1 4202496 102967 1254 0 0 293 15 0 0 25 0 1 0 733504239 429465600 102874 33554432000 4194304 4417452 140736389637568 18446744073709551615 4235978 0 0 4096 17442 0 0 0 17 1 0 0 0
/proc/24431/statm: 104850 102874 149 55 0 102736 0
Current children cumulated CPU time (s) 3.08
Current children cumulated vsize (KiB) 419400

[startup+6.30097 s]
/proc/loadavg: 7.33 5.90 3.34 6/369 24437
/proc/meminfo: memFree=20695960/32950928 swapFree=65730864/67111528
[pid=24431] ppid=24429 vsize=821428 CPUtime=6.29 cores=1,3,5,7
/proc/24431/stat : 24431 (plingeling) R 24429 24431 23830 0 -1 4202496 203576 1254 0 0 596 33 0 0 25 0 1 0 733504239 841142272 203381 33554432000 4194304 4417452 140736389637568 18446744073709551615 260905393920 0 0 4096 17442 0 0 0 17 1 0 0 0
/proc/24431/statm: 205357 203381 149 55 0 203243 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 821428

[startup+12.7008 s]
/proc/loadavg: 7.23 5.91 3.36 9/373 24442
/proc/meminfo: memFree=20086684/32950928 swapFree=65730864/67111528
[pid=24431] ppid=24429 vsize=1468276 CPUtime=15.69 cores=1,3,5,7
/proc/24431/stat : 24431 (plingeling) S 24429 24431 23830 0 -1 4202496 443851 1254 0 0 1501 68 0 0 25 0 5 0 733504239 1503514624 354857 33554432000 4194304 4417452 140736389637568 18446744073709551615 260917196773 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/24431/statm: 367069 354857 165 55 0 364955 0
[pid=24431/tid=24439] ppid=24429 vsize=1468276 CPUtime=1.01 cores=1,3,5,7
/proc/24431/task/24439/stat : 24439 (plingeling) R 24429 24431 23830 0 -1 4202560 9102 1254 0 0 99 2 0 0 25 0 5 0 733505409 1503514624 354857 33554432000 4194304 4417452 140736389637568 18446744073709551615 4282356 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=24431/tid=24440] ppid=24429 vsize=1468276 CPUtime=1 cores=1,3,5,7
/proc/24431/task/24440/stat : 24440 (plingeling) R 24429 24431 23830 0 -1 4202560 9078 1254 0 0 98 2 0 0 25 0 5 0 733505409 1503514624 354857 33554432000 4194304 4417452 140736389637568 18446744073709551615 4282467 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24431/tid=24441] ppid=24429 vsize=1468276 CPUtime=0.98 cores=1,3,5,7
/proc/24431/task/24441/stat : 24441 (plingeling) R 24429 24431 23830 0 -1 4202560 9098 1254 0 0 97 1 0 0 25 0 5 0 733505409 1503514624 354857 33554432000 4194304 4417452 140736389637568 18446744073709551615 4282361 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24431/tid=24442] ppid=24429 vsize=1468276 CPUtime=0.99 cores=1,3,5,7
/proc/24431/task/24442/stat : 24442 (plingeling) R 24429 24431 23830 0 -1 4202560 11640 1254 0 0 97 2 0 0 25 0 5 0 733505409 1503514624 354857 33554432000 4194304 4417452 140736389637568 18446744073709551615 4262481 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 15.69
Current children cumulated vsize (KiB) 1468276

[startup+25.5005 s]
/proc/loadavg: 7.35 5.97 3.41 9/373 24442
/proc/meminfo: memFree=19918288/32950928 swapFree=65730864/67111528
[pid=24431] ppid=24429 vsize=1615332 CPUtime=66.7 cores=1,3,5,7
/proc/24431/stat : 24431 (plingeling) S 24429 24431 23830 0 -1 4202496 710720 1254 0 0 6567 103 0 0 25 0 5 0 733504239 1654099968 391618 33554432000 4194304 4417452 140736389637568 18446744073709551615 260917196773 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/24431/statm: 403833 391618 178 55 0 401719 0
[pid=24431/tid=24439] ppid=24429 vsize=1615332 CPUtime=13.79 cores=1,3,5,7
/proc/24431/task/24439/stat : 24439 (plingeling) R 24429 24431 23830 0 -1 4202560 83658 1254 0 0 1368 11 0 0 25 0 5 0 733505409 1654099968 391618 33554432000 4194304 4417452 140736389637568 18446744073709551615 4265832 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=24431/tid=24440] ppid=24429 vsize=1615332 CPUtime=13.79 cores=1,3,5,7
/proc/24431/task/24440/stat : 24440 (plingeling) R 24429 24431 23830 0 -1 4202560 105510 1254 0 0 1363 16 0 0 25 0 5 0 733505409 1654099968 391618 33554432000 4194304 4417452 140736389637568 18446744073709551615 4270966 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24431/tid=24441] ppid=24429 vsize=1615332 CPUtime=13.62 cores=1,3,5,7
/proc/24431/task/24441/stat : 24441 (plingeling) R 24429 24431 23830 0 -1 4202560 100368 1254 0 0 1351 11 0 0 25 0 5 0 733505409 1654099968 391618 33554432000 4194304 4417452 140736389637568 18446744073709551615 4331746 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24431/tid=24442] ppid=24429 vsize=1615332 CPUtime=13.77 cores=1,3,5,7
/proc/24431/task/24442/stat : 24442 (plingeling) R 24429 24431 23830 0 -1 4202560 16251 1254 0 0 1375 2 0 0 25 0 5 0 733505409 1654099968 391618 33554432000 4194304 4417452 140736389637568 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 66.7
Current children cumulated vsize (KiB) 1615332

[startup+51.1008 s]
/proc/loadavg: 7.57 6.14 3.53 9/373 24443
/proc/meminfo: memFree=19727184/32950928 swapFree=65730864/67111528
[pid=24431] ppid=24429 vsize=1721804 CPUtime=168.8 cores=1,3,5,7
/proc/24431/stat : 24431 (plingeling) S 24429 24431 23830 0 -1 4202496 749951 1254 0 0 16770 110 0 0 25 0 5 0 733504239 1763127296 418249 33554432000 4194304 4417452 140736389637568 18446744073709551615 260917196773 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/24431/statm: 430451 418249 179 55 0 428337 0
[pid=24431/tid=24439] ppid=24429 vsize=1721804 CPUtime=39.39 cores=1,3,5,7
/proc/24431/task/24439/stat : 24439 (plingeling) R 24429 24431 23830 0 -1 4202560 96587 1254 0 0 3925 14 0 0 25 0 5 0 733505409 1763127296 418249 33554432000 4194304 4417452 140736389637568 18446744073709551615 4364466 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=24431/tid=24440] ppid=24429 vsize=1721804 CPUtime=39.38 cores=1,3,5,7
/proc/24431/task/24440/stat : 24440 (plingeling) R 24429 24431 23830 0 -1 4202560 118922 1254 0 0 3920 18 0 0 25 0 5 0 733505409 1763127296 418249 33554432000 4194304 4417452 140736389637568 18446744073709551615 4232654 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24431/tid=24441] ppid=24429 vsize=1721804 CPUtime=38.93 cores=1,3,5,7
/proc/24431/task/24441/stat : 24441 (plingeling) R 24429 24431 23830 0 -1 4202560 113258 1254 0 0 3879 14 0 0 25 0 5 0 733505409 1763127296 418249 33554432000 4194304 4417452 140736389637568 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24431/tid=24442] ppid=24429 vsize=1721804 CPUtime=39.37 cores=1,3,5,7
/proc/24431/task/24442/stat : 24442 (plingeling) R 24429 24431 23830 0 -1 4202560 16251 1254 0 0 3935 2 0 0 25 0 5 0 733505409 1763127296 418249 33554432000 4194304 4417452 140736389637568 18446744073709551615 4265465 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 168.8
Current children cumulated vsize (KiB) 1721804

[startup+102.3 s]
/proc/loadavg: 7.88 6.46 3.79 9/373 24444
/proc/meminfo: memFree=19679716/32950928 swapFree=65730864/67111528
[pid=24431] ppid=24429 vsize=1721804 CPUtime=372.99 cores=1,3,5,7
/proc/24431/stat : 24431 (plingeling) S 24429 24431 23830 0 -1 4202496 749951 1254 0 0 37189 110 0 0 25 0 5 0 733504239 1763127296 418249 33554432000 4194304 4417452 140736389637568 18446744073709551615 260917196773 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/24431/statm: 430451 418249 179 55 0 428337 0
[pid=24431/tid=24439] ppid=24429 vsize=1721804 CPUtime=90.59 cores=1,3,5,7
/proc/24431/task/24439/stat : 24439 (plingeling) R 24429 24431 23830 0 -1 4202560 96587 1254 0 0 9045 14 0 0 25 0 5 0 733505409 1763127296 418249 33554432000 4194304 4417452 140736389637568 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=24431/tid=24440] ppid=24429 vsize=1721804 CPUtime=90.58 cores=1,3,5,7
/proc/24431/task/24440/stat : 24440 (plingeling) R 24429 24431 23830 0 -1 4202560 118922 1254 0 0 9040 18 0 0 25 0 5 0 733505409 1763127296 418249 33554432000 4194304 4417452 140736389637568 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24431/tid=24441] ppid=24429 vsize=1721804 CPUtime=89.52 cores=1,3,5,7
/proc/24431/task/24441/stat : 24441 (plingeling) R 24429 24431 23830 0 -1 4202560 113258 1254 0 0 8938 14 0 0 25 0 5 0 733505409 1763127296 418249 33554432000 4194304 4417452 140736389637568 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24431/tid=24442] ppid=24429 vsize=1721804 CPUtime=90.57 cores=1,3,5,7
/proc/24431/task/24442/stat : 24442 (plingeling) R 24429 24431 23830 0 -1 4202560 16251 1254 0 0 9055 2 0 0 25 0 5 0 733505409 1763127296 418249 33554432000 4194304 4417452 140736389637568 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 372.99
Current children cumulated vsize (KiB) 1721804

[startup+162.3 s]
/proc/loadavg: 7.99 6.75 4.05 9/373 24446
/proc/meminfo: memFree=19661008/32950928 swapFree=65730864/67111528
[pid=24431] ppid=24429 vsize=1760348 CPUtime=612.31 cores=1,3,5,7
/proc/24431/stat : 24431 (plingeling) S 24429 24431 23830 0 -1 4202496 759587 1254 0 0 61121 110 0 0 25 0 5 0 733504239 1802596352 427885 33554432000 4194304 4417452 140736389637568 18446744073709551615 260917196773 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/24431/statm: 440087 427885 179 55 0 437973 0
[pid=24431/tid=24439] ppid=24429 vsize=1760348 CPUtime=150.6 cores=1,3,5,7
/proc/24431/task/24439/stat : 24439 (plingeling) R 24429 24431 23830 0 -1 4202560 96587 1254 0 0 15046 14 0 0 25 0 5 0 733505409 1802596352 427885 33554432000 4194304 4417452 140736389637568 18446744073709551615 4265429 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=24431/tid=24440] ppid=24429 vsize=1760348 CPUtime=150.59 cores=1,3,5,7
/proc/24431/task/24440/stat : 24440 (plingeling) R 24429 24431 23830 0 -1 4202560 123502 1254 0 0 15041 18 0 0 25 0 5 0 733505409 1802596352 427885 33554432000 4194304 4417452 140736389637568 18446744073709551615 4235565 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24431/tid=24441] ppid=24429 vsize=1760348 CPUtime=148.82 cores=1,3,5,7
/proc/24431/task/24441/stat : 24441 (plingeling) R 24429 24431 23830 0 -1 4202560 118314 1254 0 0 14868 14 0 0 25 0 5 0 733505409 1802596352 427885 33554432000 4194304 4417452 140736389637568 18446744073709551615 4364629 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24431/tid=24442] ppid=24429 vsize=1760348 CPUtime=150.58 cores=1,3,5,7
/proc/24431/task/24442/stat : 24442 (plingeling) R 24429 24431 23830 0 -1 4202560 16251 1254 0 0 15056 2 0 0 25 0 5 0 733505409 1802596352 427885 33554432000 4194304 4417452 140736389637568 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 612.31
Current children cumulated vsize (KiB) 1760348

[startup+222.3 s]
/proc/loadavg: 8.08 7.01 4.31 9/373 24447
/proc/meminfo: memFree=19626384/32950928 swapFree=65730864/67111528
[pid=24431] ppid=24429 vsize=1761948 CPUtime=851.58 cores=1,3,5,7
/proc/24431/stat : 24431 (plingeling) S 24429 24431 23830 0 -1 4202496 892358 1254 0 0 85029 129 0 0 25 0 5 0 733504239 1804234752 428285 33554432000 4194304 4417452 140736389637568 18446744073709551615 260917196773 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/24431/statm: 440487 428285 179 55 0 438373 0
[pid=24431/tid=24439] ppid=24429 vsize=1761948 CPUtime=210.59 cores=1,3,5,7
/proc/24431/task/24439/stat : 24439 (plingeling) R 24429 24431 23830 0 -1 4202560 162672 1254 0 0 21036 23 0 0 25 0 5 0 733505409 1804234752 428285 33554432000 4194304 4417452 140736389637568 18446744073709551615 4281668 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=24431/tid=24440] ppid=24429 vsize=1761948 CPUtime=210.58 cores=1,3,5,7
/proc/24431/task/24440/stat : 24440 (plingeling) R 24429 24431 23830 0 -1 4202560 189612 1254 0 0 21031 27 0 0 25 0 5 0 733505409 1804234752 428285 33554432000 4194304 4417452 140736389637568 18446744073709551615 4284289 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24431/tid=24441] ppid=24429 vsize=1761948 CPUtime=208.11 cores=1,3,5,7
/proc/24431/task/24441/stat : 24441 (plingeling) R 24429 24431 23830 0 -1 4202560 118890 1254 0 0 20797 14 0 0 25 0 5 0 733505409 1804234752 428285 33554432000 4194304 4417452 140736389637568 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24431/tid=24442] ppid=24429 vsize=1761948 CPUtime=210.57 cores=1,3,5,7
/proc/24431/task/24442/stat : 24442 (plingeling) R 24429 24431 23830 0 -1 4202560 16251 1254 0 0 21055 2 0 0 25 0 5 0 733505409 1804234752 428285 33554432000 4194304 4417452 140736389637568 18446744073709551615 4364576 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 851.58
Current children cumulated vsize (KiB) 1761948

[startup+282.314 s]
/proc/loadavg: 8.06 7.20 4.54 9/373 24449
/proc/meminfo: memFree=19671488/32950928 swapFree=65730864/67111528
[pid=24431] ppid=24429 vsize=1651092 CPUtime=1090.93 cores=1,3,5,7
/proc/24431/stat : 24431 (plingeling) S 24429 24431 23830 0 -1 4202496 1004344 1254 0 0 108949 144 0 0 25 0 5 0 733504239 1690718208 400571 33554432000 4194304 4417452 140736389637568 18446744073709551615 260917196773 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/24431/statm: 412773 400571 179 55 0 410659 0
[pid=24431/tid=24439] ppid=24429 vsize=1651092 CPUtime=270.6 cores=1,3,5,7

################
# More data... #
################

[pid=24431/tid=24440] ppid=24429 vsize=1447484 CPUtime=750.59 cores=1,3,5,7
/proc/24431/task/24440/stat : 24440 (plingeling) R 24429 24431 23830 0 -1 4202560 239964 1254 0 0 75024 35 0 0 25 0 5 0 733505409 1482223616 349676 33554432000 4194304 4417452 140736389637568 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24431/tid=24441] ppid=24429 vsize=1447484 CPUtime=741.74 cores=1,3,5,7
/proc/24431/task/24441/stat : 24441 (plingeling) R 24429 24431 23830 0 -1 4202560 250239 1254 0 0 74143 31 0 0 25 0 5 0 733505409 1482223616 349676 33554432000 4194304 4417452 140736389637568 18446744073709551615 4353539 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24431/tid=24442] ppid=24429 vsize=1447484 CPUtime=750.59 cores=1,3,5,7
/proc/24431/task/24442/stat : 24442 (plingeling) R 24429 24431 23830 0 -1 4202560 20889 1254 0 0 75056 3 0 0 25 0 5 0 733505409 1482223616 349676 33554432000 4194304 4417452 140736389637568 18446744073709551615 4232804 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3005.25
Current children cumulated vsize (KiB) 1447484

[startup+822.301 s]
/proc/loadavg: 8.08 7.94 6.09 9/373 24463
/proc/meminfo: memFree=19524472/32950928 swapFree=65730864/67111528
[pid=24431] ppid=24429 vsize=1446156 CPUtime=3244.58 cores=1,3,5,7
/proc/24431/stat : 24431 (plingeling) S 24429 24431 23830 0 -1 4202496 1129103 1254 0 0 324297 161 0 0 25 0 5 0 733504239 1480863744 349344 33554432000 4194304 4417452 140736389637568 18446744073709551615 260917196773 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/24431/statm: 361539 349344 180 55 0 359425 0
[pid=24431/tid=24439] ppid=24429 vsize=1446156 CPUtime=810.62 cores=1,3,5,7
/proc/24431/task/24439/stat : 24439 (plingeling) R 24429 24431 23830 0 -1 4202560 213078 1254 0 0 81032 30 0 0 25 0 5 0 733505409 1480863744 349344 33554432000 4194304 4417452 140736389637568 18446744073709551615 4265429 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=24431/tid=24440] ppid=24429 vsize=1446156 CPUtime=810.6 cores=1,3,5,7
/proc/24431/task/24440/stat : 24440 (plingeling) R 24429 24431 23830 0 -1 4202560 239964 1254 0 0 81025 35 0 0 25 0 5 0 733505409 1480863744 349344 33554432000 4194304 4417452 140736389637568 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24431/tid=24441] ppid=24429 vsize=1446156 CPUtime=801.04 cores=1,3,5,7
/proc/24431/task/24441/stat : 24441 (plingeling) R 24429 24431 23830 0 -1 4202560 250239 1254 0 0 80073 31 0 0 25 0 5 0 733505409 1480863744 349344 33554432000 4194304 4417452 140736389637568 18446744073709551615 4265429 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24431/tid=24442] ppid=24429 vsize=1446156 CPUtime=810.6 cores=1,3,5,7
/proc/24431/task/24442/stat : 24442 (plingeling) R 24429 24431 23830 0 -1 4202560 20889 1254 0 0 81057 3 0 0 25 0 5 0 733505409 1480863744 349344 33554432000 4194304 4417452 140736389637568 18446744073709551615 4281668 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3244.58
Current children cumulated vsize (KiB) 1446156

[startup+882.301 s]
/proc/loadavg: 6.00 7.42 6.03 5/363 24483
/proc/meminfo: memFree=21145320/32950928 swapFree=65730864/67111528
[pid=24431] ppid=24429 vsize=1444420 CPUtime=3483.85 cores=1,3,5,7
/proc/24431/stat : 24431 (plingeling) S 24429 24431 23830 0 -1 4202496 1129103 1254 0 0 348224 161 0 0 25 0 5 0 733504239 1479086080 348910 33554432000 4194304 4417452 140736389637568 18446744073709551615 260917196773 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/24431/statm: 361105 348910 180 55 0 358991 0
[pid=24431/tid=24439] ppid=24429 vsize=1444420 CPUtime=870.61 cores=1,3,5,7
/proc/24431/task/24439/stat : 24439 (plingeling) R 24429 24431 23830 0 -1 4202560 213078 1254 0 0 87031 30 0 0 25 0 5 0 733505409 1479086080 348910 33554432000 4194304 4417452 140736389637568 18446744073709551615 4281668 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=24431/tid=24440] ppid=24429 vsize=1444420 CPUtime=870.59 cores=1,3,5,7
/proc/24431/task/24440/stat : 24440 (plingeling) R 24429 24431 23830 0 -1 4202560 239964 1254 0 0 87024 35 0 0 25 0 5 0 733505409 1479086080 348910 33554432000 4194304 4417452 140736389637568 18446744073709551615 4237189 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24431/tid=24441] ppid=24429 vsize=1444420 CPUtime=860.33 cores=1,3,5,7
/proc/24431/task/24441/stat : 24441 (plingeling) R 24429 24431 23830 0 -1 4202560 250239 1254 0 0 86002 31 0 0 25 0 5 0 733505409 1479086080 348910 33554432000 4194304 4417452 140736389637568 18446744073709551615 4237189 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24431/tid=24442] ppid=24429 vsize=1444420 CPUtime=870.59 cores=1,3,5,7
/proc/24431/task/24442/stat : 24442 (plingeling) R 24429 24431 23830 0 -1 4202560 20889 1254 0 0 87056 3 0 0 25 0 5 0 733505409 1479086080 348910 33554432000 4194304 4417452 140736389637568 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3483.85
Current children cumulated vsize (KiB) 1444420

[startup+942.314 s]
/proc/loadavg: 4.77 6.81 5.90 5/363 24484
/proc/meminfo: memFree=21122820/32950928 swapFree=65730864/67111528
[pid=24431] ppid=24429 vsize=1467540 CPUtime=3723.2 cores=1,3,5,7
/proc/24431/stat : 24431 (plingeling) S 24429 24431 23830 0 -1 4202496 1135152 1254 0 0 372158 162 0 0 25 0 5 0 733504239 1502760960 354690 33554432000 4194304 4417452 140736389637568 18446744073709551615 260917196773 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/24431/statm: 366885 354690 180 55 0 364771 0
[pid=24431/tid=24439] ppid=24429 vsize=1467540 CPUtime=930.62 cores=1,3,5,7
/proc/24431/task/24439/stat : 24439 (plingeling) R 24429 24431 23830 0 -1 4202560 213078 1254 0 0 93032 30 0 0 25 0 5 0 733505409 1502760960 354690 33554432000 4194304 4417452 140736389637568 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=24431/tid=24440] ppid=24429 vsize=1467540 CPUtime=930.61 cores=1,3,5,7
/proc/24431/task/24440/stat : 24440 (plingeling) R 24429 24431 23830 0 -1 4202560 239964 1254 0 0 93026 35 0 0 25 0 5 0 733505409 1502760960 354690 33554432000 4194304 4417452 140736389637568 18446744073709551615 4364466 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24431/tid=24441] ppid=24429 vsize=1467540 CPUtime=919.63 cores=1,3,5,7
/proc/24431/task/24441/stat : 24441 (plingeling) R 24429 24431 23830 0 -1 4202560 256288 1254 0 0 91931 32 0 0 25 0 5 0 733505409 1502760960 354690 33554432000 4194304 4417452 140736389637568 18446744073709551615 4237189 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24431/tid=24442] ppid=24429 vsize=1467540 CPUtime=930.61 cores=1,3,5,7
/proc/24431/task/24442/stat : 24442 (plingeling) R 24429 24431 23830 0 -1 4202560 20889 1254 0 0 93058 3 0 0 25 0 5 0 733505409 1502760960 354690 33554432000 4194304 4417452 140736389637568 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3723.2
Current children cumulated vsize (KiB) 1467540

[startup+1002.3 s]
/proc/loadavg: 4.33 6.31 5.79 5/363 24486
/proc/meminfo: memFree=21126608/32950928 swapFree=65730864/67111528
[pid=24431] ppid=24429 vsize=1463668 CPUtime=3962.45 cores=1,3,5,7
/proc/24431/stat : 24431 (plingeling) S 24429 24431 23830 0 -1 4202496 1135722 1254 0 0 396083 162 0 0 25 0 5 0 733504239 1498796032 353722 33554432000 4194304 4417452 140736389637568 18446744073709551615 260917196773 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/24431/statm: 365917 353722 180 55 0 363803 0
[pid=24431/tid=24439] ppid=24429 vsize=1463668 CPUtime=990.61 cores=1,3,5,7
/proc/24431/task/24439/stat : 24439 (plingeling) R 24429 24431 23830 0 -1 4202560 213370 1254 0 0 99031 30 0 0 25 0 5 0 733505409 1498796032 353722 33554432000 4194304 4417452 140736389637568 18446744073709551615 4281668 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=24431/tid=24440] ppid=24429 vsize=1463668 CPUtime=990.6 cores=1,3,5,7
/proc/24431/task/24440/stat : 24440 (plingeling) R 24429 24431 23830 0 -1 4202560 240242 1254 0 0 99025 35 0 0 25 0 5 0 733505409 1498796032 353722 33554432000 4194304 4417452 140736389637568 18446744073709551615 4357902 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24431/tid=24441] ppid=24429 vsize=1463668 CPUtime=978.92 cores=1,3,5,7
/proc/24431/task/24441/stat : 24441 (plingeling) R 24429 24431 23830 0 -1 4202560 256288 1254 0 0 97860 32 0 0 25 0 5 0 733505409 1498796032 353722 33554432000 4194304 4417452 140736389637568 18446744073709551615 4265465 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24431/tid=24442] ppid=24429 vsize=1463668 CPUtime=990.6 cores=1,3,5,7
/proc/24431/task/24442/stat : 24442 (plingeling) R 24429 24431 23830 0 -1 4202560 20889 1254 0 0 99057 3 0 0 25 0 5 0 733505409 1498796032 353722 33554432000 4194304 4417452 140736389637568 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3962.45
Current children cumulated vsize (KiB) 1463668

[startup+1062.3 s]
/proc/loadavg: 4.12 5.88 5.67 5/363 24488
/proc/meminfo: memFree=21105076/32950928 swapFree=65730864/67111528
[pid=24431] ppid=24429 vsize=1485864 CPUtime=4201.74 cores=1,3,5,7
/proc/24431/stat : 24431 (plingeling) S 24429 24431 23830 0 -1 4202496 1141286 1254 0 0 420012 162 0 0 25 0 5 0 733504239 1521524736 359271 33554432000 4194304 4417452 140736389637568 18446744073709551615 260917196773 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/24431/statm: 371466 359271 180 55 0 369352 0
[pid=24431/tid=24439] ppid=24429 vsize=1485864 CPUtime=1050.61 cores=1,3,5,7
/proc/24431/task/24439/stat : 24439 (plingeling) R 24429 24431 23830 0 -1 4202560 213370 1254 0 0 105031 30 0 0 25 0 5 0 733505409 1521524736 359271 33554432000 4194304 4417452 140736389637568 18446744073709551615 4237221 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=24431/tid=24440] ppid=24429 vsize=1485864 CPUtime=1050.59 cores=1,3,5,7
/proc/24431/task/24440/stat : 24440 (plingeling) R 24429 24431 23830 0 -1 4202560 245806 1254 0 0 105024 35 0 0 25 0 5 0 733505409 1521524736 359271 33554432000 4194304 4417452 140736389637568 18446744073709551615 4364548 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24431/tid=24441] ppid=24429 vsize=1485864 CPUtime=1038.2 cores=1,3,5,7
/proc/24431/task/24441/stat : 24441 (plingeling) R 24429 24431 23830 0 -1 4202560 256288 1254 0 0 103788 32 0 0 25 0 5 0 733505409 1521524736 359271 33554432000 4194304 4417452 140736389637568 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24431/tid=24442] ppid=24429 vsize=1485864 CPUtime=1050.6 cores=1,3,5,7
/proc/24431/task/24442/stat : 24442 (plingeling) R 24429 24431 23830 0 -1 4202560 20889 1254 0 0 105057 3 0 0 25 0 5 0 733505409 1521524736 359271 33554432000 4194304 4417452 140736389637568 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4201.74
Current children cumulated vsize (KiB) 1485864

[startup+1122.3 s]
/proc/loadavg: 4.08 5.55 5.57 5/363 24489
/proc/meminfo: memFree=21121124/32950928 swapFree=65730864/67111528
[pid=24431] ppid=24429 vsize=1469296 CPUtime=4441.05 cores=1,3,5,7
/proc/24431/stat : 24431 (plingeling) S 24429 24431 23830 0 -1 4202496 1159879 1254 0 0 443940 165 0 0 25 0 5 0 733504239 1504559104 355129 33554432000 4194304 4417452 140736389637568 18446744073709551615 260917196773 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/24431/statm: 367324 355129 180 55 0 365210 0
[pid=24431/tid=24439] ppid=24429 vsize=1469296 CPUtime=1110.62 cores=1,3,5,7
/proc/24431/task/24439/stat : 24439 (plingeling) R 24429 24431 23830 0 -1 4202560 213370 1254 0 0 111032 30 0 0 25 0 5 0 733505409 1504559104 355129 33554432000 4194304 4417452 140736389637568 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=24431/tid=24440] ppid=24429 vsize=1469296 CPUtime=1110.59 cores=1,3,5,7
/proc/24431/task/24440/stat : 24440 (plingeling) R 24429 24431 23830 0 -1 4202560 245806 1254 0 0 111024 35 0 0 25 0 5 0 733505409 1504559104 355129 33554432000 4194304 4417452 140736389637568 18446744073709551615 4232680 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24431/tid=24441] ppid=24429 vsize=1469296 CPUtime=1097.5 cores=1,3,5,7
/proc/24431/task/24441/stat : 24441 (plingeling) R 24429 24431 23830 0 -1 4202560 274881 1254 0 0 109716 34 0 0 25 0 5 0 733505409 1504559104 355129 33554432000 4194304 4417452 140736389637568 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24431/tid=24442] ppid=24429 vsize=1469296 CPUtime=1110.6 cores=1,3,5,7
/proc/24431/task/24442/stat : 24442 (plingeling) R 24429 24431 23830 0 -1 4202560 20889 1254 0 0 111057 3 0 0 25 0 5 0 733505409 1504559104 355129 33554432000 4194304 4417452 140736389637568 18446744073709551615 4236986 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4441.05
Current children cumulated vsize (KiB) 1469296

[startup+1182.3 s]
/proc/loadavg: 4.03 5.27 5.47 5/363 24491
/proc/meminfo: memFree=21082484/32950928 swapFree=65730864/67111528
[pid=24431] ppid=24429 vsize=1508472 CPUtime=4680.34 cores=1,3,5,7
/proc/24431/stat : 24431 (plingeling) S 24429 24431 23830 0 -1 4202496 1203035 1254 0 0 467863 171 0 0 25 0 5 0 733504239 1544675328 364923 33554432000 4194304 4417452 140736389637568 18446744073709551615 260917196773 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/24431/statm: 377118 364923 180 55 0 375004 0
[pid=24431/tid=24439] ppid=24429 vsize=1508472 CPUtime=1170.62 cores=1,3,5,7
/proc/24431/task/24439/stat : 24439 (plingeling) R 24429 24431 23830 0 -1 4202560 230026 1254 0 0 117029 33 0 0 25 0 5 0 733505409 1544675328 364923 33554432000 4194304 4417452 140736389637568 18446744073709551615 4265465 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=24431/tid=24440] ppid=24429 vsize=1508472 CPUtime=1170.6 cores=1,3,5,7
/proc/24431/task/24440/stat : 24440 (plingeling) R 24429 24431 23830 0 -1 4202560 264187 1254 0 0 117022 38 0 0 25 0 5 0 733505409 1544675328 364923 33554432000 4194304 4417452 140736389637568 18446744073709551615 4265429 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24431/tid=24441] ppid=24429 vsize=1508472 CPUtime=1156.79 cores=1,3,5,7
/proc/24431/task/24441/stat : 24441 (plingeling) R 24429 24431 23830 0 -1 4202560 280400 1254 0 0 115644 35 0 0 25 0 5 0 733505409 1544675328 364923 33554432000 4194304 4417452 140736389637568 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24431/tid=24442] ppid=24429 vsize=1508472 CPUtime=1170.61 cores=1,3,5,7
/proc/24431/task/24442/stat : 24442 (plingeling) R 24429 24431 23830 0 -1 4202560 23489 1254 0 0 117057 4 0 0 25 0 5 0 733505409 1544675328 364923 33554432000 4194304 4417452 140736389637568 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4680.34
Current children cumulated vsize (KiB) 1508472



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1212.4 s]
/proc/loadavg: 4.07 5.16 5.43 5/363 24492
/proc/meminfo: memFree=21083816/32950928 swapFree=65730864/67111528
[pid=24431] ppid=24429 vsize=1506584 CPUtime=4800.41 cores=1,3,5,7
/proc/24431/stat : 24431 (plingeling) S 24429 24431 23830 0 -1 4202496 1203035 1254 0 0 479870 171 0 0 25 0 5 0 733504239 1542742016 364451 33554432000 4194304 4417452 140736389637568 18446744073709551615 260917196773 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/24431/statm: 376646 364451 180 55 0 374532 0
[pid=24431/tid=24439] ppid=24429 vsize=1506584 CPUtime=1200.73 cores=1,3,5,7
/proc/24431/task/24439/stat : 24439 (plingeling) R 24429 24431 23830 0 -1 4202560 230026 1254 0 0 120040 33 0 0 25 0 5 0 733505409 1542742016 364451 33554432000 4194304 4417452 140736389637568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=24431/tid=24440] ppid=24429 vsize=1506584 CPUtime=1200.71 cores=1,3,5,7
/proc/24431/task/24440/stat : 24440 (plingeling) R 24429 24431 23830 0 -1 4202560 264187 1254 0 0 120033 38 0 0 25 0 5 0 733505409 1542742016 364451 33554432000 4194304 4417452 140736389637568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24431/tid=24441] ppid=24429 vsize=1506584 CPUtime=1186.54 cores=1,3,5,7
/proc/24431/task/24441/stat : 24441 (plingeling) R 24429 24431 23830 0 -1 4202560 280400 1254 0 0 118619 35 0 0 25 0 5 0 733505409 1542742016 364451 33554432000 4194304 4417452 140736389637568 18446744073709551615 4237189 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24431/tid=24442] ppid=24429 vsize=1506584 CPUtime=1200.72 cores=1,3,5,7
/proc/24431/task/24442/stat : 24442 (plingeling) R 24429 24431 23830 0 -1 4202560 23489 1254 0 0 120068 4 0 0 25 0 5 0 733505409 1542742016 364451 33554432000 4194304 4417452 140736389637568 18446744073709551615 4232827 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.41
Current children cumulated vsize (KiB) 1506584

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1212.4 s]
/proc/loadavg: 4.07 5.16 5.43 5/363 24492
/proc/meminfo: memFree=21083816/32950928 swapFree=65730864/67111528
[pid=24431] ppid=24429 vsize=1506584 CPUtime=4800.41 cores=1,3,5,7
/proc/24431/stat : 24431 (plingeling) S 24429 24431 23830 0 -1 4202496 1203035 1254 0 0 479870 171 0 0 25 0 5 0 733504239 1542742016 364451 33554432000 4194304 4417452 140736389637568 18446744073709551615 260917196773 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/24431/statm: 376646 364451 180 55 0 374532 0
[pid=24431/tid=24439] ppid=24429 vsize=1506584 CPUtime=1200.73 cores=1,3,5,7
/proc/24431/task/24439/stat : 24439 (plingeling) R 24429 24431 23830 0 -1 4202560 230026 1254 0 0 120040 33 0 0 25 0 5 0 733505409 1542742016 364451 33554432000 4194304 4417452 140736389637568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=24431/tid=24440] ppid=24429 vsize=1506584 CPUtime=1200.71 cores=1,3,5,7
/proc/24431/task/24440/stat : 24440 (plingeling) R 24429 24431 23830 0 -1 4202560 264187 1254 0 0 120033 38 0 0 25 0 5 0 733505409 1542742016 364451 33554432000 4194304 4417452 140736389637568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24431/tid=24441] ppid=24429 vsize=1506584 CPUtime=1186.54 cores=1,3,5,7
/proc/24431/task/24441/stat : 24441 (plingeling) R 24429 24431 23830 0 -1 4202560 280400 1254 0 0 118619 35 0 0 25 0 5 0 733505409 1542742016 364451 33554432000 4194304 4417452 140736389637568 18446744073709551615 4237189 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24431/tid=24442] ppid=24429 vsize=1506584 CPUtime=1200.72 cores=1,3,5,7
/proc/24431/task/24442/stat : 24442 (plingeling) R 24429 24431 23830 0 -1 4202560 23489 1254 0 0 120068 4 0 0 25 0 5 0 733505409 1542742016 364451 33554432000 4194304 4417452 140736389637568 18446744073709551615 4232827 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.41
Current children cumulated vsize (KiB) 1506584

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1212.47
CPU time (s): 4800.51
CPU user time (s): 4798.74
CPU system time (s): 1.76973
CPU usage (%): 395.928
Max. virtual memory (cumulated for all children) (KiB): 1913172

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4798.74
system time used= 1.76973
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1204289
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 143
involuntary context switches= 22349

runsolver used 3.88141 second user time and 10.4574 second system time

The end

Launcher Data 

Begin job on node116 at 2011-05-15 06:12:16
IDJOB=3371290
IDBENCH=88158
IDSOLVER=1848
FILE ID=node116/3371290-1305432733
RUNJOBID= node116-1305432370-24250
PBS_JOBID= 13324608
Free space on /tmp= 72952 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT-Race-2010-CNF/hardware-verification/velev/1dlx_c_iq60_a.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3371290-1305432733/watcher-3371290-1305432733 -o /tmp/evaluation-result-3371290-1305432733/solver-3371290-1305432733 -C 4800 -W 1300 -M 15500  plingeling -t 4 HOME/instance-3371290-1305432733.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 1e60a63f1ba68b9b3c423ca413b4fa2b
RANDOM SEED=636491669

node116.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      21575180 kB
Buffers:         81248 kB
Cached:         818632 kB
SwapCached:      12356 kB
Active:       10630476 kB
Inactive:       526888 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      21575180 kB
SwapTotal:    67111528 kB
SwapFree:     65730864 kB
Dirty:          224968 kB
Writeback:           0 kB
AnonPages:    10255600 kB
Mapped:          16280 kB
Slab:           107608 kB
PageTables:      46080 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 83773788 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72952 MiB
End job on node116 at 2011-05-15 06:32:30