Trace number 3498288

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-seq-industrial (fixed)? (TO) 5000.1 5009.86

General information on the benchmark

NameAPPLICATIONS/bitverif/countbitsrotate/
countbitsrotate032.cnf
MD5SUM8d76aa747359ea8ccf15726792949452
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables8527
Number of clauses25484
Sum of the clauses size59460
Maximum clause length3
Minimum clause length1
Number of clauses of size 12
Number of clauses of size 216988
Number of clauses of size 38494
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.42/16.29	c INFO: loaded portfolio from HOME/etc/borg-mix+class.industrial.pickle
0.42/16.29	c INFO: solving HOME/instance-3498288-1307953119.cnf
0.42/16.29	c INFO: computing task features
0.45/16.36	c INFO: computed variable-clause graph statistics
0.45/16.36	c INFO: computed clause balance statistics
0.85/16.75	c INFO: computed variable balance statistics
1.15/17.04	c INFO: computed small-clause counts
1.15/17.04	c INFO: computed Horn-clause counts
1.25/17.16	c INFO: computed variable graph statistics
1.25/17.16	c INFO: collected features for HOME/instance-3498288-1307953119.cnf in 0.87 s
1.32/17.21	c INFO: running lingeling-276@0 for 97 with 4999 remaining (b = 0.29)
101.51/117.34	c INFO: running lingeling-276@100 for 485 with 4899 remaining (b = 0.40)
587.58/602.84	c INFO: running SApperloT@0 for 97 with 4413 remaining (b = 0.16)
687.79/702.97	c INFO: running cryptominisat-2.9.0@0 for 485 with 4313 remaining (b = 0.40)
1176.05/1190.62	c INFO: running precosat-570@0 for 388 with 3825 remaining (b = 0.29)
1564.89/1579.04	c INFO: running lingeling-276@0 for 582 with 3437 remaining (b = 0.36)
2152.15/2165.65	c INFO: running lingeling-276@0 for 97 with 2850 remaining (b = 0.09)
2251.35/2264.78	c INFO: running cryptominisat-2.9.0@487 for 97 with 2751 remaining (b = 0.07)
2348.69/2361.91	c INFO: running lingeling-276@99 for 97 with 2654 remaining (b = 0.06)
2445.89/2459.03	c INFO: running cryptominisat-2.9.0@0 for 485 with 2557 remaining (b = 0.22)
2934.67/2947.13	c INFO: running precosat-570@0 for 194 with 2069 remaining (b = 0.08)
3131.10/3143.39	c INFO: running lingeling-276@585 for 485 with 1873 remaining (b = 0.17)
3619.32/3630.93	c INFO: running cryptominisat-2.9.0@584 for 1385 with 1385 remaining (b = 0.39)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3498288-1307953119/watcher-3498288-1307953119 -o /tmp/evaluation-result-3498288-1307953119/solver-3498288-1307953119 -C 5000 -W 5100 -M 15500 python HOME/solve HOME/etc/borg-mix+class.industrial.pickle HOME/instance-3498288-1307953119.cnf -seed 1079452364 -budget 5000 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 5030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.00 0.00 0.00 2/174 12635
/proc/meminfo: memFree=31167936/32951132 swapFree=67052544/67111528
[pid=12635] ppid=12631 vsize=11804 CPUtime=0 cores=1,3,5,7
/proc/12635/stat : 12635 (runsolver) S 12631 12635 12590 0 -1 4202560 42 0 0 0 0 0 0 0 20 0 1 0 906910888 12087296 50 33554432000 4194304 5436572 140734257019216 18446744073709551615 4883175 0 2147483391 4096 24578 18446744071699870519 0 0 17 3 0 0 0
/proc/12635/statm: 2951 50 31 304 0 2643 0

[startup+0.0258261 s]
/proc/loadavg: 0.00 0.00 0.00 2/174 12635
/proc/meminfo: memFree=31167936/32951132 swapFree=67052544/67111528
[pid=12635] ppid=12631 vsize=11804 CPUtime=0 cores=1,3,5,7
/proc/12635/stat : 12635 (runsolver) S 12631 12635 12590 0 -1 4202560 47 0 0 0 0 0 0 0 21 0 1 0 906910888 12087296 55 33554432000 4194304 5436572 140734257019216 18446744073709551615 5057063 0 2147483391 4096 24578 18446744071699109054 0 0 17 3 0 0 0
/proc/12635/statm: 2951 55 36 304 0 2643 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 11804

[startup+0.100813 s]
/proc/loadavg: 0.00 0.00 0.00 2/174 12635
/proc/meminfo: memFree=31167936/32951132 swapFree=67052544/67111528
[pid=12635] ppid=12631 vsize=6240 CPUtime=0 cores=1,3,5,7
/proc/12635/stat : 12635 (python) D 12631 12635 12590 0 -1 4194304 101 0 0 0 0 0 0 0 21 0 1 0 906910888 6389760 52 33554432000 4194304 5683644 140734789549920 18446744073709551615 242965626599 0 0 4096 0 18446744071696489144 0 0 17 5 0 0 0
/proc/12635/statm: 1560 52 41 364 0 42 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 6240

[startup+0.300786 s]
/proc/loadavg: 0.00 0.00 0.00 2/174 12635
/proc/meminfo: memFree=31167936/32951132 swapFree=67052544/67111528
[pid=12635] ppid=12631 vsize=15184 CPUtime=0 cores=1,3,5,7
/proc/12635/stat : 12635 (python) S 12631 12635 12590 0 -1 4202496 714 0 0 0 0 0 0 0 16 0 1 0 906910888 15548416 604 33554432000 4194304 5683644 140734789549920 18446744073709551615 242970548224 0 2147483391 16781312 2 18446744071699109054 0 0 17 5 0 0 0
/proc/12635/statm: 3796 604 319 364 0 283 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15184

[startup+0.700726 s]
/proc/loadavg: 0.00 0.00 0.00 2/174 12635
/proc/meminfo: memFree=31167936/32951132 swapFree=67052544/67111528
[pid=12635] ppid=12631 vsize=15776 CPUtime=0 cores=1,3,5,7
/proc/12635/stat : 12635 (python) S 12631 12635 12590 0 -1 4202496 897 0 0 0 0 0 0 0 15 0 1 0 906910888 16154624 777 33554432000 4194304 5683644 140734789549920 18446744073709551615 242970548224 0 2147483391 16781312 2 18446744071699109054 0 0 17 5 0 0 0
/proc/12635/statm: 3944 777 343 364 0 431 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15776

[startup+1.50057 s]
/proc/loadavg: 0.00 0.00 0.00 1/176 12637
/proc/meminfo: memFree=31162588/32951132 swapFree=67052544/67111528
[pid=12635] ppid=12631 vsize=16396 CPUtime=0.01 cores=1,3,5,7
/proc/12635/stat : 12635 (python) S 12631 12635 12590 0 -1 4202496 1125 0 0 0 1 0 0 0 15 0 1 0 906910888 16789504 990 33554432000 4194304 5683644 140734789549920 18446744073709551615 242970548224 0 2147483391 16781312 2 18446744071699855507 0 0 17 5 0 0 0
/proc/12635/statm: 4099 990 413 364 0 586 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16396

[startup+9.05328 s]
/proc/loadavg: 0.00 0.00 0.00 1/176 12637
/proc/meminfo: memFree=31162588/32951132 swapFree=67052544/67111528
[pid=12635] ppid=12631 vsize=124240 CPUtime=0.14 cores=1,3,5,7
/proc/12635/stat : 12635 (python) S 12631 12635 12590 0 -1 4202496 4313 0 1 0 9 5 0 0 15 0 1 0 906910888 127221760 3881 33554432000 4194304 5683644 140734789549920 18446744073709551615 242970550016 0 0 16781312 2 18446744071562362840 0 0 17 5 0 0 0
/proc/12635/statm: 31060 3881 997 364 0 2871 0
Current children cumulated CPU time (s) 0.14
Current children cumulated vsize (KiB) 124240
heavy processes:

[startup+9.12226 s]
/proc/loadavg: 0.85 0.18 0.06 2/178 12639
/proc/meminfo: memFree=31141340/32951132 swapFree=67052544/67111528
[pid=12635] ppid=12631 vsize=124240 CPUtime=0.14 cores=1,3,5,7
/proc/12635/stat : 12635 (python) S 12631 12635 12590 0 -1 4202496 4313 0 1 0 9 5 0 0 15 0 1 0 906910888 127221760 3881 33554432000 4194304 5683644 140734789549920 18446744073709551615 242970550016 0 0 16781312 2 18446744071562362840 0 0 17 5 0 0 0
/proc/12635/statm: 31060 3881 997 364 0 2871 0
[pid=12638] ppid=12635 vsize=124236 CPUtime=0 cores=1,3,5,7
/proc/12638/stat : 12638 (python) D 12635 12635 12590 0 -1 4202560 26 0 0 0 0 0 0 0 15 0 1 0 906911680 127217664 2950 33554432000 4194304 5683644 140734789549920 18446744073709551615 242970371063 0 0 16781312 2 18446744071696489144 0 0 17 7 0 0 0
/proc/12638/statm: 31059 2950 66 364 0 2870 0
Current children cumulated CPU time (s) 0.14
Current children cumulated vsize (KiB) 248476
heavy processes:

[startup+12.7006 s]
/proc/loadavg: 0.85 0.18 0.06 2/178 12639
/proc/meminfo: memFree=31141296/32951132 swapFree=67052544/67111528
[pid=12635] ppid=12631 vsize=135584 CPUtime=0.18 cores=1,3,5,7
/proc/12635/stat : 12635 (python) S 12631 12635 12590 0 -1 4202496 5811 934 1 0 13 5 0 0 15 0 1 0 906910888 138838016 4121 33554432000 4194304 5683644 140734789549920 18446744073709551615 242970548224 0 2147483391 16781312 2 18446744071699855507 0 0 17 5 0 0 0
/proc/12635/statm: 33896 4121 1031 364 0 3082 0
Current children cumulated CPU time (s) 0.18
Current children cumulated vsize (KiB) 135584
heavy processes:

[startup+25.5004 s]
/proc/loadavg: 1.33 0.32 0.11 3/179 12650
/proc/meminfo: memFree=30962764/32951132 swapFree=67052544/67111528
[pid=12635] ppid=12631 vsize=290952 CPUtime=1.33 cores=1,3,5,7
/proc/12635/stat : 12635 (python) S 12631 12635 12590 0 -1 4202496 11493 934 1 0 118 15 0 0 18 0 2 0 906910888 297934848 7609 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982376907 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/12635/statm: 72738 7609 1875 364 0 8656 0
[pid=12646] ppid=12635 vsize=280708 CPUtime=0 cores=1,3,5,7
/proc/12646/stat : 12646 (python) S 12635 12635 12590 0 -1 4202560 965 0 0 0 0 0 0 0 18 0 1 0 906912609 287444992 6111 33554432000 4194304 5683644 140734789549920 18446744073709551615 242970570255 0 0 16781312 514 18212332322131247368 0 0 17 5 0 0 0
/proc/12646/statm: 70177 6111 381 364 0 6095 0
[pid=12648] ppid=12646 vsize=6204 CPUtime=8.27 cores=1,3,5,7
/proc/12648/stat : 12648 (lingeling) R 12646 12648 12648 0 -1 4202496 1541 0 0 0 827 0 0 0 25 0 1 0 906912610 6352896 720 33554432000 4194304 4352236 140736267596336 18446744073709551615 4230988 0 0 16781312 17506 0 0 0 17 3 0 0 0
/proc/12648/statm: 1551 720 129 39 0 631 0
[pid=12635/tid=12647] ppid=12631 vsize=290952 CPUtime=0 cores=1,3,5,7
/proc/12635/task/12647/stat : 12647 (python) S 12631 12635 12590 0 -1 4202560 28 934 0 0 0 0 0 0 18 0 2 0 906912610 297934848 7609 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 2 18446604471137258432 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 9.6
Current children cumulated vsize (KiB) 577864
heavy processes:

[startup+51.101 s]
/proc/loadavg: 1.56 0.45 0.15 3/179 12650
/proc/meminfo: memFree=30962280/32951132 swapFree=67052544/67111528
[pid=12635] ppid=12631 vsize=290952 CPUtime=1.33 cores=1,3,5,7
/proc/12635/stat : 12635 (python) S 12631 12635 12590 0 -1 4202496 11493 934 1 0 118 15 0 0 18 0 2 0 906910888 297934848 7609 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982376907 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/12635/statm: 72738 7609 1875 364 0 8656 0
[pid=12646] ppid=12635 vsize=280708 CPUtime=0.03 cores=1,3,5,7
/proc/12646/stat : 12646 (python) S 12635 12635 12590 0 -1 4202560 977 0 0 0 1 2 0 0 18 0 1 0 906912609 287444992 6112 33554432000 4194304 5683644 140734789549920 18446744073709551615 242970570255 0 0 16781312 514 8362041015934091528 0 0 17 5 0 0 0
/proc/12646/statm: 70177 6112 382 364 0 6095 0
[pid=12648] ppid=12646 vsize=8820 CPUtime=33.87 cores=1,3,5,7
/proc/12648/stat : 12648 (lingeling) R 12646 12648 12648 0 -1 4202496 2297 0 0 0 3387 0 0 0 25 0 1 0 906912610 9031680 1374 33554432000 4194304 4352236 140736267596336 18446744073709551615 4277105 0 0 16781312 17506 0 0 0 17 3 0 0 0
/proc/12648/statm: 2205 1374 129 39 0 1285 0
[pid=12635/tid=12647] ppid=12631 vsize=290952 CPUtime=0 cores=1,3,5,7
/proc/12635/task/12647/stat : 12647 (python) S 12631 12635 12590 0 -1 4202560 28 934 0 0 0 0 0 0 18 0 2 0 906912610 297934848 7609 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 2 18446604471137258432 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 35.23
Current children cumulated vsize (KiB) 580480
heavy processes:

[startup+102.3 s]
/proc/loadavg: 1.81 0.69 0.25 3/179 12652
/proc/meminfo: memFree=30962624/32951132 swapFree=67052544/67111528
[pid=12635] ppid=12631 vsize=290952 CPUtime=1.33 cores=1,3,5,7
/proc/12635/stat : 12635 (python) S 12631 12635 12590 0 -1 4202496 11493 934 1 0 118 15 0 0 18 0 2 0 906910888 297934848 7609 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982376907 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/12635/statm: 72738 7609 1875 364 0 8656 0
[pid=12646] ppid=12635 vsize=280708 CPUtime=0.09 cores=1,3,5,7
/proc/12646/stat : 12646 (python) S 12635 12635 12590 0 -1 4202560 998 0 0 0 4 5 0 0 18 0 1 0 906912609 287444992 6112 33554432000 4194304 5683644 140734789549920 18446744073709551615 242970570255 0 0 16781312 514 8362041015934091528 0 0 17 5 0 0 0
/proc/12646/statm: 70177 6112 382 364 0 6095 0
[pid=12648] ppid=12646 vsize=8564 CPUtime=85.07 cores=1,3,5,7
/proc/12648/stat : 12648 (lingeling) R 12646 12648 12648 0 -1 4202496 2371 0 0 0 8507 0 0 0 25 0 1 0 906912610 8769536 1320 33554432000 4194304 4352236 140736267596336 18446744073709551615 4208094 0 0 16781312 17506 0 0 0 17 3 0 0 0
/proc/12648/statm: 2141 1320 129 39 0 1221 0
[pid=12635/tid=12647] ppid=12631 vsize=290952 CPUtime=0 cores=1,3,5,7
/proc/12635/task/12647/stat : 12647 (python) S 12631 12635 12590 0 -1 4202560 28 934 0 0 0 0 0 0 18 0 2 0 906912610 297934848 7609 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 2 18446604471137258432 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 86.49
Current children cumulated vsize (KiB) 580224

[startup+162.301 s]
/proc/loadavg: 1.93 0.93 0.36 3/183 12656
/proc/meminfo: memFree=30850748/32951132 swapFree=67052544/67111528
[pid=12635] ppid=12631 vsize=290952 CPUtime=1.35 cores=1,3,5,7
/proc/12635/stat : 12635 (python) S 12631 12635 12590 0 -1 4202496 11980 934 1 0 120 15 0 0 18 0 2 0 906910888 297934848 7610 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982376907 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/12635/statm: 72738 7610 1876 364 0 8656 0
[pid=12646] ppid=12635 vsize=290952 CPUtime=0.16 cores=1,3,5,7
/proc/12646/stat : 12646 (python) S 12635 12635 12590 0 -1 4202560 1179 0 0 0 8 8 0 0 18 0 2 0 906912609 297934848 6151 33554432000 4194304 5683644 140734789549920 18446744073709551615 242970570310 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/12646/statm: 72738 6151 416 364 0 8656 0
[pid=12648] ppid=12646 vsize=8776 CPUtime=145.05 cores=1,3,5,7
/proc/12648/stat : 12648 (lingeling) R 12646 12648 12648 0 -1 4202496 2542 0 0 0 14505 0 0 0 25 0 1 0 906912610 8986624 1363 33554432000 4194304 4352236 140736267596336 18446744073709551615 4276488 0 0 16781312 17506 0 0 0 17 3 0 0 0
/proc/12648/statm: 2194 1363 129 39 0 1274 0
[pid=12646/tid=12656] ppid=12635 vsize=290952 CPUtime=0 cores=1,3,5,7
/proc/12646/task/12656/stat : 12656 (python) S 12635 12635 12590 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 906922620 297934848 6151 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 514 18446604465743016896 0 0 -1 7 0 0 0
[pid=12635/tid=12647] ppid=12631 vsize=290952 CPUtime=0 cores=1,3,5,7
/proc/12635/task/12647/stat : 12647 (python) S 12631 12635 12590 0 -1 4202560 28 934 0 0 0 0 0 0 18 0 2 0 906912610 297934848 7610 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 2 18446604471137258432 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 146.56
Current children cumulated vsize (KiB) 590680

[startup+222.301 s]
/proc/loadavg: 1.97 1.12 0.46 3/184 12659
/proc/meminfo: memFree=30839200/32951132 swapFree=67052544/67111528
[pid=12635] ppid=12631 vsize=290952 CPUtime=1.35 cores=1,3,5,7
/proc/12635/stat : 12635 (python) S 12631 12635 12590 0 -1 4202496 11980 934 1 0 120 15 0 0 18 0 2 0 906910888 297934848 7610 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982376907 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/12635/statm: 72738 7610 1876 364 0 8656 0
[pid=12646] ppid=12635 vsize=290952 CPUtime=0.23 cores=1,3,5,7
/proc/12646/stat : 12646 (python) S 12635 12635 12590 0 -1 4202560 1194 0 0 0 12 11 0 0 18 0 2 0 906912609 297934848 6151 33554432000 4194304 5683644 140734789549920 18446744073709551615 242970570310 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/12646/statm: 72738 6151 416 364 0 8656 0
[pid=12648] ppid=12646 vsize=9292 CPUtime=205.05 cores=1,3,5,7
/proc/12648/stat : 12648 (lingeling) R 12646 12648 12648 0 -1 4202496 3124 0 0 0 20505 0 0 0 25 0 1 0 906912610 9515008 1497 33554432000 4194304 4352236 140736267596336 18446744073709551615 4328090 0 0 16781312 17506 0 0 0 17 3 0 0 0
/proc/12648/statm: 2323 1497 129 39 0 1403 0
[pid=12646/tid=12656] ppid=12635 vsize=290952 CPUtime=0 cores=1,3,5,7
/proc/12646/task/12656/stat : 12656 (python) S 12635 12635 12590 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 906922620 297934848 6151 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 514 18446604465743016896 0 0 -1 7 0 0 0
[pid=12635/tid=12647] ppid=12631 vsize=290952 CPUtime=0 cores=1,3,5,7
/proc/12635/task/12647/stat : 12647 (python) S 12631 12635 12590 0 -1 4202560 28 934 0 0 0 0 0 0 18 0 2 0 906912610 297934848 7610 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 2 18446604471137258432 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 206.63
Current children cumulated vsize (KiB) 591196

[startup+282.3 s]
/proc/loadavg: 2.03 1.29 0.55 3/184 12661
/proc/meminfo: memFree=30838816/32951132 swapFree=67052544/67111528

################
# More data... #
################

[pid=12694] ppid=12635 vsize=311476 CPUtime=0.45 cores=1,3,5,7
/proc/12694/stat : 12694 (python) S 12635 12635 12590 0 -1 4202560 1146 0 0 0 22 23 0 0 16 0 2 0 907029951 318951424 6158 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/12694/statm: 77869 6158 400 364 0 13778 0
[pid=12696] ppid=12694 vsize=20612 CPUtime=388.38 cores=1,3,5,7
/proc/12696/stat : 12696 (precosat) T 12694 12696 12696 0 -1 4202496 5906 0 0 0 38837 1 0 0 25 0 1 0 907029951 21106688 4859 33554432000 4194304 4890832 140733788528832 18446744073709551615 4238599 0 0 16781312 17442 18446744071562691360 0 0 17 5 0 0 0
/proc/12696/statm: 5153 4859 86 171 0 4980 0
[pid=12694/tid=12714] ppid=12635 vsize=311476 CPUtime=0 cores=1,3,5,7
/proc/12694/task/12714/stat : 12714 (python) S 12635 12635 12590 0 -1 4202560 45 0 0 0 0 0 0 0 16 0 2 0 907068790 318951424 6158 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 514 18446604471082437632 0 0 -1 3 0 0 0
[pid=12715] ppid=12635 vsize=321732 CPUtime=0.68 cores=1,3,5,7
/proc/12715/stat : 12715 (python) S 12635 12635 12590 0 -1 4202560 1150 0 0 0 39 29 0 0 19 0 2 0 907068792 329453568 6162 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/12715/statm: 80433 6162 400 364 0 16339 0
[pid=12717] ppid=12715 vsize=10400 CPUtime=586.57 cores=1,3,5,7
/proc/12717/stat : 12717 (lingeling) T 12715 12717 12717 0 -1 4202496 6153 0 0 0 58657 0 0 0 25 0 1 0 907068793 10649600 1770 33554432000 4194304 4352236 140737135784576 18446744073709551615 4279075 0 0 16781312 17506 18446744071562691360 0 0 17 5 0 0 0
/proc/12717/statm: 2600 1770 129 39 0 1680 0
[pid=12715/tid=12734] ppid=12635 vsize=321732 CPUtime=0 cores=1,3,5,7
/proc/12715/task/12734/stat : 12734 (python) S 12635 12635 12590 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 907127452 329453568 6162 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 514 18446604470990686400 0 0 -1 3 0 0 0
[pid=12735] ppid=12635 vsize=332392 CPUtime=0.24 cores=1,3,5,7
/proc/12735/stat : 12735 (python) S 12635 12635 12590 0 -1 4202560 1140 0 0 0 11 13 0 0 18 0 2 0 907127453 340369408 6252 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/12735/statm: 83098 6252 400 364 0 19001 0
[pid=12737] ppid=12735 vsize=9120 CPUtime=196.2 cores=1,3,5,7
/proc/12737/stat : 12737 (lingeling) T 12735 12737 12737 0 -1 4202496 3027 0 0 0 19620 0 0 0 25 0 1 0 907127454 9338880 1461 33554432000 4194304 4352236 140736127328320 18446744073709551615 4231006 0 0 16781312 17506 18446744071562691360 0 0 17 1 0 0 0
/proc/12737/statm: 2280 1461 129 39 0 1360 0
[pid=12735/tid=12740] ppid=12635 vsize=332392 CPUtime=0 cores=1,3,5,7
/proc/12735/task/12740/stat : 12740 (python) S 12635 12635 12590 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 907137365 340369408 6252 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 514 18446604471137257536 0 0 -1 3 0 0 0
[pid=12747] ppid=12635 vsize=342648 CPUtime=0.66 cores=1,3,5,7
/proc/12747/stat : 12747 (python) S 12635 12635 12590 0 -1 4202560 1167 0 0 0 36 30 0 0 16 0 2 0 907156791 350871552 6257 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/12747/statm: 85662 6257 401 364 0 21562 0
[pid=12749] ppid=12747 vsize=89752 CPUtime=488.07 cores=1,3,5,7
/proc/12749/stat : 12749 (cryptominisat-2) T 12747 12749 12749 0 -1 4202496 63863 0 0 0 48796 11 0 0 25 0 1 0 907156792 91906048 14736 33554432000 4194304 4677096 140734685082576 18446744073709551615 4319558 0 0 16781312 2 18446744071562691360 0 0 17 1 0 0 0
/proc/12749/statm: 22438 14736 393 118 0 17397 0
[pid=12747/tid=12775] ppid=12635 vsize=342648 CPUtime=0 cores=1,3,5,7
/proc/12747/task/12775/stat : 12775 (python) S 12635 12635 12590 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 907205600 350871552 6257 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 514 18446604467085317952 0 0 -1 5 0 0 0
[pid=12776] ppid=12635 vsize=418440 CPUtime=0.26 cores=1,3,5,7
/proc/12776/stat : 12776 (python) S 12635 12635 12590 0 -1 4202560 1188 0 0 0 13 13 0 0 16 0 2 0 907205602 428482560 6268 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/12776/statm: 104610 6268 406 364 0 40507 0
[pid=12778] ppid=12776 vsize=16544 CPUtime=196.22 cores=1,3,5,7
/proc/12778/stat : 12778 (precosat) T 12776 12778 12778 0 -1 4202496 4846 0 0 0 19622 0 0 0 25 0 1 0 907205602 16941056 3864 33554432000 4194304 4890832 140733665619984 18446744073709551615 4235497 0 0 16781312 17442 18446744071562691360 0 0 17 1 0 0 0
/proc/12778/statm: 4136 3864 86 171 0 3963 0
[pid=12776/tid=12785] ppid=12635 vsize=418440 CPUtime=0 cores=1,3,5,7
/proc/12776/task/12785/stat : 12785 (python) S 12635 12635 12590 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 907225226 428482560 6268 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 514 18446604470985984768 0 0 -1 3 0 0 0
[pid=12635/tid=12647] ppid=12631 vsize=428684 CPUtime=0 cores=1,3,5,7
/proc/12635/task/12647/stat : 12647 (python) S 12631 12635 12590 0 -1 4202560 32 934 0 0 0 0 0 0 18 0 9 0 906912610 438972416 7755 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 2 18446604471137258432 0 0 -1 5 0 0 0
[pid=12635/tid=12671] ppid=12631 vsize=428684 CPUtime=0 cores=1,3,5,7
/proc/12635/task/12671/stat : 12671 (python) S 12631 12635 12590 0 -1 4202560 32 934 0 0 0 0 0 0 20 0 9 0 906971173 438972416 7755 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 2 18446604471137258432 0 0 -1 3 0 0 0
[pid=12635/tid=12679] ppid=12631 vsize=428684 CPUtime=0 cores=1,3,5,7
/proc/12635/task/12679/stat : 12679 (python) S 12631 12635 12590 0 -1 4202560 37 934 0 0 0 0 0 0 18 0 9 0 906981186 438972416 7755 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 2 18446604471137258432 0 0 -1 3 0 0 0
[pid=12635/tid=12695] ppid=12631 vsize=428684 CPUtime=0 cores=1,3,5,7
/proc/12635/task/12695/stat : 12695 (python) S 12631 12635 12590 0 -1 4202560 30 934 0 0 0 0 0 0 20 0 9 0 907029951 438972416 7755 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 2 18446604471137258432 0 0 -1 3 0 0 0
[pid=12635/tid=12716] ppid=12631 vsize=428684 CPUtime=0 cores=1,3,5,7
/proc/12635/task/12716/stat : 12716 (python) S 12631 12635 12590 0 -1 4202560 30 934 0 0 0 0 0 0 20 0 9 0 907068793 438972416 7755 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 2 18446604471137258432 0 0 -1 7 0 0 0
[pid=12635/tid=12736] ppid=12631 vsize=428684 CPUtime=0 cores=1,3,5,7
/proc/12635/task/12736/stat : 12736 (python) S 12631 12635 12590 0 -1 4202560 29 934 0 0 0 0 0 0 18 0 9 0 907127454 438972416 7755 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 2 18446604471137258432 0 0 -1 7 0 0 0
[pid=12635/tid=12748] ppid=12631 vsize=428684 CPUtime=0 cores=1,3,5,7
/proc/12635/task/12748/stat : 12748 (python) S 12631 12635 12590 0 -1 4202560 22 934 0 0 0 0 0 0 20 0 9 0 907156791 438972416 7755 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 2 18446604471137258432 0 0 -1 5 0 0 0
[pid=12635/tid=12777] ppid=12631 vsize=428684 CPUtime=0 cores=1,3,5,7
/proc/12635/task/12777/stat : 12777 (python) S 12631 12635 12590 0 -1 4202560 30 934 0 0 0 0 0 0 20 0 9 0 907205602 438972416 7755 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 2 18446604471137258432 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4952.54
Current children cumulated vsize (KiB) 3351116



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5009.8 s]
/proc/loadavg: 2.05 2.05 2.00 4/239 12854
/proc/meminfo: memFree=25279632/32951132 swapFree=67052544/67111528
[pid=12635] ppid=12631 vsize=428684 CPUtime=1.55 cores=1,3,5,7
/proc/12635/stat : 12635 (python) S 12631 12635 12590 0 -1 4202496 17766 934 1 0 138 17 0 0 18 0 9 0 906910888 438972416 7755 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982376907 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/12635/statm: 107171 7755 1890 364 0 43068 0
[pid=12646] ppid=12635 vsize=290952 CPUtime=1.29 cores=1,3,5,7
/proc/12646/stat : 12646 (python) S 12635 12635 12590 0 -1 4202560 1200 0 0 0 70 59 0 0 18 0 2 0 906912609 297934848 6151 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/12646/statm: 72738 6151 416 364 0 8656 0
[pid=12648] ppid=12646 vsize=14588 CPUtime=1073.13 cores=1,3,5,7
/proc/12648/stat : 12648 (lingeling) T 12646 12648 12648 0 -1 4202496 11867 0 0 0 107311 2 0 0 25 0 1 0 906912610 14938112 2816 33554432000 4194304 4352236 140736267596336 18446744073709551615 4207597 0 0 16781312 17506 18446744071562691360 0 0 17 3 0 0 0
/proc/12648/statm: 3647 2816 129 39 0 2727 0
[pid=12646/tid=12656] ppid=12635 vsize=290952 CPUtime=0 cores=1,3,5,7
/proc/12646/task/12656/stat : 12656 (python) S 12635 12635 12590 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 906922620 297934848 6151 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 514 18446604465743016896 0 0 -1 7 0 0 0
[pid=12670] ppid=12635 vsize=290964 CPUtime=0.11 cores=1,3,5,7
/proc/12670/stat : 12670 (python) S 12635 12635 12590 0 -1 4202560 1096 0 0 0 8 3 0 0 19 0 2 0 906971173 297947136 6135 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/12670/statm: 72741 6135 400 364 0 8656 0
[pid=12672] ppid=12670 vsize=10344 CPUtime=100.09 cores=1,3,5,7
/proc/12672/stat : 12672 (SApperloT-base) T 12670 12672 12672 0 -1 4202496 2497 0 0 0 10009 0 0 0 25 0 1 0 906971173 10592256 2233 33554432000 134512640 135674940 4290028880 18446744073709551615 134551104 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/12672/statm: 2586 2233 100 284 0 2300 0
[pid=12670/tid=12677] ppid=12635 vsize=290964 CPUtime=0 cores=1,3,5,7
/proc/12670/task/12677/stat : 12677 (python) S 12635 12635 12590 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 906981183 297947136 6135 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 514 18446604471124310784 0 0 -1 5 0 0 0
[pid=12678] ppid=12635 vsize=301220 CPUtime=2.63 cores=1,3,5,7
/proc/12678/stat : 12678 (python) S 12635 12635 12590 0 -1 4202560 1166 0 0 0 140 123 0 0 15 0 2 0 906981186 308449280 6140 33554432000 4194304 5683644 140734789549920 18446744073709551615 242970570310 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/12678/statm: 75305 6140 401 364 0 11217 0
[pid=12680] ppid=12678 vsize=141280 CPUtime=1963.57 cores=1,3,5,7
/proc/12680/stat : 12680 (cryptominisat-2) R 12678 12680 12680 0 -1 4202496 143625 0 0 0 196328 29 0 0 25 0 1 0 906981186 144670720 27230 33554432000 4194304 4677096 140733790776240 18446744073709551615 4320002 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/12680/statm: 35320 27230 393 118 0 30279 0
[pid=12678/tid=12693] ppid=12635 vsize=301220 CPUtime=0 cores=1,3,5,7
/proc/12678/task/12693/stat : 12693 (python) S 12635 12635 12590 0 -1 4202560 45 0 0 0 0 0 0 0 15 0 2 0 907029949 308449280 6140 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 514 18446604465743016000 0 0 -1 1 0 0 0
[pid=12694] ppid=12635 vsize=311476 CPUtime=0.45 cores=1,3,5,7
/proc/12694/stat : 12694 (python) S 12635 12635 12590 0 -1 4202560 1146 0 0 0 22 23 0 0 16 0 2 0 907029951 318951424 6158 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/12694/statm: 77869 6158 400 364 0 13778 0
[pid=12696] ppid=12694 vsize=20612 CPUtime=388.38 cores=1,3,5,7
/proc/12696/stat : 12696 (precosat) T 12694 12696 12696 0 -1 4202496 5906 0 0 0 38837 1 0 0 25 0 1 0 907029951 21106688 4859 33554432000 4194304 4890832 140733788528832 18446744073709551615 4238599 0 0 16781312 17442 18446744071562691360 0 0 17 5 0 0 0
/proc/12696/statm: 5153 4859 86 171 0 4980 0
[pid=12694/tid=12714] ppid=12635 vsize=311476 CPUtime=0 cores=1,3,5,7
/proc/12694/task/12714/stat : 12714 (python) S 12635 12635 12590 0 -1 4202560 45 0 0 0 0 0 0 0 16 0 2 0 907068790 318951424 6158 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 514 18446604471082437632 0 0 -1 3 0 0 0
[pid=12715] ppid=12635 vsize=321732 CPUtime=0.68 cores=1,3,5,7
/proc/12715/stat : 12715 (python) S 12635 12635 12590 0 -1 4202560 1150 0 0 0 39 29 0 0 19 0 2 0 907068792 329453568 6162 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/12715/statm: 80433 6162 400 364 0 16339 0
[pid=12717] ppid=12715 vsize=10400 CPUtime=586.57 cores=1,3,5,7
/proc/12717/stat : 12717 (lingeling) T 12715 12717 12717 0 -1 4202496 6153 0 0 0 58657 0 0 0 25 0 1 0 907068793 10649600 1770 33554432000 4194304 4352236 140737135784576 18446744073709551615 4279075 0 0 16781312 17506 18446744071562691360 0 0 17 5 0 0 0
/proc/12717/statm: 2600 1770 129 39 0 1680 0
[pid=12715/tid=12734] ppid=12635 vsize=321732 CPUtime=0 cores=1,3,5,7
/proc/12715/task/12734/stat : 12734 (python) S 12635 12635 12590 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 907127452 329453568 6162 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 514 18446604470990686400 0 0 -1 3 0 0 0
[pid=12735] ppid=12635 vsize=332392 CPUtime=0.24 cores=1,3,5,7
/proc/12735/stat : 12735 (python) S 12635 12635 12590 0 -1 4202560 1140 0 0 0 11 13 0 0 18 0 2 0 907127453 340369408 6252 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/12735/statm: 83098 6252 400 364 0 19001 0
[pid=12737] ppid=12735 vsize=9120 CPUtime=196.2 cores=1,3,5,7
/proc/12737/stat : 12737 (lingeling) T 12735 12737 12737 0 -1 4202496 3027 0 0 0 19620 0 0 0 25 0 1 0 907127454 9338880 1461 33554432000 4194304 4352236 140736127328320 18446744073709551615 4231006 0 0 16781312 17506 18446744071562691360 0 0 17 1 0 0 0
/proc/12737/statm: 2280 1461 129 39 0 1360 0
[pid=12735/tid=12740] ppid=12635 vsize=332392 CPUtime=0 cores=1,3,5,7
/proc/12735/task/12740/stat : 12740 (python) S 12635 12635 12590 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 907137365 340369408 6252 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 514 18446604471137257536 0 0 -1 3 0 0 0
[pid=12747] ppid=12635 vsize=342648 CPUtime=0.66 cores=1,3,5,7
/proc/12747/stat : 12747 (python) S 12635 12635 12590 0 -1 4202560 1167 0 0 0 36 30 0 0 16 0 2 0 907156791 350871552 6257 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/12747/statm: 85662 6257 401 364 0 21562 0
[pid=12749] ppid=12747 vsize=89752 CPUtime=488.07 cores=1,3,5,7
/proc/12749/stat : 12749 (cryptominisat-2) T 12747 12749 12749 0 -1 4202496 63863 0 0 0 48796 11 0 0 25 0 1 0 907156792 91906048 14736 33554432000 4194304 4677096 140734685082576 18446744073709551615 4319558 0 0 16781312 2 18446744071562691360 0 0 17 1 0 0 0
/proc/12749/statm: 22438 14736 393 118 0 17397 0
[pid=12747/tid=12775] ppid=12635 vsize=342648 CPUtime=0 cores=1,3,5,7
/proc/12747/task/12775/stat : 12775 (python) S 12635 12635 12590 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 907205600 350871552 6257 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 514 18446604467085317952 0 0 -1 5 0 0 0
[pid=12776] ppid=12635 vsize=418440 CPUtime=0.26 cores=1,3,5,7
/proc/12776/stat : 12776 (python) S 12635 12635 12590 0 -1 4202560 1188 0 0 0 13 13 0 0 16 0 2 0 907205602 428482560 6268 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/12776/statm: 104610 6268 406 364 0 40507 0
[pid=12778] ppid=12776 vsize=16544 CPUtime=196.22 cores=1,3,5,7
/proc/12778/stat : 12778 (precosat) T 12776 12778 12778 0 -1 4202496 4846 0 0 0 19622 0 0 0 25 0 1 0 907205602 16941056 3864 33554432000 4194304 4890832 140733665619984 18446744073709551615 4235497 0 0 16781312 17442 18446744071562691360 0 0 17 1 0 0 0
/proc/12778/statm: 4136 3864 86 171 0 3963 0
[pid=12776/tid=12785] ppid=12635 vsize=418440 CPUtime=0 cores=1,3,5,7
/proc/12776/task/12785/stat : 12785 (python) S 12635 12635 12590 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 907225226 428482560 6268 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 514 18446604470985984768 0 0 -1 3 0 0 0
[pid=12635/tid=12647] ppid=12631 vsize=428684 CPUtime=0 cores=1,3,5,7
/proc/12635/task/12647/stat : 12647 (python) S 12631 12635 12590 0 -1 4202560 32 934 0 0 0 0 0 0 18 0 9 0 906912610 438972416 7755 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 2 18446604471137258432 0 0 -1 5 0 0 0
[pid=12635/tid=12671] ppid=12631 vsize=428684 CPUtime=0 cores=1,3,5,7
/proc/12635/task/12671/stat : 12671 (python) S 12631 12635 12590 0 -1 4202560 32 934 0 0 0 0 0 0 20 0 9 0 906971173 438972416 7755 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 2 18446604471137258432 0 0 -1 3 0 0 0
[pid=12635/tid=12679] ppid=12631 vsize=428684 CPUtime=0 cores=1,3,5,7
/proc/12635/task/12679/stat : 12679 (python) S 12631 12635 12590 0 -1 4202560 37 934 0 0 0 0 0 0 18 0 9 0 906981186 438972416 7755 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 2 18446604471137258432 0 0 -1 3 0 0 0
[pid=12635/tid=12695] ppid=12631 vsize=428684 CPUtime=0 cores=1,3,5,7
/proc/12635/task/12695/stat : 12695 (python) S 12631 12635 12590 0 -1 4202560 30 934 0 0 0 0 0 0 20 0 9 0 907029951 438972416 7755 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 2 18446604471137258432 0 0 -1 3 0 0 0
[pid=12635/tid=12716] ppid=12631 vsize=428684 CPUtime=0 cores=1,3,5,7
/proc/12635/task/12716/stat : 12716 (python) S 12631 12635 12590 0 -1 4202560 30 934 0 0 0 0 0 0 20 0 9 0 907068793 438972416 7755 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 2 18446604471137258432 0 0 -1 7 0 0 0
[pid=12635/tid=12736] ppid=12631 vsize=428684 CPUtime=0 cores=1,3,5,7
/proc/12635/task/12736/stat : 12736 (python) S 12631 12635 12590 0 -1 4202560 29 934 0 0 0 0 0 0 18 0 9 0 907127454 438972416 7755 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 2 18446604471137258432 0 0 -1 7 0 0 0
[pid=12635/tid=12748] ppid=12631 vsize=428684 CPUtime=0 cores=1,3,5,7
/proc/12635/task/12748/stat : 12748 (python) S 12631 12635 12590 0 -1 4202560 22 934 0 0 0 0 0 0 20 0 9 0 907156791 438972416 7755 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 2 18446604471137258432 0 0 -1 5 0 0 0
[pid=12635/tid=12777] ppid=12631 vsize=428684 CPUtime=0 cores=1,3,5,7
/proc/12635/task/12777/stat : 12777 (python) S 12631 12635 12590 0 -1 4202560 30 934 0 0 0 0 0 0 20 0 9 0 907205602 438972416 7755 33554432000 4194304 5683644 140734789549920 18446744073709551615 242982373809 0 0 16781312 2 18446604471137258432 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 5000.1
Current children cumulated vsize (KiB) 3351148

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 12635 and gives
#  childrusage.ru_utime.tv_sec=1
#  childrusage.ru_utime.tv_usec=387789
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=181972
# CPU time returned by wait4() is 1.56976
# while last known CPU time is 5000.1
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5009.86
CPU time (s): 5000.1
CPU user time (s): 4996.57
CPU system time (s): 3.53
CPU usage (%): 99.8052
Max. virtual memory (cumulated for all children) (KiB): 3401516

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1.38779
system time used= 0.181972
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 18704
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 5153
involuntary context switches= 27

runsolver used 15.2417 second user time and 31.5562 second system time

The end

Launcher Data 

Begin job on node128 at 2011-06-13 10:18:39
IDJOB=3498288
IDBENCH=70775
IDSOLVER=1991
FILE ID=node128/3498288-1307953119
RUNJOBID= node128-1307953119-12609
PBS_JOBID= 13559700
Free space on /tmp= 74080 MiB

SOLVER NAME= borg-sat 11.03.02-seq-industrial (fixed)
BENCH NAME= SAT09/APPLICATIONS/bitverif/countbitsrotate/countbitsrotate032.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.industrial.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3498288-1307953119/watcher-3498288-1307953119 -o /tmp/evaluation-result-3498288-1307953119/solver-3498288-1307953119 -C 5000 -W 5100 -M 15500  python HOME/solve HOME/etc/borg-mix+class.industrial.pickle HOME/instance-3498288-1307953119.cnf -seed 1079452364 -budget 5000

TIME LIMIT= 5000 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 8d76aa747359ea8ccf15726792949452
RANDOM SEED=1079452364

node128.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5434.49
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      31169084 kB
Buffers:        322548 kB
Cached:        1209332 kB
SwapCached:      11368 kB
Active:         599072 kB
Inactive:       952420 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      31169084 kB
SwapTotal:    67111528 kB
SwapFree:     67052544 kB
Dirty:           77344 kB
Writeback:           0 kB
AnonPages:       17780 kB
Mapped:           9624 kB
Slab:           166304 kB
PageTables:       4652 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS:   188576 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 74004 MiB
End job on node128 at 2011-06-13 11:42:14