Trace number 3317878

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-industrial? (TO) 4800.13 1218.76

General information on the benchmark

Nameindustrial/jarvisalo/
eq.atree.braun.12.unsat.cnf
MD5SUM8d6acb9f6b752ad76c024133f5175c69
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables1694
Number of clauses5726
Sum of the clauses size14874
Maximum clause length23
Minimum clause length1
Number of clauses of size 13
Number of clauses of size 22490
Number of clauses of size 33143
Number of clauses of size 439
Number of clauses of size 538
Number of clauses of size over 513

Solver Data 

0.56/16.40	c INFO: loaded portfolio from HOME/etc/borg-mix+class.industrial.pickle
0.56/16.40	c INFO: solving HOME/instance-3317878-1304267285.cnf
0.56/16.40	c INFO: computing task features
0.56/16.42	c INFO: computed variable-clause graph statistics
0.56/16.42	c INFO: computed clause balance statistics
0.66/16.53	c INFO: computed variable balance statistics
0.76/16.61	c INFO: computed small-clause counts
0.76/16.61	c INFO: computed Horn-clause counts
0.76/16.62	c INFO: computed variable graph statistics
0.76/16.62	c INFO: collected features for HOME/instance-3317878-1304267285.cnf in 0.22 s
0.76/16.65	c INFO: running lingeling-276@0 for 97 with 4799 remaining (b = 0.28)
0.76/16.68	c INFO: running cryptominisat-2.9.0@0 for 485 with 4799 remaining (b = 0.36)
0.85/16.70	c INFO: running lingeling-276@0 for 873 with 4799 remaining (b = 0.45)
0.85/16.72	c INFO: running lingeling-276@0 for 1261 with 4799 remaining (b = 0.47)
400.64/116.80	c INFO: running cryptominisat-2.9.0@0 for 485 with 4699 remaining (b = 0.26)
1958.47/506.84	c INFO: running precosat-570@0 for 388 with 4309 remaining (b = 0.20)
2372.16/610.40	c INFO: running lingeling-276@0 for 485 with 4206 remaining (b = 0.23)
3515.55/896.75	c INFO: running lingeling-276@0 for 97 with 3919 remaining (b = 0.13)
3533.88/901.33	c INFO: running cryptominisat-2.9.0@0 for 388 with 3915 remaining (b = 0.15)
3915.59/996.90	c INFO: running cryptominisat-2.9.0@486 for 97 with 3819 remaining (b = 0.05)
4294.08/1091.70	c INFO: running lingeling-276@391 for 194 with 3724 remaining (b = 0.09)
4307.23/1095.01	c INFO: running SApperloT@0 for 97 with 3721 remaining (b = 0.08)
4706.07/1195.18	c INFO: running precosat-570@0 for 291 with 3621 remaining (b = 0.12)
4800.13/1218.75	c INFO: running cryptominisat-2.9.0@0 for 97 with 3597 remaining (b = 0.04)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317878-1304267285/watcher-3317878-1304267285 -o /tmp/evaluation-result-3317878-1304267285/solver-3317878-1304267285 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.industrial.pickle HOME/instance-3317878-1304267285.cnf -seed 344144970 -budget 4800 -cores 4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.00 0.00 0.00 2/174 23592
/proc/meminfo: memFree=23003956/32950928 swapFree=67111524/67111528
[pid=23592] ppid=23588 vsize=11800 CPUtime=0 cores=0,2,4,6
/proc/23592/stat : 23592 (runsolver) S 23588 23592 23557 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 1 0 538328558 12083200 55 33554432000 4194304 5434984 140735453966592 18446744073709551615 5055495 0 2147483391 4096 24578 18446744071699215550 0 0 17 6 0 0 0
/proc/23592/statm: 2950 55 34 303 0 2644 0

[startup+0.104545 s]
/proc/loadavg: 0.00 0.00 0.00 2/174 23592
/proc/meminfo: memFree=23003956/32950928 swapFree=67111524/67111528
[pid=23592] ppid=23588 vsize=14820 CPUtime=0 cores=0,2,4,6
/proc/23592/stat : 23592 (python) S 23588 23592 23557 0 -1 4202496 506 0 6 0 0 0 0 0 17 0 1 0 538328558 15175680 419 33554432000 4194304 5683644 140733384550480 18446744073709551615 274473963365 0 2147483391 4096 0 18446744071699215550 0 0 17 4 0 0 4
/proc/23592/statm: 3705 419 259 364 0 192 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 14820

[startup+0.200526 s]
/proc/loadavg: 0.00 0.00 0.00 2/174 23592
/proc/meminfo: memFree=23003956/32950928 swapFree=67111524/67111528
[pid=23592] ppid=23588 vsize=15084 CPUtime=0 cores=0,2,4,6
/proc/23592/stat : 23592 (python) D 23588 23592 23557 0 -1 4202496 619 0 6 0 0 0 0 0 17 0 1 0 538328558 15446016 521 33554432000 4194304 5683644 140733384550480 18446744073709551615 274473967360 0 0 16781312 2 18446744071562234166 0 0 17 4 0 0 5
/proc/23592/statm: 3771 521 302 364 0 258 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15084

[startup+0.300506 s]
/proc/loadavg: 0.00 0.00 0.00 2/174 23592
/proc/meminfo: memFree=23003956/32950928 swapFree=67111524/67111528
[pid=23592] ppid=23588 vsize=15184 CPUtime=0 cores=0,2,4,6
/proc/23592/stat : 23592 (python) S 23588 23592 23557 0 -1 4202496 719 0 6 0 0 0 0 0 16 0 1 0 538328558 15548416 615 33554432000 4194304 5683644 140733384550480 18446744073709551615 274473965568 0 2147483391 16781312 2 18446744071699215550 0 0 17 4 0 0 6
/proc/23592/statm: 3796 615 321 364 0 283 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15184

[startup+0.700438 s]
/proc/loadavg: 0.00 0.00 0.00 2/174 23592
/proc/meminfo: memFree=23003956/32950928 swapFree=67111524/67111528
[pid=23592] ppid=23588 vsize=15752 CPUtime=0 cores=0,2,4,6
/proc/23592/stat : 23592 (python) D 23588 23592 23557 0 -1 4202496 834 0 6 0 0 0 0 0 15 0 1 0 538328558 16130048 723 33554432000 4194304 5683644 140733384550480 18446744073709551615 274473965568 0 0 16781312 2 0 0 0 17 4 0 0 11
/proc/23592/statm: 3938 723 334 364 0 425 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15752

[startup+1.5003 s]
/proc/loadavg: 0.00 0.00 0.00 1/178 23596
/proc/meminfo: memFree=22996984/32950928 swapFree=67111524/67111528
[pid=23592] ppid=23588 vsize=16396 CPUtime=0.01 cores=0,2,4,6
/proc/23592/stat : 23592 (python) D 23588 23592 23557 0 -1 4202496 1111 0 6 0 1 0 0 0 16 0 1 0 538328558 16789504 984 33554432000 4194304 5683644 140733384550480 18446744073709551615 274473963365 0 0 16781312 2 0 0 0 17 4 0 0 24
/proc/23592/statm: 4099 984 409 364 0 586 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16396

[startup+3.1009 s]
/proc/loadavg: 0.08 0.02 0.01 3/178 23596
/proc/meminfo: memFree=22993540/32950928 swapFree=67111524/67111528
[pid=23592] ppid=23588 vsize=84604 CPUtime=0.1 cores=0,2,4,6
/proc/23592/stat : 23592 (python) D 23588 23592 23557 0 -1 4202496 2389 0 21 0 6 4 0 0 16 0 1 0 538328558 86634496 2136 33554432000 4194304 5683644 140733384550480 18446744073709551615 274473963365 0 0 16781312 2 0 0 0 17 4 0 0 33
/proc/23592/statm: 21151 2136 707 364 0 1368 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 84604
heavy processes:

[startup+6.30645 s]
/proc/loadavg: 0.08 0.02 0.01 1/178 23596
/proc/meminfo: memFree=22971056/32950928 swapFree=67111524/67111528
[pid=23592] ppid=23588 vsize=116604 CPUtime=0.19 cores=0,2,4,6
/proc/23592/stat : 23592 (python) S 23588 23592 23557 0 -1 4202496 3821 0 35 0 10 9 0 0 15 0 1 0 538328558 119402496 3491 33554432000 4194304 5683644 140733384550480 18446744073709551615 274473965568 0 2147483391 16781312 2 18446744071699215550 0 0 17 4 0 0 80
/proc/23592/statm: 29151 3491 895 364 0 2652 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 116604
heavy processes:

[startup+12.7003 s]
/proc/loadavg: 0.22 0.05 0.02 1/178 23604
/proc/meminfo: memFree=22957092/32950928 swapFree=67111524/67111528
[pid=23592] ppid=23588 vsize=168672 CPUtime=0.36 cores=0,2,4,6
/proc/23592/stat : 23592 (python) S 23588 23592 23557 0 -1 4202496 6664 927 61 8 20 16 0 0 15 0 1 0 538328558 172720128 4797 33554432000 4194304 5683644 140733384550480 18446744073709551615 274473965568 0 2147483391 16781312 2 18446744071699215550 0 0 17 4 0 0 154
/proc/23592/statm: 42168 4797 1331 364 0 3453 0
Current children cumulated CPU time (s) 0.36
Current children cumulated vsize (KiB) 168672
heavy processes:

[startup+25.501 s]
/proc/loadavg: 0.90 0.20 0.06 9/202 23629
/proc/meminfo: memFree=22834668/32950928 swapFree=67111524/67111528
[pid=23592] ppid=23588 vsize=320144 CPUtime=0.88 cores=0,2,4,6
/proc/23592/stat : 23592 (python) S 23588 23592 23557 0 -1 4202496 12093 927 117 8 63 25 0 0 24 0 5 0 538328558 327827456 7261 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485794251 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 217
/proc/23592/statm: 80036 7261 1881 364 0 15945 0
[pid=23611] ppid=23592 vsize=279132 CPUtime=0 cores=0,2,4,6
/proc/23611/stat : 23611 (python) S 23592 23592 23557 0 -1 4202560 970 0 0 0 0 0 0 0 15 0 1 0 538330224 285831168 5743 33554432000 4194304 5683644 140733384550480 18446744073709551615 274473987599 0 0 16781312 514 4176639508181123332 0 0 17 6 0 0 0
/proc/23611/statm: 69783 5743 381 364 0 5701 0
[pid=23613] ppid=23611 vsize=9696 CPUtime=8.82 cores=0,2,4,6
/proc/23613/stat : 23613 (lingeling) R 23611 23613 23613 0 -1 4202496 4144 0 0 0 882 0 0 0 25 0 1 0 538330225 9928704 1627 33554432000 4194304 4352236 140735544987184 18446744073709551615 4328116 0 0 16781312 17506 0 0 0 17 4 0 0 0
/proc/23613/statm: 2424 1627 129 39 0 1504 0
[pid=23617] ppid=23592 vsize=289388 CPUtime=0 cores=0,2,4,6
/proc/23617/stat : 23617 (python) S 23592 23592 23557 0 -1 4202560 906 0 0 0 0 0 0 0 15 0 1 0 538330226 296333312 5736 33554432000 4194304 5683644 140733384550480 18446744073709551615 274473987654 0 0 16781312 514 5601785463166697732 0 0 17 2 0 0 0
/proc/23617/statm: 72347 5736 370 364 0 8262 0
[pid=23619] ppid=23617 vsize=41236 CPUtime=8.74 cores=0,2,4,6
/proc/23619/stat : 23619 (cryptominisat-2) R 23617 23619 23619 0 -1 4202496 7650 0 0 0 874 0 0 0 25 0 1 0 538330227 42225664 3546 33554432000 4194304 4677096 140733808106976 18446744073709551615 4335980 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/23619/statm: 10309 3546 392 118 0 5268 0
[pid=23623] ppid=23592 vsize=299644 CPUtime=0 cores=0,2,4,6
/proc/23623/stat : 23623 (python) S 23592 23592 23557 0 -1 4202560 919 0 0 0 0 0 0 0 18 0 1 0 538330229 306835456 5742 33554432000 4194304 5683644 140733384550480 18446744073709551615 274473987654 0 0 16781312 514 4103716048736321796 0 0 17 0 0 0 0
/proc/23623/statm: 74911 5742 370 364 0 10823 0
[pid=23625] ppid=23623 vsize=9136 CPUtime=8.77 cores=0,2,4,6
/proc/23625/stat : 23625 (lingeling) R 23623 23625 23625 0 -1 4202496 3398 0 0 0 877 0 0 0 25 0 1 0 538330229 9355264 1486 33554432000 4194304 4352236 140735766811232 18446744073709551615 4327985 0 0 16781312 17506 0 0 0 17 2 0 0 0
/proc/23625/statm: 2284 1486 129 39 0 1364 0
[pid=23626] ppid=23592 vsize=309900 CPUtime=0 cores=0,2,4,6
/proc/23626/stat : 23626 (python) S 23592 23592 23557 0 -1 4202560 929 0 0 0 0 0 0 0 18 0 1 0 538330231 317337600 5747 33554432000 4194304 5683644 140733384550480 18446744073709551615 274473987654 0 0 16781312 514 4596386593299726596 0 0 17 0 0 0 0
/proc/23626/statm: 77475 5747 371 364 0 13384 0
[pid=23628] ppid=23626 vsize=9200 CPUtime=8.74 cores=0,2,4,6
/proc/23628/stat : 23628 (lingeling) R 23626 23628 23628 0 -1 4202496 3255 0 0 0 874 0 0 0 25 0 1 0 538330231 9420800 1502 33554432000 4194304 4352236 140734004180464 18446744073709551615 4328035 0 0 16781312 17506 0 0 0 17 0 0 0 0
/proc/23628/statm: 2300 1502 129 39 0 1380 0
[pid=23592/tid=23612] ppid=23588 vsize=320144 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23612/stat : 23612 (python) S 23588 23592 23557 0 -1 4202560 25 927 0 8 0 0 0 0 18 0 5 0 538330224 327827456 7261 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 0 0 0 0
[pid=23592/tid=23618] ppid=23588 vsize=320144 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23618/stat : 23618 (python) S 23588 23592 23557 0 -1 4202560 22 927 0 8 0 0 0 0 20 0 5 0 538330226 327827456 7261 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 2 0 0 0
[pid=23592/tid=23624] ppid=23588 vsize=320144 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23624/stat : 23624 (python) S 23588 23592 23557 0 -1 4202560 35 927 0 8 0 0 0 0 22 0 5 0 538330229 327827456 7261 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 0 0 0 0
[pid=23592/tid=23627] ppid=23588 vsize=320144 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23627/stat : 23627 (python) S 23588 23592 23557 0 -1 4202560 25 927 0 8 0 0 0 0 24 0 5 0 538330231 327827456 7261 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 35.95
Current children cumulated vsize (KiB) 1567476

[startup+51.1014 s]
/proc/loadavg: 3.46 0.85 0.28 9/202 23629
/proc/meminfo: memFree=22795180/32950928 swapFree=67111524/67111528
[pid=23592] ppid=23588 vsize=320144 CPUtime=0.88 cores=0,2,4,6
/proc/23592/stat : 23592 (python) S 23588 23592 23557 0 -1 4202496 12093 927 117 8 63 25 0 0 24 0 5 0 538328558 327827456 7261 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485794251 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 217
/proc/23592/statm: 80036 7261 1881 364 0 15945 0
[pid=23611] ppid=23592 vsize=279132 CPUtime=0.03 cores=0,2,4,6
/proc/23611/stat : 23611 (python) S 23592 23592 23557 0 -1 4202560 985 0 0 0 1 2 0 0 16 0 1 0 538330224 285831168 5744 33554432000 4194304 5683644 140733384550480 18446744073709551615 274473987599 0 0 16781312 514 16679925644896731396 0 0 17 6 0 0 0
/proc/23611/statm: 69783 5744 382 364 0 5701 0
[pid=23613] ppid=23611 vsize=12136 CPUtime=34.42 cores=0,2,4,6
/proc/23613/stat : 23613 (lingeling) R 23611 23613 23613 0 -1 4202496 9682 0 0 0 3441 1 0 0 25 0 1 0 538330225 12427264 2237 33554432000 4194304 4352236 140735544987184 18446744073709551615 4328194 0 0 16781312 17506 0 0 0 17 4 0 0 0
/proc/23613/statm: 3034 2237 129 39 0 2114 0
[pid=23617] ppid=23592 vsize=289388 CPUtime=0.05 cores=0,2,4,6
/proc/23617/stat : 23617 (python) S 23592 23592 23557 0 -1 4202560 920 0 0 0 3 2 0 0 15 0 1 0 538330226 296333312 5737 33554432000 4194304 5683644 140733384550480 18446744073709551615 274473987654 0 0 16781312 514 16679925644896731396 0 0 17 2 0 0 0
/proc/23617/statm: 72347 5737 371 364 0 8262 0
[pid=23619] ppid=23617 vsize=52500 CPUtime=34.17 cores=0,2,4,6
/proc/23619/stat : 23619 (cryptominisat-2) R 23617 23619 23619 0 -1 4202496 18235 0 0 0 3415 2 0 0 25 0 1 0 538330227 53760000 6287 33554432000 4194304 4677096 140733808106976 18446744073709551615 4320152 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/23619/statm: 13125 6287 392 118 0 8084 0
[pid=23623] ppid=23592 vsize=299644 CPUtime=0.03 cores=0,2,4,6
/proc/23623/stat : 23623 (python) S 23592 23592 23557 0 -1 4202560 933 0 0 0 2 1 0 0 18 0 1 0 538330229 306835456 5743 33554432000 4194304 5683644 140733384550480 18446744073709551615 274473987654 0 0 16781312 514 16679925644896731396 0 0 17 0 0 0 0
/proc/23623/statm: 74911 5743 371 364 0 10823 0
[pid=23625] ppid=23623 vsize=12864 CPUtime=34.32 cores=0,2,4,6
/proc/23625/stat : 23625 (lingeling) R 23623 23625 23625 0 -1 4202496 9802 0 0 0 3431 1 0 0 25 0 1 0 538330229 13172736 2418 33554432000 4194304 4352236 140735766811232 18446744073709551615 4231316 0 0 16781312 17506 0 0 0 17 2 0 0 0
/proc/23625/statm: 3216 2418 129 39 0 2296 0
[pid=23626] ppid=23592 vsize=309900 CPUtime=0.04 cores=0,2,4,6
/proc/23626/stat : 23626 (python) S 23592 23592 23557 0 -1 4202560 943 0 0 0 3 1 0 0 18 0 1 0 538330231 317337600 5747 33554432000 4194304 5683644 140733384550480 18446744073709551615 274473987654 0 0 16781312 514 16679925644896731396 0 0 17 0 0 0 0
/proc/23626/statm: 77475 5747 371 364 0 13384 0
[pid=23628] ppid=23626 vsize=13688 CPUtime=34.26 cores=0,2,4,6
/proc/23628/stat : 23628 (lingeling) R 23626 23628 23628 0 -1 4202496 9401 0 0 0 3425 1 0 0 25 0 1 0 538330231 14016512 2624 33554432000 4194304 4352236 140734004180464 18446744073709551615 4328894 0 0 16781312 17506 0 0 0 17 0 0 0 0
/proc/23628/statm: 3422 2624 129 39 0 2502 0
[pid=23592/tid=23612] ppid=23588 vsize=320144 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23612/stat : 23612 (python) S 23588 23592 23557 0 -1 4202560 25 927 0 8 0 0 0 0 18 0 5 0 538330224 327827456 7261 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 0 0 0 0
[pid=23592/tid=23618] ppid=23588 vsize=320144 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23618/stat : 23618 (python) S 23588 23592 23557 0 -1 4202560 22 927 0 8 0 0 0 0 20 0 5 0 538330226 327827456 7261 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 2 0 0 0
[pid=23592/tid=23624] ppid=23588 vsize=320144 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23624/stat : 23624 (python) S 23588 23592 23557 0 -1 4202560 35 927 0 8 0 0 0 0 22 0 5 0 538330229 327827456 7261 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 0 0 0 0
[pid=23592/tid=23627] ppid=23588 vsize=320144 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23627/stat : 23627 (python) S 23588 23592 23557 0 -1 4202560 25 927 0 8 0 0 0 0 24 0 5 0 538330231 327827456 7261 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 138.2
Current children cumulated vsize (KiB) 1589396

[startup+102.301 s]
/proc/loadavg: 6.32 2.08 0.73 9/202 23631
/proc/meminfo: memFree=22748836/32950928 swapFree=67111524/67111528
[pid=23592] ppid=23588 vsize=320144 CPUtime=0.88 cores=0,2,4,6
/proc/23592/stat : 23592 (python) S 23588 23592 23557 0 -1 4202496 12093 927 117 8 63 25 0 0 24 0 5 0 538328558 327827456 7261 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485794251 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 217
/proc/23592/statm: 80036 7261 1881 364 0 15945 0
[pid=23611] ppid=23592 vsize=279132 CPUtime=0.11 cores=0,2,4,6
/proc/23611/stat : 23611 (python) S 23592 23592 23557 0 -1 4202560 1005 0 0 0 5 6 0 0 15 0 1 0 538330224 285831168 5744 33554432000 4194304 5683644 140733384550480 18446744073709551615 274473987599 0 0 16781312 514 16679925644896731396 0 0 17 6 0 0 0
/proc/23611/statm: 69783 5744 382 364 0 5701 0
[pid=23613] ppid=23611 vsize=18668 CPUtime=85.63 cores=0,2,4,6
/proc/23613/stat : 23613 (lingeling) R 23611 23613 23613 0 -1 4202496 18067 0 0 0 8560 3 0 0 25 0 1 0 538330225 19116032 3870 33554432000 4194304 4352236 140735544987184 18446744073709551615 4329086 0 0 16781312 17506 0 0 0 17 4 0 0 0
/proc/23613/statm: 4667 3870 129 39 0 3747 0
[pid=23617] ppid=23592 vsize=289388 CPUtime=0.12 cores=0,2,4,6
/proc/23617/stat : 23617 (python) S 23592 23592 23557 0 -1 4202560 945 0 0 0 7 5 0 0 15 0 1 0 538330226 296333312 5737 33554432000 4194304 5683644 140733384550480 18446744073709551615 274473987654 0 0 16781312 514 16093468137168535812 0 0 17 2 0 0 0
/proc/23617/statm: 72347 5737 371 364 0 8262 0

################
# More data... #
################

/proc/23637/statm: 80039 5781 400 364 0 15945 0
[pid=23639] ppid=23637 vsize=100072 CPUtime=585.27 cores=0,2,4,6
/proc/23639/stat : 23639 (cryptominisat-2) T 23637 23639 23639 0 -1 4202496 118004 0 0 0 58488 39 0 0 25 0 1 0 538340239 102473728 17821 33554432000 4194304 4677096 140735509246912 18446744073709551615 4318902 0 0 16781312 2 18446744071562691360 0 0 17 6 0 0 0
/proc/23639/statm: 25018 17821 402 118 0 19977 0
[pid=23637/tid=23662] ppid=23592 vsize=320156 CPUtime=0 cores=0,2,4,6
/proc/23637/task/23662/stat : 23662 (python) S 23592 23592 23557 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 538389596 327839744 5781 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 514 18446604453860693056 0 0 -1 0 0 0 0
[pid=23652] ppid=23592 vsize=330412 CPUtime=0.52 cores=0,2,4,6
/proc/23652/stat : 23652 (python) S 23592 23592 23557 0 -1 4202560 1158 0 0 0 29 23 0 0 19 0 2 0 538379242 338341888 5785 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485794251 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/23652/statm: 82603 5785 400 364 0 18506 0
[pid=23654] ppid=23652 vsize=31136 CPUtime=391.64 cores=0,2,4,6
/proc/23654/stat : 23654 (precosat) T 23652 23654 23654 0 -1 4202496 8401 0 0 0 39161 3 0 0 25 0 1 0 538379243 31883264 7592 33554432000 4194304 4890832 140736199698032 18446744073709551615 4238577 0 0 16781312 17442 18446744071562691360 0 0 17 6 0 0 0
/proc/23654/statm: 7784 7592 86 171 0 7611 0
[pid=23652/tid=23679] ppid=23592 vsize=330412 CPUtime=0 cores=0,2,4,6
/proc/23652/task/23679/stat : 23679 (python) S 23592 23592 23557 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 538418689 338341888 5785 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 514 18446604453892854976 0 0 -1 0 0 0 0
[pid=23676] ppid=23592 vsize=340660 CPUtime=0.31 cores=0,2,4,6
/proc/23676/stat : 23676 (python) S 23592 23592 23557 0 -1 4202560 1167 0 0 0 18 13 0 0 18 0 2 0 538418234 348835840 5759 33554432000 4194304 5683644 140733384550480 18446744073709551615 274473987654 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/23676/statm: 85165 5759 400 364 0 21065 0
[pid=23678] ppid=23676 vsize=20996 CPUtime=226.76 cores=0,2,4,6
/proc/23678/stat : 23678 (lingeling) R 23676 23678 23678 0 -1 4202496 33439 0 0 0 22670 6 0 0 25 0 1 0 538418234 21499904 4451 33554432000 4194304 4352236 140735600916912 18446744073709551615 4328075 0 0 16781312 17506 0 0 0 17 0 0 0 0
/proc/23678/statm: 5249 4451 129 39 0 4329 0
[pid=23676/tid=23684] ppid=23592 vsize=340660 CPUtime=0 cores=0,2,4,6
/proc/23676/task/23684/stat : 23684 (python) S 23592 23592 23557 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 538428247 348835840 5759 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 514 18446604453867416256 0 0 -1 0 0 0 0
[pid=23688] ppid=23592 vsize=351384 CPUtime=0.15 cores=0,2,4,6
/proc/23688/stat : 23688 (python) S 23592 23592 23557 0 -1 4202560 1192 0 0 0 7 8 0 0 19 0 2 0 538438060 359817216 5886 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485794251 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/23688/statm: 87846 5886 406 364 0 23743 0
[pid=23690] ppid=23688 vsize=7388 CPUtime=99.67 cores=0,2,4,6
/proc/23690/stat : 23690 (SApperloT-base) T 23688 23690 23690 0 -1 4202496 1840 0 0 0 9965 2 0 0 25 0 1 0 538438060 7565312 1566 33554432000 134512640 135674940 4294578944 18446744073709551615 134564250 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/23690/statm: 1847 1566 100 284 0 1561 0
[pid=23688/tid=23697] ppid=23592 vsize=351384 CPUtime=0 cores=0,2,4,6
/proc/23688/task/23697/stat : 23697 (python) S 23592 23592 23557 0 -1 4202560 51 0 0 0 0 0 0 0 19 0 2 0 538448074 359817216 5886 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 514 18446604440278547456 0 0 -1 0 0 0 0
[pid=23698] ppid=23592 vsize=361640 CPUtime=0.03 cores=0,2,4,6
/proc/23698/stat : 23698 (python) S 23592 23592 23557 0 -1 4202560 955 0 0 0 2 1 0 0 18 0 1 0 538448076 370319360 5854 33554432000 4194304 5683644 140733384550480 18446744073709551615 274473987654 0 0 16781312 514 16679925644896731394 0 0 17 4 0 0 0
/proc/23698/statm: 90410 5854 371 364 0 26304 0
[pid=23700] ppid=23698 vsize=11116 CPUtime=23.48 cores=0,2,4,6
/proc/23700/stat : 23700 (precosat) R 23698 23700 23700 0 -1 4202496 2917 0 0 0 2348 0 0 0 25 0 1 0 538448077 11382784 2634 33554432000 4194304 4890832 140733234578528 18446744073709551615 4238599 0 0 16781312 17442 0 0 0 17 4 0 0 0
/proc/23700/statm: 2779 2634 86 171 0 2606 0
[pid=23592/tid=23612] ppid=23588 vsize=371884 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23612/stat : 23612 (python) S 23588 23592 23557 0 -1 4202560 30 927 0 8 0 0 0 0 18 0 10 0 538330224 380809216 7379 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 0 0 0 0
[pid=23592/tid=23618] ppid=23588 vsize=371884 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23618/stat : 23618 (python) S 23588 23592 23557 0 -1 4202560 28 927 0 8 0 0 0 0 18 0 10 0 538330226 380809216 7379 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 2 0 0 0
[pid=23592/tid=23624] ppid=23588 vsize=371884 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23624/stat : 23624 (python) S 23588 23592 23557 0 -1 4202560 35 927 0 8 0 0 0 0 22 0 10 0 538330229 380809216 7379 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 0 0 0 0
[pid=23592/tid=23627] ppid=23588 vsize=371884 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23627/stat : 23627 (python) S 23588 23592 23557 0 -1 4202560 25 927 0 8 0 0 0 0 24 0 10 0 538330231 380809216 7379 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 0 0 0 0
[pid=23592/tid=23638] ppid=23588 vsize=371884 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23638/stat : 23638 (python) S 23588 23592 23557 0 -1 4202560 34 927 0 8 0 0 0 0 18 0 10 0 538340239 380809216 7379 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 0 0 0 0
[pid=23592/tid=23653] ppid=23588 vsize=371884 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23653/stat : 23653 (python) S 23588 23592 23557 0 -1 4202560 32 927 0 8 0 0 0 0 20 0 10 0 538379243 380809216 7379 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 0 0 0 0
[pid=23592/tid=23677] ppid=23588 vsize=371884 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23677/stat : 23677 (python) S 23588 23592 23557 0 -1 4202560 30 927 0 8 0 0 0 0 18 0 10 0 538418234 380809216 7379 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 0 0 0 0
[pid=23592/tid=23689] ppid=23588 vsize=371884 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23689/stat : 23689 (python) S 23588 23592 23557 0 -1 4202560 31 927 0 8 0 0 0 0 20 0 10 0 538438060 380809216 7379 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 0 0 0 0
[pid=23592/tid=23699] ppid=23588 vsize=371884 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23699/stat : 23699 (python) S 23588 23592 23557 0 -1 4202560 27 927 0 8 0 0 0 0 20 0 10 0 538448077 380809216 7379 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.13
Current children cumulated vsize (KiB) 3659072

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1218.7 s]
/proc/loadavg: 8.12 7.97 5.87 9/246 23704
/proc/meminfo: memFree=22201920/32950928 swapFree=67111524/67111528
[pid=23592] ppid=23588 vsize=371884 CPUtime=1.1 cores=0,2,4,6
/proc/23592/stat : 23592 (python) S 23588 23592 23557 0 -1 4202496 16369 927 117 8 83 27 0 0 19 0 10 0 538328558 380809216 7379 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485794251 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 217
/proc/23592/statm: 92971 7379 1892 364 0 28865 0
[pid=23611] ppid=23592 vsize=289376 CPUtime=0.78 cores=0,2,4,6
/proc/23611/stat : 23611 (python) S 23592 23592 23557 0 -1 4202560 1204 0 0 0 43 35 0 0 15 0 2 0 538330224 296321024 5783 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485794251 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/23611/statm: 72344 5783 416 364 0 8262 0
[pid=23613] ppid=23611 vsize=32364 CPUtime=584.71 cores=0,2,4,6
/proc/23613/stat : 23613 (lingeling) T 23611 23613 23613 0 -1 4202496 61651 0 0 0 58461 10 0 0 25 0 1 0 538330225 33140736 7294 33554432000 4194304 4352236 140735544987184 18446744073709551615 4328194 0 0 16781312 17506 18446744071562691360 0 0 17 4 0 0 0
/proc/23613/statm: 8091 7294 129 39 0 7171 0
[pid=23611/tid=23636] ppid=23592 vsize=289376 CPUtime=0 cores=0,2,4,6
/proc/23611/task/23636/stat : 23636 (python) S 23592 23592 23557 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 538340237 296321024 5783 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 514 18446604453885142912 0 0 -1 6 0 0 0
[pid=23617] ppid=23592 vsize=289388 CPUtime=1.12 cores=0,2,4,6
/proc/23617/stat : 23617 (python) S 23592 23592 23557 0 -1 4202560 1163 0 0 0 62 50 0 0 15 0 2 0 538330226 296333312 5767 33554432000 4194304 5683644 140733384550480 18446744073709551615 274473987654 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/23617/statm: 72347 5767 400 364 0 8262 0
[pid=23619] ppid=23617 vsize=122092 CPUtime=801.64 cores=0,2,4,6
/proc/23619/stat : 23619 (cryptominisat-2) R 23617 23619 23619 0 -1 4202496 156512 0 0 0 80116 48 0 0 25 0 1 0 538330227 125022208 24096 33554432000 4194304 4677096 140733808106976 18446744073709551615 4337464 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/23619/statm: 30523 24096 392 118 0 25482 0
[pid=23617/tid=23651] ppid=23592 vsize=289388 CPUtime=0 cores=0,2,4,6
/proc/23617/task/23651/stat : 23651 (python) S 23592 23592 23557 0 -1 4202560 45 0 0 0 0 0 0 0 16 0 2 0 538379240 296333312 5767 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 514 18446604453846264512 0 0 -1 2 0 0 0
[pid=23623] ppid=23592 vsize=299644 CPUtime=1.16 cores=0,2,4,6
/proc/23623/stat : 23623 (python) S 23592 23592 23557 0 -1 4202560 1140 0 0 0 64 52 0 0 19 0 2 0 538330229 306835456 5773 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485794251 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/23623/statm: 74911 5773 400 364 0 10823 0
[pid=23625] ppid=23623 vsize=34776 CPUtime=879.23 cores=0,2,4,6
/proc/23625/stat : 23625 (lingeling) T 23623 23625 23625 0 -1 4202496 81740 0 0 0 87911 12 0 0 25 0 1 0 538330229 35610624 7896 33554432000 4194304 4352236 140735766811232 18446744073709551615 4328194 0 0 16781312 17506 18446744071562691360 0 0 17 2 0 0 0
/proc/23625/statm: 8694 7896 129 39 0 7774 0
[pid=23623/tid=23675] ppid=23592 vsize=299644 CPUtime=0 cores=0,2,4,6
/proc/23623/task/23675/stat : 23675 (python) S 23592 23592 23557 0 -1 4202560 44 0 0 0 0 0 0 0 19 0 2 0 538418232 306835456 5773 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 514 18446604453868164096 0 0 -1 0 0 0 0
[pid=23626] ppid=23592 vsize=309900 CPUtime=1.61 cores=0,2,4,6
/proc/23626/stat : 23626 (python) S 23592 23592 23557 0 -1 4202560 998 0 0 0 87 74 0 0 18 0 1 0 538330231 317337600 5747 33554432000 4194304 5683644 140733384550480 18446744073709551615 274473987654 0 0 16781312 514 16679925644896731396 0 0 17 0 0 0 0
/proc/23626/statm: 77475 5747 371 364 0 13384 0
[pid=23628] ppid=23626 vsize=34688 CPUtime=1200.12 cores=0,2,4,6
/proc/23628/stat : 23628 (lingeling) R 23626 23628 23628 0 -1 4202496 103099 0 0 0 119997 15 0 0 25 0 1 0 538330231 35520512 7874 33554432000 4194304 4352236 140734004180464 18446744073709551615 4328304 0 0 16781312 17506 0 0 0 17 2 0 0 0
/proc/23628/statm: 8672 7874 129 39 0 7752 0
[pid=23637] ppid=23592 vsize=320156 CPUtime=0.83 cores=0,2,4,6
/proc/23637/stat : 23637 (python) S 23592 23592 23557 0 -1 4202560 1174 0 0 0 45 38 0 0 15 0 2 0 538340239 327839744 5781 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485794251 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/23637/statm: 80039 5781 400 364 0 15945 0
[pid=23639] ppid=23637 vsize=100072 CPUtime=585.27 cores=0,2,4,6
/proc/23639/stat : 23639 (cryptominisat-2) T 23637 23639 23639 0 -1 4202496 118004 0 0 0 58488 39 0 0 25 0 1 0 538340239 102473728 17821 33554432000 4194304 4677096 140735509246912 18446744073709551615 4318902 0 0 16781312 2 18446744071562691360 0 0 17 6 0 0 0
/proc/23639/statm: 25018 17821 402 118 0 19977 0
[pid=23637/tid=23662] ppid=23592 vsize=320156 CPUtime=0 cores=0,2,4,6
/proc/23637/task/23662/stat : 23662 (python) S 23592 23592 23557 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 538389596 327839744 5781 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 514 18446604453860693056 0 0 -1 0 0 0 0
[pid=23652] ppid=23592 vsize=330412 CPUtime=0.52 cores=0,2,4,6
/proc/23652/stat : 23652 (python) S 23592 23592 23557 0 -1 4202560 1158 0 0 0 29 23 0 0 19 0 2 0 538379242 338341888 5785 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485794251 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/23652/statm: 82603 5785 400 364 0 18506 0
[pid=23654] ppid=23652 vsize=31136 CPUtime=391.64 cores=0,2,4,6
/proc/23654/stat : 23654 (precosat) T 23652 23654 23654 0 -1 4202496 8401 0 0 0 39161 3 0 0 25 0 1 0 538379243 31883264 7592 33554432000 4194304 4890832 140736199698032 18446744073709551615 4238577 0 0 16781312 17442 18446744071562691360 0 0 17 6 0 0 0
/proc/23654/statm: 7784 7592 86 171 0 7611 0
[pid=23652/tid=23679] ppid=23592 vsize=330412 CPUtime=0 cores=0,2,4,6
/proc/23652/task/23679/stat : 23679 (python) S 23592 23592 23557 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 538418689 338341888 5785 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 514 18446604453892854976 0 0 -1 0 0 0 0
[pid=23676] ppid=23592 vsize=340660 CPUtime=0.31 cores=0,2,4,6
/proc/23676/stat : 23676 (python) S 23592 23592 23557 0 -1 4202560 1167 0 0 0 18 13 0 0 18 0 2 0 538418234 348835840 5759 33554432000 4194304 5683644 140733384550480 18446744073709551615 274473987654 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/23676/statm: 85165 5759 400 364 0 21065 0
[pid=23678] ppid=23676 vsize=20996 CPUtime=226.76 cores=0,2,4,6
/proc/23678/stat : 23678 (lingeling) R 23676 23678 23678 0 -1 4202496 33439 0 0 0 22670 6 0 0 25 0 1 0 538418234 21499904 4451 33554432000 4194304 4352236 140735600916912 18446744073709551615 4328075 0 0 16781312 17506 0 0 0 17 0 0 0 0
/proc/23678/statm: 5249 4451 129 39 0 4329 0
[pid=23676/tid=23684] ppid=23592 vsize=340660 CPUtime=0 cores=0,2,4,6
/proc/23676/task/23684/stat : 23684 (python) S 23592 23592 23557 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 538428247 348835840 5759 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 514 18446604453867416256 0 0 -1 0 0 0 0
[pid=23688] ppid=23592 vsize=351384 CPUtime=0.15 cores=0,2,4,6
/proc/23688/stat : 23688 (python) S 23592 23592 23557 0 -1 4202560 1192 0 0 0 7 8 0 0 19 0 2 0 538438060 359817216 5886 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485794251 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/23688/statm: 87846 5886 406 364 0 23743 0
[pid=23690] ppid=23688 vsize=7388 CPUtime=99.67 cores=0,2,4,6
/proc/23690/stat : 23690 (SApperloT-base) T 23688 23690 23690 0 -1 4202496 1840 0 0 0 9965 2 0 0 25 0 1 0 538438060 7565312 1566 33554432000 134512640 135674940 4294578944 18446744073709551615 134564250 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/23690/statm: 1847 1566 100 284 0 1561 0
[pid=23688/tid=23697] ppid=23592 vsize=351384 CPUtime=0 cores=0,2,4,6
/proc/23688/task/23697/stat : 23697 (python) S 23592 23592 23557 0 -1 4202560 51 0 0 0 0 0 0 0 19 0 2 0 538448074 359817216 5886 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 514 18446604440278547456 0 0 -1 0 0 0 0
[pid=23698] ppid=23592 vsize=361640 CPUtime=0.03 cores=0,2,4,6
/proc/23698/stat : 23698 (python) S 23592 23592 23557 0 -1 4202560 955 0 0 0 2 1 0 0 18 0 1 0 538448076 370319360 5854 33554432000 4194304 5683644 140733384550480 18446744073709551615 274473987654 0 0 16781312 514 16679925644896731394 0 0 17 4 0 0 0
/proc/23698/statm: 90410 5854 371 364 0 26304 0
[pid=23700] ppid=23698 vsize=11116 CPUtime=23.48 cores=0,2,4,6
/proc/23700/stat : 23700 (precosat) R 23698 23700 23700 0 -1 4202496 2917 0 0 0 2348 0 0 0 25 0 1 0 538448077 11382784 2634 33554432000 4194304 4890832 140733234578528 18446744073709551615 4238599 0 0 16781312 17442 0 0 0 17 4 0 0 0
/proc/23700/statm: 2779 2634 86 171 0 2606 0
[pid=23592/tid=23612] ppid=23588 vsize=371884 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23612/stat : 23612 (python) S 23588 23592 23557 0 -1 4202560 30 927 0 8 0 0 0 0 18 0 10 0 538330224 380809216 7379 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 0 0 0 0
[pid=23592/tid=23618] ppid=23588 vsize=371884 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23618/stat : 23618 (python) S 23588 23592 23557 0 -1 4202560 28 927 0 8 0 0 0 0 18 0 10 0 538330226 380809216 7379 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 2 0 0 0
[pid=23592/tid=23624] ppid=23588 vsize=371884 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23624/stat : 23624 (python) S 23588 23592 23557 0 -1 4202560 35 927 0 8 0 0 0 0 22 0 10 0 538330229 380809216 7379 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 0 0 0 0
[pid=23592/tid=23627] ppid=23588 vsize=371884 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23627/stat : 23627 (python) S 23588 23592 23557 0 -1 4202560 25 927 0 8 0 0 0 0 24 0 10 0 538330231 380809216 7379 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 0 0 0 0
[pid=23592/tid=23638] ppid=23588 vsize=371884 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23638/stat : 23638 (python) S 23588 23592 23557 0 -1 4202560 34 927 0 8 0 0 0 0 18 0 10 0 538340239 380809216 7379 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 0 0 0 0
[pid=23592/tid=23653] ppid=23588 vsize=371884 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23653/stat : 23653 (python) S 23588 23592 23557 0 -1 4202560 32 927 0 8 0 0 0 0 20 0 10 0 538379243 380809216 7379 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 0 0 0 0
[pid=23592/tid=23677] ppid=23588 vsize=371884 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23677/stat : 23677 (python) S 23588 23592 23557 0 -1 4202560 30 927 0 8 0 0 0 0 18 0 10 0 538418234 380809216 7379 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 0 0 0 0
[pid=23592/tid=23689] ppid=23588 vsize=371884 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23689/stat : 23689 (python) S 23588 23592 23557 0 -1 4202560 31 927 0 8 0 0 0 0 20 0 10 0 538438060 380809216 7379 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 0 0 0 0
[pid=23592/tid=23699] ppid=23588 vsize=371884 CPUtime=0 cores=0,2,4,6
/proc/23592/task/23699/stat : 23699 (python) S 23588 23592 23557 0 -1 4202560 27 927 0 8 0 0 0 0 20 0 10 0 538448077 380809216 7379 33554432000 4194304 5683644 140733384550480 18446744073709551615 274485791153 0 0 16781312 2 18446604453885142016 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.13
Current children cumulated vsize (KiB) 3659072

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 23592 and gives
#  childrusage.ru_utime.tv_sec=1204
#  childrusage.ru_utime.tv_usec=445896
#  childrusage.ru_stime.tv_sec=3
#  childrusage.ru_stime.tv_usec=427478
# CPU time returned by wait4() is 1207.87
# while last known CPU time is 4800.13
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1218.76
CPU time (s): 4800.13
CPU user time (s): 4795.57
CPU system time (s): 4.56
CPU usage (%): 393.853
Max. virtual memory (cumulated for all children) (KiB): 3705380

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1204.45
system time used= 3.42748
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 131000
page faults= 125
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 28247
involuntary context switches= 4323

runsolver used 2.58661 second user time and 6.05308 second system time

The end

Launcher Data 

Begin job on node125 at 2011-05-01 18:28:06
IDJOB=3317878
IDBENCH=20468
IDSOLVER=1706
FILE ID=node125/3317878-1304267285
RUNJOBID= node125-1304267283-23575
PBS_JOBID= 13173954
Free space on /tmp= 73120 MiB

SOLVER NAME= borg-sat 11.03.02-industrial
BENCH NAME= SAT07/industrial/jarvisalo/eq.atree.braun.12.unsat.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.industrial.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317878-1304267285/watcher-3317878-1304267285 -o /tmp/evaluation-result-3317878-1304267285/solver-3317878-1304267285 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.industrial.pickle HOME/instance-3317878-1304267285.cnf -seed 344144970 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 8d6acb9f6b752ad76c024133f5175c69
RANDOM SEED=344144970

node125.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5266.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      23004624 kB
Buffers:       2138420 kB
Cached:        7216824 kB
SwapCached:          4 kB
Active:        2648472 kB
Inactive:      6770708 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      23004624 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:           64392 kB
Writeback:           0 kB
AnonPages:       63872 kB
Mapped:          14544 kB
Slab:           463620 kB
PageTables:       4620 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   186096 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73056 MiB
End job on node125 at 2011-05-01 18:48:30