Trace number 3317731

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-industrial? (MO) 3840.08 1068.9

General information on the benchmark

NameSAT-Race-2010-CNF/hardware-verification/
velev/11pipe_11_ooo.cnf
MD5SUMf603f11fbfe855df381282b7bce010ea
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables88289
Number of clauses4187694
Sum of the clauses size12410354
Maximum clause length341
Minimum clause length1
Number of clauses of size 11
Number of clauses of size 23770000
Number of clauses of size 3347602
Number of clauses of size 41622
Number of clauses of size 51351
Number of clauses of size over 567118

Solver Data 

0.38/0.77	c INFO: loaded portfolio from HOME/etc/borg-mix+class.industrial.pickle
0.38/0.77	c INFO: solving HOME/instance-3317731-1304271185.cnf
0.38/0.77	c INFO: computing task features
14.45/14.88	c INFO: computed variable-clause graph statistics
14.94/15.38	c INFO: computed clause balance statistics
80.94/81.30	c INFO: computed variable balance statistics
124.64/125.00	c INFO: computed small-clause counts
125.25/125.60	c INFO: computed Horn-clause counts
140.65/141.00	c INFO: computed variable graph statistics
140.65/141.01	c INFO: collected features for HOME/instance-3317731-1304271185.cnf in 139.62 s
140.65/141.04	c INFO: running lingeling-276@0 for 97 with 4659 remaining (b = 0.54)
140.65/141.07	c INFO: running SApperloT@0 for 97 with 4659 remaining (b = 0.08)
140.75/141.11	c INFO: running cryptominisat-2.9.0@0 for 485 with 4659 remaining (b = 0.33)
140.75/141.15	c INFO: running lingeling-276@0 for 485 with 4659 remaining (b = 0.30)
540.30/241.25	c INFO: running SApperloT@0 for 97 with 4559 remaining (b = 0.04)
540.69/241.30	c INFO: running cryptominisat-2.9.0@0 for 485 with 4559 remaining (b = 0.20)
940.33/341.47	c INFO: running lingeling-276@0 for 1164 with 4459 remaining (b = 0.33)
2096.59/631.08	c INFO: running lingeling-276@0 for 97 with 4169 remaining (b = 0.32)
2103.75/632.85	c INFO: running precosat-570@0 for 291 with 4167 remaining (b = 0.13)
2496.06/731.32	c INFO: running precosat-570@0 for 97 with 4069 remaining (b = 0.06)
2514.73/736.04	c INFO: running lingeling-276@489 for 776 with 4064 remaining (b = 0.15)
2891.48/830.60	c INFO: running lingeling-276@0 for 97 with 3970 remaining (b = 0.09)
3286.50/929.85	c INFO: running cryptominisat-2.9.0@0 for 388 with 3870 remaining (b = 0.07)
3292.89/931.41	c INFO: running cryptominisat-2.9.0@99 for 1552 with 3869 remaining (b = 0.31)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317731-1304271185/watcher-3317731-1304271185 -o /tmp/evaluation-result-3317731-1304271185/solver-3317731-1304271185 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.industrial.pickle HOME/instance-3317731-1304271185.cnf -seed 1199711537 -budget 4800 -cores 4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.17 0.89 3.50 2/227 2733
/proc/meminfo: memFree=28341972/32950928 swapFree=67111448/67111528
[pid=2732] ppid=2730 vsize=15184 CPUtime=0 cores=0,2,4,6
/proc/2732/stat : 2732 (python) S 2730 2732 1708 0 -1 4202496 728 0 0 0 0 0 0 0 20 0 1 0 538717523 15548416 617 33554432000 4194304 5683644 140737215696368 18446744073709551615 251814238208 0 2147483391 16781312 2 18446744071699215550 0 0 17 4 0 0 0
/proc/2732/statm: 3796 617 322 364 0 283 0

[startup+0.082335 s]
/proc/loadavg: 0.17 0.89 3.50 2/227 2733
/proc/meminfo: memFree=28341972/32950928 swapFree=67111448/67111528
[pid=2732] ppid=2730 vsize=36508 CPUtime=0.04 cores=0,2,4,6
/proc/2732/stat : 2732 (python) S 2730 2732 1708 0 -1 4202496 1614 0 0 0 3 1 0 0 17 0 1 0 538717523 37384192 1445 33554432000 4194304 5683644 140737215696368 18446744073709551615 251814238208 0 2147483391 16781312 2 18446744071699215550 0 0 17 4 0 0 0
/proc/2732/statm: 9127 1445 452 364 0 944 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 36508

[startup+0.10028 s]
/proc/loadavg: 0.17 0.89 3.50 2/227 2733
/proc/meminfo: memFree=28341972/32950928 swapFree=67111448/67111528
[pid=2732] ppid=2730 vsize=52412 CPUtime=0.05 cores=0,2,4,6
/proc/2732/stat : 2732 (python) S 2730 2732 1708 0 -1 4202496 2016 0 0 0 3 2 0 0 17 0 1 0 538717523 53669888 1775 33554432000 4194304 5683644 140737215696368 18446744073709551615 251814238208 0 2147483391 16781312 2 18446744071699215550 0 0 17 4 0 0 0
/proc/2732/statm: 13103 1775 484 364 0 1284 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 52412

[startup+0.300315 s]
/proc/loadavg: 0.17 0.89 3.50 2/227 2733
/proc/meminfo: memFree=28341972/32950928 swapFree=67111448/67111528
[pid=2732] ppid=2730 vsize=135588 CPUtime=0.19 cores=0,2,4,6
/proc/2732/stat : 2732 (python) R 2730 2732 1708 0 -1 4202496 5887 934 1 0 13 6 0 0 17 0 1 0 538717523 138842112 4131 33554432000 4194304 5683644 140737215696368 18446744073709551615 251814238208 0 2147483391 16781312 2 0 0 0 17 4 0 0 0
/proc/2732/statm: 33897 4131 1031 364 0 3083 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 135588

[startup+0.700297 s]
/proc/loadavg: 0.17 0.89 3.50 2/227 2733
/proc/meminfo: memFree=28341972/32950928 swapFree=67111448/67111528
[pid=2732] ppid=2730 vsize=246576 CPUtime=0.38 cores=0,2,4,6
/proc/2732/stat : 2732 (python) R 2730 2732 1708 0 -1 4202496 8227 934 1 0 26 12 0 0 15 0 1 0 538717523 252493824 6124 33554432000 4194304 5683644 140737215696368 18446744073709551615 4609872 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/2732/statm: 61644 6124 1621 364 0 4646 0
Current children cumulated CPU time (s) 0.38
Current children cumulated vsize (KiB) 246576

[startup+1.50116 s]
/proc/loadavg: 0.17 0.89 3.50 3/232 2749
/proc/meminfo: memFree=28002864/32950928 swapFree=67111448/67111528
[pid=2732] ppid=2730 vsize=404416 CPUtime=1.14 cores=0,2,4,6
/proc/2732/stat : 2732 (python) R 2730 2732 1708 0 -1 4202496 40607 934 1 0 95 19 0 0 16 0 1 0 538717523 414121984 38288 33554432000 4194304 5683644 140737215696368 18446744073709551615 4445115 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/2732/statm: 101104 38288 1761 364 0 37028 0
Current children cumulated CPU time (s) 1.14
Current children cumulated vsize (KiB) 404416

[startup+3.10078 s]
/proc/loadavg: 0.17 0.89 3.50 4/232 2749
/proc/meminfo: memFree=27857536/32950928 swapFree=67111448/67111528
[pid=2732] ppid=2730 vsize=507180 CPUtime=2.74 cores=0,2,4,6
/proc/2732/stat : 2732 (python) R 2730 2732 1708 0 -1 4202496 66328 934 1 0 251 23 0 0 18 0 1 0 538717523 519352320 64009 33554432000 4194304 5683644 140737215696368 18446744073709551615 47998531269897 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/2732/statm: 126795 64009 1761 364 0 62719 0
Current children cumulated CPU time (s) 2.74
Current children cumulated vsize (KiB) 507180

[startup+6.30024 s]
/proc/loadavg: 0.48 0.94 3.50 3/232 2749
/proc/meminfo: memFree=27314832/32950928 swapFree=67111448/67111528
[pid=2732] ppid=2730 vsize=731372 CPUtime=5.95 cores=0,2,4,6
/proc/2732/stat : 2732 (python) R 2730 2732 1708 0 -1 4202496 119764 934 1 0 564 31 0 0 25 0 1 0 538717523 748924928 117445 33554432000 4194304 5683644 140737215696368 18446744073709551615 251813900997 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/2732/statm: 182843 117445 1761 364 0 118767 0
Current children cumulated CPU time (s) 5.95
Current children cumulated vsize (KiB) 731372

[startup+12.7002 s]
/proc/loadavg: 0.60 0.95 3.49 3/232 2749
/proc/meminfo: memFree=26631664/32950928 swapFree=67111448/67111528
[pid=2732] ppid=2730 vsize=1118028 CPUtime=12.35 cores=0,2,4,6
/proc/2732/stat : 2732 (python) R 2730 2732 1708 0 -1 4202496 215203 934 1 0 1190 45 0 0 25 0 1 0 538717523 1144860672 212884 33554432000 4194304 5683644 140737215696368 18446744073709551615 4436508 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/2732/statm: 279507 212884 1761 364 0 215431 0
Current children cumulated CPU time (s) 12.35
Current children cumulated vsize (KiB) 1118028

[startup+25.501 s]
/proc/loadavg: 0.91 1.01 3.47 3/232 2750
/proc/meminfo: memFree=25795540/32950928 swapFree=67111448/67111528
[pid=2732] ppid=2730 vsize=1104448 CPUtime=25.14 cores=0,2,4,6
/proc/2732/stat : 2732 (python) R 2730 2732 1708 0 -1 4202496 426329 934 1 0 2445 69 0 0 25 0 1 0 538717523 1130954752 213529 33554432000 4194304 5683644 140737215696368 18446744073709551615 47998363193672 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/2732/statm: 276112 213529 1826 364 0 212036 0
Current children cumulated CPU time (s) 25.14
Current children cumulated vsize (KiB) 1104448

[startup+51.1008 s]
/proc/loadavg: 1.28 1.08 3.43 3/232 2751
/proc/meminfo: memFree=25891808/32950928 swapFree=67111448/67111528
[pid=2732] ppid=2730 vsize=1104448 CPUtime=50.74 cores=0,2,4,6
/proc/2732/stat : 2732 (python) R 2730 2732 1708 0 -1 4202496 426329 934 1 0 5005 69 0 0 25 0 1 0 538717523 1130954752 213529 33554432000 4194304 5683644 140737215696368 18446744073709551615 4540565 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/2732/statm: 276112 213529 1826 364 0 212036 0
Current children cumulated CPU time (s) 50.74
Current children cumulated vsize (KiB) 1104448

[startup+102.3 s]
/proc/loadavg: 1.69 1.22 3.35 3/232 2752
/proc/meminfo: memFree=25891696/32950928 swapFree=67111448/67111528
[pid=2732] ppid=2730 vsize=1104448 CPUtime=101.94 cores=0,2,4,6
/proc/2732/stat : 2732 (python) R 2730 2732 1708 0 -1 4202496 426330 934 1 0 10125 69 0 0 25 0 1 0 538717523 1130954752 213530 33554432000 4194304 5683644 140737215696368 18446744073709551615 47998363181426 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/2732/statm: 276112 213530 1827 364 0 212036 0
Current children cumulated CPU time (s) 101.94
Current children cumulated vsize (KiB) 1104448

[startup+162.301 s]
/proc/loadavg: 2.73 1.56 3.33 6/244 2766
/proc/meminfo: memFree=24963900/32950928 swapFree=67111448/67111528
[pid=2732] ppid=2730 vsize=777652 CPUtime=140.8 cores=0,2,4,6
/proc/2732/stat : 2732 (python) S 2730 2732 1708 0 -1 4202496 478048 934 1 0 13998 82 0 0 23 0 5 0 538717523 796315648 121642 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826066891 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/2732/statm: 194413 121642 1881 364 0 130322 0
[pid=2754] ppid=2732 vsize=736640 CPUtime=0.03 cores=0,2,4,6
/proc/2754/stat : 2754 (python) S 2732 2732 1708 0 -1 4202560 996 0 0 0 1 2 0 0 18 0 1 0 538731628 754319360 120127 33554432000 4194304 5683644 140737215696368 18446744073709551615 251814260239 0 0 16781312 514 8804396534020931842 0 0 17 2 0 0 0
/proc/2754/statm: 184160 120127 382 364 0 120078 0
[pid=2756] ppid=2754 vsize=179156 CPUtime=21.15 cores=0,2,4,6
/proc/2756/stat : 2756 (lingeling) R 2754 2756 2756 0 -1 4202496 87336 0 0 0 2102 13 0 0 25 0 1 0 538731629 183455744 43959 33554432000 4194304 4352236 140733793707296 18446744073709551615 4328026 0 0 16781312 17506 0 0 0 17 4 0 0 0
/proc/2756/statm: 44789 43959 129 39 0 43869 0
[pid=2757] ppid=2732 vsize=746896 CPUtime=0.03 cores=0,2,4,6
/proc/2757/stat : 2757 (python) S 2732 2732 1708 0 -1 4202560 916 0 0 0 1 2 0 0 18 0 1 0 538731631 764821504 120119 33554432000 4194304 5683644 140737215696368 18446744073709551615 251814260294 0 0 16781312 514 8804396534020931843 0 0 17 4 0 0 0
/proc/2757/statm: 186724 120119 370 364 0 122639 0
[pid=2759] ppid=2757 vsize=180448 CPUtime=21.14 cores=0,2,4,6
/proc/2759/stat : 2759 (SApperloT-base) R 2757 2759 2759 0 -1 4202496 39922 0 0 0 2103 11 0 0 25 0 1 0 538731634 184778752 39461 33554432000 134512640 135674940 4287089888 18446744073709551615 134556795 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/2759/statm: 45112 39461 99 284 0 44826 0
[pid=2760] ppid=2732 vsize=757152 CPUtime=0.04 cores=0,2,4,6
/proc/2760/stat : 2760 (python) S 2732 2732 1708 0 -1 4202560 929 0 0 0 1 3 0 0 15 0 1 0 538731635 775323648 120123 33554432000 4194304 5683644 140737215696368 18446744073709551615 251814260294 0 0 16781312 514 6729156301818921218 0 0 17 0 0 0 0
/proc/2760/statm: 189288 120123 370 364 0 125200 0
[pid=2763] ppid=2760 vsize=929988 CPUtime=21.11 cores=0,2,4,6
/proc/2763/stat : 2763 (cryptominisat-2) R 2760 2763 2763 0 -1 4202496 194796 0 0 0 2083 28 0 0 25 0 1 0 538731637 952307712 192366 33554432000 4194304 4677096 140736649612192 18446744073709551615 4586829 0 0 16781312 2 0 0 0 17 0 0 0 0
/proc/2763/statm: 232497 192366 372 118 0 227456 0
[pid=2764] ppid=2732 vsize=767408 CPUtime=0.03 cores=0,2,4,6
/proc/2764/stat : 2764 (python) S 2732 2732 1708 0 -1 4202560 944 0 0 0 1 2 0 0 18 0 1 0 538731639 785825792 120127 33554432000 4194304 5683644 140737215696368 18446744073709551615 251814260294 0 0 16781312 514 8804396534020931844 0 0 17 4 0 0 0
/proc/2764/statm: 191852 120127 370 364 0 127761 0
[pid=2766] ppid=2764 vsize=179156 CPUtime=20.96 cores=0,2,4,6
/proc/2766/stat : 2766 (lingeling) R 2764 2766 2766 0 -1 4202496 87347 0 0 0 2084 12 0 0 25 0 1 0 538731640 183455744 43959 33554432000 4194304 4352236 140734494992560 18446744073709551615 4256428 0 0 16781312 17506 0 0 0 17 6 0 0 0
/proc/2766/statm: 44789 43959 129 39 0 43869 0
[pid=2732/tid=2755] ppid=2730 vsize=777652 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2755/stat : 2755 (python) S 2730 2732 1708 0 -1 4202560 25 934 0 0 0 0 0 0 25 0 5 0 538731629 796315648 121642 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 6 0 0 0
[pid=2732/tid=2758] ppid=2730 vsize=777652 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2758/stat : 2758 (python) S 2730 2732 1708 0 -1 4202560 24 934 0 0 0 0 0 0 20 0 5 0 538731633 796315648 121642 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 2 0 0 0
[pid=2732/tid=2762] ppid=2730 vsize=777652 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2762/stat : 2762 (python) S 2730 2732 1708 0 -1 4202560 22 934 0 0 0 0 0 0 22 0 5 0 538731636 796315648 121642 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 4 0 0 0
[pid=2732/tid=2765] ppid=2730 vsize=777652 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2765/stat : 2765 (python) S 2730 2732 1708 0 -1 4202560 25 934 0 0 0 0 0 0 24 0 5 0 538731640 796315648 121642 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 225.29
Current children cumulated vsize (KiB) 5254496

[startup+222.301 s]
/proc/loadavg: 4.17 2.18 3.43 6/244 2768
/proc/meminfo: memFree=24528248/32950928 swapFree=67111448/67111528
[pid=2732] ppid=2730 vsize=777652 CPUtime=140.8 cores=0,2,4,6
/proc/2732/stat : 2732 (python) S 2730 2732 1708 0 -1 4202496 478048 934 1 0 13998 82 0 0 23 0 5 0 538717523 796315648 121642 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826066891 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/2732/statm: 194413 121642 1881 364 0 130322 0
[pid=2754] ppid=2732 vsize=736640 CPUtime=0.13 cores=0,2,4,6
/proc/2754/stat : 2754 (python) S 2732 2732 1708 0 -1 4202560 1023 0 0 0 6 7 0 0 18 0 1 0 538731628 754319360 120127 33554432000 4194304 5683644 140737215696368 18446744073709551615 251814260239 0 0 16781312 514 8804396534020931842 0 0 17 2 0 0 0
/proc/2754/statm: 184160 120127 382 364 0 120078 0
[pid=2756] ppid=2754 vsize=189188 CPUtime=80.95 cores=0,2,4,6
/proc/2756/stat : 2756 (lingeling) R 2754 2756 2756 0 -1 4202496 125798 0 0 0 8077 18 0 0 25 0 1 0 538731629 193728512 46467 33554432000 4194304 4352236 140733793707296 18446744073709551615 4278667 0 0 16781312 17506 0 0 0 17 4 0 0 0
/proc/2756/statm: 47297 46467 129 39 0 46377 0
[pid=2757] ppid=2732 vsize=746896 CPUtime=0.13 cores=0,2,4,6
/proc/2757/stat : 2757 (python) S 2732 2732 1708 0 -1 4202560 937 0 0 0 7 6 0 0 18 0 1 0 538731631 764821504 120119 33554432000 4194304 5683644 140737215696368 18446744073709551615 251814260294 0 0 16781312 514 8804396534020931843 0 0 17 4 0 0 0
/proc/2757/statm: 186724 120119 370 364 0 122639 0
[pid=2759] ppid=2757 vsize=200720 CPUtime=81.03 cores=0,2,4,6
/proc/2759/stat : 2759 (SApperloT-base) R 2757 2759 2759 0 -1 4202496 44928 0 0 0 8090 13 0 0 25 0 1 0 538731634 205537280 44402 33554432000 134512640 135674940 4287089888 18446744073709551615 134564017 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/2759/statm: 50180 44402 100 284 0 49894 0
[pid=2760] ppid=2732 vsize=757152 CPUtime=0.16 cores=0,2,4,6
/proc/2760/stat : 2760 (python) S 2732 2732 1708 0 -1 4202560 959 0 0 0 8 8 0 0 15 0 1 0 538731635 775323648 120123 33554432000 4194304 5683644 140737215696368 18446744073709551615 251814260294 0 0 16781312 514 8804396534020931842 0 0 17 0 0 0 0
/proc/2760/statm: 189288 120123 370 364 0 125200 0
[pid=2763] ppid=2760 vsize=1510568 CPUtime=80.99 cores=0,2,4,6
/proc/2763/stat : 2763 (cryptominisat-2) R 2760 2763 2763 0 -1 4202496 298773 0 0 0 8057 42 0 0 25 0 1 0 538731637 1546821632 284624 33554432000 4194304 4677096 140736649612192 18446744073709551615 4502912 0 0 16781312 2 0 0 0 17 0 0 0 0
/proc/2763/statm: 377642 284624 391 118 0 372601 0
[pid=2764] ppid=2732 vsize=767408 CPUtime=0.14 cores=0,2,4,6
/proc/2764/stat : 2764 (python) S 2732 2732 1708 0 -1 4202560 968 0 0 0 7 7 0 0 18 0 1 0 538731639 785825792 120127 33554432000 4194304 5683644 140737215696368 18446744073709551615 251814260294 0 0 16781312 514 8804396534020931844 0 0 17 4 0 0 0
/proc/2764/statm: 191852 120127 370 364 0 127761 0
[pid=2766] ppid=2764 vsize=210964 CPUtime=80.51 cores=0,2,4,6
/proc/2766/stat : 2766 (lingeling) R 2764 2766 2766 0 -1 4202496 131386 0 0 0 8034 17 0 0 25 0 1 0 538731640 216027136 51911 33554432000 4194304 4352236 140734494992560 18446744073709551615 4328100 0 0 16781312 17506 0 0 0 17 6 0 0 0
/proc/2766/statm: 52741 51911 129 39 0 51821 0
[pid=2732/tid=2755] ppid=2730 vsize=777652 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2755/stat : 2755 (python) S 2730 2732 1708 0 -1 4202560 25 934 0 0 0 0 0 0 25 0 5 0 538731629 796315648 121642 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 6 0 0 0

################
# More data... #
################

/proc/2799/stat : 2799 (precosat) T 2797 2799 2799 0 -1 4202496 139660 0 0 0 29465 22 0 0 25 0 1 0 538780810 573902848 137814 33554432000 4194304 4890832 140737250156160 18446744073709551615 4273723 0 0 16781312 17442 18446744071562691360 0 0 17 6 0 0 0
/proc/2799/statm: 140113 137814 86 171 0 139940 0
[pid=2797/tid=2837] ppid=2732 vsize=808432 CPUtime=0 cores=0,2,4,6
/proc/2797/task/2837/stat : 2837 (python) S 2732 2732 1708 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538810662 827834368 120179 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 514 18446604453869824768 0 0 -1 4 0 0 0
[pid=2817] ppid=2732 vsize=818688 CPUtime=0.19 cores=0,2,4,6
/proc/2817/stat : 2817 (python) S 2732 2732 1708 0 -1 4202560 1120 0 0 0 9 10 0 0 19 0 2 0 538790656 838336512 120177 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/2817/statm: 204672 120177 399 364 0 140566 0
[pid=2819] ppid=2817 vsize=560440 CPUtime=99.01 cores=0,2,4,6
/proc/2819/stat : 2819 (precosat) T 2817 2819 2819 0 -1 4202496 139415 0 0 0 9877 24 0 0 25 0 1 0 538790657 573890560 137589 33554432000 4194304 4890832 140735606623424 18446744073709551615 4293300 0 0 16781312 17442 18446744071562691360 0 0 17 4 0 0 0
/proc/2819/statm: 140110 137589 86 171 0 139937 0
[pid=2817/tid=2829] ppid=2732 vsize=818688 CPUtime=0 cores=0,2,4,6
/proc/2817/task/2829/stat : 2829 (python) S 2732 2732 1708 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 538800581 838336512 120177 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 514 18446604448700623616 0 0 -1 2 0 0 0
[pid=2830] ppid=2732 vsize=828944 CPUtime=0.2 cores=0,2,4,6
/proc/2830/stat : 2830 (python) S 2732 2732 1708 0 -1 4202560 1131 0 0 0 9 11 0 0 19 0 2 0 538800584 848838656 120181 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/2830/statm: 207236 120181 399 364 0 143127 0
[pid=2832] ppid=2830 vsize=205216 CPUtime=97.25 cores=0,2,4,6
/proc/2832/stat : 2832 (lingeling) T 2830 2832 2832 0 -1 4202496 129849 0 0 0 9706 19 0 0 25 0 1 0 538800586 210141184 50474 33554432000 4194304 4352236 140734624896240 18446744073709551615 4309132 0 0 16781312 17506 18446744071562691360 0 0 17 6 0 0 0
/proc/2832/statm: 51304 50474 129 39 0 50384 0
[pid=2830/tid=2836] ppid=2732 vsize=828944 CPUtime=0 cores=0,2,4,6
/proc/2830/task/2836/stat : 2836 (python) S 2732 2732 1708 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538810506 848838656 120181 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 514 18446604445571144768 0 0 -1 6 0 0 0
[pid=2732/tid=2755] ppid=2730 vsize=839188 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2755/stat : 2755 (python) S 2730 2732 1708 0 -1 4202560 30 934 0 0 0 0 0 0 18 0 11 0 538731629 859328512 121679 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 6 0 0 0
[pid=2732/tid=2758] ppid=2730 vsize=839188 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2758/stat : 2758 (python) S 2730 2732 1708 0 -1 4202560 24 934 0 0 0 0 0 0 20 0 11 0 538731633 859328512 121679 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 2 0 0 0
[pid=2732/tid=2762] ppid=2730 vsize=839188 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2762/stat : 2762 (python) S 2730 2732 1708 0 -1 4202560 25 934 0 0 0 0 0 0 18 0 11 0 538731636 859328512 121679 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 4 0 0 0
[pid=2732/tid=2765] ppid=2730 vsize=839188 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2765/stat : 2765 (python) S 2730 2732 1708 0 -1 4202560 25 934 0 0 0 0 0 0 24 0 11 0 538731640 859328512 121679 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 4 0 0 0
[pid=2732/tid=2772] ppid=2730 vsize=839188 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2772/stat : 2772 (python) S 2730 2732 1708 0 -1 4202560 28 934 0 0 0 0 0 0 20 0 11 0 538741651 859328512 121679 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 4 0 0 0
[pid=2732/tid=2778] ppid=2730 vsize=839188 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2778/stat : 2778 (python) S 2730 2732 1708 0 -1 4202560 32 934 0 0 0 0 0 0 18 0 11 0 538741656 859328512 121679 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 4 0 0 0
[pid=2732/tid=2794] ppid=2730 vsize=839188 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2794/stat : 2794 (python) S 2730 2732 1708 0 -1 4202560 33 934 0 0 0 0 0 0 18 0 11 0 538780634 859328512 121679 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 4 0 0 0
[pid=2732/tid=2798] ppid=2730 vsize=839188 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2798/stat : 2798 (python) S 2730 2732 1708 0 -1 4202560 31 934 0 0 0 0 0 0 20 0 11 0 538780810 859328512 121679 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 4 0 0 0
[pid=2732/tid=2818] ppid=2730 vsize=839188 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2818/stat : 2818 (python) S 2730 2732 1708 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 11 0 538790657 859328512 121679 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 4 0 0 0
[pid=2732/tid=2831] ppid=2730 vsize=839188 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2831/stat : 2831 (python) S 2730 2732 1708 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 11 0 538800585 859328512 121679 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3840.08
Current children cumulated vsize (KiB) 15903232

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1068.8 s]
/proc/loadavg: 6.90 6.93 5.41 5/269 2874
/proc/meminfo: memFree=20116824/32950928 swapFree=67111448/67111528
[pid=2732] ppid=2730 vsize=839188 CPUtime=141.15 cores=0,2,4,6
/proc/2732/stat : 2732 (python) S 2730 2732 1708 0 -1 4202496 483821 934 1 0 14023 92 0 0 18 0 11 0 538717523 859328512 121679 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826066891 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/2732/statm: 209797 121679 1894 364 0 145688 0
[pid=2754] ppid=2732 vsize=746884 CPUtime=1.46 cores=0,2,4,6
/proc/2754/stat : 2754 (python) S 2732 2732 1708 0 -1 4202560 1220 0 0 0 77 69 0 0 18 0 2 0 538731628 764809216 120166 33554432000 4194304 5683644 140737215696368 18446744073709551615 251814260294 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/2754/statm: 186721 120166 416 364 0 122639 0
[pid=2756] ppid=2754 vsize=230244 CPUtime=823.27 cores=0,2,4,6
/proc/2756/stat : 2756 (lingeling) R 2754 2756 2756 0 -1 4202496 367712 0 0 0 82281 46 0 0 25 0 1 0 538731629 235769856 56731 33554432000 4194304 4352236 140733793707296 18446744073709551615 4328026 0 0 16781312 17506 0 0 0 17 6 0 0 0
/proc/2756/statm: 57561 56731 129 39 0 56641 0
[pid=2754/tid=2770] ppid=2732 vsize=746884 CPUtime=0 cores=0,2,4,6
/proc/2754/task/2770/stat : 2770 (python) S 2732 2732 1708 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 538741647 764809216 120166 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 514 18446604453869823872 0 0 -1 2 0 0 0
[pid=2757] ppid=2732 vsize=746896 CPUtime=0.17 cores=0,2,4,6
/proc/2757/stat : 2757 (python) S 2732 2732 1708 0 -1 4202560 1095 0 0 0 9 8 0 0 19 0 2 0 538731631 764821504 120149 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/2757/statm: 186724 120149 399 364 0 122639 0
[pid=2759] ppid=2757 vsize=200856 CPUtime=99.96 cores=0,2,4,6
/proc/2759/stat : 2759 (SApperloT-base) T 2757 2759 2759 0 -1 4202496 44967 0 0 0 9983 13 0 0 25 0 1 0 538731634 205676544 44441 33554432000 134512640 135674940 4287089888 18446744073709551615 134564017 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/2759/statm: 50214 44441 100 284 0 49928 0
[pid=2757/tid=2775] ppid=2732 vsize=746896 CPUtime=0 cores=0,2,4,6
/proc/2757/task/2775/stat : 2775 (python) S 2732 2732 1708 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 538741652 764821504 120149 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 514 18446604452414601152 0 0 -1 4 0 0 0
[pid=2760] ppid=2732 vsize=757152 CPUtime=1.2 cores=0,2,4,6
/proc/2760/stat : 2760 (python) S 2732 2732 1708 0 -1 4202560 1183 0 0 0 65 55 0 0 15 0 2 0 538731635 775323648 120154 33554432000 4194304 5683644 140737215696368 18446744073709551615 251814260294 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/2760/statm: 189288 120154 400 364 0 125200 0
[pid=2763] ppid=2760 vsize=2333420 CPUtime=629.45 cores=0,2,4,6
/proc/2763/stat : 2763 (cryptominisat-2) R 2760 2763 2763 0 -1 4202496 637300 0 0 0 62836 109 0 0 25 0 1 0 538731637 2389422080 484994 33554432000 4194304 4677096 140736649612192 18446744073709551615 4564132 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/2763/statm: 583355 484994 391 118 0 578314 0
[pid=2760/tid=2796] ppid=2732 vsize=757152 CPUtime=0 cores=0,2,4,6
/proc/2760/task/2796/stat : 2796 (python) S 2732 2732 1708 0 -1 4202560 49 0 0 0 0 0 0 0 16 0 2 0 538780806 775323648 120154 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 514 18446604447628206336 0 0 -1 0 0 0 0
[pid=2764] ppid=2732 vsize=767408 CPUtime=0.82 cores=0,2,4,6
/proc/2764/stat : 2764 (python) S 2732 2732 1708 0 -1 4202560 1166 0 0 0 47 35 0 0 19 0 2 0 538731639 785825792 120157 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/2764/statm: 191852 120157 399 364 0 127761 0
[pid=2766] ppid=2764 vsize=221872 CPUtime=488.29 cores=0,2,4,6
/proc/2766/stat : 2766 (lingeling) T 2764 2766 2766 0 -1 4202496 299291 0 0 0 48791 38 0 0 25 0 1 0 538731640 227196928 54638 33554432000 4194304 4352236 140734494992560 18446744073709551615 4328026 0 0 16781312 17506 18446744071562691360 0 0 17 2 0 0 0
/proc/2766/statm: 55468 54638 129 39 0 54548 0
[pid=2764/tid=2792] ppid=2732 vsize=767408 CPUtime=0 cores=0,2,4,6
/proc/2764/task/2792/stat : 2792 (python) S 2732 2732 1708 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538780630 785825792 120157 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 514 18446604453857020864 0 0 -1 4 0 0 0
[pid=2771] ppid=2732 vsize=777664 CPUtime=0.17 cores=0,2,4,6
/proc/2771/stat : 2771 (python) S 2732 2732 1708 0 -1 4202560 1133 0 0 0 9 8 0 0 19 0 2 0 538741649 796327936 120161 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/2771/statm: 194416 120161 399 364 0 130322 0
[pid=2773] ppid=2771 vsize=210988 CPUtime=99.98 cores=0,2,4,6
/proc/2773/stat : 2773 (SApperloT-base) T 2771 2773 2773 0 -1 4202496 47388 0 0 0 9984 14 0 0 25 0 1 0 538741651 216051712 46925 33554432000 134512640 135674940 4290855584 18446744073709551615 134564631 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/2773/statm: 52747 46925 100 284 0 52461 0
[pid=2771/tid=2783] ppid=2732 vsize=777664 CPUtime=0 cores=0,2,4,6
/proc/2771/task/2783/stat : 2783 (python) S 2732 2732 1708 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 538751668 796327936 120161 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 514 18446604448700621824 0 0 -1 2 0 0 0
[pid=2776] ppid=2732 vsize=787920 CPUtime=1.2 cores=0,2,4,6
/proc/2776/stat : 2776 (python) S 2732 2732 1708 0 -1 4202560 1197 0 0 0 64 56 0 0 15 0 2 0 538741654 806830080 120166 33554432000 4194304 5683644 140737215696368 18446744073709551615 251814260294 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/2776/statm: 196980 120166 400 364 0 132883 0
[pid=2779] ppid=2776 vsize=2483532 CPUtime=628.92 cores=0,2,4,6
/proc/2779/stat : 2779 (cryptominisat-2) R 2776 2779 2779 0 -1 4202496 623385 0 0 0 62780 112 0 0 25 0 1 0 538741656 2543136768 479762 33554432000 4194304 4677096 140733963224528 18446744073709551615 4550317 0 0 16781312 2 0 0 0 17 0 0 0 0
/proc/2779/statm: 620883 479762 391 118 0 615842 0
[pid=2776/tid=2820] ppid=2732 vsize=787920 CPUtime=0 cores=0,2,4,6
/proc/2776/task/2820/stat : 2820 (python) S 2732 2732 1708 0 -1 4202560 51 0 0 0 0 0 0 0 16 0 2 0 538791125 806830080 120166 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 514 18446604448700620928 0 0 -1 4 0 0 0
[pid=2793] ppid=2732 vsize=798176 CPUtime=0.81 cores=0,2,4,6
/proc/2793/stat : 2793 (python) S 2732 2732 1708 0 -1 4202560 1161 0 0 0 41 40 0 0 18 0 2 0 538780632 817332224 120169 33554432000 4194304 5683644 140737215696368 18446744073709551615 251814260294 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/2793/statm: 199544 120169 399 364 0 135444 0
[pid=2795] ppid=2793 vsize=218860 CPUtime=431.16 cores=0,2,4,6
/proc/2795/stat : 2795 (lingeling) R 2793 2795 2795 0 -1 4202496 273774 0 0 0 43080 36 0 0 25 0 1 0 538780634 224112640 53885 33554432000 4194304 4352236 140736370345920 18446744073709551615 4328009 0 0 16781312 17506 0 0 0 17 2 0 0 0
/proc/2795/statm: 54715 53885 129 39 0 53795 0
[pid=2793/tid=2816] ppid=2732 vsize=798176 CPUtime=0 cores=0,2,4,6
/proc/2793/task/2816/stat : 2816 (python) S 2732 2732 1708 0 -1 4202560 51 0 0 0 0 0 0 0 19 0 2 0 538790653 817332224 120169 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 514 18446604453882025152 0 0 -1 2 0 0 0
[pid=2797] ppid=2732 vsize=808432 CPUtime=0.55 cores=0,2,4,6
/proc/2797/stat : 2797 (python) S 2732 2732 1708 0 -1 4202560 1211 0 0 0 28 27 0 0 19 0 2 0 538780809 827834368 120179 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/2797/statm: 202108 120179 405 364 0 138005 0
[pid=2799] ppid=2797 vsize=560452 CPUtime=294.87 cores=0,2,4,6
/proc/2799/stat : 2799 (precosat) T 2797 2799 2799 0 -1 4202496 139660 0 0 0 29465 22 0 0 25 0 1 0 538780810 573902848 137814 33554432000 4194304 4890832 140737250156160 18446744073709551615 4273723 0 0 16781312 17442 18446744071562691360 0 0 17 6 0 0 0
/proc/2799/statm: 140113 137814 86 171 0 139940 0
[pid=2797/tid=2837] ppid=2732 vsize=808432 CPUtime=0 cores=0,2,4,6
/proc/2797/task/2837/stat : 2837 (python) S 2732 2732 1708 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538810662 827834368 120179 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 514 18446604453869824768 0 0 -1 4 0 0 0
[pid=2817] ppid=2732 vsize=818688 CPUtime=0.19 cores=0,2,4,6
/proc/2817/stat : 2817 (python) S 2732 2732 1708 0 -1 4202560 1120 0 0 0 9 10 0 0 19 0 2 0 538790656 838336512 120177 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/2817/statm: 204672 120177 399 364 0 140566 0
[pid=2819] ppid=2817 vsize=560440 CPUtime=99.01 cores=0,2,4,6
/proc/2819/stat : 2819 (precosat) T 2817 2819 2819 0 -1 4202496 139415 0 0 0 9877 24 0 0 25 0 1 0 538790657 573890560 137589 33554432000 4194304 4890832 140735606623424 18446744073709551615 4293300 0 0 16781312 17442 18446744071562691360 0 0 17 4 0 0 0
/proc/2819/statm: 140110 137589 86 171 0 139937 0
[pid=2817/tid=2829] ppid=2732 vsize=818688 CPUtime=0 cores=0,2,4,6
/proc/2817/task/2829/stat : 2829 (python) S 2732 2732 1708 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 538800581 838336512 120177 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 514 18446604448700623616 0 0 -1 2 0 0 0
[pid=2830] ppid=2732 vsize=828944 CPUtime=0.2 cores=0,2,4,6
/proc/2830/stat : 2830 (python) S 2732 2732 1708 0 -1 4202560 1131 0 0 0 9 11 0 0 19 0 2 0 538800584 848838656 120181 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/2830/statm: 207236 120181 399 364 0 143127 0
[pid=2832] ppid=2830 vsize=205216 CPUtime=97.25 cores=0,2,4,6
/proc/2832/stat : 2832 (lingeling) T 2830 2832 2832 0 -1 4202496 129849 0 0 0 9706 19 0 0 25 0 1 0 538800586 210141184 50474 33554432000 4194304 4352236 140734624896240 18446744073709551615 4309132 0 0 16781312 17506 18446744071562691360 0 0 17 6 0 0 0
/proc/2832/statm: 51304 50474 129 39 0 50384 0
[pid=2830/tid=2836] ppid=2732 vsize=828944 CPUtime=0 cores=0,2,4,6
/proc/2830/task/2836/stat : 2836 (python) S 2732 2732 1708 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538810506 848838656 120181 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 514 18446604445571144768 0 0 -1 6 0 0 0
[pid=2732/tid=2755] ppid=2730 vsize=839188 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2755/stat : 2755 (python) S 2730 2732 1708 0 -1 4202560 30 934 0 0 0 0 0 0 18 0 11 0 538731629 859328512 121679 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 6 0 0 0
[pid=2732/tid=2758] ppid=2730 vsize=839188 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2758/stat : 2758 (python) S 2730 2732 1708 0 -1 4202560 24 934 0 0 0 0 0 0 20 0 11 0 538731633 859328512 121679 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 2 0 0 0
[pid=2732/tid=2762] ppid=2730 vsize=839188 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2762/stat : 2762 (python) S 2730 2732 1708 0 -1 4202560 25 934 0 0 0 0 0 0 18 0 11 0 538731636 859328512 121679 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 4 0 0 0
[pid=2732/tid=2765] ppid=2730 vsize=839188 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2765/stat : 2765 (python) S 2730 2732 1708 0 -1 4202560 25 934 0 0 0 0 0 0 24 0 11 0 538731640 859328512 121679 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 4 0 0 0
[pid=2732/tid=2772] ppid=2730 vsize=839188 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2772/stat : 2772 (python) S 2730 2732 1708 0 -1 4202560 28 934 0 0 0 0 0 0 20 0 11 0 538741651 859328512 121679 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 4 0 0 0
[pid=2732/tid=2778] ppid=2730 vsize=839188 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2778/stat : 2778 (python) S 2730 2732 1708 0 -1 4202560 32 934 0 0 0 0 0 0 18 0 11 0 538741656 859328512 121679 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 4 0 0 0
[pid=2732/tid=2794] ppid=2730 vsize=839188 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2794/stat : 2794 (python) S 2730 2732 1708 0 -1 4202560 33 934 0 0 0 0 0 0 18 0 11 0 538780634 859328512 121679 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 4 0 0 0
[pid=2732/tid=2798] ppid=2730 vsize=839188 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2798/stat : 2798 (python) S 2730 2732 1708 0 -1 4202560 31 934 0 0 0 0 0 0 20 0 11 0 538780810 859328512 121679 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 4 0 0 0
[pid=2732/tid=2818] ppid=2730 vsize=839188 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2818/stat : 2818 (python) S 2730 2732 1708 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 11 0 538790657 859328512 121679 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 4 0 0 0
[pid=2732/tid=2831] ppid=2730 vsize=839188 CPUtime=0 cores=0,2,4,6
/proc/2732/task/2831/stat : 2831 (python) S 2730 2732 1708 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 11 0 538800585 859328512 121679 33554432000 4194304 5683644 140737215696368 18446744073709551615 251826063793 0 0 16781312 2 18446604441936148608 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3840.08
Current children cumulated vsize (KiB) 15903232

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 2732 and gives
#  childrusage.ru_utime.tv_sec=140
#  childrusage.ru_utime.tv_usec=230681
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=959854
# CPU time returned by wait4() is 141.191
# while last known CPU time is 3840.08
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1068.9
CPU time (s): 3840.08
CPU user time (s): 3831.64
CPU system time (s): 8.44
CPU usage (%): 359.254
Max. virtual memory (cumulated for all children) (KiB): 15903232

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 140.231
system time used= 0.959854
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 484760
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 12460
involuntary context switches= 500

runsolver used 3.35649 second user time and 8.71667 second system time

The end

Launcher Data 

Begin job on node126 at 2011-05-01 19:33:08
IDJOB=3317731
IDBENCH=88163
IDSOLVER=1706
FILE ID=node126/3317731-1304271185
RUNJOBID= node126-1304271185-2718
PBS_JOBID= 13173953
Free space on /tmp= 72956 MiB

SOLVER NAME= borg-sat 11.03.02-industrial
BENCH NAME= SAT-Race-2010-CNF/hardware-verification/velev/11pipe_11_ooo.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.industrial.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317731-1304271185/watcher-3317731-1304271185 -o /tmp/evaluation-result-3317731-1304271185/solver-3317731-1304271185 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.industrial.pickle HOME/instance-3317731-1304271185.cnf -seed 1199711537 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= f603f11fbfe855df381282b7bce010ea
RANDOM SEED=1199711537

node126.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.56
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      28342144 kB
Buffers:        318636 kB
Cached:        2050396 kB
SwapCached:         76 kB
Active:        2507264 kB
Inactive:      1763560 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      28342144 kB
SwapTotal:    67111528 kB
SwapFree:     67111448 kB
Dirty:          338492 kB
Writeback:           0 kB
AnonPages:     1901940 kB
Mapped:          19392 kB
Slab:           264212 kB
PageTables:      14064 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 23542516 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73112 MiB
End job on node126 at 2011-05-01 19:50:59