Trace number 3385495

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (TO) 2343.5 5100.04

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v800-c3408-S1831593686-090.SATISFIABLE.cnf
MD5SUM7dd850f670e1c9ea63e995ef24dddf5f
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.032994
Satisfiable
(Un)Satisfiability was proved
Number of variables800
Number of clauses3408
Sum of the clauses size10224
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 33408
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.37/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.37/0.67	c INFO: solving HOME/instance-3385495-1308045448.cnf
0.37/0.67	c INFO: computing task features
0.37/0.68	c INFO: computed variable-clause graph statistics
0.37/0.68	c INFO: computed clause balance statistics
0.44/0.75	c INFO: computed variable balance statistics
0.54/0.80	c INFO: computed small-clause counts
0.54/0.80	c INFO: computed Horn-clause counts
0.54/0.80	c INFO: computed variable graph statistics
0.54/0.80	c INFO: collected features for HOME/instance-3385495-1308045448.cnf in 0.13 s
0.54/0.83	c INFO: running gnovelty+2@0 for 97 with 4999 remaining (b = 0.59)
0.54/0.86	c INFO: running gnovelty+2@0 for 97 with 4999 remaining (b = 0.08)
0.54/0.89	c INFO: running march_hi@0 for 388 with 4999 remaining (b = 0.25)
0.64/0.92	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.26)
0.64/0.96	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.23)
0.64/0.99	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.20)
1.19/1.02	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.17)
1.19/1.05	c INFO: running march_hi@0 for 388 with 4999 remaining (b = 0.13)
1.19/1.06	Traceback (most recent call last):
1.19/1.06	  File "HOME/solve", line 33, in <module>
1.19/1.06	    plac.call(main)
1.19/1.06	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
1.19/1.10	    cmd, result = parser_from(obj).consume(arglist)
1.19/1.10	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
1.19/1.10	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
1.19/1.10	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
1.19/1.10	    (_, answer, _) = solver(input_path, budget / cores, cores)
1.19/1.10	  File "HOME/borg/build/lib/borg/bilevel.py", line 226, in __call__
1.19/1.10	    process.die()
1.19/1.10	  File "HOME/borg/build/lib/borg/solvers.py", line 185, in die
1.19/1.10	    os.kill(self._process.pid, signal.SIGUSR1)
1.19/1.10	OSError: [Errno 3] No such process

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3385495-1308045448/watcher-3385495-1308045448 -o /tmp/evaluation-result-3385495-1308045448/solver-3385495-1308045448 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385495-1308045448.cnf -seed 2107038995 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.83 1.73 1.97 2/170 1580
/proc/meminfo: memFree=31471080/32950928 swapFree=67055948/67111528
[pid=1580] ppid=1578 vsize=15080 CPUtime=0 cores=0-7
/proc/1580/stat : 1580 (python) R 1578 1580 1477 0 -1 4202496 618 0 0 0 0 0 0 0 23 0 1 0 916144501 15441920 513 33554432000 4194304 5683644 140733936125264 18446744073709551615 208080228197 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/1580/statm: 3770 513 300 364 0 257 0

[startup+0.0770079 s]
/proc/loadavg: 3.83 1.73 1.97 2/170 1580
/proc/meminfo: memFree=31471080/32950928 swapFree=67055948/67111528
[pid=1580] ppid=1578 vsize=45824 CPUtime=0.05 cores=0-7
/proc/1580/stat : 1580 (python) R 1578 1580 1477 0 -1 4202496 1910 0 0 0 4 1 0 0 22 0 1 0 916144501 46923776 1682 33554432000 4194304 5683644 140733936125264 18446744073709551615 208079893537 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/1580/statm: 11456 1688 470 364 0 1205 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 45824

[startup+0.101004 s]
/proc/loadavg: 3.83 1.73 1.97 2/170 1580
/proc/meminfo: memFree=31471080/32950928 swapFree=67055948/67111528
[pid=1580] ppid=1578 vsize=91832 CPUtime=0.06 cores=0-7
/proc/1580/stat : 1580 (python) R 1578 1580 1477 0 -1 4202496 2680 0 0 0 5 1 0 0 22 0 1 0 916144501 94035968 2394 33554432000 4194304 5683644 140733936125264 18446744073709551615 208080230400 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/1580/statm: 22958 2394 755 364 0 1623 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 91832

[startup+0.300967 s]
/proc/loadavg: 3.83 1.73 1.97 2/170 1580
/proc/meminfo: memFree=31471080/32950928 swapFree=67055948/67111528
[pid=1580] ppid=1578 vsize=135752 CPUtime=0.19 cores=0-7
/proc/1580/stat : 1580 (python) S 1578 1580 1477 0 -1 4202496 5966 933 1 0 14 5 0 0 22 0 1 0 916144501 139010048 4151 33554432000 4194304 5683644 140733936125264 18446744073709551615 208080228357 0 2147483391 16781312 2 18446744071699207358 0 0 17 2 0 0 0
/proc/1580/statm: 33938 4151 1032 364 0 3124 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 135752

[startup+0.700891 s]
/proc/loadavg: 3.83 1.73 1.97 2/170 1580
/proc/meminfo: memFree=31471080/32950928 swapFree=67055948/67111528
[pid=1580] ppid=1578 vsize=278628 CPUtime=0.44 cores=0-7
/proc/1580/stat : 1580 (python) R 1578 1580 1477 0 -1 4202496 9406 933 1 0 33 11 0 0 22 0 1 0 916144501 285315072 7071 33554432000 4194304 5683644 140733936125264 18446744073709551615 47597520669498 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/1580/statm: 69657 7071 1826 364 0 5581 0
Current children cumulated CPU time (s) 0.44
Current children cumulated vsize (KiB) 278628

[startup+1.50074 s]
/proc/loadavg: 3.83 1.73 1.97 7/187 1604
/proc/meminfo: memFree=31424096/32950928 swapFree=67055948/67111528
[pid=1580] ppid=1578 vsize=375204 CPUtime=1.09 cores=0-7
/proc/1580/stat : 1580 (python) S 1578 1580 1477 0 -1 4202496 16154 4451 1 0 67 13 28 1 25 0 1 0 916144501 384208896 7208 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092062527 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 3
/proc/1580/statm: 93801 7208 1891 364 0 29698 0
[pid=1592] ppid=1580 vsize=299228 CPUtime=0 cores=0-7
/proc/1592/stat : 1592 (python) S 1580 1580 1477 0 -1 4202560 753 0 0 0 0 0 0 0 20 0 1 0 916144590 306409472 5619 33554432000 4194304 5683644 140733936125264 18446744073709551615 208080252486 0 0 16781312 514 4093780861667737860 0 0 17 0 0 0 0
/proc/1592/statm: 74807 5619 348 364 0 10719 0
[pid=1594] ppid=1592 vsize=3032 CPUtime=0.59 cores=0-7
/proc/1594/stat : 1594 (march_hi) R 1592 1594 1594 0 -1 4202496 802 0 0 0 59 0 0 0 25 0 1 0 916144591 3104768 538 33554432000 4194304 4955518 140736493981056 18446744073709551615 4227807 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/1594/statm: 758 538 104 186 0 571 0
[pid=1595] ppid=1580 vsize=310092 CPUtime=0 cores=0-7
/proc/1595/stat : 1595 (python) S 1580 1580 1477 0 -1 4202560 752 0 0 0 0 0 0 0 20 0 1 0 916144594 317534208 5743 33554432000 4194304 5683644 140733936125264 18446744073709551615 208080252486 0 0 16781312 514 4101842480922591489 0 0 17 7 0 0 0
/proc/1595/statm: 77523 5743 348 364 0 13432 0
[pid=1597] ppid=1595 vsize=3032 CPUtime=0.56 cores=0-7
/proc/1597/stat : 1597 (march_hi) R 1595 1597 1597 0 -1 4202496 802 0 0 0 56 0 0 0 25 0 1 0 916144594 3104768 536 33554432000 4194304 4955518 140735910734800 18446744073709551615 4209688 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/1597/statm: 758 536 104 186 0 571 0
[pid=1598] ppid=1580 vsize=320348 CPUtime=0 cores=0-7
/proc/1598/stat : 1598 (python) S 1580 1580 1477 0 -1 4202560 761 0 0 0 0 0 0 0 20 0 1 0 916144597 328036352 5764 33554432000 4194304 5683644 140733936125264 18446744073709551615 208080252486 0 0 16781312 514 5344567697239671042 0 0 17 7 0 0 0
/proc/1598/statm: 80087 5764 348 364 0 15993 0
[pid=1600] ppid=1598 vsize=3032 CPUtime=0.53 cores=0-7
/proc/1600/stat : 1600 (march_hi) R 1598 1600 1600 0 -1 4202496 801 0 0 0 53 0 0 0 25 0 1 0 916144597 3104768 536 33554432000 4194304 4955518 140736738226496 18446744073709551615 4261360 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/1600/statm: 758 536 104 186 0 571 0
[pid=1602] ppid=1580 vsize=330740 CPUtime=0 cores=0-7
/proc/1602/stat : 1602 (python) S 1580 1580 1477 0 -1 4202560 762 0 0 0 0 0 0 0 22 0 1 0 916144600 338677760 5785 33554432000 4194304 5683644 140733936125264 18446744073709551615 208080252486 0 0 16781312 514 11003204289588855044 0 0 17 4 0 0 0
/proc/1602/statm: 82685 5785 348 364 0 18588 0
[pid=1604] ppid=1602 vsize=3032 CPUtime=0.5 cores=0-7
/proc/1604/stat : 1604 (march_hi) R 1602 1604 1604 0 -1 4202496 802 0 0 0 50 0 0 0 25 0 1 0 916144600 3104768 535 33554432000 4194304 4955518 140737233720704 18446744073709551615 4224707 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/1604/statm: 758 535 104 186 0 571 0
Current children cumulated CPU time (s) 3.27
Current children cumulated vsize (KiB) 1647740

[startup+3.10043 s]
/proc/loadavg: 3.83 1.73 1.97 6/181 1610
/proc/meminfo: memFree=31417572/32950928 swapFree=67055948/67111528
[pid=1580] ppid=1578 vsize=375204 CPUtime=1.09 cores=0-7
/proc/1580/stat : 1580 (python) S 1578 1580 1477 0 -1 4202496 16154 4451 1 0 67 13 28 1 25 0 1 0 916144501 384208896 7208 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092062527 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 3
/proc/1580/statm: 93801 7208 1891 364 0 29698 0
[pid=1592] ppid=1580 vsize=299228 CPUtime=0 cores=0-7
/proc/1592/stat : 1592 (python) S 1580 1580 1477 0 -1 4202560 753 0 0 0 0 0 0 0 15 0 1 0 916144590 306409472 5619 33554432000 4194304 5683644 140733936125264 18446744073709551615 208080252486 0 0 16781312 514 4093780861667737860 0 0 17 0 0 0 0
/proc/1592/statm: 74807 5619 348 364 0 10719 0
[pid=1594] ppid=1592 vsize=3488 CPUtime=2.18 cores=0-7
/proc/1594/stat : 1594 (march_hi) R 1592 1594 1594 0 -1 4202496 892 0 0 0 218 0 0 0 25 0 1 0 916144591 3571712 628 33554432000 4194304 4955518 140736493981056 18446744073709551615 4227679 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/1594/statm: 872 628 104 186 0 685 0
[pid=1595] ppid=1580 vsize=310092 CPUtime=0 cores=0-7
/proc/1595/stat : 1595 (python) S 1580 1580 1477 0 -1 4202560 759 0 0 0 0 0 0 0 15 0 1 0 916144594 317534208 5743 33554432000 4194304 5683644 140733936125264 18446744073709551615 208080252486 0 0 16781312 514 4101842480922591489 0 0 17 7 0 0 0
/proc/1595/statm: 77523 5743 348 364 0 13432 0
[pid=1597] ppid=1595 vsize=3488 CPUtime=2.16 cores=0-7
/proc/1597/stat : 1597 (march_hi) R 1595 1597 1597 0 -1 4202496 892 0 0 0 216 0 0 0 25 0 1 0 916144594 3571712 626 33554432000 4194304 4955518 140735910734800 18446744073709551615 4290028 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/1597/statm: 872 626 104 186 0 685 0
[pid=1598] ppid=1580 vsize=320348 CPUtime=0 cores=0-7
/proc/1598/stat : 1598 (python) S 1580 1580 1477 0 -1 4202560 768 0 0 0 0 0 0 0 15 0 1 0 916144597 328036352 5764 33554432000 4194304 5683644 140733936125264 18446744073709551615 208080252486 0 0 16781312 514 5344567697239671042 0 0 17 7 0 0 0
/proc/1598/statm: 80087 5764 348 364 0 15993 0
[pid=1600] ppid=1598 vsize=3488 CPUtime=2.13 cores=0-7
/proc/1600/stat : 1600 (march_hi) R 1598 1600 1600 0 -1 4202496 890 0 0 0 213 0 0 0 25 0 1 0 916144597 3571712 625 33554432000 4194304 4955518 140736738226496 18446744073709551615 4286724 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/1600/statm: 872 625 104 186 0 685 0
[pid=1602] ppid=1580 vsize=330740 CPUtime=0 cores=0-7
/proc/1602/stat : 1602 (python) S 1580 1580 1477 0 -1 4202560 762 0 0 0 0 0 0 0 15 0 1 0 916144600 338677760 5785 33554432000 4194304 5683644 140733936125264 18446744073709551615 208080252486 0 0 16781312 514 11003204289588855044 0 0 17 4 0 0 0
/proc/1602/statm: 82685 5785 348 364 0 18588 0
[pid=1604] ppid=1602 vsize=3488 CPUtime=2.1 cores=0-7
/proc/1604/stat : 1604 (march_hi) R 1602 1604 1604 0 -1 4202496 891 0 0 0 210 0 0 0 25 0 1 0 916144600 3571712 624 33554432000 4194304 4955518 140737233720704 18446744073709551615 4263698 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/1604/statm: 872 624 104 186 0 685 0
[pid=1606] ppid=1580 vsize=340996 CPUtime=0 cores=0-7
/proc/1606/stat : 1606 (python) S 1580 1580 1477 0 -1 4202560 781 0 0 0 0 0 0 0 15 0 1 0 916144603 349179904 5789 33554432000 4194304 5683644 140733936125264 18446744073709551615 208080252486 0 0 16781312 514 4540217766916882692 0 0 17 5 0 0 0
/proc/1606/statm: 85249 5789 348 364 0 21149 0
[pid=1608] ppid=1606 vsize=3488 CPUtime=2.06 cores=0-7
/proc/1608/stat : 1608 (march_hi) R 1606 1608 1608 0 -1 4202496 890 0 0 0 206 0 0 0 25 0 1 0 916144603 3571712 623 33554432000 4194304 4955518 140736355248720 18446744073709551615 4314917 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/1608/statm: 872 623 104 186 0 685 0
Current children cumulated CPU time (s) 11.72
Current children cumulated vsize (KiB) 1994048

[startup+6.30082 s]
/proc/loadavg: 4.00 1.80 1.99 6/181 1610
/proc/meminfo: memFree=31415960/32950928 swapFree=67055948/67111528
[pid=1580] ppid=1578 vsize=375204 CPUtime=1.09 cores=0-7
/proc/1580/stat : 1580 (python) S 1578 1580 1477 0 -1 4202496 16154 4451 1 0 67 13 28 1 25 0 1 0 916144501 384208896 7208 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092062527 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 3
/proc/1580/statm: 93801 7208 1891 364 0 29698 0
[pid=1592] ppid=1580 vsize=299228 CPUtime=0 cores=0-7
/proc/1592/stat : 1592 (python) S 1580 1580 1477 0 -1 4202560 898 0 0 0 0 0 0 0 15 0 1 0 916144590 306409472 5642 33554432000 4194304 5683644 140733936125264 18446744073709551615 208080252486 0 0 16781312 514 4093780861667737860 0 0 17 0 0 0 0
/proc/1592/statm: 74807 5642 371 364 0 10719 0
[pid=1594] ppid=1592 vsize=4228 CPUtime=5.38 cores=0-7
/proc/1594/stat : 1594 (march_hi) R 1592 1594 1594 0 -1 4202496 1052 0 0 0 538 0 0 0 25 0 1 0 916144591 4329472 788 33554432000 4194304 4955518 140736493981056 18446744073709551615 4209727 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/1594/statm: 1057 788 104 186 0 870 0
[pid=1595] ppid=1580 vsize=310092 CPUtime=0 cores=0-7
/proc/1595/stat : 1595 (python) S 1580 1580 1477 0 -1 4202560 907 0 0 0 0 0 0 0 15 0 1 0 916144594 317534208 5766 33554432000 4194304 5683644 140733936125264 18446744073709551615 208080252486 0 0 16781312 514 4101842480922591489 0 0 17 7 0 0 0
/proc/1595/statm: 77523 5766 371 364 0 13432 0
[pid=1597] ppid=1595 vsize=4228 CPUtime=5.36 cores=0-7
/proc/1597/stat : 1597 (march_hi) R 1595 1597 1597 0 -1 4202496 1058 0 0 0 536 0 0 0 25 0 1 0 916144594 4329472 792 33554432000 4194304 4955518 140735910734800 18446744073709551615 4346482 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/1597/statm: 1057 792 104 186 0 870 0
[pid=1598] ppid=1580 vsize=320348 CPUtime=0 cores=0-7
/proc/1598/stat : 1598 (python) S 1580 1580 1477 0 -1 4202560 929 0 0 0 0 0 0 0 15 0 1 0 916144597 328036352 5787 33554432000 4194304 5683644 140733936125264 18446744073709551615 208080252486 0 0 16781312 514 5344567697239671042 0 0 17 7 0 0 0
/proc/1598/statm: 80087 5787 371 364 0 15993 0
[pid=1600] ppid=1598 vsize=4228 CPUtime=5.33 cores=0-7
/proc/1600/stat : 1600 (march_hi) R 1598 1600 1600 0 -1 4202496 1056 0 0 0 533 0 0 0 25 0 1 0 916144597 4329472 791 33554432000 4194304 4955518 140736738226496 18446744073709551615 4224773 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/1600/statm: 1057 791 104 186 0 870 0
[pid=1602] ppid=1580 vsize=330740 CPUtime=0 cores=0-7
/proc/1602/stat : 1602 (python) S 1580 1580 1477 0 -1 4202560 940 0 0 0 0 0 0 0 15 0 1 0 916144600 338677760 5808 33554432000 4194304 5683644 140733936125264 18446744073709551615 208080252486 0 0 16781312 514 11003204289588855044 0 0 17 4 0 0 0
/proc/1602/statm: 82685 5808 371 364 0 18588 0
[pid=1604] ppid=1602 vsize=4228 CPUtime=5.3 cores=0-7
/proc/1604/stat : 1604 (march_hi) R 1602 1604 1604 0 -1 4202496 1057 0 0 0 530 0 0 0 25 0 1 0 916144600 4329472 790 33554432000 4194304 4955518 140737233720704 18446744073709551615 4224549 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/1604/statm: 1057 790 104 186 0 870 0
[pid=1606] ppid=1580 vsize=340996 CPUtime=0 cores=0-7
/proc/1606/stat : 1606 (python) S 1580 1580 1477 0 -1 4202560 943 0 0 0 0 0 0 0 15 0 1 0 916144603 349179904 5812 33554432000 4194304 5683644 140733936125264 18446744073709551615 208080252486 0 0 16781312 514 4540217766916882692 0 0 17 5 0 0 0
/proc/1606/statm: 85249 5812 371 364 0 21149 0
[pid=1608] ppid=1606 vsize=4228 CPUtime=5.27 cores=0-7
/proc/1608/stat : 1608 (march_hi) R 1606 1608 1608 0 -1 4202496 1055 0 0 0 527 0 0 0 25 0 1 0 916144603 4329472 788 33554432000 4194304 4955518 140736355248720 18446744073709551615 4227814 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/1608/statm: 1057 788 104 186 0 870 0
Current children cumulated CPU time (s) 27.73
Current children cumulated vsize (KiB) 1997748

[startup+12.7006 s]
/proc/loadavg: 4.08 1.86 2.01 6/181 1610
/proc/meminfo: memFree=31410776/32950928 swapFree=67055948/67111528
[pid=1580] ppid=1578 vsize=375204 CPUtime=1.09 cores=0-7
/proc/1580/stat : 1580 (python) S 1578 1580 1477 0 -1 4202496 16154 4451 1 0 67 13 28 1 25 0 1 0 916144501 384208896 7208 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092062527 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 3
/proc/1580/statm: 93801 7208 1891 364 0 29698 0
[pid=1592] ppid=1580 vsize=299228 CPUtime=0.01 cores=0-7
/proc/1592/stat : 1592 (python) S 1580 1580 1477 0 -1 4202560 905 0 0 0 1 0 0 0 15 0 1 0 916144590 306409472 5642 33554432000 4194304 5683644 140733936125264 18446744073709551615 208080252486 0 0 16781312 514 3729525849230115076 0 0 17 0 0 0 0
/proc/1592/statm: 74807 5642 371 364 0 10719 0
[pid=1594] ppid=1592 vsize=5680 CPUtime=11.77 cores=0-7
/proc/1594/stat : 1594 (march_hi) R 1592 1594 1594 0 -1 4202496 1305 0 0 0 1177 0 0 0 25 0 1 0 916144591 5816320 1041 33554432000 4194304 4955518 140736493981056 18446744073709551615 4227824 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/1594/statm: 1420 1041 104 186 0 1233 0
[pid=1595] ppid=1580 vsize=310092 CPUtime=0 cores=0-7
/proc/1595/stat : 1595 (python) S 1580 1580 1477 0 -1 4202560 912 0 0 0 0 0 0 0 15 0 1 0 916144594 317534208 5766 33554432000 4194304 5683644 140733936125264 18446744073709551615 208080252486 0 0 16781312 514 3729525849230115073 0 0 17 7 0 0 0
/proc/1595/statm: 77523 5766 371 364 0 13432 0
[pid=1597] ppid=1595 vsize=5680 CPUtime=11.76 cores=0-7
/proc/1597/stat : 1597 (march_hi) R 1595 1597 1597 0 -1 4202496 1316 0 0 0 1176 0 0 0 25 0 1 0 916144594 5816320 1050 33554432000 4194304 4955518 140735910734800 18446744073709551615 4227814 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/1597/statm: 1420 1050 104 186 0 1233 0
[pid=1598] ppid=1580 vsize=320348 CPUtime=0 cores=0-7
/proc/1598/stat : 1598 (python) S 1580 1580 1477 0 -1 4202560 932 0 0 0 0 0 0 0 15 0 1 0 916144597 328036352 5787 33554432000 4194304 5683644 140733936125264 18446744073709551615 208080252486 0 0 16781312 514 3729525849230115074 0 0 17 7 0 0 0
/proc/1598/statm: 80087 5787 371 364 0 15993 0
[pid=1600] ppid=1598 vsize=5680 CPUtime=11.73 cores=0-7
/proc/1600/stat : 1600 (march_hi) R 1598 1600 1600 0 -1 4202496 1312 0 0 0 1173 0 0 0 25 0 1 0 916144597 5816320 1047 33554432000 4194304 4955518 140736738226496 18446744073709551615 4227679 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/1600/statm: 1420 1047 104 186 0 1233 0
[pid=1602] ppid=1580 vsize=330740 CPUtime=0.01 cores=0-7

################
# More data... #
################

/proc/meminfo: memFree=31068416/32950928 swapFree=67055948/67111528
[pid=1580] ppid=1578 vsize=375204 CPUtime=1.09 cores=0-7
/proc/1580/stat : 1580 (python) S 1578 1580 1477 0 -1 4202496 16154 4451 1 0 67 13 28 1 25 0 1 0 916144501 384208896 7208 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092062527 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 3
/proc/1580/statm: 93801 7208 1891 364 0 29698 0
[pid=1592] ppid=1580 vsize=364764 CPUtime=0.47 cores=0-7
/proc/1592/stat : 1592 (python) S 1580 1580 1477 0 -1 4202560 1133 0 0 0 28 19 0 0 16 0 2 0 916144590 373518336 5673 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092059083 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/1592/statm: 91191 5673 400 364 0 27103 0
[pid=1594] ppid=1592 vsize=73540 CPUtime=392.51 cores=0-7
/proc/1594/stat : 1594 (march_hi) T 1592 1594 1594 0 -1 4202496 15498 0 0 0 39247 4 0 0 25 0 1 0 916144591 75304960 15234 33554432000 4194304 4955518 140736493981056 18446744073709551615 4224541 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/1594/statm: 18385 15234 104 186 0 18198 0
[pid=1592/tid=1632] ppid=1580 vsize=364764 CPUtime=0 cores=0-7
/proc/1592/task/1632/stat : 1632 (python) S 1580 1580 1477 0 -1 4202560 45 0 0 0 0 0 0 0 16 0 2 0 916183882 373518336 5673 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092055985 0 0 16781312 514 18446604453876357952 0 0 -1 5 0 0 0
[pid=1595] ppid=1580 vsize=310092 CPUtime=0.57 cores=0-7
/proc/1595/stat : 1595 (python) S 1580 1580 1477 0 -1 4202560 1139 0 0 0 33 24 0 0 17 0 2 0 916144594 317534208 5796 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092059083 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/1595/statm: 77523 5796 400 364 0 13432 0
[pid=1597] ppid=1595 vsize=114180 CPUtime=488.48 cores=0-7
/proc/1597/stat : 1597 (march_hi) T 1595 1597 1597 0 -1 4202496 19057 0 0 0 48844 4 0 0 25 0 1 0 916144594 116920320 18791 33554432000 4194304 4955518 140735910734800 18446744073709551615 4224648 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/1597/statm: 28545 18791 104 186 0 28358 0
[pid=1595/tid=1640] ppid=1580 vsize=310092 CPUtime=0 cores=0-7
/proc/1595/task/1640/stat : 1640 (python) S 1580 1580 1477 0 -1 4202560 48 0 0 0 0 0 0 0 17 0 2 0 916193442 317534208 5796 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092055985 0 0 16781312 514 18446604453870746560 0 0 -1 3 0 0 0
[pid=1598] ppid=1580 vsize=320348 CPUtime=0.58 cores=0-7
/proc/1598/stat : 1598 (python) S 1580 1580 1477 0 -1 4202560 1161 0 0 0 34 24 0 0 16 0 2 0 916144597 328036352 5817 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092059083 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/1598/statm: 80087 5817 400 364 0 15993 0
[pid=1600] ppid=1598 vsize=84712 CPUtime=486.35 cores=0-7
/proc/1600/stat : 1600 (march_hi) T 1598 1600 1600 0 -1 4202496 18930 0 0 0 48631 4 0 0 25 0 1 0 916144597 86745088 18665 33554432000 4194304 4955518 140736738226496 18446744073709551615 4284802 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/1600/statm: 21178 18665 104 186 0 20991 0
[pid=1598/tid=1639] ppid=1580 vsize=320348 CPUtime=0 cores=0-7
/proc/1598/task/1639/stat : 1639 (python) S 1580 1580 1477 0 -1 4202560 49 0 0 0 0 0 0 0 16 0 2 0 916193231 328036352 5817 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092055985 0 0 16781312 514 18446604453875684288 0 0 -1 5 0 0 0
[pid=1602] ppid=1580 vsize=330740 CPUtime=0.58 cores=0-7
/proc/1602/stat : 1602 (python) S 1580 1580 1477 0 -1 4202560 1170 0 0 0 33 25 0 0 16 0 2 0 916144600 338677760 5838 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092059083 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/1602/statm: 82685 5838 400 364 0 18588 0
[pid=1604] ppid=1602 vsize=84712 CPUtime=486.24 cores=0-7
/proc/1604/stat : 1604 (march_hi) T 1602 1604 1604 0 -1 4202496 18932 0 0 0 48621 3 0 0 25 0 1 0 916144600 86745088 18665 33554432000 4194304 4955518 140737233720704 18446744073709551615 4244774 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/1604/statm: 21178 18665 104 186 0 20991 0
[pid=1602/tid=1638] ppid=1580 vsize=330740 CPUtime=0 cores=0-7
/proc/1602/task/1638/stat : 1638 (python) S 1580 1580 1477 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 916193224 338677760 5838 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092055985 0 0 16781312 514 18446604453865550848 0 0 -1 4 0 0 0
[pid=1606] ppid=1580 vsize=340996 CPUtime=0.58 cores=0-7
/proc/1606/stat : 1606 (python) S 1580 1580 1477 0 -1 4202560 1173 0 0 0 31 27 0 0 16 0 2 0 916144603 349179904 5842 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092059083 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/1606/statm: 85249 5842 400 364 0 21149 0
[pid=1608] ppid=1606 vsize=114180 CPUtime=486.05 cores=0-7
/proc/1608/stat : 1608 (march_hi) T 1606 1608 1608 0 -1 4202496 18941 0 0 0 48601 4 0 0 25 0 1 0 916144603 116920320 18674 33554432000 4194304 4955518 140736355248720 18446744073709551615 4227502 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/1608/statm: 28545 18674 104 186 0 28358 0
[pid=1606/tid=1637] ppid=1580 vsize=340996 CPUtime=0 cores=0-7
/proc/1606/task/1637/stat : 1637 (python) S 1580 1580 1477 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 916193208 349179904 5842 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092055985 0 0 16781312 514 18446604453870997632 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2343.5
Current children cumulated vsize (KiB) 2513468
heavy processes:

[startup+5082.3 s]
/proc/loadavg: 0.00 0.00 0.00 1/186 1775
/proc/meminfo: memFree=31068356/32950928 swapFree=67055948/67111528
[pid=1580] ppid=1578 vsize=375204 CPUtime=1.09 cores=0-7
/proc/1580/stat : 1580 (python) S 1578 1580 1477 0 -1 4202496 16154 4451 1 0 67 13 28 1 25 0 1 0 916144501 384208896 7208 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092062527 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 3
/proc/1580/statm: 93801 7208 1891 364 0 29698 0
[pid=1592] ppid=1580 vsize=364764 CPUtime=0.47 cores=0-7
/proc/1592/stat : 1592 (python) S 1580 1580 1477 0 -1 4202560 1133 0 0 0 28 19 0 0 16 0 2 0 916144590 373518336 5673 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092059083 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/1592/statm: 91191 5673 400 364 0 27103 0
[pid=1594] ppid=1592 vsize=73540 CPUtime=392.51 cores=0-7
/proc/1594/stat : 1594 (march_hi) T 1592 1594 1594 0 -1 4202496 15498 0 0 0 39247 4 0 0 25 0 1 0 916144591 75304960 15234 33554432000 4194304 4955518 140736493981056 18446744073709551615 4224541 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/1594/statm: 18385 15234 104 186 0 18198 0
[pid=1592/tid=1632] ppid=1580 vsize=364764 CPUtime=0 cores=0-7
/proc/1592/task/1632/stat : 1632 (python) S 1580 1580 1477 0 -1 4202560 45 0 0 0 0 0 0 0 16 0 2 0 916183882 373518336 5673 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092055985 0 0 16781312 514 18446604453876357952 0 0 -1 5 0 0 0
[pid=1595] ppid=1580 vsize=310092 CPUtime=0.57 cores=0-7
/proc/1595/stat : 1595 (python) S 1580 1580 1477 0 -1 4202560 1139 0 0 0 33 24 0 0 17 0 2 0 916144594 317534208 5796 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092059083 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/1595/statm: 77523 5796 400 364 0 13432 0
[pid=1597] ppid=1595 vsize=114180 CPUtime=488.48 cores=0-7
/proc/1597/stat : 1597 (march_hi) T 1595 1597 1597 0 -1 4202496 19057 0 0 0 48844 4 0 0 25 0 1 0 916144594 116920320 18791 33554432000 4194304 4955518 140735910734800 18446744073709551615 4224648 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/1597/statm: 28545 18791 104 186 0 28358 0
[pid=1595/tid=1640] ppid=1580 vsize=310092 CPUtime=0 cores=0-7
/proc/1595/task/1640/stat : 1640 (python) S 1580 1580 1477 0 -1 4202560 48 0 0 0 0 0 0 0 17 0 2 0 916193442 317534208 5796 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092055985 0 0 16781312 514 18446604453870746560 0 0 -1 3 0 0 0
[pid=1598] ppid=1580 vsize=320348 CPUtime=0.58 cores=0-7
/proc/1598/stat : 1598 (python) S 1580 1580 1477 0 -1 4202560 1161 0 0 0 34 24 0 0 16 0 2 0 916144597 328036352 5817 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092059083 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/1598/statm: 80087 5817 400 364 0 15993 0
[pid=1600] ppid=1598 vsize=84712 CPUtime=486.35 cores=0-7
/proc/1600/stat : 1600 (march_hi) T 1598 1600 1600 0 -1 4202496 18930 0 0 0 48631 4 0 0 25 0 1 0 916144597 86745088 18665 33554432000 4194304 4955518 140736738226496 18446744073709551615 4284802 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/1600/statm: 21178 18665 104 186 0 20991 0
[pid=1598/tid=1639] ppid=1580 vsize=320348 CPUtime=0 cores=0-7
/proc/1598/task/1639/stat : 1639 (python) S 1580 1580 1477 0 -1 4202560 49 0 0 0 0 0 0 0 16 0 2 0 916193231 328036352 5817 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092055985 0 0 16781312 514 18446604453875684288 0 0 -1 5 0 0 0
[pid=1602] ppid=1580 vsize=330740 CPUtime=0.58 cores=0-7
/proc/1602/stat : 1602 (python) S 1580 1580 1477 0 -1 4202560 1170 0 0 0 33 25 0 0 16 0 2 0 916144600 338677760 5838 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092059083 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/1602/statm: 82685 5838 400 364 0 18588 0
[pid=1604] ppid=1602 vsize=84712 CPUtime=486.24 cores=0-7
/proc/1604/stat : 1604 (march_hi) T 1602 1604 1604 0 -1 4202496 18932 0 0 0 48621 3 0 0 25 0 1 0 916144600 86745088 18665 33554432000 4194304 4955518 140737233720704 18446744073709551615 4244774 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/1604/statm: 21178 18665 104 186 0 20991 0
[pid=1602/tid=1638] ppid=1580 vsize=330740 CPUtime=0 cores=0-7
/proc/1602/task/1638/stat : 1638 (python) S 1580 1580 1477 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 916193224 338677760 5838 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092055985 0 0 16781312 514 18446604453865550848 0 0 -1 4 0 0 0
[pid=1606] ppid=1580 vsize=340996 CPUtime=0.58 cores=0-7
/proc/1606/stat : 1606 (python) S 1580 1580 1477 0 -1 4202560 1173 0 0 0 31 27 0 0 16 0 2 0 916144603 349179904 5842 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092059083 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/1606/statm: 85249 5842 400 364 0 21149 0
[pid=1608] ppid=1606 vsize=114180 CPUtime=486.05 cores=0-7
/proc/1608/stat : 1608 (march_hi) T 1606 1608 1608 0 -1 4202496 18941 0 0 0 48601 4 0 0 25 0 1 0 916144603 116920320 18674 33554432000 4194304 4955518 140736355248720 18446744073709551615 4227502 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/1608/statm: 28545 18674 104 186 0 28358 0
[pid=1606/tid=1637] ppid=1580 vsize=340996 CPUtime=0 cores=0-7
/proc/1606/task/1637/stat : 1637 (python) S 1580 1580 1477 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 916193208 349179904 5842 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092055985 0 0 16781312 514 18446604453870997632 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2343.5
Current children cumulated vsize (KiB) 2513468
heavy processes:



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 0.00 0.00 0.00 1/186 1775
/proc/meminfo: memFree=31068088/32950928 swapFree=67055948/67111528
[pid=1580] ppid=1578 vsize=375204 CPUtime=1.09 cores=0-7
/proc/1580/stat : 1580 (python) S 1578 1580 1477 0 -1 4202496 16154 4451 1 0 67 13 28 1 25 0 1 0 916144501 384208896 7208 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092062527 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 3
/proc/1580/statm: 93801 7208 1891 364 0 29698 0
[pid=1592] ppid=1580 vsize=364764 CPUtime=0.47 cores=0-7
/proc/1592/stat : 1592 (python) S 1580 1580 1477 0 -1 4202560 1133 0 0 0 28 19 0 0 16 0 2 0 916144590 373518336 5673 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092059083 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/1592/statm: 91191 5673 400 364 0 27103 0
[pid=1594] ppid=1592 vsize=73540 CPUtime=392.51 cores=0-7
/proc/1594/stat : 1594 (march_hi) T 1592 1594 1594 0 -1 4202496 15498 0 0 0 39247 4 0 0 25 0 1 0 916144591 75304960 15234 33554432000 4194304 4955518 140736493981056 18446744073709551615 4224541 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/1594/statm: 18385 15234 104 186 0 18198 0
[pid=1592/tid=1632] ppid=1580 vsize=364764 CPUtime=0 cores=0-7
/proc/1592/task/1632/stat : 1632 (python) S 1580 1580 1477 0 -1 4202560 45 0 0 0 0 0 0 0 16 0 2 0 916183882 373518336 5673 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092055985 0 0 16781312 514 18446604453876357952 0 0 -1 5 0 0 0
[pid=1595] ppid=1580 vsize=310092 CPUtime=0.57 cores=0-7
/proc/1595/stat : 1595 (python) S 1580 1580 1477 0 -1 4202560 1139 0 0 0 33 24 0 0 17 0 2 0 916144594 317534208 5796 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092059083 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/1595/statm: 77523 5796 400 364 0 13432 0
[pid=1597] ppid=1595 vsize=114180 CPUtime=488.48 cores=0-7
/proc/1597/stat : 1597 (march_hi) T 1595 1597 1597 0 -1 4202496 19057 0 0 0 48844 4 0 0 25 0 1 0 916144594 116920320 18791 33554432000 4194304 4955518 140735910734800 18446744073709551615 4224648 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/1597/statm: 28545 18791 104 186 0 28358 0
[pid=1595/tid=1640] ppid=1580 vsize=310092 CPUtime=0 cores=0-7
/proc/1595/task/1640/stat : 1640 (python) S 1580 1580 1477 0 -1 4202560 48 0 0 0 0 0 0 0 17 0 2 0 916193442 317534208 5796 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092055985 0 0 16781312 514 18446604453870746560 0 0 -1 3 0 0 0
[pid=1598] ppid=1580 vsize=320348 CPUtime=0.58 cores=0-7
/proc/1598/stat : 1598 (python) S 1580 1580 1477 0 -1 4202560 1161 0 0 0 34 24 0 0 16 0 2 0 916144597 328036352 5817 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092059083 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/1598/statm: 80087 5817 400 364 0 15993 0
[pid=1600] ppid=1598 vsize=84712 CPUtime=486.35 cores=0-7
/proc/1600/stat : 1600 (march_hi) T 1598 1600 1600 0 -1 4202496 18930 0 0 0 48631 4 0 0 25 0 1 0 916144597 86745088 18665 33554432000 4194304 4955518 140736738226496 18446744073709551615 4284802 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/1600/statm: 21178 18665 104 186 0 20991 0
[pid=1598/tid=1639] ppid=1580 vsize=320348 CPUtime=0 cores=0-7
/proc/1598/task/1639/stat : 1639 (python) S 1580 1580 1477 0 -1 4202560 49 0 0 0 0 0 0 0 16 0 2 0 916193231 328036352 5817 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092055985 0 0 16781312 514 18446604453875684288 0 0 -1 5 0 0 0
[pid=1602] ppid=1580 vsize=330740 CPUtime=0.58 cores=0-7
/proc/1602/stat : 1602 (python) S 1580 1580 1477 0 -1 4202560 1170 0 0 0 33 25 0 0 16 0 2 0 916144600 338677760 5838 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092059083 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/1602/statm: 82685 5838 400 364 0 18588 0
[pid=1604] ppid=1602 vsize=84712 CPUtime=486.24 cores=0-7
/proc/1604/stat : 1604 (march_hi) T 1602 1604 1604 0 -1 4202496 18932 0 0 0 48621 3 0 0 25 0 1 0 916144600 86745088 18665 33554432000 4194304 4955518 140737233720704 18446744073709551615 4244774 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/1604/statm: 21178 18665 104 186 0 20991 0
[pid=1602/tid=1638] ppid=1580 vsize=330740 CPUtime=0 cores=0-7
/proc/1602/task/1638/stat : 1638 (python) S 1580 1580 1477 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 916193224 338677760 5838 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092055985 0 0 16781312 514 18446604453865550848 0 0 -1 4 0 0 0
[pid=1606] ppid=1580 vsize=340996 CPUtime=0.58 cores=0-7
/proc/1606/stat : 1606 (python) S 1580 1580 1477 0 -1 4202560 1173 0 0 0 31 27 0 0 16 0 2 0 916144603 349179904 5842 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092059083 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/1606/statm: 85249 5842 400 364 0 21149 0
[pid=1608] ppid=1606 vsize=114180 CPUtime=486.05 cores=0-7
/proc/1608/stat : 1608 (march_hi) T 1606 1608 1608 0 -1 4202496 18941 0 0 0 48601 4 0 0 25 0 1 0 916144603 116920320 18674 33554432000 4194304 4955518 140736355248720 18446744073709551615 4227502 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/1608/statm: 28545 18674 104 186 0 28358 0
[pid=1606/tid=1637] ppid=1580 vsize=340996 CPUtime=0 cores=0-7
/proc/1606/task/1637/stat : 1637 (python) S 1580 1580 1477 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 916193208 349179904 5842 33554432000 4194304 5683644 140733936125264 18446744073709551615 208092055985 0 0 16781312 514 18446604453870997632 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2343.5
Current children cumulated vsize (KiB) 2513468
heavy processes:

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 1580 and gives
#  childrusage.ru_utime.tv_sec=2
#  childrusage.ru_utime.tv_usec=567609
#  childrusage.ru_stime.tv_sec=1
#  childrusage.ru_stime.tv_usec=372791
# CPU time returned by wait4() is 3.9404
# while last known CPU time is 2343.5
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5100.04
CPU time (s): 2343.5
CPU user time (s): 2341.98
CPU system time (s): 1.52
CPU usage (%): 45.9506
Max. virtual memory (cumulated for all children) (KiB): 2908084

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2.56761
system time used= 1.37279
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 28307
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 6462
involuntary context switches= 69

runsolver used 6.63899 second user time and 16.7565 second system time

The end

Launcher Data 

Begin job on node127 at 2011-06-14 11:57:28
IDJOB=3385495
IDBENCH=82760
IDSOLVER=1870
FILE ID=node127/3385495-1308045448
RUNJOBID= node127-1308045214-1491
PBS_JOBID= 13565704
Free space on /tmp= 74032 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v800-c3408-S1831593686-090.SATISFIABLE.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3385495-1308045448/watcher-3385495-1308045448 -o /tmp/evaluation-result-3385495-1308045448/solver-3385495-1308045448 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385495-1308045448.cnf -seed 2107038995 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 7dd850f670e1c9ea63e995ef24dddf5f
RANDOM SEED=2107038995

node127.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.823
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.64
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.823
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.823
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.823
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.86
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.823
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.823
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.823
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.60
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.823
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      31471268 kB
Buffers:        197904 kB
Cached:        1032452 kB
SwapCached:       9120 kB
Active:         329448 kB
Inactive:       922540 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      31471268 kB
SwapTotal:    67111528 kB
SwapFree:     67055948 kB
Dirty:           31928 kB
Writeback:           0 kB
AnonPages:       18712 kB
Mapped:          12560 kB
Slab:           157728 kB
PageTables:       4696 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   188848 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 74000 MiB
End job on node127 at 2011-06-14 13:22:30