Trace number 3317526

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (TO) 4034.15 1300.12

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v600-c2556-S506401817-075.UNKNOWN.cnf
MD5SUM2d7407b1b64ab354711b83851f166439
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables600
Number of clauses2556
Sum of the clauses size7668
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32556
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.32/0.45	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.32/0.45	c INFO: solving HOME/instance-3317526-1304269951.cnf
0.32/0.45	c INFO: computing task features
0.32/0.46	c INFO: computed variable-clause graph statistics
0.32/0.46	c INFO: computed clause balance statistics
0.40/0.50	c INFO: computed variable balance statistics
0.40/0.54	c INFO: computed small-clause counts
0.40/0.54	c INFO: computed Horn-clause counts
0.40/0.54	c INFO: computed variable graph statistics
0.40/0.54	c INFO: collected features for HOME/instance-3317526-1304269951.cnf in 0.10 s
0.40/0.57	c INFO: running gnovelty+2@0 for 97 with 4799 remaining (b = 0.58)
0.51/0.60	c INFO: running march_hi@0 for 388 with 4799 remaining (b = 0.30)
0.51/0.63	c INFO: running march_hi@0 for 97 with 4799 remaining (b = 0.08)
0.51/0.66	c INFO: running gnovelty+2@0 for 97 with 4799 remaining (b = 0.07)
334.79/105.38	c INFO: running march_hi@0 for 97 with 4695 remaining (b = 0.08)
420.71/129.91	c INFO: running march_hi@0 for 485 with 4670 remaining (b = 0.27)
513.17/160.83	c INFO: running march_hi@0 for 485 with 4639 remaining (b = 0.24)
845.93/271.64	c INFO: running march_hi@97 for 97 with 4528 remaining (b = 0.06)
1214.44/394.32	c INFO: running march_hi@0 for 291 with 4406 remaining (b = 0.16)
1328.28/432.26	c INFO: running march_hi@490 for 97 with 4368 remaining (b = 0.07)
1980.26/618.92	c INFO: running march_hi@583 for 97 with 4181 remaining (b = 0.08)
2090.60/650.54	c INFO: running march_hi@95 for 97 with 4149 remaining (b = 0.06)
2249.52/703.60	c INFO: running march_hi@99 for 97 with 4096 remaining (b = 0.07)
2450.32/770.71	c INFO: running march_hi@97 for 291 with 4029 remaining (b = 0.18)
2513.45/791.82	c INFO: running march_hi@0 for 485 with 4008 remaining (b = 0.22)
2562.81/808.39	c INFO: running march_hi@384 for 97 with 3992 remaining (b = 0.10)
2636.69/833.01	c INFO: running march_hi@586 for 97 with 3967 remaining (b = 0.06)
3122.62/995.41	c INFO: running march_hi@1072 for 291 with 3805 remaining (b = 0.17)
3147.46/1003.71	c INFO: running march_hi@479 for 97 with 3796 remaining (b = 0.06)
3363.54/1075.95	c INFO: running march_hi@193 for 291 with 3724 remaining (b = 0.14)
3555.70/1140.11	c INFO: running march_hi@0 for 388 with 3660 remaining (b = 0.19)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317526-1304269951/watcher-3317526-1304269951 -o /tmp/evaluation-result-3317526-1304269951/solver-3317526-1304269951 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317526-1304269951.cnf -seed 532702394 -budget 4800 -cores 4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.97 8.33 8.11 6/248 24615
/proc/meminfo: memFree=29497236/32951124 swapFree=67107188/67111528
[pid=24615] ppid=24613 vsize=15780 CPUtime=0 cores=0,2,4,6
/proc/24615/stat : 24615 (python) R 24613 24615 23486 0 -1 4202496 890 0 0 0 0 0 0 0 25 0 1 0 538593637 16158720 772 33554432000 4194304 5683644 140734311852032 18446744073709551615 210204308165 0 0 16781312 2 0 0 0 17 0 0 0 0
/proc/24615/statm: 3945 773 343 364 0 432 0

[startup+0.0961791 s]
/proc/loadavg: 7.97 8.33 8.11 6/248 24615
/proc/meminfo: memFree=29497236/32951124 swapFree=67107188/67111528
[pid=24615] ppid=24613 vsize=104228 CPUtime=0.06 cores=0,2,4,6
/proc/24615/stat : 24615 (python) S 24613 24615 23486 0 -1 4202496 3058 0 0 0 5 1 0 0 25 0 1 0 538593637 106729472 2741 33554432000 4194304 5683644 140734311852032 18446744073709551615 210204645376 0 2147483391 16781312 2 18446744071699207358 0 0 17 6 0 0 0
/proc/24615/statm: 26057 2741 847 364 0 1805 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 104228

[startup+0.100198 s]
/proc/loadavg: 7.97 8.33 8.11 6/248 24615
/proc/meminfo: memFree=29497236/32951124 swapFree=67107188/67111528
[pid=24615] ppid=24613 vsize=104620 CPUtime=0.06 cores=0,2,4,6
/proc/24615/stat : 24615 (python) S 24613 24615 23486 0 -1 4202496 3144 0 0 0 5 1 0 0 25 0 1 0 538593637 107130880 2822 33554432000 4194304 5683644 140734311852032 18446744073709551615 210204645376 0 2147483391 16781312 2 18446744071699207358 0 0 17 6 0 0 0
/proc/24615/statm: 26155 2822 847 364 0 1903 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 104620

[startup+0.306194 s]
/proc/loadavg: 7.97 8.33 8.11 6/248 24615
/proc/meminfo: memFree=29497236/32951124 swapFree=67107188/67111528
[pid=24615] ppid=24613 vsize=198920 CPUtime=0.24 cores=0,2,4,6
/proc/24615/stat : 24615 (python) R 24613 24615 23486 0 -1 4202496 7022 936 1 0 18 6 0 0 17 0 1 0 538593637 203694080 5053 33554432000 4194304 5683644 140734311852032 18446744073709551615 4620076 0 0 16781312 2 0 0 0 17 0 0 0 0
/proc/24615/statm: 49730 5053 1466 364 0 3757 0
Current children cumulated CPU time (s) 0.24
Current children cumulated vsize (KiB) 198920

[startup+0.701069 s]
/proc/loadavg: 7.97 8.33 8.11 6/248 24615
/proc/meminfo: memFree=29497236/32951124 swapFree=67107188/67111528
[pid=24615] ppid=24613 vsize=320324 CPUtime=0.57 cores=0,2,4,6
/proc/24615/stat : 24615 (python) S 24613 24615 23486 0 -1 4202496 12414 936 1 0 49 8 0 0 25 0 5 0 538593637 328011776 7277 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216474059 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/24615/statm: 80081 7277 1881 364 0 15990 0
Current children cumulated CPU time (s) 0.57
Current children cumulated vsize (KiB) 320324

[startup+1.50093 s]
/proc/loadavg: 7.97 8.33 8.11 10/261 24632
/proc/meminfo: memFree=29456460/32951124 swapFree=67107188/67111528
[pid=24615] ppid=24613 vsize=320324 CPUtime=0.57 cores=0,2,4,6
/proc/24615/stat : 24615 (python) S 24613 24615 23486 0 -1 4202496 12414 936 1 0 49 8 0 0 25 0 5 0 538593637 328011776 7277 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216474059 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/24615/statm: 80081 7277 1881 364 0 15990 0
[pid=24621] ppid=24615 vsize=278624 CPUtime=0 cores=0,2,4,6
/proc/24621/stat : 24621 (python) S 24615 24615 23486 0 -1 4202560 734 0 0 0 0 0 0 0 18 0 1 0 538593695 285310976 5589 33554432000 4194304 5683644 140734311852032 18446744073709551615 210204667407 0 0 16781312 514 6922239614627348479 0 0 17 6 0 0 0
/proc/24621/statm: 69656 5589 354 364 0 5574 0
[pid=24623] ppid=24621 vsize=1017932 CPUtime=0.91 cores=0,2,4,6
/proc/24623/stat : 24623 (gnovelty+2) R 24621 24623 24623 0 -1 4202496 445 0 0 0 91 0 0 0 21 0 1 0 538593695 1042362368 185 33554432000 134512640 135663704 4289877616 18446744073709551615 134545611 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/24623/statm: 254483 185 113 282 0 254199 0
[pid=24624] ppid=24615 vsize=289432 CPUtime=0 cores=0,2,4,6
/proc/24624/stat : 24624 (python) S 24615 24615 23486 0 -1 4202560 755 0 0 0 0 0 0 0 15 0 1 0 538593698 296378368 5715 33554432000 4194304 5683644 140734311852032 18446744073709551615 210204667462 0 0 16781312 514 5226664866369995012 0 0 17 2 0 0 0
/proc/24624/statm: 72358 5715 348 364 0 8273 0
[pid=24626] ppid=24624 vsize=2664 CPUtime=0.88 cores=0,2,4,6
/proc/24626/stat : 24626 (march_hi) R 24624 24626 24626 0 -1 4202496 728 0 0 0 88 0 0 0 25 0 1 0 538593698 2727936 463 33554432000 4194304 4955518 140734978868192 18446744073709551615 4244474 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/24626/statm: 666 463 104 186 0 479 0
[pid=24627] ppid=24615 vsize=299216 CPUtime=0 cores=0,2,4,6
/proc/24627/stat : 24627 (python) S 24615 24615 23486 0 -1 4202560 754 0 0 0 0 0 0 0 15 0 1 0 538593701 306397184 5616 33554432000 4194304 5683644 140734311852032 18446744073709551615 210204667462 0 0 16781312 514 4582474197795569922 0 0 17 6 0 0 0
/proc/24627/statm: 74804 5616 348 364 0 10716 0
[pid=24629] ppid=24627 vsize=2460 CPUtime=0.41 cores=0,2,4,6
/proc/24629/stat : 24629 (march_hi) R 24627 24629 24629 0 -1 4202496 684 0 0 0 41 0 0 0 25 0 1 0 538593703 2519040 419 33554432000 4194304 4955518 140736232454560 18446744073709551615 4238422 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/24629/statm: 615 419 104 186 0 428 0
[pid=24630] ppid=24615 vsize=310080 CPUtime=0 cores=0,2,4,6
/proc/24630/stat : 24630 (python) S 24615 24615 23486 0 -1 4202560 750 0 0 0 0 0 0 0 25 0 1 0 538593704 317521920 5740 33554432000 4194304 5683644 140734311852032 18446744073709551615 210204667462 0 0 16781312 514 14307646299137114111 0 0 17 6 0 0 0
/proc/24630/statm: 77520 5740 348 364 0 13429 0
[pid=24632] ppid=24630 vsize=1017932 CPUtime=0.81 cores=0,2,4,6
/proc/24632/stat : 24632 (gnovelty+2) R 24630 24632 24632 0 -1 4202496 447 0 0 0 81 0 0 0 25 0 1 0 538593704 1042362368 184 33554432000 134512640 135663704 4290612256 18446744073709551615 134545183 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/24632/statm: 254483 184 113 282 0 254199 0
[pid=24615/tid=24622] ppid=24613 vsize=320324 CPUtime=0 cores=0,2,4,6
/proc/24615/task/24622/stat : 24622 (python) S 24613 24615 23486 0 -1 4202560 27 936 0 0 0 0 0 0 19 0 5 0 538593695 328011776 7277 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 0 0 0 0
[pid=24615/tid=24625] ppid=24613 vsize=320324 CPUtime=0 cores=0,2,4,6
/proc/24615/task/24625/stat : 24625 (python) S 24613 24615 23486 0 -1 4202560 20 936 0 0 0 0 0 0 21 0 5 0 538593698 328011776 7277 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 0 0 0 0
[pid=24615/tid=24628] ppid=24613 vsize=320324 CPUtime=0 cores=0,2,4,6
/proc/24615/task/24628/stat : 24628 (python) S 24613 24615 23486 0 -1 4202560 22 936 0 0 0 0 0 0 23 0 5 0 538593701 328011776 7277 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 6 0 0 0
[pid=24615/tid=24631] ppid=24613 vsize=320324 CPUtime=0 cores=0,2,4,6
/proc/24615/task/24631/stat : 24631 (python) S 24613 24615 23486 0 -1 4202560 29 936 0 0 0 0 0 0 25 0 5 0 538593704 328011776 7277 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3.58
Current children cumulated vsize (KiB) 3538664

[startup+3.10066 s]
/proc/loadavg: 7.97 8.33 8.11 10/261 24632
/proc/meminfo: memFree=29455076/32951124 swapFree=67107188/67111528
[pid=24615] ppid=24613 vsize=320324 CPUtime=0.57 cores=0,2,4,6
/proc/24615/stat : 24615 (python) S 24613 24615 23486 0 -1 4202496 12414 936 1 0 49 8 0 0 25 0 5 0 538593637 328011776 7277 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216474059 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/24615/statm: 80081 7277 1881 364 0 15990 0
[pid=24621] ppid=24615 vsize=278624 CPUtime=0 cores=0,2,4,6
/proc/24621/stat : 24621 (python) S 24615 24615 23486 0 -1 4202560 734 0 0 0 0 0 0 0 18 0 1 0 538593695 285310976 5589 33554432000 4194304 5683644 140734311852032 18446744073709551615 210204667407 0 0 16781312 514 6922239614627315970 0 0 17 6 0 0 0
/proc/24621/statm: 69656 5589 354 364 0 5574 0
[pid=24623] ppid=24621 vsize=1017932 CPUtime=2.51 cores=0,2,4,6
/proc/24623/stat : 24623 (gnovelty+2) R 24621 24623 24623 0 -1 4202496 445 0 0 0 251 0 0 0 25 0 1 0 538593695 1042362368 185 33554432000 134512640 135663704 4289877616 18446744073709551615 134525678 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/24623/statm: 254483 185 113 282 0 254199 0
[pid=24624] ppid=24615 vsize=289432 CPUtime=0 cores=0,2,4,6
/proc/24624/stat : 24624 (python) S 24615 24615 23486 0 -1 4202560 758 0 0 0 0 0 0 0 15 0 1 0 538593698 296378368 5715 33554432000 4194304 5683644 140734311852032 18446744073709551615 210204667462 0 0 16781312 514 5226664866369995012 0 0 17 2 0 0 0
/proc/24624/statm: 72358 5715 348 364 0 8273 0
[pid=24626] ppid=24624 vsize=2916 CPUtime=2.48 cores=0,2,4,6
/proc/24626/stat : 24626 (march_hi) R 24624 24626 24626 0 -1 4202496 789 0 0 0 248 0 0 0 25 0 1 0 538593698 2985984 524 33554432000 4194304 4955518 140734978868192 18446744073709551615 4227664 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/24626/statm: 729 524 104 186 0 542 0
[pid=24627] ppid=24615 vsize=299216 CPUtime=0 cores=0,2,4,6
/proc/24627/stat : 24627 (python) S 24615 24615 23486 0 -1 4202560 756 0 0 0 0 0 0 0 15 0 1 0 538593701 306397184 5616 33554432000 4194304 5683644 140734311852032 18446744073709551615 210204667462 0 0 16781312 514 4582474197795569922 0 0 17 6 0 0 0
/proc/24627/statm: 74804 5616 348 364 0 10716 0
[pid=24629] ppid=24627 vsize=2664 CPUtime=1.21 cores=0,2,4,6
/proc/24629/stat : 24629 (march_hi) R 24627 24629 24629 0 -1 4202496 740 0 0 0 121 0 0 0 25 0 1 0 538593703 2727936 475 33554432000 4194304 4955518 140736232454560 18446744073709551615 4209600 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/24629/statm: 666 475 104 186 0 479 0
[pid=24630] ppid=24615 vsize=310080 CPUtime=0 cores=0,2,4,6
/proc/24630/stat : 24630 (python) S 24615 24615 23486 0 -1 4202560 750 0 0 0 0 0 0 0 18 0 1 0 538593704 317521920 5740 33554432000 4194304 5683644 140734311852032 18446744073709551615 210204667462 0 0 16781312 514 14307646299137081603 0 0 17 6 0 0 0
/proc/24630/statm: 77520 5740 348 364 0 13429 0
[pid=24632] ppid=24630 vsize=1017932 CPUtime=2.4 cores=0,2,4,6
/proc/24632/stat : 24632 (gnovelty+2) R 24630 24632 24632 0 -1 4202496 447 0 0 0 240 0 0 0 25 0 1 0 538593704 1042362368 184 33554432000 134512640 135663704 4290612256 18446744073709551615 134525678 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/24632/statm: 254483 184 113 282 0 254199 0
[pid=24615/tid=24622] ppid=24613 vsize=320324 CPUtime=0 cores=0,2,4,6
/proc/24615/task/24622/stat : 24622 (python) S 24613 24615 23486 0 -1 4202560 27 936 0 0 0 0 0 0 19 0 5 0 538593695 328011776 7277 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 0 0 0 0
[pid=24615/tid=24625] ppid=24613 vsize=320324 CPUtime=0 cores=0,2,4,6
/proc/24615/task/24625/stat : 24625 (python) S 24613 24615 23486 0 -1 4202560 20 936 0 0 0 0 0 0 21 0 5 0 538593698 328011776 7277 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 0 0 0 0
[pid=24615/tid=24628] ppid=24613 vsize=320324 CPUtime=0 cores=0,2,4,6
/proc/24615/task/24628/stat : 24628 (python) S 24613 24615 23486 0 -1 4202560 22 936 0 0 0 0 0 0 23 0 5 0 538593701 328011776 7277 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 6 0 0 0
[pid=24615/tid=24631] ppid=24613 vsize=320324 CPUtime=0 cores=0,2,4,6
/proc/24615/task/24631/stat : 24631 (python) S 24613 24615 23486 0 -1 4202560 29 936 0 0 0 0 0 0 25 0 5 0 538593704 328011776 7277 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 9.17
Current children cumulated vsize (KiB) 3539120

[startup+6.30113 s]
/proc/loadavg: 8.05 8.34 8.11 10/261 24632
/proc/meminfo: memFree=29450364/32951124 swapFree=67107188/67111528
[pid=24615] ppid=24613 vsize=320324 CPUtime=0.57 cores=0,2,4,6
/proc/24615/stat : 24615 (python) S 24613 24615 23486 0 -1 4202496 12414 936 1 0 49 8 0 0 25 0 5 0 538593637 328011776 7277 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216474059 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/24615/statm: 80081 7277 1881 364 0 15990 0
[pid=24621] ppid=24615 vsize=278624 CPUtime=0 cores=0,2,4,6
/proc/24621/stat : 24621 (python) S 24615 24615 23486 0 -1 4202560 900 0 0 0 0 0 0 0 18 0 1 0 538593695 285310976 5612 33554432000 4194304 5683644 140734311852032 18446744073709551615 210204667407 0 0 16781312 514 6922239614627315970 0 0 17 6 0 0 0
/proc/24621/statm: 69656 5612 377 364 0 5574 0
[pid=24623] ppid=24621 vsize=1017932 CPUtime=4.9 cores=0,2,4,6
/proc/24623/stat : 24623 (gnovelty+2) R 24621 24623 24623 0 -1 4202496 445 0 0 0 490 0 0 0 25 0 1 0 538593695 1042362368 185 33554432000 134512640 135663704 4289877616 18446744073709551615 134524857 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/24623/statm: 254483 185 113 282 0 254199 0
[pid=24624] ppid=24615 vsize=289432 CPUtime=0 cores=0,2,4,6
/proc/24624/stat : 24624 (python) S 24615 24615 23486 0 -1 4202560 902 0 0 0 0 0 0 0 15 0 1 0 538593698 296378368 5737 33554432000 4194304 5683644 140734311852032 18446744073709551615 210204667462 0 0 16781312 514 5226664866369995012 0 0 17 2 0 0 0
/proc/24624/statm: 72358 5737 370 364 0 8273 0
[pid=24626] ppid=24624 vsize=3620 CPUtime=5.67 cores=0,2,4,6
/proc/24626/stat : 24626 (march_hi) R 24624 24626 24626 0 -1 4202496 929 0 0 0 567 0 0 0 25 0 1 0 538593698 3706880 664 33554432000 4194304 4955518 140734978868192 18446744073709551615 4209730 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/24626/statm: 905 664 104 186 0 718 0
[pid=24627] ppid=24615 vsize=299216 CPUtime=0 cores=0,2,4,6
/proc/24627/stat : 24627 (python) S 24615 24615 23486 0 -1 4202560 904 0 0 0 0 0 0 0 15 0 1 0 538593701 306397184 5639 33554432000 4194304 5683644 140734311852032 18446744073709551615 210204667462 0 0 16781312 514 4582474197795569922 0 0 17 6 0 0 0
/proc/24627/statm: 74804 5639 371 364 0 10716 0
[pid=24629] ppid=24627 vsize=3280 CPUtime=3.6 cores=0,2,4,6
/proc/24629/stat : 24629 (march_hi) R 24627 24629 24629 0 -1 4202496 847 0 0 0 360 0 0 0 25 0 1 0 538593703 3358720 582 33554432000 4194304 4955518 140736232454560 18446744073709551615 4235388 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/24629/statm: 820 582 104 186 0 633 0
[pid=24630] ppid=24615 vsize=310080 CPUtime=0 cores=0,2,4,6
/proc/24630/stat : 24630 (python) S 24615 24615 23486 0 -1 4202560 915 0 0 0 0 0 0 0 18 0 1 0 538593704 317521920 5763 33554432000 4194304 5683644 140734311852032 18446744073709551615 210204667462 0 0 16781312 514 14307646299137081603 0 0 17 6 0 0 0
/proc/24630/statm: 77520 5763 371 364 0 13429 0
[pid=24632] ppid=24630 vsize=1017932 CPUtime=4.76 cores=0,2,4,6
/proc/24632/stat : 24632 (gnovelty+2) R 24630 24632 24632 0 -1 4202496 447 0 0 0 476 0 0 0 25 0 1 0 538593704 1042362368 184 33554432000 134512640 135663704 4290612256 18446744073709551615 134544252 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/24632/statm: 254483 184 113 282 0 254199 0
[pid=24615/tid=24622] ppid=24613 vsize=320324 CPUtime=0 cores=0,2,4,6
/proc/24615/task/24622/stat : 24622 (python) S 24613 24615 23486 0 -1 4202560 27 936 0 0 0 0 0 0 19 0 5 0 538593695 328011776 7277 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 0 0 0 0
[pid=24615/tid=24625] ppid=24613 vsize=320324 CPUtime=0 cores=0,2,4,6
/proc/24615/task/24625/stat : 24625 (python) S 24613 24615 23486 0 -1 4202560 20 936 0 0 0 0 0 0 21 0 5 0 538593698 328011776 7277 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 0 0 0 0
[pid=24615/tid=24628] ppid=24613 vsize=320324 CPUtime=0 cores=0,2,4,6
/proc/24615/task/24628/stat : 24628 (python) S 24613 24615 23486 0 -1 4202560 22 936 0 0 0 0 0 0 23 0 5 0 538593701 328011776 7277 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 6 0 0 0
[pid=24615/tid=24631] ppid=24613 vsize=320324 CPUtime=0 cores=0,2,4,6
/proc/24615/task/24631/stat : 24631 (python) S 24613 24615 23486 0 -1 4202560 29 936 0 0 0 0 0 0 25 0 5 0 538593704 328011776 7277 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 19.5
Current children cumulated vsize (KiB) 3540440

[startup+12.7011 s]
/proc/loadavg: 8.12 8.35 8.12 11/261 24633
/proc/meminfo: memFree=29445772/32951124 swapFree=67107188/67111528
[pid=24615] ppid=24613 vsize=320324 CPUtime=0.57 cores=0,2,4,6
/proc/24615/stat : 24615 (python) S 24613 24615 23486 0 -1 4202496 12414 936 1 0 49 8 0 0 25 0 5 0 538593637 328011776 7277 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216474059 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/24615/statm: 80081 7277 1881 364 0 15990 0
[pid=24621] ppid=24615 vsize=278624 CPUtime=0.01 cores=0,2,4,6
/proc/24621/stat : 24621 (python) S 24615 24615 23486 0 -1 4202560 904 0 0 0 1 0 0 0 18 0 1 0 538593695 285310976 5612 33554432000 4194304 5683644 140734311852032 18446744073709551615 210204667407 0 0 16781312 514 10654742662172606722 0 0 17 6 0 0 0
/proc/24621/statm: 69656 5612 377 364 0 5574 0
[pid=24623] ppid=24621 vsize=1017932 CPUtime=8.1 cores=0,2,4,6
/proc/24623/stat : 24623 (gnovelty+2) R 24621 24623 24623 0 -1 4202496 445 0 0 0 810 0 0 0 25 0 1 0 538593695 1042362368 185 33554432000 134512640 135663704 4289877616 18446744073709551615 134544218 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/24623/statm: 254483 185 113 282 0 254199 0
[pid=24624] ppid=24615 vsize=289432 CPUtime=0.01 cores=0,2,4,6

################
# More data... #
################

[pid=24627] ppid=24615 vsize=299216 CPUtime=1.74 cores=0,2,4,6
/proc/24627/stat : 24627 (python) S 24615 24615 23486 0 -1 4202560 1155 0 0 0 94 80 0 0 15 0 2 0 538593701 306397184 5669 33554432000 4194304 5683644 140734311852032 18446744073709551615 210204667462 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/24627/statm: 74804 5669 400 364 0 10716 0
[pid=24629] ppid=24627 vsize=108764 CPUtime=698.69 cores=0,2,4,6
/proc/24629/stat : 24629 (march_hi) R 24627 24629 24629 0 -1 4202496 22954 0 0 0 69864 5 0 0 25 0 1 0 538593703 111374336 22689 33554432000 4194304 4955518 140736232454560 18446744073709551615 4227664 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/24629/statm: 27191 22689 104 186 0 27004 0
[pid=24627/tid=24637] ppid=24615 vsize=299216 CPUtime=0 cores=0,2,4,6
/proc/24627/task/24637/stat : 24637 (python) S 24615 24615 23486 0 -1 4202560 49 0 0 0 0 0 0 0 15 0 2 0 538604173 306397184 5669 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 514 18446604451948985344 0 0 -1 6 0 0 0
[pid=24630] ppid=24615 vsize=310080 CPUtime=0.19 cores=0,2,4,6
/proc/24630/stat : 24630 (python) S 24615 24615 23486 0 -1 4202560 1113 0 0 0 10 9 0 0 19 0 2 0 538593704 317521920 5793 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/24630/statm: 77520 5793 400 364 0 13429 0
[pid=24632] ppid=24630 vsize=1017932 CPUtime=97.85 cores=0,2,4,6
/proc/24632/stat : 24632 (gnovelty+2) T 24630 24632 24632 0 -1 4202496 447 0 0 0 9785 0 0 0 25 0 1 0 538593704 1042362368 184 33554432000 134512640 135663704 4290612256 18446744073709551615 134545572 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/24632/statm: 254483 184 113 282 0 254199 0
[pid=24630/tid=24638] ppid=24615 vsize=310080 CPUtime=0 cores=0,2,4,6
/proc/24630/task/24638/stat : 24638 (python) S 24615 24615 23486 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 538606626 317521920 5793 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 514 18446604451948984448 0 0 -1 6 0 0 0
[pid=24639] ppid=24615 vsize=319944 CPUtime=1.87 cores=0,2,4,6
/proc/24639/stat : 24639 (python) S 24615 24615 23486 0 -1 4202560 1179 0 0 0 107 80 0 0 15 0 2 0 538606629 327622656 5709 33554432000 4194304 5683644 140734311852032 18446744073709551615 210204667462 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/24639/statm: 79986 5709 409 364 0 15892 0
[pid=24641] ppid=24639 vsize=170464 CPUtime=892.75 cores=0,2,4,6
/proc/24641/stat : 24641 (march_hi) R 24639 24641 24641 0 -1 4202496 28749 0 0 0 89268 7 0 0 25 0 1 0 538606629 174555136 28483 33554432000 4194304 4955518 140733969023200 18446744073709551615 4224691 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/24641/statm: 42616 28483 104 186 0 42429 0
[pid=24639/tid=25075] ppid=24615 vsize=319944 CPUtime=0 cores=0,2,4,6
/proc/24639/task/25075/stat : 25075 (python) S 24615 24615 23486 0 -1 4202560 50 0 0 0 0 0 0 0 15 0 2 0 538670706 327622656 5709 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 514 18446604450998127424 0 0 -1 2 0 0 0
[pid=24691] ppid=24615 vsize=330684 CPUtime=1.66 cores=0,2,4,6
/proc/24691/stat : 24691 (python) S 24615 24615 23486 0 -1 4202560 1177 0 0 0 96 70 0 0 15 0 2 0 538609721 338620416 5824 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/24691/statm: 82671 5824 400 364 0 18574 0
[pid=24693] ppid=24691 vsize=126264 CPUtime=780.8 cores=0,2,4,6
/proc/24693/stat : 24693 (march_hi) T 24691 24693 24693 0 -1 4202496 25192 0 0 0 78072 8 0 0 25 0 1 0 538609721 129294336 24926 33554432000 4194304 4955518 140736666741024 18446744073709551615 4227752 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/24693/statm: 31566 24926 104 186 0 31379 0
[pid=24691/tid=25051] ppid=24615 vsize=330684 CPUtime=0 cores=0,2,4,6
/proc/24691/task/25051/stat : 25051 (python) S 24615 24615 23486 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 538658689 338620416 5824 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 514 18446604453931590016 0 0 -1 0 0 0 0
[pid=25076] ppid=24615 vsize=340920 CPUtime=0.77 cores=0,2,4,6
/proc/25076/stat : 25076 (python) S 24615 24615 23486 0 -1 4202560 1009 0 0 0 40 37 0 0 15 0 1 0 538672819 349102080 5793 33554432000 4194304 5683644 140734311852032 18446744073709551615 210204667462 0 0 16781312 514 10654742662172606724 0 0 17 0 0 0 0
/proc/25076/statm: 85230 5793 371 364 0 21130 0
[pid=25078] ppid=25076 vsize=50768 CPUtime=356.81 cores=0,2,4,6
/proc/25078/stat : 25078 (march_hi) R 25076 25078 25078 0 -1 4202496 12303 0 0 0 35677 4 0 0 25 0 1 0 538672820 51986432 12037 33554432000 4194304 4955518 140733934013952 18446744073709551615 4209692 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/25078/statm: 12692 12037 104 186 0 12505 0
[pid=25108] ppid=24615 vsize=351268 CPUtime=0.15 cores=0,2,4,6
/proc/25108/stat : 25108 (python) S 24615 24615 23486 0 -1 4202560 1009 0 0 0 8 7 0 0 15 0 1 0 538707649 359698432 5832 33554432000 4194304 5683644 140734311852032 18446744073709551615 210204667462 0 0 16781312 514 10654742662172606724 0 0 17 6 0 0 0
/proc/25108/statm: 87817 5832 377 364 0 23714 0
[pid=25110] ppid=25108 vsize=11448 CPUtime=50.99 cores=0,2,4,6
/proc/25110/stat : 25110 (march_hi) R 25108 25110 25110 0 -1 4202496 2537 0 0 0 5099 0 0 0 25 0 1 0 538707649 11722752 2269 33554432000 4194304 4955518 140734123783728 18446744073709551615 4224651 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/25110/statm: 2862 2269 104 186 0 2675 0
[pid=24615/tid=24622] ppid=24613 vsize=361512 CPUtime=0 cores=0,2,4,6
/proc/24615/task/24622/stat : 24622 (python) S 24613 24615 23486 0 -1 4202560 27 936 0 0 0 0 0 0 19 0 9 0 538593695 370188288 7349 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 0 0 0 0
[pid=24615/tid=24625] ppid=24613 vsize=361512 CPUtime=0 cores=0,2,4,6
/proc/24615/task/24625/stat : 24625 (python) S 24613 24615 23486 0 -1 4202560 28 936 0 0 0 0 0 0 18 0 9 0 538593698 370188288 7349 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 6 0 0 0
[pid=24615/tid=24628] ppid=24613 vsize=361512 CPUtime=0 cores=0,2,4,6
/proc/24615/task/24628/stat : 24628 (python) S 24613 24615 23486 0 -1 4202560 34 936 0 0 0 0 0 0 18 0 9 0 538593701 370188288 7349 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 6 0 0 0
[pid=24615/tid=24631] ppid=24613 vsize=361512 CPUtime=0 cores=0,2,4,6
/proc/24615/task/24631/stat : 24631 (python) S 24613 24615 23486 0 -1 4202560 29 936 0 0 0 0 0 0 25 0 9 0 538593704 370188288 7349 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 6 0 0 0
[pid=24615/tid=24640] ppid=24613 vsize=361512 CPUtime=0 cores=0,2,4,6
/proc/24615/task/24640/stat : 24640 (python) S 24613 24615 23486 0 -1 4202560 36 936 0 0 0 0 0 0 18 0 9 0 538606629 370188288 7349 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 6 0 0 0
[pid=24615/tid=24692] ppid=24613 vsize=361512 CPUtime=0 cores=0,2,4,6
/proc/24615/task/24692/stat : 24692 (python) S 24613 24615 23486 0 -1 4202560 29 936 0 0 0 0 0 0 18 0 9 0 538609721 370188288 7349 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 6 0 0 0
[pid=24615/tid=25077] ppid=24613 vsize=361512 CPUtime=0 cores=0,2,4,6
/proc/24615/task/25077/stat : 25077 (python) S 24613 24615 23486 0 -1 4202560 31 936 0 0 0 0 0 0 20 0 9 0 538672820 370188288 7349 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 6 0 0 0
[pid=24615/tid=25109] ppid=24613 vsize=361512 CPUtime=0 cores=0,2,4,6
/proc/24615/task/25109/stat : 25109 (python) S 24613 24615 23486 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 9 0 538707649 370188288 7349 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3861.46
Current children cumulated vsize (KiB) 5565960



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+1300 s]
/proc/loadavg: 5.16 6.76 7.87 6/246 25114
/proc/meminfo: memFree=29302828/32951124 swapFree=67107188/67111528
[pid=24615] ppid=24613 vsize=361512 CPUtime=1.05 cores=0,2,4,6
/proc/24615/stat : 24615 (python) S 24613 24615 23486 0 -1 4202496 16782 936 1 0 96 9 0 0 19 0 9 0 538593637 370188288 7349 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216474059 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/24615/statm: 90378 7349 1890 364 0 26275 0
[pid=24621] ppid=24615 vsize=288868 CPUtime=0.24 cores=0,2,4,6
/proc/24621/stat : 24621 (python) S 24615 24615 23486 0 -1 4202560 1100 0 0 0 13 11 0 0 19 0 2 0 538593695 295800832 5650 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/24621/statm: 72217 5650 410 364 0 8135 0
[pid=24623] ppid=24621 vsize=1017932 CPUtime=97.26 cores=0,2,4,6
/proc/24623/stat : 24623 (gnovelty+2) T 24621 24623 24623 0 -1 4202496 446 0 0 0 9726 0 0 0 25 0 1 0 538593695 1042362368 186 33554432000 134512640 135663704 4289877616 18446744073709551615 134525361 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/24623/statm: 254483 186 113 282 0 254199 0
[pid=24621/tid=24690] ppid=24615 vsize=288868 CPUtime=0 cores=0,2,4,6
/proc/24621/task/24690/stat : 24690 (python) S 24615 24615 23486 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 538609718 295800832 5650 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 514 18446604441639500864 0 0 -1 6 0 0 0
[pid=24624] ppid=24615 vsize=289432 CPUtime=1.71 cores=0,2,4,6
/proc/24624/stat : 24624 (python) S 24615 24615 23486 0 -1 4202560 1157 0 0 0 97 74 0 0 15 0 2 0 538593698 296378368 5768 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/24624/statm: 72358 5768 400 364 0 8273 0
[pid=24626] ppid=24624 vsize=170464 CPUtime=876.93 cores=0,2,4,6
/proc/24626/stat : 24626 (march_hi) T 24624 24626 24626 0 -1 4202496 28253 0 0 0 87676 17 0 0 25 0 1 0 538593698 174555136 27988 33554432000 4194304 4955518 140734978868192 18446744073709551615 4224648 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/24626/statm: 42616 27988 104 186 0 42429 0
[pid=24624/tid=24982] ppid=24615 vsize=289432 CPUtime=0 cores=0,2,4,6
/proc/24624/task/24982/stat : 24982 (python) S 24615 24615 23486 0 -1 4202560 44 0 0 0 0 0 0 0 15 0 2 0 538633066 296378368 5768 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 514 18446604441639498176 0 0 -1 2 0 0 0
[pid=24627] ppid=24615 vsize=299216 CPUtime=1.83 cores=0,2,4,6
/proc/24627/stat : 24627 (python) S 24615 24615 23486 0 -1 4202560 1155 0 0 0 100 83 0 0 15 0 2 0 538593701 306397184 5669 33554432000 4194304 5683644 140734311852032 18446744073709551615 210204667462 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/24627/statm: 74804 5669 400 364 0 10716 0
[pid=24629] ppid=24627 vsize=126264 CPUtime=755.77 cores=0,2,4,6
/proc/24629/stat : 24629 (march_hi) R 24627 24629 24629 0 -1 4202496 24565 0 0 0 75571 6 0 0 25 0 1 0 538593703 129294336 24300 33554432000 4194304 4955518 140736232454560 18446744073709551615 4261393 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/24629/statm: 31566 24300 104 186 0 31379 0
[pid=24627/tid=24637] ppid=24615 vsize=299216 CPUtime=0 cores=0,2,4,6
/proc/24627/task/24637/stat : 24637 (python) S 24615 24615 23486 0 -1 4202560 49 0 0 0 0 0 0 0 15 0 2 0 538604173 306397184 5669 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 514 18446604451948985344 0 0 -1 6 0 0 0
[pid=24630] ppid=24615 vsize=310080 CPUtime=0.19 cores=0,2,4,6
/proc/24630/stat : 24630 (python) S 24615 24615 23486 0 -1 4202560 1113 0 0 0 10 9 0 0 19 0 2 0 538593704 317521920 5793 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/24630/statm: 77520 5793 400 364 0 13429 0
[pid=24632] ppid=24630 vsize=1017932 CPUtime=97.85 cores=0,2,4,6
/proc/24632/stat : 24632 (gnovelty+2) T 24630 24632 24632 0 -1 4202496 447 0 0 0 9785 0 0 0 25 0 1 0 538593704 1042362368 184 33554432000 134512640 135663704 4290612256 18446744073709551615 134545572 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/24632/statm: 254483 184 113 282 0 254199 0
[pid=24630/tid=24638] ppid=24615 vsize=310080 CPUtime=0 cores=0,2,4,6
/proc/24630/task/24638/stat : 24638 (python) S 24615 24615 23486 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 538606626 317521920 5793 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 514 18446604451948984448 0 0 -1 6 0 0 0
[pid=24639] ppid=24615 vsize=319944 CPUtime=1.96 cores=0,2,4,6
/proc/24639/stat : 24639 (python) S 24615 24615 23486 0 -1 4202560 1179 0 0 0 113 83 0 0 15 0 2 0 538606629 327622656 5709 33554432000 4194304 5683644 140734311852032 18446744073709551615 210204667462 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/24639/statm: 79986 5709 409 364 0 15892 0
[pid=24641] ppid=24639 vsize=170464 CPUtime=921.56 cores=0,2,4,6
/proc/24641/stat : 24641 (march_hi) R 24639 24641 24641 0 -1 4202496 29644 0 0 0 92149 7 0 0 25 0 1 0 538606629 174555136 29378 33554432000 4194304 4955518 140733969023200 18446744073709551615 4224628 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/24641/statm: 42616 29378 104 186 0 42429 0
[pid=24639/tid=25075] ppid=24615 vsize=319944 CPUtime=0 cores=0,2,4,6
/proc/24639/task/25075/stat : 25075 (python) S 24615 24615 23486 0 -1 4202560 50 0 0 0 0 0 0 0 15 0 2 0 538670706 327622656 5709 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 514 18446604450998127424 0 0 -1 2 0 0 0
[pid=24691] ppid=24615 vsize=330684 CPUtime=1.66 cores=0,2,4,6
/proc/24691/stat : 24691 (python) S 24615 24615 23486 0 -1 4202560 1177 0 0 0 96 70 0 0 15 0 2 0 538609721 338620416 5824 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/24691/statm: 82671 5824 400 364 0 18574 0
[pid=24693] ppid=24691 vsize=126264 CPUtime=780.8 cores=0,2,4,6
/proc/24693/stat : 24693 (march_hi) T 24691 24693 24693 0 -1 4202496 25192 0 0 0 78072 8 0 0 25 0 1 0 538609721 129294336 24926 33554432000 4194304 4955518 140736666741024 18446744073709551615 4227752 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/24693/statm: 31566 24926 104 186 0 31379 0
[pid=24691/tid=25051] ppid=24615 vsize=330684 CPUtime=0 cores=0,2,4,6
/proc/24691/task/25051/stat : 25051 (python) S 24615 24615 23486 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 538658689 338620416 5824 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 514 18446604453931590016 0 0 -1 0 0 0 0
[pid=25076] ppid=24615 vsize=340920 CPUtime=0.87 cores=0,2,4,6
/proc/25076/stat : 25076 (python) S 24615 24615 23486 0 -1 4202560 1009 0 0 0 46 41 0 0 15 0 1 0 538672819 349102080 5793 33554432000 4194304 5683644 140734311852032 18446744073709551615 210204667462 0 0 16781312 514 10654742662172606724 0 0 17 0 0 0 0
/proc/25076/statm: 85230 5793 371 364 0 21130 0
[pid=25078] ppid=25076 vsize=79296 CPUtime=414.43 cores=0,2,4,6
/proc/25078/stat : 25078 (march_hi) R 25076 25078 25078 0 -1 4202496 14104 0 0 0 41438 5 0 0 25 0 1 0 538672820 81199104 13838 33554432000 4194304 4955518 140733934013952 18446744073709551615 4244774 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/25078/statm: 19824 13838 104 186 0 19637 0
[pid=25108] ppid=24615 vsize=351268 CPUtime=0.24 cores=0,2,4,6
/proc/25108/stat : 25108 (python) S 24615 24615 23486 0 -1 4202560 1027 0 0 0 12 12 0 0 15 0 1 0 538707649 359698432 5832 33554432000 4194304 5683644 140734311852032 18446744073709551615 210204667462 0 0 16781312 514 10654742662172606724 0 0 17 6 0 0 0
/proc/25108/statm: 87817 5832 377 364 0 23714 0
[pid=25110] ppid=25108 vsize=16424 CPUtime=79.8 cores=0,2,4,6
/proc/25110/stat : 25110 (march_hi) R 25108 25110 25110 0 -1 4202496 3547 0 0 0 7979 1 0 0 25 0 1 0 538707649 16818176 3279 33554432000 4194304 4955518 140734123783728 18446744073709551615 4227679 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/25110/statm: 4106 3279 104 186 0 3919 0
[pid=24615/tid=24622] ppid=24613 vsize=361512 CPUtime=0 cores=0,2,4,6
/proc/24615/task/24622/stat : 24622 (python) S 24613 24615 23486 0 -1 4202560 27 936 0 0 0 0 0 0 19 0 9 0 538593695 370188288 7349 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 0 0 0 0
[pid=24615/tid=24625] ppid=24613 vsize=361512 CPUtime=0 cores=0,2,4,6
/proc/24615/task/24625/stat : 24625 (python) S 24613 24615 23486 0 -1 4202560 28 936 0 0 0 0 0 0 18 0 9 0 538593698 370188288 7349 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 6 0 0 0
[pid=24615/tid=24628] ppid=24613 vsize=361512 CPUtime=0 cores=0,2,4,6
/proc/24615/task/24628/stat : 24628 (python) S 24613 24615 23486 0 -1 4202560 34 936 0 0 0 0 0 0 18 0 9 0 538593701 370188288 7349 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 6 0 0 0
[pid=24615/tid=24631] ppid=24613 vsize=361512 CPUtime=0 cores=0,2,4,6
/proc/24615/task/24631/stat : 24631 (python) S 24613 24615 23486 0 -1 4202560 29 936 0 0 0 0 0 0 25 0 9 0 538593704 370188288 7349 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 6 0 0 0
[pid=24615/tid=24640] ppid=24613 vsize=361512 CPUtime=0 cores=0,2,4,6
/proc/24615/task/24640/stat : 24640 (python) S 24613 24615 23486 0 -1 4202560 36 936 0 0 0 0 0 0 18 0 9 0 538606629 370188288 7349 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 6 0 0 0
[pid=24615/tid=24692] ppid=24613 vsize=361512 CPUtime=0 cores=0,2,4,6
/proc/24615/task/24692/stat : 24692 (python) S 24613 24615 23486 0 -1 4202560 29 936 0 0 0 0 0 0 18 0 9 0 538609721 370188288 7349 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 6 0 0 0
[pid=24615/tid=25077] ppid=24613 vsize=361512 CPUtime=0 cores=0,2,4,6
/proc/24615/task/25077/stat : 25077 (python) S 24613 24615 23486 0 -1 4202560 31 936 0 0 0 0 0 0 20 0 9 0 538672820 370188288 7349 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 6 0 0 0
[pid=24615/tid=25109] ppid=24613 vsize=361512 CPUtime=0 cores=0,2,4,6
/proc/24615/task/25109/stat : 25109 (python) S 24613 24615 23486 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 9 0 538707649 370188288 7349 33554432000 4194304 5683644 140734311852032 18446744073709551615 210216470961 0 0 16781312 2 18446604441639499072 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4034.15
Current children cumulated vsize (KiB) 5616964

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 24615 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=978851
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=102984
# CPU time returned by wait4() is 1.08183
# while last known CPU time is 4034.15
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1300.12
CPU time (s): 4034.15
CPU user time (s): 4029.79
CPU system time (s): 4.36
CPU usage (%): 310.29
Max. virtual memory (cumulated for all children) (KiB): 5616964

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0.978851
system time used= 0.102984
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 18156
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1057
involuntary context switches= 55

runsolver used 3.09653 second user time and 6.555 second system time

The end

Launcher Data 

Begin job on node130 at 2011-05-01 19:12:31
IDJOB=3317526
IDBENCH=82991
IDSOLVER=1705
FILE ID=node130/3317526-1304269951
RUNJOBID= node130-1304267282-23504
PBS_JOBID= 13173949
Free space on /tmp= 72668 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v600-c2556-S506401817-075.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317526-1304269951/watcher-3317526-1304269951 -o /tmp/evaluation-result-3317526-1304269951/solver-3317526-1304269951 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317526-1304269951.cnf -seed 532702394 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 2d7407b1b64ab354711b83851f166439
RANDOM SEED=532702394

node130.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      29497276 kB
Buffers:        235364 kB
Cached:        1495180 kB
SwapCached:       1948 kB
Active:        1907720 kB
Inactive:      1248344 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      29497276 kB
SwapTotal:    67111528 kB
SwapFree:     67107188 kB
Dirty:           36960 kB
Writeback:          52 kB
AnonPages:     1424548 kB
Mapped:          24116 kB
Slab:           221080 kB
PageTables:      17732 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 19468740 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72668 MiB
End job on node130 at 2011-05-01 19:34:14