Trace number 3317590

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (TO) 4800.21 1203.67

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v550-c2343-S1693168158-005.UNKNOWN.cnf
MD5SUMc2b69177f59f5d1d2093247dc6de56f3
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables550
Number of clauses2343
Sum of the clauses size7029
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32343
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.34/0.68	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.34/0.68	c INFO: solving HOME/instance-3317590-1304270240.cnf
0.34/0.68	c INFO: computing task features
0.34/0.69	c INFO: computed variable-clause graph statistics
0.34/0.69	c INFO: computed clause balance statistics
0.42/0.73	c INFO: computed variable balance statistics
0.42/0.76	c INFO: computed small-clause counts
0.42/0.76	c INFO: computed Horn-clause counts
0.42/0.77	c INFO: computed variable graph statistics
0.42/0.77	c INFO: collected features for HOME/instance-3317590-1304270240.cnf in 0.09 s
0.42/0.80	c INFO: running gnovelty+2@0 for 97 with 4799 remaining (b = 0.57)
0.52/0.83	c INFO: running march_hi@0 for 388 with 4799 remaining (b = 0.32)
0.52/0.85	c INFO: running gnovelty+2@0 for 97 with 4799 remaining (b = 0.07)
0.52/0.88	c INFO: running march_hi@0 for 388 with 4799 remaining (b = 0.30)
400.03/100.98	c INFO: running march_hi@0 for 485 with 4699 remaining (b = 0.31)
400.41/101.05	c INFO: running march_hi@0 for 485 with 4699 remaining (b = 0.28)
1560.45/391.68	c INFO: running march_hi@99 for 97 with 4409 remaining (b = 0.07)
1569.24/393.82	c INFO: running march_hi@0 for 97 with 4406 remaining (b = 0.06)
1947.20/488.52	c INFO: running march_hi@94 for 291 with 4312 remaining (b = 0.19)
1959.20/491.50	c INFO: running march_hi@492 for 97 with 4309 remaining (b = 0.07)
2342.36/587.55	c INFO: running march_hi@587 for 97 with 4213 remaining (b = 0.08)
2349.15/589.28	c INFO: running march_hi@100 for 97 with 4211 remaining (b = 0.07)
2360.31/592.02	c INFO: running march_hi@388 for 97 with 4208 remaining (b = 0.08)
2732.72/685.38	c INFO: running march_hi@586 for 97 with 4115 remaining (b = 0.08)
2742.66/687.88	c INFO: running march_hi@0 for 485 with 4112 remaining (b = 0.25)
2748.64/689.37	c INFO: running march_hi@483 for 97 with 4111 remaining (b = 0.06)
3117.01/781.67	c INFO: running march_hi@1075 for 291 with 4019 remaining (b = 0.17)
3132.99/785.60	c INFO: running march_hi@194 for 291 with 4015 remaining (b = 0.16)
3138.58/787.09	c INFO: running march_hi@0 for 388 with 4013 remaining (b = 0.20)
4274.88/1071.72	c INFO: running kcnfs-2006@0 for 582 with 3728 remaining (b = 0.23)
4293.19/1076.35	c INFO: running march_hi@291 for 97 with 3724 remaining (b = 0.07)
4685.53/1174.90	c INFO: running march_hi@389 for 97 with 3625 remaining (b = 0.04)
4695.46/1177.38	c INFO: running march_hi@289 for 194 with 3623 remaining (b = 0.09)
4716.21/1182.55	c INFO: running kcnfs-2006@0 for 97 with 3618 remaining (b = 0.04)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317590-1304270240/watcher-3317590-1304270240 -o /tmp/evaluation-result-3317590-1304270240/solver-3317590-1304270240 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317590-1304270240.cnf -seed 1693679091 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.28 8.18 7.77 6/251 13884
/proc/meminfo: memFree=28037624/32950928 swapFree=67111364/67111528
[pid=13884] ppid=13882 vsize=15380 CPUtime=0 cores=1,3,5,7
/proc/13884/stat : 13884 (python) R 13882 13884 12773 0 -1 4202496 810 0 0 0 0 0 0 0 25 0 1 0 617254738 15749120 695 33554432000 4194304 5683644 140736052847632 18446744073709551615 4433760 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/13884/statm: 3845 696 334 364 0 332 0

[startup+0.0199381 s]
/proc/loadavg: 8.28 8.18 7.77 6/251 13884
/proc/meminfo: memFree=28037624/32950928 swapFree=67111364/67111528
[pid=13884] ppid=13882 vsize=16240 CPUtime=0 cores=1,3,5,7
/proc/13884/stat : 13884 (python) R 13882 13884 12773 0 -1 4202496 1043 0 0 0 0 0 0 0 25 0 1 0 617254738 16629760 914 33554432000 4194304 5683644 140736052847632 18446744073709551615 5187280 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/13884/statm: 4060 914 381 364 0 547 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 16240

[startup+0.100936 s]
/proc/loadavg: 8.28 8.18 7.77 6/251 13884
/proc/meminfo: memFree=28037624/32950928 swapFree=67111364/67111528
[pid=13884] ppid=13882 vsize=85620 CPUtime=0.06 cores=1,3,5,7
/proc/13884/stat : 13884 (python) S 13882 13884 12773 0 -1 4202496 2631 0 0 0 5 1 0 0 17 0 1 0 617254738 87674880 2348 33554432000 4194304 5683644 140736052847632 18446744073709551615 256410467975 0 2147483391 16781312 2 18446744071699199166 0 0 17 5 0 0 0
/proc/13884/statm: 21405 2348 713 364 0 1622 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 85620

[startup+0.300896 s]
/proc/loadavg: 8.28 8.18 7.77 6/251 13884
/proc/meminfo: memFree=28037624/32950928 swapFree=67111364/67111528
[pid=13884] ppid=13882 vsize=135748 CPUtime=0.19 cores=1,3,5,7
/proc/13884/stat : 13884 (python) S 13882 13884 12773 0 -1 4202496 5962 933 1 0 14 5 0 0 17 0 1 0 617254738 139005952 4151 33554432000 4194304 5683644 140736052847632 18446744073709551615 256415387653 0 2147483391 16781312 2 18446744071699199166 0 0 17 5 0 0 0
/proc/13884/statm: 33937 4151 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 135748

[startup+0.700817 s]
/proc/loadavg: 8.28 8.18 7.77 6/251 13884
/proc/meminfo: memFree=28037624/32950928 swapFree=67111364/67111528
[pid=13884] ppid=13882 vsize=278444 CPUtime=0.42 cores=1,3,5,7
/proc/13884/stat : 13884 (python) R 13882 13884 12773 0 -1 4202496 9332 933 1 0 31 11 0 0 15 0 1 0 617254738 285126656 7003 33554432000 4194304 5683644 140736052847632 18446744073709551615 5003231 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/13884/statm: 69611 7003 1826 364 0 5535 0
Current children cumulated CPU time (s) 0.42
Current children cumulated vsize (KiB) 278444

[startup+1.50059 s]
/proc/loadavg: 8.28 8.18 7.77 9/264 13901
/proc/meminfo: memFree=27998304/32950928 swapFree=67111364/67111528
[pid=13884] ppid=13882 vsize=320292 CPUtime=0.61 cores=1,3,5,7
/proc/13884/stat : 13884 (python) S 13882 13884 12773 0 -1 4202496 12334 933 1 0 49 12 0 0 19 0 5 0 617254738 327979008 7270 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427218379 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/13884/statm: 80073 7270 1881 364 0 15982 0
[pid=13890] ppid=13884 vsize=278608 CPUtime=0 cores=1,3,5,7
/proc/13890/stat : 13890 (python) S 13884 13884 12773 0 -1 4202560 734 0 0 0 0 0 0 0 18 0 1 0 617254818 285294592 5576 33554432000 4194304 5683644 140736052847632 18446744073709551615 256415411727 0 0 16781312 514 15351833525460402175 0 0 17 1 0 0 0
/proc/13890/statm: 69652 5576 354 364 0 5570 0
[pid=13892] ppid=13890 vsize=1017800 CPUtime=0.69 cores=1,3,5,7
/proc/13892/stat : 13892 (gnovelty+2) R 13890 13892 13892 0 -1 4202496 444 0 0 0 69 0 0 0 21 0 1 0 617254818 1042227200 181 33554432000 134512640 135663704 4293700720 18446744073709551615 134545529 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/13892/statm: 254450 181 113 282 0 254166 0
[pid=13893] ppid=13884 vsize=289404 CPUtime=0 cores=1,3,5,7
/proc/13893/stat : 13893 (python) S 13884 13884 12773 0 -1 4202560 745 0 0 0 0 0 0 0 16 0 1 0 617254821 296349696 5709 33554432000 4194304 5683644 140736052847632 18446744073709551615 256415411782 0 0 16781312 514 3431971618769830151 0 0 17 5 0 0 0
/proc/13893/statm: 72351 5709 348 364 0 8266 0
[pid=13895] ppid=13893 vsize=2500 CPUtime=0.66 cores=1,3,5,7
/proc/13895/stat : 13895 (march_hi) R 13893 13895 13895 0 -1 4202496 698 0 0 0 66 0 0 0 24 0 1 0 617254821 2560000 433 33554432000 4194304 4955518 140735475406272 18446744073709551615 4227502 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/13895/statm: 625 433 104 186 0 438 0
[pid=13896] ppid=13884 vsize=299188 CPUtime=0 cores=1,3,5,7
/proc/13896/stat : 13896 (python) S 13884 13884 12773 0 -1 4202560 743 0 0 0 0 0 0 0 19 0 1 0 617254824 306368512 5610 33554432000 4194304 5683644 140736052847632 18446744073709551615 256415411782 0 0 16781312 514 4502148826072612863 0 0 17 1 0 0 0
/proc/13896/statm: 74797 5610 348 364 0 10709 0
[pid=13898] ppid=13896 vsize=1017800 CPUtime=0.63 cores=1,3,5,7
/proc/13898/stat : 13898 (gnovelty+2) R 13896 13898 13898 0 -1 4202496 444 0 0 0 63 0 0 0 22 0 1 0 617254824 1042227200 181 33554432000 134512640 135663704 4291398992 18446744073709551615 134524697 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/13898/statm: 254450 181 113 282 0 254166 0
[pid=13899] ppid=13884 vsize=310048 CPUtime=0 cores=1,3,5,7
/proc/13899/stat : 13899 (python) S 13884 13884 12773 0 -1 4202560 770 0 0 0 0 0 0 0 15 0 1 0 617254827 317489152 5733 33554432000 4194304 5683644 140736052847632 18446744073709551615 256415411782 0 0 16781312 514 18146190375528661256 0 0 17 5 0 0 0
/proc/13899/statm: 77512 5733 348 364 0 13421 0
[pid=13901] ppid=13899 vsize=2500 CPUtime=0.59 cores=1,3,5,7
/proc/13901/stat : 13901 (march_hi) R 13899 13901 13901 0 -1 4202496 686 0 0 0 59 0 0 0 22 0 1 0 617254827 2560000 425 33554432000 4194304 4955518 140734879974256 18446744073709551615 4209717 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/13901/statm: 625 425 104 186 0 438 0
[pid=13884/tid=13891] ppid=13882 vsize=320292 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13891/stat : 13891 (python) S 13882 13884 12773 0 -1 4202560 25 933 0 0 0 0 0 0 18 0 5 0 617254818 327979008 7270 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 7 0 0 0
[pid=13884/tid=13894] ppid=13882 vsize=320292 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13894/stat : 13894 (python) S 13882 13884 12773 0 -1 4202560 21 933 0 0 0 0 0 0 20 0 5 0 617254821 327979008 7270 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 5 0 0 0
[pid=13884/tid=13897] ppid=13882 vsize=320292 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13897/stat : 13897 (python) S 13882 13884 12773 0 -1 4202560 20 933 0 0 0 0 0 0 20 0 5 0 617254824 327979008 7270 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 1 0 0 0
[pid=13884/tid=13900] ppid=13882 vsize=320292 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13900/stat : 13900 (python) S 13882 13884 12773 0 -1 4202560 25 933 0 0 0 0 0 0 20 0 5 0 617254827 327979008 7270 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3.18
Current children cumulated vsize (KiB) 3538140

[startup+3.10129 s]
/proc/loadavg: 8.26 8.18 7.77 9/264 13901
/proc/meminfo: memFree=27996816/32950928 swapFree=67111364/67111528
[pid=13884] ppid=13882 vsize=320292 CPUtime=0.61 cores=1,3,5,7
/proc/13884/stat : 13884 (python) S 13882 13884 12773 0 -1 4202496 12334 933 1 0 49 12 0 0 19 0 5 0 617254738 327979008 7270 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427218379 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/13884/statm: 80073 7270 1881 364 0 15982 0
[pid=13890] ppid=13884 vsize=278608 CPUtime=0 cores=1,3,5,7
/proc/13890/stat : 13890 (python) S 13884 13884 12773 0 -1 4202560 734 0 0 0 0 0 0 0 18 0 1 0 617254818 285294592 5576 33554432000 4194304 5683644 140736052847632 18446744073709551615 256415411727 0 0 16781312 514 15351833525460369672 0 0 17 1 0 0 0
/proc/13890/statm: 69652 5576 354 364 0 5570 0
[pid=13892] ppid=13890 vsize=1017800 CPUtime=2.29 cores=1,3,5,7
/proc/13892/stat : 13892 (gnovelty+2) R 13890 13892 13892 0 -1 4202496 444 0 0 0 229 0 0 0 25 0 1 0 617254818 1042227200 181 33554432000 134512640 135663704 4293700720 18446744073709551615 134525278 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/13892/statm: 254450 181 113 282 0 254166 0
[pid=13893] ppid=13884 vsize=289404 CPUtime=0 cores=1,3,5,7
/proc/13893/stat : 13893 (python) S 13884 13884 12773 0 -1 4202560 752 0 0 0 0 0 0 0 15 0 1 0 617254821 296349696 5709 33554432000 4194304 5683644 140736052847632 18446744073709551615 256415411782 0 0 16781312 514 3431971618769830151 0 0 17 5 0 0 0
/proc/13893/statm: 72351 5709 348 364 0 8266 0
[pid=13895] ppid=13893 vsize=2820 CPUtime=2.26 cores=1,3,5,7
/proc/13895/stat : 13895 (march_hi) R 13893 13895 13895 0 -1 4202496 755 0 0 0 226 0 0 0 25 0 1 0 617254821 2887680 490 33554432000 4194304 4955518 140735475406272 18446744073709551615 4227502 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/13895/statm: 705 490 104 186 0 518 0
[pid=13896] ppid=13884 vsize=299188 CPUtime=0 cores=1,3,5,7
/proc/13896/stat : 13896 (python) S 13884 13884 12773 0 -1 4202560 743 0 0 0 0 0 0 0 18 0 1 0 617254824 306368512 5610 33554432000 4194304 5683644 140736052847632 18446744073709551615 256415411782 0 0 16781312 514 4502148826072580359 0 0 17 1 0 0 0
/proc/13896/statm: 74797 5610 348 364 0 10709 0
[pid=13898] ppid=13896 vsize=1017800 CPUtime=2.23 cores=1,3,5,7
/proc/13898/stat : 13898 (gnovelty+2) R 13896 13898 13898 0 -1 4202496 444 0 0 0 223 0 0 0 25 0 1 0 617254824 1042227200 181 33554432000 134512640 135663704 4291398992 18446744073709551615 134546746 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/13898/statm: 254450 181 113 282 0 254166 0
[pid=13899] ppid=13884 vsize=310048 CPUtime=0 cores=1,3,5,7
/proc/13899/stat : 13899 (python) S 13884 13884 12773 0 -1 4202560 775 0 0 0 0 0 0 0 15 0 1 0 617254827 317489152 5733 33554432000 4194304 5683644 140736052847632 18446744073709551615 256415411782 0 0 16781312 514 18146190375528661256 0 0 17 5 0 0 0
/proc/13899/statm: 77512 5733 348 364 0 13421 0
[pid=13901] ppid=13899 vsize=2820 CPUtime=2.19 cores=1,3,5,7
/proc/13901/stat : 13901 (march_hi) R 13899 13901 13901 0 -1 4202496 747 0 0 0 219 0 0 0 25 0 1 0 617254827 2887680 486 33554432000 4194304 4955518 140734879974256 18446744073709551615 4209692 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/13901/statm: 705 486 104 186 0 518 0
[pid=13884/tid=13891] ppid=13882 vsize=320292 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13891/stat : 13891 (python) S 13882 13884 12773 0 -1 4202560 25 933 0 0 0 0 0 0 18 0 5 0 617254818 327979008 7270 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 7 0 0 0
[pid=13884/tid=13894] ppid=13882 vsize=320292 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13894/stat : 13894 (python) S 13882 13884 12773 0 -1 4202560 21 933 0 0 0 0 0 0 20 0 5 0 617254821 327979008 7270 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 5 0 0 0
[pid=13884/tid=13897] ppid=13882 vsize=320292 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13897/stat : 13897 (python) S 13882 13884 12773 0 -1 4202560 20 933 0 0 0 0 0 0 20 0 5 0 617254824 327979008 7270 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 1 0 0 0
[pid=13884/tid=13900] ppid=13882 vsize=320292 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13900/stat : 13900 (python) S 13882 13884 12773 0 -1 4202560 25 933 0 0 0 0 0 0 20 0 5 0 617254827 327979008 7270 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 9.58
Current children cumulated vsize (KiB) 3538780

[startup+6.30068 s]
/proc/loadavg: 8.26 8.18 7.77 9/264 13901
/proc/meminfo: memFree=27995940/32950928 swapFree=67111364/67111528
[pid=13884] ppid=13882 vsize=320292 CPUtime=0.61 cores=1,3,5,7
/proc/13884/stat : 13884 (python) S 13882 13884 12773 0 -1 4202496 12334 933 1 0 49 12 0 0 19 0 5 0 617254738 327979008 7270 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427218379 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/13884/statm: 80073 7270 1881 364 0 15982 0
[pid=13890] ppid=13884 vsize=278608 CPUtime=0 cores=1,3,5,7
/proc/13890/stat : 13890 (python) S 13884 13884 12773 0 -1 4202560 889 0 0 0 0 0 0 0 18 0 1 0 617254818 285294592 5598 33554432000 4194304 5683644 140736052847632 18446744073709551615 256415411727 0 0 16781312 514 15351833525460369672 0 0 17 1 0 0 0
/proc/13890/statm: 69652 5598 376 364 0 5570 0
[pid=13892] ppid=13890 vsize=1017800 CPUtime=5.47 cores=1,3,5,7
/proc/13892/stat : 13892 (gnovelty+2) R 13890 13892 13892 0 -1 4202496 444 0 0 0 547 0 0 0 25 0 1 0 617254818 1042227200 181 33554432000 134512640 135663704 4293700720 18446744073709551615 134525361 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/13892/statm: 254450 181 113 282 0 254166 0
[pid=13893] ppid=13884 vsize=289404 CPUtime=0 cores=1,3,5,7
/proc/13893/stat : 13893 (python) S 13884 13884 12773 0 -1 4202560 897 0 0 0 0 0 0 0 15 0 1 0 617254821 296349696 5732 33554432000 4194304 5683644 140736052847632 18446744073709551615 256415411782 0 0 16781312 514 3218960482341716231 0 0 17 5 0 0 0
/proc/13893/statm: 72351 5732 371 364 0 8266 0
[pid=13895] ppid=13893 vsize=3612 CPUtime=5.44 cores=1,3,5,7
/proc/13895/stat : 13895 (march_hi) R 13893 13895 13895 0 -1 4202496 891 0 0 0 544 0 0 0 25 0 1 0 617254821 3698688 626 33554432000 4194304 4955518 140735475406272 18446744073709551615 4224682 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/13895/statm: 903 626 104 186 0 716 0
[pid=13896] ppid=13884 vsize=299188 CPUtime=0 cores=1,3,5,7
/proc/13896/stat : 13896 (python) S 13884 13884 12773 0 -1 4202560 886 0 0 0 0 0 0 0 18 0 1 0 617254824 306368512 5633 33554432000 4194304 5683644 140736052847632 18446744073709551615 256415411782 0 0 16781312 514 4502148826072580359 0 0 17 1 0 0 0
/proc/13896/statm: 74797 5633 371 364 0 10709 0
[pid=13898] ppid=13896 vsize=1017800 CPUtime=5.43 cores=1,3,5,7
/proc/13898/stat : 13898 (gnovelty+2) R 13896 13898 13898 0 -1 4202496 444 0 0 0 543 0 0 0 25 0 1 0 617254824 1042227200 181 33554432000 134512640 135663704 4291398992 18446744073709551615 134545657 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/13898/statm: 254450 181 113 282 0 254166 0
[pid=13899] ppid=13884 vsize=310048 CPUtime=0 cores=1,3,5,7
/proc/13899/stat : 13899 (python) S 13884 13884 12773 0 -1 4202560 935 0 0 0 0 0 0 0 15 0 1 0 617254827 317489152 5755 33554432000 4194304 5683644 140736052847632 18446744073709551615 256415411782 0 0 16781312 514 3218952785760321800 0 0 17 5 0 0 0
/proc/13899/statm: 77512 5755 370 364 0 13421 0
[pid=13901] ppid=13899 vsize=3612 CPUtime=5.36 cores=1,3,5,7
/proc/13901/stat : 13901 (march_hi) R 13899 13901 13901 0 -1 4202496 883 0 0 0 536 0 0 0 25 0 1 0 617254827 3698688 622 33554432000 4194304 4955518 140734879974256 18446744073709551615 4245462 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/13901/statm: 903 622 104 186 0 716 0
[pid=13884/tid=13891] ppid=13882 vsize=320292 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13891/stat : 13891 (python) S 13882 13884 12773 0 -1 4202560 25 933 0 0 0 0 0 0 18 0 5 0 617254818 327979008 7270 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 7 0 0 0
[pid=13884/tid=13894] ppid=13882 vsize=320292 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13894/stat : 13894 (python) S 13882 13884 12773 0 -1 4202560 21 933 0 0 0 0 0 0 20 0 5 0 617254821 327979008 7270 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 5 0 0 0
[pid=13884/tid=13897] ppid=13882 vsize=320292 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13897/stat : 13897 (python) S 13882 13884 12773 0 -1 4202560 20 933 0 0 0 0 0 0 20 0 5 0 617254824 327979008 7270 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 1 0 0 0
[pid=13884/tid=13900] ppid=13882 vsize=320292 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13900/stat : 13900 (python) S 13882 13884 12773 0 -1 4202560 25 933 0 0 0 0 0 0 20 0 5 0 617254827 327979008 7270 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 22.31
Current children cumulated vsize (KiB) 3540364

[startup+12.7005 s]
/proc/loadavg: 8.22 8.17 7.77 9/280 13918
/proc/meminfo: memFree=27966100/32950928 swapFree=67111364/67111528
[pid=13884] ppid=13882 vsize=320292 CPUtime=0.61 cores=1,3,5,7
/proc/13884/stat : 13884 (python) S 13882 13884 12773 0 -1 4202496 12334 933 1 0 49 12 0 0 19 0 5 0 617254738 327979008 7270 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427218379 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/13884/statm: 80073 7270 1881 364 0 15982 0
[pid=13890] ppid=13884 vsize=278608 CPUtime=0.01 cores=1,3,5,7
/proc/13890/stat : 13890 (python) S 13884 13884 12773 0 -1 4202560 893 0 0 0 1 0 0 0 18 0 1 0 617254818 285294592 5599 33554432000 4194304 5683644 140736052847632 18446744073709551615 256415411727 0 0 16781312 514 12961025476037411080 0 0 17 1 0 0 0
/proc/13890/statm: 69652 5599 377 364 0 5570 0
[pid=13892] ppid=13890 vsize=1017800 CPUtime=11.85 cores=1,3,5,7
/proc/13892/stat : 13892 (gnovelty+2) R 13890 13892 13892 0 -1 4202496 444 0 0 0 1185 0 0 0 25 0 1 0 617254818 1042227200 181 33554432000 134512640 135663704 4293700720 18446744073709551615 134545786 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/13892/statm: 254450 181 113 282 0 254166 0
[pid=13893] ppid=13884 vsize=289404 CPUtime=0.02 cores=1,3,5,7

################
# More data... #
################

[pid=13926/tid=13985] ppid=13884 vsize=330560 CPUtime=0 cores=1,3,5,7
/proc/13926/task/13985/stat : 13985 (python) S 13884 13884 12773 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 617313938 338493440 5796 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 514 18446604454223080512 0 0 -1 7 0 0 0
[pid=13991] ppid=13884 vsize=341000 CPUtime=0.98 cores=1,3,5,7
/proc/13991/stat : 13991 (python) S 13884 13884 12773 0 -1 4202560 1034 0 0 0 48 50 0 0 15 0 1 0 617323526 349184000 5819 33554432000 4194304 5683644 140736052847632 18446744073709551615 256415411782 0 0 16781312 514 4667919445413298440 0 0 17 5 0 0 0
/proc/13991/statm: 85250 5819 371 364 0 21150 0
[pid=13993] ppid=13991 vsize=78032 CPUtime=488.91 cores=1,3,5,7
/proc/13993/stat : 13993 (march_hi) R 13991 13993 13993 0 -1 4202496 15742 0 0 0 48889 2 0 0 25 0 1 0 617323527 79904768 15479 33554432000 4194304 4955518 140736289574800 18446744073709551615 4307184 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/13993/statm: 19508 15479 104 186 0 19321 0
[pid=13997] ppid=13884 vsize=351256 CPUtime=0.77 cores=1,3,5,7
/proc/13997/stat : 13997 (python) S 13884 13884 12773 0 -1 4202560 1204 0 0 0 39 38 0 0 15 0 2 0 617333447 359686144 5875 33554432000 4194304 5683644 140736052847632 18446744073709551615 256415411782 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/13997/statm: 87814 5875 406 364 0 23711 0
[pid=13999] ppid=13997 vsize=58388 CPUtime=393.4 cores=1,3,5,7
/proc/13999/stat : 13999 (march_hi) R 13997 13999 13999 0 -1 4202496 13038 0 0 0 39338 2 0 0 25 0 1 0 617333448 59789312 12772 33554432000 4194304 4955518 140737283038288 18446744073709551615 4238478 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/13999/statm: 14597 12772 104 186 0 14410 0
[pid=13997/tid=14037] ppid=13884 vsize=351256 CPUtime=0 cores=1,3,5,7
/proc/13997/task/14037/stat : 14037 (python) S 13884 13884 12773 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 617372474 359686144 5875 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 514 18446604455026473792 0 0 -1 3 0 0 0
[pid=14031] ppid=13884 vsize=361512 CPUtime=0.19 cores=1,3,5,7
/proc/14031/stat : 14031 (python) S 13884 13884 12773 0 -1 4202560 989 0 0 0 9 10 0 0 18 0 1 0 617361910 370188288 5843 33554432000 4194304 5683644 140736052847632 18446744073709551615 256415411782 0 0 16781312 514 12961025476037411080 0 0 17 5 0 0 0
/proc/14031/statm: 90378 5843 371 364 0 26272 0
[pid=14033] ppid=14031 vsize=182448 CPUtime=110.35 cores=1,3,5,7
/proc/14033/stat : 14033 (kcnfs-2006) R 14031 14033 14033 0 -1 4202496 616 0 0 0 11035 0 0 0 25 0 1 0 617361911 186826752 350 33554432000 134512640 135163436 4293077824 18446744073709551615 134572201 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/14033/statm: 45612 350 77 159 0 45452 0
[pid=13884/tid=13891] ppid=13882 vsize=371496 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13891/stat : 13891 (python) S 13882 13884 12773 0 -1 4202560 25 933 0 0 0 0 0 0 18 0 10 0 617254818 380411904 7282 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 7 0 0 0
[pid=13884/tid=13894] ppid=13882 vsize=371496 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13894/stat : 13894 (python) S 13882 13884 12773 0 -1 4202560 36 933 0 0 0 0 0 0 18 0 10 0 617254821 380411904 7282 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 5 0 0 0
[pid=13884/tid=13897] ppid=13882 vsize=371496 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13897/stat : 13897 (python) S 13882 13884 12773 0 -1 4202560 20 933 0 0 0 0 0 0 20 0 10 0 617254824 380411904 7282 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 1 0 0 0
[pid=13884/tid=13900] ppid=13882 vsize=371496 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13900/stat : 13900 (python) S 13882 13884 12773 0 -1 4202560 37 933 0 0 0 0 0 0 18 0 10 0 617254827 380411904 7282 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 5 0 0 0
[pid=13884/tid=13923] ppid=13882 vsize=371496 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13923/stat : 13923 (python) S 13882 13884 12773 0 -1 4202560 34 933 0 0 0 0 0 0 18 0 10 0 617264837 380411904 7282 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 5 0 0 0
[pid=13884/tid=13927] ppid=13882 vsize=371496 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13927/stat : 13927 (python) S 13882 13884 12773 0 -1 4202560 30 933 0 0 0 0 0 0 18 0 10 0 617264843 380411904 7282 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 5 0 0 0
[pid=13884/tid=13992] ppid=13882 vsize=371496 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13992/stat : 13992 (python) S 13882 13884 12773 0 -1 4202560 30 933 0 0 0 0 0 0 20 0 10 0 617323526 380411904 7282 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 5 0 0 0
[pid=13884/tid=13998] ppid=13882 vsize=371496 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13998/stat : 13998 (python) S 13882 13884 12773 0 -1 4202560 33 933 0 0 0 0 0 0 18 0 10 0 617333448 380411904 7282 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 5 0 0 0
[pid=13884/tid=14032] ppid=13882 vsize=371496 CPUtime=0 cores=1,3,5,7
/proc/13884/task/14032/stat : 14032 (python) S 13882 13884 12773 0 -1 4202560 27 933 0 0 0 0 0 0 20 0 10 0 617361911 380411904 7282 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4715.38
Current children cumulated vsize (KiB) 6157776



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.6 s]
/proc/loadavg: 4.03 5.07 6.55 5/287 14041
/proc/meminfo: memFree=27425400/32950928 swapFree=67111364/67111528
[pid=13884] ppid=13882 vsize=382128 CPUtime=1.19 cores=1,3,5,7
/proc/13884/stat : 13884 (python) S 13882 13884 12773 0 -1 4202496 18019 933 1 0 106 13 0 0 19 0 11 0 617254738 391299072 7382 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427218379 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/13884/statm: 95532 7382 1894 364 0 31423 0
[pid=13890] ppid=13884 vsize=288852 CPUtime=0.16 cores=1,3,5,7
/proc/13890/stat : 13890 (python) S 13884 13884 12773 0 -1 4202560 1084 0 0 0 9 7 0 0 19 0 2 0 617254818 295784448 5637 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427218379 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/13890/statm: 72213 5637 410 364 0 8131 0
[pid=13892] ppid=13890 vsize=1017800 CPUtime=99.82 cores=1,3,5,7
/proc/13892/stat : 13892 (gnovelty+2) T 13890 13892 13892 0 -1 4202496 444 0 0 0 9982 0 0 0 25 0 1 0 617254818 1042227200 181 33554432000 134512640 135663704 4293700720 18446744073709551615 134544212 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/13892/statm: 254450 181 113 282 0 254166 0
[pid=13890/tid=13921] ppid=13884 vsize=288852 CPUtime=0 cores=1,3,5,7
/proc/13890/task/13921/stat : 13921 (python) S 13884 13884 12773 0 -1 4202560 51 0 0 0 0 0 0 0 19 0 2 0 617264834 295784448 5637 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 514 18446604469371972416 0 0 -1 1 0 0 0
[pid=13893] ppid=13884 vsize=289404 CPUtime=2.41 cores=1,3,5,7
/proc/13893/stat : 13893 (python) S 13884 13884 12773 0 -1 4202560 1160 0 0 0 126 115 0 0 15 0 2 0 617254821 296349696 5762 33554432000 4194304 5683644 140736052847632 18446744073709551615 256415411782 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/13893/statm: 72351 5762 400 364 0 8266 0
[pid=13895] ppid=13893 vsize=168072 CPUtime=1196.37 cores=1,3,5,7
/proc/13895/stat : 13895 (march_hi) R 13893 13895 13895 0 -1 4202496 34973 0 0 0 119629 8 0 0 25 0 1 0 617254821 172105728 34708 33554432000 4194304 4955518 140735475406272 18446744073709551615 4224682 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/13895/statm: 42018 34708 104 186 0 41831 0
[pid=13893/tid=13967] ppid=13884 vsize=289404 CPUtime=0 cores=1,3,5,7
/proc/13893/task/13967/stat : 13967 (python) S 13884 13884 12773 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 617294117 296349696 5762 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 514 18446604470842035072 0 0 -1 5 0 0 0
[pid=13896] ppid=13884 vsize=299188 CPUtime=0.16 cores=1,3,5,7
/proc/13896/stat : 13896 (python) S 13884 13884 12773 0 -1 4202560 1078 0 0 0 8 8 0 0 19 0 2 0 617254824 306368512 5663 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427218379 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/13896/statm: 74797 5663 400 364 0 10709 0
[pid=13898] ppid=13896 vsize=1017800 CPUtime=100.15 cores=1,3,5,7
/proc/13898/stat : 13898 (gnovelty+2) T 13896 13898 13898 0 -1 4202496 444 0 0 0 10015 0 0 0 25 0 1 0 617254824 1042227200 181 33554432000 134512640 135663704 4291398992 18446744073709551615 134545653 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/13898/statm: 254450 181 113 282 0 254166 0
[pid=13896/tid=13925] ppid=13884 vsize=299188 CPUtime=0 cores=1,3,5,7
/proc/13896/task/13925/stat : 13925 (python) S 13884 13884 12773 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 617264840 306368512 5663 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 514 18446604471140800640 0 0 -1 1 0 0 0
[pid=13899] ppid=13884 vsize=310048 CPUtime=1.98 cores=1,3,5,7
/proc/13899/stat : 13899 (python) S 13884 13884 12773 0 -1 4202560 1178 0 0 0 101 97 0 0 15 0 2 0 617254827 317489152 5786 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427218379 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/13899/statm: 77512 5786 400 364 0 13421 0
[pid=13901] ppid=13899 vsize=123872 CPUtime=974.29 cores=1,3,5,7
/proc/13901/stat : 13901 (march_hi) T 13899 13901 13901 0 -1 4202496 29143 0 0 0 97422 7 0 0 25 0 1 0 617254827 126844928 28882 33554432000 4194304 4955518 140734879974256 18446744073709551615 4261360 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/13901/statm: 30968 28882 104 186 0 30781 0
[pid=13899/tid=13966] ppid=13884 vsize=310048 CPUtime=0 cores=1,3,5,7
/proc/13899/task/13966/stat : 13966 (python) S 13884 13884 12773 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 617293903 317489152 5786 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 514 18446604470842033280 0 0 -1 5 0 0 0
[pid=13922] ppid=13884 vsize=320304 CPUtime=1.44 cores=1,3,5,7
/proc/13922/stat : 13922 (python) S 13884 13884 12773 0 -1 4202560 1173 0 0 0 77 67 0 0 15 0 2 0 617264837 327991296 5790 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427218379 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/13922/statm: 80076 5790 400 364 0 15982 0
[pid=13924] ppid=13922 vsize=123872 CPUtime=682.25 cores=1,3,5,7
/proc/13924/stat : 13924 (march_hi) T 13922 13924 13924 0 -1 4202496 21232 0 0 0 68220 5 0 0 25 0 1 0 617264837 126844928 20967 33554432000 4194304 4955518 140736427282208 18446744073709551615 4227502 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/13924/statm: 30968 20967 104 186 0 30781 0
[pid=13922/tid=13984] ppid=13884 vsize=320304 CPUtime=0 cores=1,3,5,7
/proc/13922/task/13984/stat : 13984 (python) S 13884 13884 12773 0 -1 4202560 45 0 0 0 0 0 0 0 15 0 2 0 617313664 327991296 5790 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 514 18446604467442902848 0 0 -1 3 0 0 0
[pid=13926] ppid=13884 vsize=330560 CPUtime=1.43 cores=1,3,5,7
/proc/13926/stat : 13926 (python) S 13884 13884 12773 0 -1 4202560 1172 0 0 0 74 69 0 0 16 0 2 0 617264843 338493440 5796 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427218379 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/13926/statm: 82640 5796 400 364 0 18543 0
[pid=13928] ppid=13926 vsize=123872 CPUtime=680.47 cores=1,3,5,7
/proc/13928/stat : 13928 (march_hi) T 13926 13928 13928 0 -1 4202496 21116 0 0 0 68041 6 0 0 25 0 1 0 617264843 126844928 20851 33554432000 4194304 4955518 140736459796112 18446744073709551615 4238968 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/13928/statm: 30968 20851 104 186 0 30781 0
[pid=13926/tid=13985] ppid=13884 vsize=330560 CPUtime=0 cores=1,3,5,7
/proc/13926/task/13985/stat : 13985 (python) S 13884 13884 12773 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 617313938 338493440 5796 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 514 18446604454223080512 0 0 -1 7 0 0 0
[pid=13991] ppid=13884 vsize=341000 CPUtime=0.98 cores=1,3,5,7
/proc/13991/stat : 13991 (python) S 13884 13884 12773 0 -1 4202560 1189 0 0 0 48 50 0 0 16 0 2 0 617323526 349184000 5849 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427218379 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/13991/statm: 85250 5849 400 364 0 21150 0
[pid=13993] ppid=13991 vsize=78032 CPUtime=489.12 cores=1,3,5,7
/proc/13993/stat : 13993 (march_hi) T 13991 13993 13993 0 -1 4202496 15747 0 0 0 48910 2 0 0 25 0 1 0 617323527 79904768 15484 33554432000 4194304 4955518 140736289574800 18446744073709551615 4238449 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/13993/statm: 19508 15484 104 186 0 19321 0
[pid=13991/tid=14038] ppid=13884 vsize=341000 CPUtime=0 cores=1,3,5,7
/proc/13991/task/14038/stat : 14038 (python) S 13884 13884 12773 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 617372991 349184000 5849 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 514 18446604471140798848 0 0 -1 5 0 0 0
[pid=13997] ppid=13884 vsize=351256 CPUtime=0.81 cores=1,3,5,7
/proc/13997/stat : 13997 (python) S 13884 13884 12773 0 -1 4202560 1207 0 0 0 41 40 0 0 15 0 2 0 617333447 359686144 5875 33554432000 4194304 5683644 140736052847632 18446744073709551615 256415411782 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/13997/statm: 87814 5875 406 364 0 23711 0
[pid=13999] ppid=13997 vsize=78032 CPUtime=414.33 cores=1,3,5,7
/proc/13999/stat : 13999 (march_hi) R 13997 13999 13999 0 -1 4202496 13649 0 0 0 41431 2 0 0 25 0 1 0 617333448 79904768 13383 33554432000 4194304 4955518 140737283038288 18446744073709551615 4209742 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/13999/statm: 19508 13383 104 186 0 19321 0
[pid=13997/tid=14037] ppid=13884 vsize=351256 CPUtime=0 cores=1,3,5,7
/proc/13997/task/14037/stat : 14037 (python) S 13884 13884 12773 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 617372474 359686144 5875 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 514 18446604455026473792 0 0 -1 3 0 0 0
[pid=14031] ppid=13884 vsize=361512 CPUtime=0.24 cores=1,3,5,7
/proc/14031/stat : 14031 (python) S 13884 13884 12773 0 -1 4202560 996 0 0 0 12 12 0 0 18 0 1 0 617361910 370188288 5843 33554432000 4194304 5683644 140736052847632 18446744073709551615 256415411782 0 0 16781312 514 12961025476037411080 0 0 17 5 0 0 0
/proc/14031/statm: 90378 5843 371 364 0 26272 0
[pid=14033] ppid=14031 vsize=182448 CPUtime=131.62 cores=1,3,5,7
/proc/14033/stat : 14033 (kcnfs-2006) R 14031 14033 14033 0 -1 4202496 616 0 0 0 13162 0 0 0 25 0 1 0 617361911 186826752 350 33554432000 134512640 135163436 4293077824 18446744073709551615 134675210 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/14033/statm: 45612 350 77 159 0 45452 0
[pid=14039] ppid=13884 vsize=371884 CPUtime=0.02 cores=1,3,5,7
/proc/14039/stat : 14039 (python) S 13884 13884 12773 0 -1 4202560 962 0 0 0 1 1 0 0 18 0 1 0 617372994 380809216 5855 33554432000 4194304 5683644 140736052847632 18446744073709551615 256415411782 0 0 16781312 514 12961025476037411079 0 0 17 5 0 0 0
/proc/14039/statm: 92971 5855 371 364 0 28862 0
[pid=14041] ppid=14039 vsize=182452 CPUtime=20.97 cores=1,3,5,7
/proc/14041/stat : 14041 (kcnfs-2006) R 14039 14041 14041 0 -1 4202496 615 0 0 0 2097 0 0 0 25 0 1 0 617372994 186830848 350 33554432000 134512640 135163436 4289592000 18446744073709551615 134564103 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/14041/statm: 45613 350 77 159 0 45453 0
[pid=13884/tid=13891] ppid=13882 vsize=382128 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13891/stat : 13891 (python) S 13882 13884 12773 0 -1 4202560 25 933 0 0 0 0 0 0 18 0 11 0 617254818 391299072 7382 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 7 0 0 0
[pid=13884/tid=13894] ppid=13882 vsize=382128 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13894/stat : 13894 (python) S 13882 13884 12773 0 -1 4202560 36 933 0 0 0 0 0 0 18 0 11 0 617254821 391299072 7382 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 5 0 0 0
[pid=13884/tid=13897] ppid=13882 vsize=382128 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13897/stat : 13897 (python) S 13882 13884 12773 0 -1 4202560 20 933 0 0 0 0 0 0 20 0 11 0 617254824 391299072 7382 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 1 0 0 0
[pid=13884/tid=13900] ppid=13882 vsize=382128 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13900/stat : 13900 (python) S 13882 13884 12773 0 -1 4202560 37 933 0 0 0 0 0 0 18 0 11 0 617254827 391299072 7382 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 5 0 0 0
[pid=13884/tid=13923] ppid=13882 vsize=382128 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13923/stat : 13923 (python) S 13882 13884 12773 0 -1 4202560 34 933 0 0 0 0 0 0 18 0 11 0 617264837 391299072 7382 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 5 0 0 0
[pid=13884/tid=13927] ppid=13882 vsize=382128 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13927/stat : 13927 (python) S 13882 13884 12773 0 -1 4202560 30 933 0 0 0 0 0 0 18 0 11 0 617264843 391299072 7382 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 5 0 0 0
[pid=13884/tid=13992] ppid=13882 vsize=382128 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13992/stat : 13992 (python) S 13882 13884 12773 0 -1 4202560 30 933 0 0 0 0 0 0 20 0 11 0 617323526 391299072 7382 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 5 0 0 0
[pid=13884/tid=13998] ppid=13882 vsize=382128 CPUtime=0 cores=1,3,5,7
/proc/13884/task/13998/stat : 13998 (python) S 13882 13884 12773 0 -1 4202560 33 933 0 0 0 0 0 0 18 0 11 0 617333448 391299072 7382 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 5 0 0 0
[pid=13884/tid=14032] ppid=13882 vsize=382128 CPUtime=0 cores=1,3,5,7
/proc/13884/task/14032/stat : 14032 (python) S 13882 13884 12773 0 -1 4202560 27 933 0 0 0 0 0 0 20 0 11 0 617361911 391299072 7382 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 5 0 0 0
[pid=13884/tid=14040] ppid=13882 vsize=382128 CPUtime=0 cores=1,3,5,7
/proc/13884/task/14040/stat : 14040 (python) S 13882 13884 12773 0 -1 4202560 28 933 0 0 0 0 0 0 20 0 11 0 617372994 391299072 7382 33554432000 4194304 5683644 140736052847632 18446744073709551615 256427215281 0 0 16781312 2 18446604459192072384 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.21
Current children cumulated vsize (KiB) 6742388

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 13884 and gives
#  childrusage.ru_utime.tv_sec=1
#  childrusage.ru_utime.tv_usec=82835
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=140978
# CPU time returned by wait4() is 1.22381
# while last known CPU time is 4800.21
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1203.67
CPU time (s): 4800.21
CPU user time (s): 4795.12
CPU system time (s): 5.09
CPU usage (%): 398.798
Max. virtual memory (cumulated for all children) (KiB): 6742388

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1.08283
system time used= 0.140978
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 19447
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2941
involuntary context switches= 40

runsolver used 3.38248 second user time and 8.04078 second system time

The end

Launcher Data 

Begin job on node115 at 2011-05-01 19:17:21
IDJOB=3317590
IDBENCH=83055
IDSOLVER=1705
FILE ID=node115/3317590-1304270240
RUNJOBID= node115-1304267285-12792
PBS_JOBID= 13173964
Free space on /tmp= 73100 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v550-c2343-S1693168158-005.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317590-1304270240/watcher-3317590-1304270240 -o /tmp/evaluation-result-3317590-1304270240/solver-3317590-1304270240 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317590-1304270240.cnf -seed 1693679091 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= c2b69177f59f5d1d2093247dc6de56f3
RANDOM SEED=1693679091

node115.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.62
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      28037780 kB
Buffers:        879212 kB
Cached:        2130176 kB
SwapCached:          0 kB
Active:        1846760 kB
Inactive:      2611956 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      28037780 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:           31900 kB
Writeback:           0 kB
AnonPages:     1448952 kB
Mapped:          25120 kB
Slab:           373556 kB
PageTables:      18036 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 30818208 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73092 MiB
End job on node115 at 2011-05-01 19:37:26