Trace number 3385478

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (TO) 2448.21 5100.04

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v700-c2982-S900569154-026.SATISFIABLE.cnf
MD5SUM90af97b0aabddb9e07bb7ada8a5dcbc1
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.024996
Satisfiable
(Un)Satisfiability was proved
Number of variables700
Number of clauses2982
Sum of the clauses size8946
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32982
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.35/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.35/0.67	c INFO: solving HOME/instance-3385478-1308045232.cnf
0.35/0.67	c INFO: computing task features
0.35/0.68	c INFO: computed variable-clause graph statistics
0.35/0.68	c INFO: computed clause balance statistics
0.42/0.74	c INFO: computed variable balance statistics
0.42/0.78	c INFO: computed small-clause counts
0.42/0.78	c INFO: computed Horn-clause counts
0.42/0.78	c INFO: computed variable graph statistics
0.42/0.78	c INFO: collected features for HOME/instance-3385478-1308045232.cnf in 0.11 s
0.52/0.81	c INFO: running gnovelty+2@0 for 97 with 4999 remaining (b = 0.58)
0.52/0.84	c INFO: running march_hi@0 for 97 with 4999 remaining (b = 0.07)
0.52/0.87	c INFO: running gnovelty+2@0 for 97 with 4999 remaining (b = 0.08)
0.63/0.91	c INFO: running march_hi@0 for 388 with 4999 remaining (b = 0.23)
0.63/0.94	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.25)
0.63/0.98	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.22)
1.08/1.01	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.20)
1.08/1.04	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.17)
1.08/1.05	Traceback (most recent call last):
1.08/1.05	  File "HOME/solve", line 33, in <module>
1.08/1.05	    plac.call(main)
1.08/1.05	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
1.08/1.05	    cmd, result = parser_from(obj).consume(arglist)
1.08/1.05	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
1.08/1.05	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
1.08/1.05	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
1.08/1.05	    (_, answer, _) = solver(input_path, budget / cores, cores)
1.08/1.05	  File "HOME/borg/build/lib/borg/bilevel.py", line 226, in __call__
1.08/1.05	    process.die()
1.08/1.05	  File "HOME/borg/build/lib/borg/solvers.py", line 185, in die
1.08/1.05	    os.kill(self._process.pid, signal.SIGUSR1)
1.08/1.05	OSError: [Errno 3] No such process

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3385478-1308045232/watcher-3385478-1308045232 -o /tmp/evaluation-result-3385478-1308045232/solver-3385478-1308045232 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385478-1308045232.cnf -seed 919075938 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 4.09 4.19 3.83 2/168 21220
/proc/meminfo: memFree=28645772/32951132 swapFree=67052544/67111528
[pid=21220] ppid=21218 vsize=15216 CPUtime=0 cores=0-7
/proc/21220/stat : 21220 (python) R 21218 21220 21190 0 -1 4202496 659 0 0 0 0 0 0 0 25 0 1 0 916121851 15581184 554 33554432000 4194304 5683644 140736420045120 18446744073709551615 5288080 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/21220/statm: 3804 554 314 364 0 291 0

[startup+0.055292 s]
/proc/loadavg: 4.09 4.19 3.83 2/168 21220
/proc/meminfo: memFree=28645772/32951132 swapFree=67052544/67111528
[pid=21220] ppid=21218 vsize=27344 CPUtime=0.02 cores=0-7
/proc/21220/stat : 21220 (python) R 21218 21220 21190 0 -1 4202496 1351 0 0 0 2 0 0 0 25 0 1 0 916121851 28000256 1196 33554432000 4194304 5683644 140736420045120 18446744073709551615 242970549904 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/21220/statm: 6836 1196 440 364 0 723 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 27344

[startup+0.100264 s]
/proc/loadavg: 4.09 4.19 3.83 2/168 21220
/proc/meminfo: memFree=28645772/32951132 swapFree=67052544/67111528
[pid=21220] ppid=21218 vsize=87700 CPUtime=0.06 cores=0-7
/proc/21220/stat : 21220 (python) R 21218 21220 21190 0 -1 4202496 2637 0 0 0 4 2 0 0 25 0 1 0 916121851 89804800 2353 33554432000 4194304 5683644 140736420045120 18446744073709551615 242970548224 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/21220/statm: 21925 2353 717 364 0 1621 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 87700

[startup+0.300244 s]
/proc/loadavg: 4.09 4.19 3.83 2/168 21220
/proc/meminfo: memFree=28645772/32951132 swapFree=67052544/67111528
[pid=21220] ppid=21218 vsize=135744 CPUtime=0.19 cores=0-7
/proc/21220/stat : 21220 (python) R 21218 21220 21190 0 -1 4202496 5961 934 1 0 14 5 0 0 17 0 1 0 916121851 139001856 4150 33554432000 4194304 5683644 140736420045120 18446744073709551615 4620076 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/21220/statm: 33936 4150 1032 364 0 3122 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 135744

[startup+0.701155 s]
/proc/loadavg: 4.09 4.19 3.83 2/168 21220
/proc/meminfo: memFree=28645772/32951132 swapFree=67052544/67111528
[pid=21220] ppid=21218 vsize=278600 CPUtime=0.42 cores=0-7
/proc/21220/stat : 21220 (python) R 21218 21220 21190 0 -1 4202496 9371 934 1 0 32 10 0 0 16 0 1 0 916121851 285286400 7041 33554432000 4194304 5683644 140736420045120 18446744073709551615 47145180069199 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/21220/statm: 69650 7041 1826 364 0 5574 0
Current children cumulated CPU time (s) 0.42
Current children cumulated vsize (KiB) 278600

[startup+1.50095 s]
/proc/loadavg: 4.09 4.19 3.83 6/183 21245
/proc/meminfo: memFree=28601152/32951132 swapFree=67052544/67111528
[pid=21220] ppid=21218 vsize=310116 CPUtime=0.81 cores=0-7
/proc/21220/stat : 21220 (python) S 21218 21220 21190 0 -1 4202496 15996 3995 1 0 64 12 5 0 21 0 1 0 916121851 317558784 7297 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982380351 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/21220/statm: 77529 7297 1891 364 0 13426 0
[pid=21230] ppid=21220 vsize=289420 CPUtime=0 cores=0-7
/proc/21230/stat : 21230 (python) S 21220 21220 21190 0 -1 4202560 750 0 0 0 0 0 0 0 15 0 1 0 916121936 296366080 5713 33554432000 4194304 5683644 140736420045120 18446744073709551615 242970570310 0 0 16781312 514 5630823350507897091 0 0 17 1 0 0 0
/proc/21230/statm: 72355 5713 348 364 0 8270 0
[pid=21232] ppid=21230 vsize=2816 CPUtime=0.64 cores=0-7
/proc/21232/stat : 21232 (march_hi) R 21230 21232 21232 0 -1 4202496 756 0 0 0 64 0 0 0 21 0 1 0 916121936 2883584 493 33554432000 4194304 4955518 140735616547856 18446744073709551615 4227824 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/21232/statm: 704 493 104 186 0 517 0
[pid=21236] ppid=21220 vsize=310068 CPUtime=0 cores=0-7
/proc/21236/stat : 21236 (python) S 21220 21220 21190 0 -1 4202560 766 0 0 0 0 0 0 0 15 0 1 0 916121942 317509632 5737 33554432000 4194304 5683644 140736420045120 18446744073709551615 242970570310 0 0 16781312 514 484295293911662851 0 0 17 0 0 0 0
/proc/21236/statm: 77517 5737 348 364 0 13426 0
[pid=21238] ppid=21236 vsize=2816 CPUtime=0.57 cores=0-7
/proc/21238/stat : 21238 (march_hi) R 21236 21238 21238 0 -1 4202496 751 0 0 0 57 0 0 0 22 0 1 0 916121942 2883584 490 33554432000 4194304 4955518 140735116186944 18446744073709551615 4235403 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/21238/statm: 704 490 104 186 0 517 0
[pid=21240] ppid=21220 vsize=320324 CPUtime=0 cores=0-7
/proc/21240/stat : 21240 (python) S 21220 21220 21190 0 -1 4202560 771 0 0 0 0 0 0 0 16 0 1 0 916121945 328011776 5741 33554432000 4194304 5683644 140736420045120 18446744073709551615 242970570310 0 0 16781312 514 3528957340432695556 0 0 17 4 0 0 0
/proc/21240/statm: 80081 5741 348 364 0 15987 0
[pid=21242] ppid=21240 vsize=2816 CPUtime=0.54 cores=0-7
/proc/21242/stat : 21242 (march_hi) R 21240 21242 21242 0 -1 4202496 753 0 0 0 54 0 0 0 25 0 1 0 916121946 2883584 488 33554432000 4194304 4955518 140737390713680 18446744073709551615 4286687 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/21242/statm: 704 488 104 186 0 517 0
[pid=21243] ppid=21220 vsize=330580 CPUtime=0 cores=0-7
/proc/21243/stat : 21243 (python) S 21220 21220 21190 0 -1 4202560 783 0 0 0 0 0 0 0 19 0 1 0 916121949 338513920 5745 33554432000 4194304 5683644 140736420045120 18446744073709551615 242970570310 0 0 16781312 514 1588860277128921347 0 0 17 0 0 0 0
/proc/21243/statm: 82645 5745 348 364 0 18548 0
[pid=21245] ppid=21243 vsize=2816 CPUtime=0.51 cores=0-7
/proc/21245/stat : 21245 (march_hi) R 21243 21245 21245 0 -1 4202496 753 0 0 0 51 0 0 0 25 0 1 0 916121949 2883584 487 33554432000 4194304 4955518 140736818080160 18446744073709551615 4227109 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/21245/statm: 704 487 104 186 0 517 0
Current children cumulated CPU time (s) 3.07
Current children cumulated vsize (KiB) 1571772

[startup+3.10073 s]
/proc/loadavg: 4.09 4.19 3.83 7/181 21251
/proc/meminfo: memFree=28587520/32951132 swapFree=67052544/67111528
[pid=21220] ppid=21218 vsize=310116 CPUtime=0.81 cores=0-7
/proc/21220/stat : 21220 (python) S 21218 21220 21190 0 -1 4202496 15996 3995 1 0 64 12 5 0 21 0 1 0 916121851 317558784 7297 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982380351 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/21220/statm: 77529 7297 1891 364 0 13426 0
[pid=21230] ppid=21220 vsize=289420 CPUtime=0 cores=0-7
/proc/21230/stat : 21230 (python) S 21220 21220 21190 0 -1 4202560 752 0 0 0 0 0 0 0 15 0 1 0 916121936 296366080 5713 33554432000 4194304 5683644 140736420045120 18446744073709551615 242970570310 0 0 16781312 514 4030072134297452543 0 0 17 1 0 0 0
/proc/21230/statm: 72355 5713 348 364 0 8270 0
[pid=21232] ppid=21230 vsize=3420 CPUtime=2.24 cores=0-7
/proc/21232/stat : 21232 (march_hi) R 21230 21232 21232 0 -1 4202496 879 0 0 0 224 0 0 0 25 0 1 0 916121936 3502080 616 33554432000 4194304 4955518 140735616547856 18446744073709551615 4224738 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/21232/statm: 855 616 104 186 0 668 0
[pid=21236] ppid=21220 vsize=310068 CPUtime=0 cores=0-7
/proc/21236/stat : 21236 (python) S 21220 21220 21190 0 -1 4202560 767 0 0 0 0 0 0 0 15 0 1 0 916121942 317509632 5737 33554432000 4194304 5683644 140736420045120 18446744073709551615 242970570310 0 0 16781312 514 8957338880617480191 0 0 17 0 0 0 0
/proc/21236/statm: 77517 5737 348 364 0 13426 0
[pid=21238] ppid=21236 vsize=3420 CPUtime=2.17 cores=0-7
/proc/21238/stat : 21238 (march_hi) R 21236 21238 21238 0 -1 4202496 874 0 0 0 217 0 0 0 25 0 1 0 916121942 3502080 613 33554432000 4194304 4955518 140735116186944 18446744073709551615 4298062 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/21238/statm: 855 613 104 186 0 668 0
[pid=21240] ppid=21220 vsize=320324 CPUtime=0 cores=0-7
/proc/21240/stat : 21240 (python) S 21220 21220 21190 0 -1 4202560 773 0 0 0 0 0 0 0 15 0 1 0 916121945 328011776 5741 33554432000 4194304 5683644 140736420045120 18446744073709551615 242970570310 0 0 16781312 514 1078410079788597247 0 0 17 4 0 0 0
/proc/21240/statm: 80081 5741 348 364 0 15987 0
[pid=21242] ppid=21240 vsize=3420 CPUtime=2.14 cores=0-7
/proc/21242/stat : 21242 (march_hi) R 21240 21242 21242 0 -1 4202496 877 0 0 0 214 0 0 0 25 0 1 0 916121946 3502080 612 33554432000 4194304 4955518 140737390713680 18446744073709551615 4209727 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/21242/statm: 855 612 104 186 0 668 0
[pid=21243] ppid=21220 vsize=330580 CPUtime=0 cores=0-7
/proc/21243/stat : 21243 (python) S 21220 21220 21190 0 -1 4202560 784 0 0 0 0 0 0 0 15 0 1 0 916121949 338513920 5745 33554432000 4194304 5683644 140736420045120 18446744073709551615 242970570310 0 0 16781312 514 8957331184036085759 0 0 17 0 0 0 0
/proc/21243/statm: 82645 5745 348 364 0 18548 0
[pid=21245] ppid=21243 vsize=3420 CPUtime=2.11 cores=0-7
/proc/21245/stat : 21245 (march_hi) R 21243 21245 21245 0 -1 4202496 877 0 0 0 211 0 0 0 25 0 1 0 916121949 3502080 611 33554432000 4194304 4955518 140736818080160 18446744073709551615 4227679 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/21245/statm: 855 611 104 186 0 668 0
[pid=21246] ppid=21220 vsize=340836 CPUtime=0 cores=0-7
/proc/21246/stat : 21246 (python) S 21220 21220 21190 0 -1 4202560 787 0 0 0 0 0 0 0 15 0 1 0 916121952 349016064 5749 33554432000 4194304 5683644 140736420045120 18446744073709551615 242970570310 0 0 16781312 514 3526793776427171839 0 0 17 0 0 0 0
/proc/21246/statm: 85209 5749 348 364 0 21109 0
[pid=21248] ppid=21246 vsize=3420 CPUtime=2.07 cores=0-7
/proc/21248/stat : 21248 (march_hi) R 21246 21248 21248 0 -1 4202496 875 0 0 0 207 0 0 0 25 0 1 0 916121953 3502080 609 33554432000 4194304 4955518 140733393224016 18446744073709551615 4227873 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/21248/statm: 855 609 104 186 0 668 0
[pid=21249] ppid=21220 vsize=351092 CPUtime=0 cores=0-7
/proc/21249/stat : 21249 (python) S 21220 21220 21190 0 -1 4202560 806 0 0 0 0 0 0 0 15 0 1 0 916121955 359518208 5776 33554432000 4194304 5683644 140736420045120 18446744073709551615 242970570310 0 0 16781312 514 4362223327054397439 0 0 17 0 0 0 0
/proc/21249/statm: 87773 5776 354 364 0 23670 0
[pid=21251] ppid=21249 vsize=3420 CPUtime=2.04 cores=0-7
/proc/21251/stat : 21251 (march_hi) R 21249 21251 21251 0 -1 4202496 877 0 0 0 204 0 0 0 25 0 1 0 916121956 3502080 605 33554432000 4194304 4955518 140735693511776 18446744073709551615 4295598 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/21251/statm: 855 605 104 186 0 668 0
Current children cumulated CPU time (s) 13.58
Current children cumulated vsize (KiB) 2272956

[startup+6.30115 s]
/proc/loadavg: 4.24 4.22 3.84 7/181 21251
/proc/meminfo: memFree=28581196/32951132 swapFree=67052544/67111528
[pid=21220] ppid=21218 vsize=310116 CPUtime=0.81 cores=0-7
/proc/21220/stat : 21220 (python) S 21218 21220 21190 0 -1 4202496 15996 3995 1 0 64 12 5 0 21 0 1 0 916121851 317558784 7297 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982380351 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/21220/statm: 77529 7297 1891 364 0 13426 0
[pid=21230] ppid=21220 vsize=289420 CPUtime=0 cores=0-7
/proc/21230/stat : 21230 (python) S 21220 21220 21190 0 -1 4202560 906 0 0 0 0 0 0 0 15 0 1 0 916121936 296366080 5736 33554432000 4194304 5683644 140736420045120 18446744073709551615 242970570310 0 0 16781312 514 5630823350507897091 0 0 17 1 0 0 0
/proc/21230/statm: 72355 5736 371 364 0 8270 0
[pid=21232] ppid=21230 vsize=3988 CPUtime=5.44 cores=0-7
/proc/21232/stat : 21232 (march_hi) R 21230 21232 21232 0 -1 4202496 1013 0 0 0 544 0 0 0 25 0 1 0 916121936 4083712 750 33554432000 4194304 4955518 140735616547856 18446744073709551615 4227079 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/21232/statm: 997 750 104 186 0 810 0
[pid=21236] ppid=21220 vsize=310068 CPUtime=0 cores=0-7
/proc/21236/stat : 21236 (python) S 21220 21220 21190 0 -1 4202560 939 0 0 0 0 0 0 0 15 0 1 0 916121942 317509632 5760 33554432000 4194304 5683644 140736420045120 18446744073709551615 242970570310 0 0 16781312 514 484295293911662851 0 0 17 0 0 0 0
/proc/21236/statm: 77517 5760 371 364 0 13426 0
[pid=21238] ppid=21236 vsize=3988 CPUtime=5.28 cores=0-7
/proc/21238/stat : 21238 (march_hi) R 21236 21238 21238 0 -1 4202496 1000 0 0 0 528 0 0 0 25 0 1 0 916121942 4083712 739 33554432000 4194304 4955518 140735116186944 18446744073709551615 4227502 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/21238/statm: 997 739 104 186 0 810 0
[pid=21240] ppid=21220 vsize=320324 CPUtime=0 cores=0-7
/proc/21240/stat : 21240 (python) S 21220 21220 21190 0 -1 4202560 940 0 0 0 0 0 0 0 15 0 1 0 916121945 328011776 5764 33554432000 4194304 5683644 140736420045120 18446744073709551615 242970570310 0 0 16781312 514 3528957340432695556 0 0 17 4 0 0 0
/proc/21240/statm: 80081 5764 371 364 0 15987 0
[pid=21242] ppid=21240 vsize=3988 CPUtime=5.34 cores=0-7
/proc/21242/stat : 21242 (march_hi) R 21240 21242 21242 0 -1 4202496 1010 0 0 0 534 0 0 0 25 0 1 0 916121946 4083712 745 33554432000 4194304 4955518 140737390713680 18446744073709551615 4314057 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/21242/statm: 997 745 104 186 0 810 0
[pid=21243] ppid=21220 vsize=330580 CPUtime=0 cores=0-7
/proc/21243/stat : 21243 (python) S 21220 21220 21190 0 -1 4202560 947 0 0 0 0 0 0 0 15 0 1 0 916121949 338513920 5767 33554432000 4194304 5683644 140736420045120 18446744073709551615 242970570310 0 0 16781312 514 1588860277128921347 0 0 17 0 0 0 0
/proc/21243/statm: 82645 5767 370 364 0 18548 0
[pid=21245] ppid=21243 vsize=3988 CPUtime=5.25 cores=0-7
/proc/21245/stat : 21245 (march_hi) R 21243 21245 21245 0 -1 4202496 1005 0 0 0 525 0 0 0 25 0 1 0 916121949 4083712 739 33554432000 4194304 4955518 140736818080160 18446744073709551615 4227827 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/21245/statm: 997 739 104 186 0 810 0
[pid=21246] ppid=21220 vsize=340836 CPUtime=0 cores=0-7
/proc/21246/stat : 21246 (python) S 21220 21220 21190 0 -1 4202560 951 0 0 0 0 0 0 0 15 0 1 0 916121952 349016064 5771 33554432000 4194304 5683644 140736420045120 18446744073709551615 242970570310 0 0 16781312 514 934277024648757506 0 0 17 0 0 0 0
/proc/21246/statm: 85209 5771 370 364 0 21109 0
[pid=21248] ppid=21246 vsize=3988 CPUtime=5.27 cores=0-7
/proc/21248/stat : 21248 (march_hi) R 21246 21248 21248 0 -1 4202496 1010 0 0 0 527 0 0 0 25 0 1 0 916121953 4083712 744 33554432000 4194304 4955518 140733393224016 18446744073709551615 4227502 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/21248/statm: 997 744 104 186 0 810 0
[pid=21249] ppid=21220 vsize=351092 CPUtime=0 cores=0-7
/proc/21249/stat : 21249 (python) S 21220 21220 21190 0 -1 4202560 966 0 0 0 0 0 0 0 15 0 1 0 916121955 359518208 5799 33554432000 4194304 5683644 140736420045120 18446744073709551615 242970570310 0 0 16781312 514 4211670498397880578 0 0 17 0 0 0 0
/proc/21249/statm: 87773 5799 377 364 0 23670 0
[pid=21251] ppid=21249 vsize=3988 CPUtime=5.23 cores=0-7
/proc/21251/stat : 21251 (march_hi) R 21249 21251 21251 0 -1 4202496 1010 0 0 0 523 0 0 0 25 0 1 0 916121956 4083712 738 33554432000 4194304 4955518 140735693511776 18446744073709551615 4227664 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/21251/statm: 997 738 104 186 0 810 0
Current children cumulated CPU time (s) 32.62
Current children cumulated vsize (KiB) 2276364

[startup+12.701 s]
/proc/loadavg: 4.54 4.28 3.86 7/181 21252
/proc/meminfo: memFree=28576044/32951132 swapFree=67052544/67111528
[pid=21220] ppid=21218 vsize=310116 CPUtime=0.81 cores=0-7
/proc/21220/stat : 21220 (python) S 21218 21220 21190 0 -1 4202496 15996 3995 1 0 64 12 5 0 21 0 1 0 916121851 317558784 7297 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982380351 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/21220/statm: 77529 7297 1891 364 0 13426 0
[pid=21230] ppid=21220 vsize=289420 CPUtime=0 cores=0-7
/proc/21230/stat : 21230 (python) S 21220 21220 21190 0 -1 4202560 908 0 0 0 0 0 0 0 15 0 1 0 916121936 296366080 5736 33554432000 4194304 5683644 140736420045120 18446744073709551615 242970570310 0 0 16781312 514 5630823350507897091 0 0 17 0 0 0 0
/proc/21230/statm: 72355 5736 371 364 0 8270 0
[pid=21232] ppid=21230 vsize=5424 CPUtime=11.84 cores=0-7
/proc/21232/stat : 21232 (march_hi) R 21230 21232 21232 0 -1 4202496 1311 0 0 0 1184 0 0 0 25 0 1 0 916121936 5554176 1014 33554432000 4194304 4955518 140735616547856 18446744073709551615 4209669 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/21232/statm: 1356 1014 104 186 0 1169 0
[pid=21236] ppid=21220 vsize=310068 CPUtime=0.01 cores=0-7

################
# More data... #
################

/proc/21242/stat : 21242 (march_hi) T 21240 21242 21242 0 -1 4202496 18667 0 0 0 48555 4 0 0 25 0 1 0 916121946 83853312 18368 33554432000 4194304 4955518 140737390713680 18446744073709551615 4227831 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/21242/statm: 20472 18368 104 186 0 20285 0
[pid=21240/tid=21277] ppid=21220 vsize=385860 CPUtime=0 cores=0-7
/proc/21240/task/21277/stat : 21277 (python) S 21220 21220 21190 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 916170507 395120640 5795 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982373809 0 0 16781312 514 18446604440278878272 0 0 -1 3 0 0 0
[pid=21243] ppid=21220 vsize=396116 CPUtime=0.6 cores=0-7
/proc/21243/stat : 21243 (python) S 21220 21220 21190 0 -1 4202560 1180 0 0 0 35 25 0 0 16 0 2 0 916121949 405622784 5799 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/21243/statm: 99029 5799 400 364 0 34932 0
[pid=21245] ppid=21243 vsize=81888 CPUtime=485.93 cores=0-7
/proc/21245/stat : 21245 (march_hi) T 21243 21245 21245 0 -1 4202496 18668 0 0 0 48588 5 0 0 25 0 1 0 916121949 83853312 18368 33554432000 4194304 4955518 140736818080160 18446744073709551615 4209701 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/21245/statm: 20472 18368 104 186 0 20285 0
[pid=21243/tid=21278] ppid=21220 vsize=396116 CPUtime=0 cores=0-7
/proc/21243/task/21278/stat : 21278 (python) S 21220 21220 21190 0 -1 4202560 47 0 0 0 0 0 0 0 17 0 2 0 916170554 405622784 5799 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982373809 0 0 16781312 514 18446604450703267136 0 0 -1 3 0 0 0
[pid=21246] ppid=21220 vsize=340836 CPUtime=0.62 cores=0-7
/proc/21246/stat : 21246 (python) S 21220 21220 21190 0 -1 4202560 1189 0 0 0 38 24 0 0 16 0 2 0 916121952 349016064 5802 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/21246/statm: 85209 5802 400 364 0 21109 0
[pid=21248] ppid=21246 vsize=81888 CPUtime=490.87 cores=0-7
/proc/21248/stat : 21248 (march_hi) T 21246 21248 21248 0 -1 4202496 18876 0 0 0 49082 5 0 0 25 0 1 0 916121953 83853312 18576 33554432000 4194304 4955518 140733393224016 18446744073709551615 4263648 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/21248/statm: 20472 18576 104 186 0 20285 0
[pid=21246/tid=21280] ppid=21220 vsize=340836 CPUtime=0 cores=0-7
/proc/21246/task/21280/stat : 21280 (python) S 21220 21220 21190 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 916171040 349016064 5802 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982373809 0 0 16781312 514 18446604453733838912 0 0 -1 1 0 0 0
[pid=21249] ppid=21220 vsize=351092 CPUtime=0.61 cores=0-7
/proc/21249/stat : 21249 (python) S 21220 21220 21190 0 -1 4202560 1205 0 0 0 36 25 0 0 16 0 2 0 916121955 359518208 5829 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/21249/statm: 87773 5829 406 364 0 23670 0
[pid=21251] ppid=21249 vsize=81888 CPUtime=489.56 cores=0-7
/proc/21251/stat : 21251 (march_hi) T 21249 21251 21251 0 -1 4202496 18594 0 0 0 48950 6 0 0 25 0 1 0 916121956 83853312 18288 33554432000 4194304 4955518 140735693511776 18446744073709551615 4227654 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/21251/statm: 20472 18288 104 186 0 20285 0
[pid=21249/tid=21279] ppid=21220 vsize=351092 CPUtime=0 cores=0-7
/proc/21249/task/21279/stat : 21279 (python) S 21220 21220 21190 0 -1 4202560 51 0 0 0 0 0 0 0 16 0 2 0 916170946 359518208 5829 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982373809 0 0 16781312 514 18446604445338417920 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 2448.21
Current children cumulated vsize (KiB) 2883828
heavy processes:

[startup+5082.3 s]
/proc/loadavg: 0.00 0.00 0.00 1/187 21417
/proc/meminfo: memFree=28226004/32951132 swapFree=67052544/67111528
[pid=21220] ppid=21218 vsize=310116 CPUtime=0.81 cores=0-7
/proc/21220/stat : 21220 (python) S 21218 21220 21190 0 -1 4202496 15996 3995 1 0 64 12 5 0 21 0 1 0 916121851 317558784 7297 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982380351 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/21220/statm: 77529 7297 1891 364 0 13426 0
[pid=21230] ppid=21220 vsize=354956 CPUtime=0.12 cores=0-7
/proc/21230/stat : 21230 (python) S 21220 21220 21190 0 -1 4202560 1097 0 0 0 7 5 0 0 17 0 2 0 916121936 363474944 5767 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/21230/statm: 88739 5767 400 364 0 24654 0
[pid=21232] ppid=21230 vsize=25344 CPUtime=99.05 cores=0-7
/proc/21232/stat : 21232 (march_hi) T 21230 21232 21232 0 -1 4202496 4659 0 0 0 9904 1 0 0 25 0 1 0 916121936 25952256 4362 33554432000 4194304 4955518 140735616547856 18446744073709551615 4290855 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/21232/statm: 6336 4362 104 186 0 6149 0
[pid=21230/tid=21256] ppid=21220 vsize=354956 CPUtime=0 cores=0-7
/proc/21230/task/21256/stat : 21256 (python) S 21220 21220 21190 0 -1 4202560 45 0 0 0 0 0 0 0 17 0 2 0 916131852 363474944 5767 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982373809 0 0 16781312 514 18446604453851017536 0 0 -1 3 0 0 0
[pid=21236] ppid=21220 vsize=310068 CPUtime=0.48 cores=0-7
/proc/21236/stat : 21236 (python) S 21220 21220 21190 0 -1 4202560 1173 0 0 0 29 19 0 0 16 0 2 0 916121942 317509632 5790 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/21236/statm: 77517 5790 400 364 0 13426 0
[pid=21238] ppid=21236 vsize=81888 CPUtime=393.39 cores=0-7
/proc/21238/stat : 21238 (march_hi) T 21236 21238 21238 0 -1 4202496 15265 0 0 0 39331 8 0 0 25 0 1 0 916121942 83853312 14970 33554432000 4194304 4955518 140735116186944 18446744073709551615 4238735 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/21238/statm: 20472 14970 104 186 0 20285 0
[pid=21236/tid=21265] ppid=21220 vsize=310068 CPUtime=0 cores=0-7
/proc/21236/task/21265/stat : 21265 (python) S 21220 21220 21190 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 916161385 317509632 5790 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982373809 0 0 16781312 514 18446604450703268928 0 0 -1 3 0 0 0
[pid=21240] ppid=21220 vsize=385860 CPUtime=0.58 cores=0-7
/proc/21240/stat : 21240 (python) S 21220 21220 21190 0 -1 4202560 1175 0 0 0 32 26 0 0 16 0 2 0 916121945 395120640 5795 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/21240/statm: 96465 5795 400 364 0 32371 0
[pid=21242] ppid=21240 vsize=81888 CPUtime=485.59 cores=0-7
/proc/21242/stat : 21242 (march_hi) T 21240 21242 21242 0 -1 4202496 18667 0 0 0 48555 4 0 0 25 0 1 0 916121946 83853312 18368 33554432000 4194304 4955518 140737390713680 18446744073709551615 4227831 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/21242/statm: 20472 18368 104 186 0 20285 0
[pid=21240/tid=21277] ppid=21220 vsize=385860 CPUtime=0 cores=0-7
/proc/21240/task/21277/stat : 21277 (python) S 21220 21220 21190 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 916170507 395120640 5795 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982373809 0 0 16781312 514 18446604440278878272 0 0 -1 3 0 0 0
[pid=21243] ppid=21220 vsize=396116 CPUtime=0.6 cores=0-7
/proc/21243/stat : 21243 (python) S 21220 21220 21190 0 -1 4202560 1180 0 0 0 35 25 0 0 16 0 2 0 916121949 405622784 5799 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/21243/statm: 99029 5799 400 364 0 34932 0
[pid=21245] ppid=21243 vsize=81888 CPUtime=485.93 cores=0-7
/proc/21245/stat : 21245 (march_hi) T 21243 21245 21245 0 -1 4202496 18668 0 0 0 48588 5 0 0 25 0 1 0 916121949 83853312 18368 33554432000 4194304 4955518 140736818080160 18446744073709551615 4209701 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/21245/statm: 20472 18368 104 186 0 20285 0
[pid=21243/tid=21278] ppid=21220 vsize=396116 CPUtime=0 cores=0-7
/proc/21243/task/21278/stat : 21278 (python) S 21220 21220 21190 0 -1 4202560 47 0 0 0 0 0 0 0 17 0 2 0 916170554 405622784 5799 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982373809 0 0 16781312 514 18446604450703267136 0 0 -1 3 0 0 0
[pid=21246] ppid=21220 vsize=340836 CPUtime=0.62 cores=0-7
/proc/21246/stat : 21246 (python) S 21220 21220 21190 0 -1 4202560 1189 0 0 0 38 24 0 0 16 0 2 0 916121952 349016064 5802 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/21246/statm: 85209 5802 400 364 0 21109 0
[pid=21248] ppid=21246 vsize=81888 CPUtime=490.87 cores=0-7
/proc/21248/stat : 21248 (march_hi) T 21246 21248 21248 0 -1 4202496 18876 0 0 0 49082 5 0 0 25 0 1 0 916121953 83853312 18576 33554432000 4194304 4955518 140733393224016 18446744073709551615 4263648 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/21248/statm: 20472 18576 104 186 0 20285 0
[pid=21246/tid=21280] ppid=21220 vsize=340836 CPUtime=0 cores=0-7
/proc/21246/task/21280/stat : 21280 (python) S 21220 21220 21190 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 916171040 349016064 5802 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982373809 0 0 16781312 514 18446604453733838912 0 0 -1 1 0 0 0
[pid=21249] ppid=21220 vsize=351092 CPUtime=0.61 cores=0-7
/proc/21249/stat : 21249 (python) S 21220 21220 21190 0 -1 4202560 1205 0 0 0 36 25 0 0 16 0 2 0 916121955 359518208 5829 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/21249/statm: 87773 5829 406 364 0 23670 0
[pid=21251] ppid=21249 vsize=81888 CPUtime=489.56 cores=0-7
/proc/21251/stat : 21251 (march_hi) T 21249 21251 21251 0 -1 4202496 18594 0 0 0 48950 6 0 0 25 0 1 0 916121956 83853312 18288 33554432000 4194304 4955518 140735693511776 18446744073709551615 4227654 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/21251/statm: 20472 18288 104 186 0 20285 0
[pid=21249/tid=21279] ppid=21220 vsize=351092 CPUtime=0 cores=0-7
/proc/21249/task/21279/stat : 21279 (python) S 21220 21220 21190 0 -1 4202560 51 0 0 0 0 0 0 0 16 0 2 0 916170946 359518208 5829 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982373809 0 0 16781312 514 18446604445338417920 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 2448.21
Current children cumulated vsize (KiB) 2883828
heavy processes:



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 0.00 0.00 0.00 1/187 21417
/proc/meminfo: memFree=28226004/32951132 swapFree=67052544/67111528
[pid=21220] ppid=21218 vsize=310116 CPUtime=0.81 cores=0-7
/proc/21220/stat : 21220 (python) S 21218 21220 21190 0 -1 4202496 15996 3995 1 0 64 12 5 0 21 0 1 0 916121851 317558784 7297 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982380351 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/21220/statm: 77529 7297 1891 364 0 13426 0
[pid=21230] ppid=21220 vsize=354956 CPUtime=0.12 cores=0-7
/proc/21230/stat : 21230 (python) S 21220 21220 21190 0 -1 4202560 1097 0 0 0 7 5 0 0 17 0 2 0 916121936 363474944 5767 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/21230/statm: 88739 5767 400 364 0 24654 0
[pid=21232] ppid=21230 vsize=25344 CPUtime=99.05 cores=0-7
/proc/21232/stat : 21232 (march_hi) T 21230 21232 21232 0 -1 4202496 4659 0 0 0 9904 1 0 0 25 0 1 0 916121936 25952256 4362 33554432000 4194304 4955518 140735616547856 18446744073709551615 4290855 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/21232/statm: 6336 4362 104 186 0 6149 0
[pid=21230/tid=21256] ppid=21220 vsize=354956 CPUtime=0 cores=0-7
/proc/21230/task/21256/stat : 21256 (python) S 21220 21220 21190 0 -1 4202560 45 0 0 0 0 0 0 0 17 0 2 0 916131852 363474944 5767 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982373809 0 0 16781312 514 18446604453851017536 0 0 -1 3 0 0 0
[pid=21236] ppid=21220 vsize=310068 CPUtime=0.48 cores=0-7
/proc/21236/stat : 21236 (python) S 21220 21220 21190 0 -1 4202560 1173 0 0 0 29 19 0 0 16 0 2 0 916121942 317509632 5790 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/21236/statm: 77517 5790 400 364 0 13426 0
[pid=21238] ppid=21236 vsize=81888 CPUtime=393.39 cores=0-7
/proc/21238/stat : 21238 (march_hi) T 21236 21238 21238 0 -1 4202496 15265 0 0 0 39331 8 0 0 25 0 1 0 916121942 83853312 14970 33554432000 4194304 4955518 140735116186944 18446744073709551615 4238735 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/21238/statm: 20472 14970 104 186 0 20285 0
[pid=21236/tid=21265] ppid=21220 vsize=310068 CPUtime=0 cores=0-7
/proc/21236/task/21265/stat : 21265 (python) S 21220 21220 21190 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 916161385 317509632 5790 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982373809 0 0 16781312 514 18446604450703268928 0 0 -1 3 0 0 0
[pid=21240] ppid=21220 vsize=385860 CPUtime=0.58 cores=0-7
/proc/21240/stat : 21240 (python) S 21220 21220 21190 0 -1 4202560 1175 0 0 0 32 26 0 0 16 0 2 0 916121945 395120640 5795 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/21240/statm: 96465 5795 400 364 0 32371 0
[pid=21242] ppid=21240 vsize=81888 CPUtime=485.59 cores=0-7
/proc/21242/stat : 21242 (march_hi) T 21240 21242 21242 0 -1 4202496 18667 0 0 0 48555 4 0 0 25 0 1 0 916121946 83853312 18368 33554432000 4194304 4955518 140737390713680 18446744073709551615 4227831 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/21242/statm: 20472 18368 104 186 0 20285 0
[pid=21240/tid=21277] ppid=21220 vsize=385860 CPUtime=0 cores=0-7
/proc/21240/task/21277/stat : 21277 (python) S 21220 21220 21190 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 916170507 395120640 5795 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982373809 0 0 16781312 514 18446604440278878272 0 0 -1 3 0 0 0
[pid=21243] ppid=21220 vsize=396116 CPUtime=0.6 cores=0-7
/proc/21243/stat : 21243 (python) S 21220 21220 21190 0 -1 4202560 1180 0 0 0 35 25 0 0 16 0 2 0 916121949 405622784 5799 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/21243/statm: 99029 5799 400 364 0 34932 0
[pid=21245] ppid=21243 vsize=81888 CPUtime=485.93 cores=0-7
/proc/21245/stat : 21245 (march_hi) T 21243 21245 21245 0 -1 4202496 18668 0 0 0 48588 5 0 0 25 0 1 0 916121949 83853312 18368 33554432000 4194304 4955518 140736818080160 18446744073709551615 4209701 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/21245/statm: 20472 18368 104 186 0 20285 0
[pid=21243/tid=21278] ppid=21220 vsize=396116 CPUtime=0 cores=0-7
/proc/21243/task/21278/stat : 21278 (python) S 21220 21220 21190 0 -1 4202560 47 0 0 0 0 0 0 0 17 0 2 0 916170554 405622784 5799 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982373809 0 0 16781312 514 18446604450703267136 0 0 -1 3 0 0 0
[pid=21246] ppid=21220 vsize=340836 CPUtime=0.62 cores=0-7
/proc/21246/stat : 21246 (python) S 21220 21220 21190 0 -1 4202560 1189 0 0 0 38 24 0 0 16 0 2 0 916121952 349016064 5802 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/21246/statm: 85209 5802 400 364 0 21109 0
[pid=21248] ppid=21246 vsize=81888 CPUtime=490.87 cores=0-7
/proc/21248/stat : 21248 (march_hi) T 21246 21248 21248 0 -1 4202496 18876 0 0 0 49082 5 0 0 25 0 1 0 916121953 83853312 18576 33554432000 4194304 4955518 140733393224016 18446744073709551615 4263648 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/21248/statm: 20472 18576 104 186 0 20285 0
[pid=21246/tid=21280] ppid=21220 vsize=340836 CPUtime=0 cores=0-7
/proc/21246/task/21280/stat : 21280 (python) S 21220 21220 21190 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 916171040 349016064 5802 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982373809 0 0 16781312 514 18446604453733838912 0 0 -1 1 0 0 0
[pid=21249] ppid=21220 vsize=351092 CPUtime=0.61 cores=0-7
/proc/21249/stat : 21249 (python) S 21220 21220 21190 0 -1 4202560 1205 0 0 0 36 25 0 0 16 0 2 0 916121955 359518208 5829 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/21249/statm: 87773 5829 406 364 0 23670 0
[pid=21251] ppid=21249 vsize=81888 CPUtime=489.56 cores=0-7
/proc/21251/stat : 21251 (march_hi) T 21249 21251 21251 0 -1 4202496 18594 0 0 0 48950 6 0 0 25 0 1 0 916121956 83853312 18288 33554432000 4194304 4955518 140735693511776 18446744073709551615 4227654 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/21251/statm: 20472 18288 104 186 0 20285 0
[pid=21249/tid=21279] ppid=21220 vsize=351092 CPUtime=0 cores=0-7
/proc/21249/task/21279/stat : 21279 (python) S 21220 21220 21190 0 -1 4202560 51 0 0 0 0 0 0 0 16 0 2 0 916170946 359518208 5829 33554432000 4194304 5683644 140736420045120 18446744073709551615 242982373809 0 0 16781312 514 18446604445338417920 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 2448.21
Current children cumulated vsize (KiB) 2883828
heavy processes:

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 21220 and gives
#  childrusage.ru_utime.tv_sec=2
#  childrusage.ru_utime.tv_usec=516617
#  childrusage.ru_stime.tv_sec=1
#  childrusage.ru_stime.tv_usec=423783
# CPU time returned by wait4() is 3.9404
# while last known CPU time is 2448.21
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5100.04
CPU time (s): 2448.21
CPU user time (s): 2446.56
CPU system time (s): 1.65
CPU usage (%): 48.0037
Max. virtual memory (cumulated for all children) (KiB): 2883828

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2.51662
system time used= 1.42378
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 28941
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 6622
involuntary context switches= 53

runsolver used 6.63099 second user time and 16.8444 second system time

The end

Launcher Data 

Begin job on node128 at 2011-06-14 11:53:52
IDJOB=3385478
IDBENCH=82743
IDSOLVER=1870
FILE ID=node128/3385478-1308045232
RUNJOBID= node128-1308045232-21204
PBS_JOBID= 13565708
Free space on /tmp= 73992 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v700-c2982-S900569154-026.SATISFIABLE.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3385478-1308045232/watcher-3385478-1308045232 -o /tmp/evaluation-result-3385478-1308045232/solver-3385478-1308045232 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385478-1308045232.cnf -seed 919075938 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 90af97b0aabddb9e07bb7ada8a5dcbc1
RANDOM SEED=919075938

node128.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5434.49
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      28646208 kB
Buffers:        342748 kB
Cached:        3710340 kB
SwapCached:      11368 kB
Active:         873996 kB
Inactive:      3197732 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      28646208 kB
SwapTotal:    67111528 kB
SwapFree:     67052544 kB
Dirty:           31824 kB
Writeback:           0 kB
AnonPages:       17448 kB
Mapped:           8876 kB
Slab:           166888 kB
PageTables:       4260 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS:   187564 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73960 MiB
End job on node128 at 2011-06-14 13:18:54