Trace number 3322099

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (exit code) 288.829 49.3951

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v400-c1704-S1050579435-065.SATISFIABLE.cnf
MD5SUMa9aff6c7ce1658eb3edc84aa7f38478c
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.007998
Satisfiable
(Un)Satisfiability was proved
Number of variables400
Number of clauses1704
Sum of the clauses size5112
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 31704
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.35/0.64	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.35/0.64	c INFO: solving HOME/instance-3322099-1304374280.cnf
0.35/0.64	c INFO: computing task features
0.35/0.65	c INFO: computed variable-clause graph statistics
0.35/0.65	c INFO: computed clause balance statistics
0.35/0.68	c INFO: computed variable balance statistics
0.43/0.70	c INFO: computed small-clause counts
0.43/0.70	c INFO: computed Horn-clause counts
0.43/0.71	c INFO: computed variable graph statistics
0.43/0.71	c INFO: collected features for HOME/instance-3322099-1304374280.cnf in 0.06 s
0.43/0.74	c INFO: running gnovelty+2@0 for 97 with 39999 remaining (b = 0.56)
0.43/0.78	c INFO: running march_hi@0 for 388 with 39999 remaining (b = 0.39)
0.52/0.82	c INFO: running march_hi@0 for 485 with 39999 remaining (b = 0.39)
0.52/0.86	c INFO: running march_hi@0 for 485 with 39999 remaining (b = 0.35)
0.66/0.90	c INFO: running march_hi@0 for 97 with 39999 remaining (b = 0.11)
0.66/0.94	c INFO: running TNM@0 for 97 with 39999 remaining (b = 0.07)
0.66/0.98	c INFO: running march_hi@0 for 485 with 39999 remaining (b = 0.24)
1.41/1.02	c INFO: running march_hi@0 for 485 with 39999 remaining (b = 0.21)
1.41/1.02	Traceback (most recent call last):
1.41/1.02	  File "HOME/solve", line 33, in <module>
1.41/1.02	    plac.call(main)
1.41/1.02	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
1.41/1.02	    cmd, result = parser_from(obj).consume(arglist)
1.41/1.02	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
1.41/1.02	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
1.41/1.02	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
1.41/1.02	    (_, answer, _) = solver(input_path, budget, cores)
1.41/1.02	  File "HOME/borg/build/lib/borg/bilevel.py", line 226, in __call__
1.41/1.02	    process.die()
1.41/1.02	  File "HOME/borg/build/lib/borg/solvers.py", line 185, in die
1.41/1.02	    os.kill(self._process.pid, signal.SIGUSR1)
1.41/1.02	OSError: [Errno 3] No such process

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3322099-1304374280/watcher-3322099-1304374280 -o /tmp/evaluation-result-3322099-1304374280/solver-3322099-1304374280 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3322099-1304374280.cnf -seed 350712018 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 9.08 9.17 9.11 3/342 13811
/proc/meminfo: memFree=17588684/32951124 swapFree=67111524/67111528
[pid=13811] ppid=13809 vsize=15776 CPUtime=0 cores=0-7
/proc/13811/stat : 13811 (python) S 13809 13811 12854 0 -1 4202496 904 0 0 0 0 0 0 0 20 0 1 0 529843666 16154624 784 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726403072 0 2147483391 16781312 2 18446744071699207358 0 0 17 2 0 0 0
/proc/13811/statm: 3944 784 346 364 0 431 0

[startup+0.0875679 s]
/proc/loadavg: 9.08 9.17 9.11 3/342 13811
/proc/meminfo: memFree=17588684/32951124 swapFree=67111524/67111528
[pid=13811] ppid=13809 vsize=56780 CPUtime=0.05 cores=0-7
/proc/13811/stat : 13811 (python) R 13809 13811 12854 0 -1 4202496 2103 0 0 0 4 1 0 0 20 0 1 0 529843666 58142720 1857 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726403072 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/13811/statm: 14195 1857 499 364 0 1318 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 56780

[startup+0.100552 s]
/proc/loadavg: 9.08 9.17 9.11 3/342 13811
/proc/meminfo: memFree=17588684/32951124 swapFree=67111524/67111528
[pid=13811] ppid=13809 vsize=91828 CPUtime=0.05 cores=0-7
/proc/13811/stat : 13811 (python) R 13809 13811 12854 0 -1 4202496 2678 0 0 0 4 1 0 0 20 0 1 0 529843666 94031872 2393 33554432000 4194304 5683644 140735921285008 18446744073709551615 4695995 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/13811/statm: 22957 2393 754 364 0 1622 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 91828

[startup+0.30055 s]
/proc/loadavg: 9.08 9.17 9.11 3/342 13811
/proc/meminfo: memFree=17588684/32951124 swapFree=67111524/67111528
[pid=13811] ppid=13809 vsize=135748 CPUtime=0.18 cores=0-7
/proc/13811/stat : 13811 (python) S 13809 13811 12854 0 -1 4202496 5963 935 1 0 14 4 0 0 20 0 1 0 529843666 139005952 4151 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726401029 0 2147483391 16781312 2 0 0 0 17 2 0 0 0
/proc/13811/statm: 33937 4151 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.18
Current children cumulated vsize (KiB) 135748

[startup+0.700466 s]
/proc/loadavg: 9.08 9.17 9.11 3/342 13811
/proc/meminfo: memFree=17588684/32951124 swapFree=67111524/67111528
[pid=13811] ppid=13809 vsize=278288 CPUtime=0.43 cores=0-7
/proc/13811/stat : 13811 (python) R 13809 13811 12854 0 -1 4202496 9297 935 1 0 35 8 0 0 19 0 1 0 529843666 284966912 6966 33554432000 4194304 5683644 140735921285008 18446744073709551615 4602978 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/13811/statm: 69572 6966 1827 364 0 5496 0
Current children cumulated CPU time (s) 0.43
Current children cumulated vsize (KiB) 278288

[startup+1.50125 s]
/proc/loadavg: 9.08 9.17 9.11 9/360 13840
/proc/meminfo: memFree=17544892/32951124 swapFree=67111524/67111528
[pid=13811] ppid=13809 vsize=374980 CPUtime=0.79 cores=0-7
/proc/13811/stat : 13811 (python) S 13809 13811 12854 0 -1 4202496 15537 3913 1 0 64 10 4 1 25 0 1 0 529843666 383979520 7327 33554432000 4194304 5683644 140735921285008 18446744073709551615 264738235199 0 0 16781312 2 18446744071562232103 0 0 17 0 0 0 0
/proc/13811/statm: 93745 7327 1891 364 0 29642 0
[pid=13821] ppid=13811 vsize=354420 CPUtime=0 cores=0-7
/proc/13821/stat : 13821 (python) S 13811 13811 12854 0 -1 4202560 702 0 0 0 0 0 0 0 15 0 1 0 529843745 362926080 5751 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 11556179473522983170 0 0 17 6 0 0 0
/proc/13821/statm: 88605 5751 348 364 0 24520 0
[pid=13824] ppid=13821 vsize=2120 CPUtime=0.7 cores=0-7
/proc/13824/stat : 13824 (march_hi) R 13821 13824 13824 0 -1 4202496 606 0 0 0 70 0 0 0 25 0 1 0 529843745 2170880 355 33554432000 4194304 4955518 140736287561264 18446744073709551615 4421280 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/13824/statm: 530 355 104 186 0 343 0
[pid=13825] ppid=13811 vsize=364676 CPUtime=0 cores=0-7
/proc/13825/stat : 13825 (python) S 13811 13811 12854 0 -1 4202560 718 0 0 0 0 0 0 0 15 0 1 0 529843748 373428224 5729 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 4119153191990296833 0 0 17 0 0 0 0
/proc/13825/statm: 91169 5729 348 364 0 27081 0
[pid=13827] ppid=13825 vsize=2120 CPUtime=0.66 cores=0-7
/proc/13827/stat : 13827 (march_hi) R 13825 13827 13827 0 -1 4202496 608 0 0 0 66 0 0 0 21 0 1 0 529843749 2170880 353 33554432000 4194304 4955518 140736658624736 18446744073709551615 4209727 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/13827/statm: 530 353 104 186 0 343 0
[pid=13828] ppid=13811 vsize=374932 CPUtime=0 cores=0-7
/proc/13828/stat : 13828 (python) S 13811 13811 12854 0 -1 4202560 723 0 0 0 0 0 0 0 15 0 1 0 529843752 383930368 5750 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 3538492305268769027 0 0 17 6 0 0 0
/proc/13828/statm: 93733 5750 348 364 0 29642 0
[pid=13830] ppid=13828 vsize=2120 CPUtime=0.61 cores=0-7
/proc/13830/stat : 13830 (march_hi) R 13828 13830 13830 0 -1 4202496 604 0 0 0 61 0 0 0 24 0 1 0 529843752 2170880 352 33554432000 4194304 4955518 140736745965488 18446744073709551615 4224738 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/13830/statm: 530 352 104 186 0 343 0
[pid=13831] ppid=13811 vsize=385188 CPUtime=0 cores=0-7
/proc/13831/stat : 13831 (python) S 13811 13811 12854 0 -1 4202560 735 0 0 0 0 0 0 0 15 0 1 0 529843756 394432512 5756 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 3600157315400958209 0 0 17 6 0 0 0
/proc/13831/statm: 96297 5756 348 364 0 32203 0
[pid=13833] ppid=13831 vsize=2120 CPUtime=0.59 cores=0-7
/proc/13833/stat : 13833 (march_hi) R 13831 13833 13833 0 -1 4202496 609 0 0 0 59 0 0 0 22 0 1 0 529843757 2170880 351 33554432000 4194304 4955518 140733714810320 18446744073709551615 4224648 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/13833/statm: 530 351 104 186 0 343 0
[pid=13838] ppid=13811 vsize=405700 CPUtime=0 cores=0-7
/proc/13838/stat : 13838 (python) S 13811 13811 12854 0 -1 4202560 753 0 0 0 0 0 0 0 19 0 1 0 529843764 415436800 5865 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 9407892082523078916 0 0 17 0 0 0 0
/proc/13838/statm: 101425 5865 348 364 0 37325 0
[pid=13840] ppid=13838 vsize=1984 CPUtime=0.5 cores=0-7
/proc/13840/stat : 13840 (march_hi) R 13838 13840 13840 0 -1 4202496 585 0 0 0 50 0 0 0 25 0 1 0 529843765 2031616 325 33554432000 4194304 4955518 140734988190928 18446744073709551615 4209717 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/13840/statm: 496 325 104 186 0 309 0
Current children cumulated CPU time (s) 3.85
Current children cumulated vsize (KiB) 2270360

[startup+3.10095 s]
/proc/loadavg: 8.99 9.15 9.11 8/355 13843
/proc/meminfo: memFree=17538476/32951124 swapFree=67111524/67111528
[pid=13811] ppid=13809 vsize=374980 CPUtime=0.79 cores=0-7
/proc/13811/stat : 13811 (python) S 13809 13811 12854 0 -1 4202496 15537 3913 1 0 64 10 4 1 25 0 1 0 529843666 383979520 7327 33554432000 4194304 5683644 140735921285008 18446744073709551615 264738235199 0 0 16781312 2 18446744071562232103 0 0 17 0 0 0 0
/proc/13811/statm: 93745 7327 1891 364 0 29642 0
[pid=13821] ppid=13811 vsize=354420 CPUtime=0 cores=0-7
/proc/13821/stat : 13821 (python) S 13811 13811 12854 0 -1 4202560 704 0 0 0 0 0 0 0 15 0 1 0 529843745 362926080 5751 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 11556179473522983170 0 0 17 6 0 0 0
/proc/13821/statm: 88605 5751 348 364 0 24520 0
[pid=13824] ppid=13821 vsize=2628 CPUtime=2.3 cores=0-7
/proc/13824/stat : 13824 (march_hi) R 13821 13824 13824 0 -1 4202496 683 0 0 0 230 0 0 0 25 0 1 0 529843745 2691072 432 33554432000 4194304 4955518 140736287561264 18446744073709551615 4238438 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/13824/statm: 657 432 104 186 0 470 0
[pid=13825] ppid=13811 vsize=364676 CPUtime=0 cores=0-7
/proc/13825/stat : 13825 (python) S 13811 13811 12854 0 -1 4202560 718 0 0 0 0 0 0 0 15 0 1 0 529843748 373428224 5729 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 4119153191990296833 0 0 17 0 0 0 0
/proc/13825/statm: 91169 5729 348 364 0 27081 0
[pid=13827] ppid=13825 vsize=2628 CPUtime=2.26 cores=0-7
/proc/13827/stat : 13827 (march_hi) R 13825 13827 13827 0 -1 4202496 684 0 0 0 226 0 0 0 25 0 1 0 529843749 2691072 429 33554432000 4194304 4955518 140736658624736 18446744073709551615 4238727 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/13827/statm: 657 429 104 186 0 470 0
[pid=13828] ppid=13811 vsize=374932 CPUtime=0 cores=0-7
/proc/13828/stat : 13828 (python) S 13811 13811 12854 0 -1 4202560 724 0 0 0 0 0 0 0 15 0 1 0 529843752 383930368 5750 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 3538492305268769027 0 0 17 6 0 0 0
/proc/13828/statm: 93733 5750 348 364 0 29642 0
[pid=13830] ppid=13828 vsize=2628 CPUtime=2.2 cores=0-7
/proc/13830/stat : 13830 (march_hi) R 13828 13830 13830 0 -1 4202496 679 0 0 0 220 0 0 0 25 0 1 0 529843752 2691072 427 33554432000 4194304 4955518 140736745965488 18446744073709551615 4238469 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/13830/statm: 657 427 104 186 0 470 0
[pid=13831] ppid=13811 vsize=385188 CPUtime=0 cores=0-7
/proc/13831/stat : 13831 (python) S 13811 13811 12854 0 -1 4202560 735 0 0 0 0 0 0 0 15 0 1 0 529843756 394432512 5756 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 3600157315400958209 0 0 17 6 0 0 0
/proc/13831/statm: 96297 5756 348 364 0 32203 0
[pid=13833] ppid=13831 vsize=2628 CPUtime=2.19 cores=0-7
/proc/13833/stat : 13833 (march_hi) R 13831 13833 13833 0 -1 4202496 685 0 0 0 219 0 0 0 25 0 1 0 529843757 2691072 427 33554432000 4194304 4955518 140733714810320 18446744073709551615 4235355 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/13833/statm: 657 427 104 186 0 470 0
[pid=13838] ppid=13811 vsize=405700 CPUtime=0 cores=0-7
/proc/13838/stat : 13838 (python) S 13811 13811 12854 0 -1 4202560 754 0 0 0 0 0 0 0 15 0 1 0 529843764 415436800 5865 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 9407892082523078916 0 0 17 0 0 0 0
/proc/13838/statm: 101425 5865 348 364 0 37325 0
[pid=13840] ppid=13838 vsize=2628 CPUtime=2.1 cores=0-7
/proc/13840/stat : 13840 (march_hi) R 13838 13840 13840 0 -1 4202496 684 0 0 0 210 0 0 0 25 0 1 0 529843765 2691072 424 33554432000 4194304 4955518 140734988190928 18446744073709551615 4224679 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/13840/statm: 657 424 104 186 0 470 0
[pid=13841] ppid=13811 vsize=415956 CPUtime=0 cores=0-7
/proc/13841/stat : 13841 (python) S 13811 13811 12854 0 -1 4202560 787 0 0 0 0 0 0 0 15 0 1 0 529843768 425938944 5806 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 18105574415998615809 0 0 17 0 0 0 0
/proc/13841/statm: 103989 5806 354 364 0 39886 0
[pid=13843] ppid=13841 vsize=2476 CPUtime=2.06 cores=0-7
/proc/13843/stat : 13843 (march_hi) R 13841 13843 13843 0 -1 4202496 682 0 0 0 206 0 0 0 25 0 1 0 529843769 2535424 419 33554432000 4194304 4955518 140737061776736 18446744073709551615 4227116 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/13843/statm: 619 419 104 186 0 432 0
Current children cumulated CPU time (s) 13.9
Current children cumulated vsize (KiB) 2691468

[startup+6.30137 s]
/proc/loadavg: 8.99 9.15 9.11 8/355 13843
/proc/meminfo: memFree=17535252/32951124 swapFree=67111524/67111528
[pid=13811] ppid=13809 vsize=374980 CPUtime=0.79 cores=0-7
/proc/13811/stat : 13811 (python) S 13809 13811 12854 0 -1 4202496 15537 3913 1 0 64 10 4 1 25 0 1 0 529843666 383979520 7327 33554432000 4194304 5683644 140735921285008 18446744073709551615 264738235199 0 0 16781312 2 18446744071562232103 0 0 17 0 0 0 0
/proc/13811/statm: 93745 7327 1891 364 0 29642 0
[pid=13821] ppid=13811 vsize=354420 CPUtime=0.02 cores=0-7
/proc/13821/stat : 13821 (python) S 13811 13811 12854 0 -1 4202560 780 0 0 0 1 1 0 0 15 0 1 0 529843745 362926080 5774 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 11556179473522983170 0 0 17 6 0 0 0
/proc/13821/statm: 88605 5774 371 364 0 24520 0
[pid=13824] ppid=13821 vsize=3480 CPUtime=5.5 cores=0-7
/proc/13824/stat : 13824 (march_hi) R 13821 13824 13824 0 -1 4202496 857 0 0 0 550 0 0 0 25 0 1 0 529843745 3563520 592 33554432000 4194304 4955518 140736287561264 18446744073709551615 4209643 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/13824/statm: 870 592 104 186 0 683 0
[pid=13825] ppid=13811 vsize=364676 CPUtime=0.01 cores=0-7
/proc/13825/stat : 13825 (python) S 13811 13811 12854 0 -1 4202560 794 0 0 0 1 0 0 0 15 0 1 0 529843748 373428224 5752 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 4119153191990296833 0 0 17 0 0 0 0
/proc/13825/statm: 91169 5752 371 364 0 27081 0
[pid=13827] ppid=13825 vsize=3480 CPUtime=5.46 cores=0-7
/proc/13827/stat : 13827 (march_hi) R 13825 13827 13827 0 -1 4202496 859 0 0 0 546 0 0 0 25 0 1 0 529843749 3563520 590 33554432000 4194304 4955518 140736658624736 18446744073709551615 4227615 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/13827/statm: 870 590 104 186 0 683 0
[pid=13828] ppid=13811 vsize=374932 CPUtime=0.01 cores=0-7
/proc/13828/stat : 13828 (python) S 13811 13811 12854 0 -1 4202560 801 0 0 0 1 0 0 0 15 0 1 0 529843752 383930368 5773 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 3538492305268769027 0 0 17 6 0 0 0
/proc/13828/statm: 93733 5773 371 364 0 29642 0
[pid=13830] ppid=13828 vsize=3480 CPUtime=5.39 cores=0-7
/proc/13830/stat : 13830 (march_hi) R 13828 13830 13830 0 -1 4202496 854 0 0 0 539 0 0 0 25 0 1 0 529843752 3563520 588 33554432000 4194304 4955518 140736745965488 18446744073709551615 4263701 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/13830/statm: 870 588 104 186 0 683 0
[pid=13831] ppid=13811 vsize=385188 CPUtime=0.01 cores=0-7
/proc/13831/stat : 13831 (python) S 13811 13811 12854 0 -1 4202560 814 0 0 0 1 0 0 0 15 0 1 0 529843756 394432512 5779 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 3600157315400958209 0 0 17 1 0 0 0
/proc/13831/statm: 96297 5779 371 364 0 32203 0
[pid=13833] ppid=13831 vsize=3480 CPUtime=5.38 cores=0-7
/proc/13833/stat : 13833 (march_hi) R 13831 13833 13833 0 -1 4202496 858 0 0 0 538 0 0 0 25 0 1 0 529843757 3563520 586 33554432000 4194304 4955518 140733714810320 18446744073709551615 4227640 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/13833/statm: 870 586 104 186 0 683 0
[pid=13838] ppid=13811 vsize=405700 CPUtime=0.01 cores=0-7
/proc/13838/stat : 13838 (python) S 13811 13811 12854 0 -1 4202560 832 0 0 0 0 1 0 0 15 0 1 0 529843764 415436800 5888 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 9407892082523078916 0 0 17 0 0 0 0
/proc/13838/statm: 101425 5888 371 364 0 37325 0
[pid=13840] ppid=13838 vsize=3480 CPUtime=5.26 cores=0-7
/proc/13840/stat : 13840 (march_hi) R 13838 13840 13840 0 -1 4202496 857 0 0 0 526 0 0 0 25 0 1 0 529843765 3563520 583 33554432000 4194304 4955518 140734988190928 18446744073709551615 4290765 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/13840/statm: 870 583 104 186 0 683 0
[pid=13841] ppid=13811 vsize=415956 CPUtime=0.02 cores=0-7
/proc/13841/stat : 13841 (python) S 13811 13811 12854 0 -1 4202560 870 0 0 0 1 1 0 0 15 0 1 0 529843768 425938944 5829 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 18105574415998615809 0 0 17 0 0 0 0
/proc/13841/statm: 103989 5829 377 364 0 39886 0
[pid=13843] ppid=13841 vsize=3196 CPUtime=5.22 cores=0-7
/proc/13843/stat : 13843 (march_hi) R 13841 13843 13843 0 -1 4202496 808 0 0 0 522 0 0 0 25 0 1 0 529843769 3272704 545 33554432000 4194304 4955518 140737061776736 18446744073709551615 4227502 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/13843/statm: 799 545 104 186 0 612 0
Current children cumulated CPU time (s) 33.08
Current children cumulated vsize (KiB) 2696448

[startup+12.7012 s]
/proc/loadavg: 8.76 9.09 9.09 8/355 13843
/proc/meminfo: memFree=17530112/32951124 swapFree=67111524/67111528
[pid=13811] ppid=13809 vsize=374980 CPUtime=0.79 cores=0-7
/proc/13811/stat : 13811 (python) S 13809 13811 12854 0 -1 4202496 15537 3913 1 0 64 10 4 1 25 0 1 0 529843666 383979520 7327 33554432000 4194304 5683644 140735921285008 18446744073709551615 264738235199 0 0 16781312 2 18446744071562232103 0 0 17 0 0 0 0
/proc/13811/statm: 93745 7327 1891 364 0 29642 0
[pid=13821] ppid=13811 vsize=354420 CPUtime=0.03 cores=0-7

################
# More data... #
################

[pid=13824] ppid=13821 vsize=10220 CPUtime=43.9 cores=0-7
/proc/13824/stat : 13824 (march_hi) R 13821 13824 13824 0 -1 4202496 2144 0 0 0 4390 0 0 0 25 0 1 0 529843745 10465280 1879 33554432000 4194304 4955518 140736287561264 18446744073709551615 4224656 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/13824/statm: 2555 1879 104 186 0 2368 0
[pid=13825] ppid=13811 vsize=364676 CPUtime=0.14 cores=0-7
/proc/13825/stat : 13825 (python) S 13811 13811 12854 0 -1 4202560 797 0 0 0 9 5 0 0 15 0 1 0 529843748 373428224 5752 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 4119153191990296833 0 0 17 0 0 0 0
/proc/13825/statm: 91169 5752 371 364 0 27081 0
[pid=13827] ppid=13825 vsize=10220 CPUtime=43.85 cores=0-7
/proc/13827/stat : 13827 (march_hi) R 13825 13827 13827 0 -1 4202496 2146 0 0 0 4385 0 0 0 25 0 1 0 529843749 10465280 1877 33554432000 4194304 4955518 140736658624736 18446744073709551615 4209737 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/13827/statm: 2555 1877 104 186 0 2368 0
[pid=13828] ppid=13811 vsize=374932 CPUtime=0.13 cores=0-7
/proc/13828/stat : 13828 (python) S 13811 13811 12854 0 -1 4202560 806 0 0 0 7 6 0 0 15 0 1 0 529843752 383930368 5773 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 3538492305268769027 0 0 17 6 0 0 0
/proc/13828/statm: 93733 5773 371 364 0 29642 0
[pid=13830] ppid=13828 vsize=10220 CPUtime=43.7 cores=0-7
/proc/13830/stat : 13830 (march_hi) R 13828 13830 13830 0 -1 4202496 2121 0 0 0 4369 1 0 0 25 0 1 0 529843752 10465280 1855 33554432000 4194304 4955518 140736745965488 18446744073709551615 4227079 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/13830/statm: 2555 1855 104 186 0 2368 0
[pid=13831] ppid=13811 vsize=385188 CPUtime=0.13 cores=0-7
/proc/13831/stat : 13831 (python) S 13811 13811 12854 0 -1 4202560 816 0 0 0 8 5 0 0 15 0 1 0 529843756 394432512 5779 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 3600157315400958209 0 0 17 1 0 0 0
/proc/13831/statm: 96297 5779 371 364 0 32203 0
[pid=13833] ppid=13831 vsize=10220 CPUtime=43.59 cores=0-7
/proc/13833/stat : 13833 (march_hi) R 13831 13833 13833 0 -1 4202496 2131 0 0 0 4359 0 0 0 25 0 1 0 529843757 10465280 1859 33554432000 4194304 4955518 140733714810320 18446744073709551615 4224631 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/13833/statm: 2555 1859 104 186 0 2368 0
[pid=13838] ppid=13811 vsize=405700 CPUtime=0.13 cores=0-7
/proc/13838/stat : 13838 (python) S 13811 13811 12854 0 -1 4202560 837 0 0 0 7 6 0 0 15 0 1 0 529843764 415436800 5888 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 9407892082523078916 0 0 17 0 0 0 0
/proc/13838/statm: 101425 5888 371 364 0 37325 0
[pid=13840] ppid=13838 vsize=10220 CPUtime=43.46 cores=0-7
/proc/13840/stat : 13840 (march_hi) R 13838 13840 13840 0 -1 4202496 2141 0 0 0 4345 1 0 0 25 0 1 0 529843765 10465280 1867 33554432000 4194304 4955518 140734988190928 18446744073709551615 4261438 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/13840/statm: 2555 1867 104 186 0 2368 0
[pid=13841] ppid=13811 vsize=415956 CPUtime=0.12 cores=0-7
/proc/13841/stat : 13841 (python) S 13811 13811 12854 0 -1 4202560 871 0 0 0 7 5 0 0 15 0 1 0 529843768 425938944 5829 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 18105574415998615809 0 0 17 6 0 0 0
/proc/13841/statm: 103989 5829 377 364 0 39886 0
[pid=13843] ppid=13841 vsize=10220 CPUtime=43.53 cores=0-7
/proc/13843/stat : 13843 (march_hi) R 13841 13843 13843 0 -1 4202496 2140 0 0 0 4353 0 0 0 25 0 1 0 529843769 10465280 1863 33554432000 4194304 4955518 140737061776736 18446744073709551615 4227671 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/13843/statm: 2555 1863 104 186 0 2368 0
Current children cumulated CPU time (s) 263.6
Current children cumulated vsize (KiB) 2737172

[startup+47.9005 s]
/proc/loadavg: 8.03 8.88 9.02 8/355 13844
/proc/meminfo: memFree=17504760/32951124 swapFree=67111524/67111528
[pid=13811] ppid=13809 vsize=374980 CPUtime=0.79 cores=0-7
/proc/13811/stat : 13811 (python) S 13809 13811 12854 0 -1 4202496 15537 3913 1 0 64 10 4 1 25 0 1 0 529843666 383979520 7327 33554432000 4194304 5683644 140735921285008 18446744073709551615 264738235199 0 0 16781312 2 18446744071562232103 0 0 17 0 0 0 0
/proc/13811/statm: 93745 7327 1891 364 0 29642 0
[pid=13821] ppid=13811 vsize=354420 CPUtime=0.15 cores=0-7
/proc/13821/stat : 13821 (python) S 13811 13811 12854 0 -1 4202560 783 0 0 0 8 7 0 0 15 0 1 0 529843745 362926080 5774 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 2800763979201216770 0 0 17 6 0 0 0
/proc/13821/statm: 88605 5774 371 364 0 24520 0
[pid=13824] ppid=13821 vsize=10220 CPUtime=47.1 cores=0-7
/proc/13824/stat : 13824 (march_hi) R 13821 13824 13824 0 -1 4202496 2243 0 0 0 4710 0 0 0 25 0 1 0 529843745 10465280 1978 33554432000 4194304 4955518 140736287561264 18446744073709551615 4209699 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/13824/statm: 2555 1978 104 186 0 2368 0
[pid=13825] ppid=13811 vsize=364676 CPUtime=0.16 cores=0-7
/proc/13825/stat : 13825 (python) S 13811 13811 12854 0 -1 4202560 797 0 0 0 10 6 0 0 15 0 1 0 529843748 373428224 5752 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 2800763979201216769 0 0 17 0 0 0 0
/proc/13825/statm: 91169 5752 371 364 0 27081 0
[pid=13827] ppid=13825 vsize=10220 CPUtime=47.05 cores=0-7
/proc/13827/stat : 13827 (march_hi) R 13825 13827 13827 0 -1 4202496 2244 0 0 0 4705 0 0 0 25 0 1 0 529843749 10465280 1975 33554432000 4194304 4955518 140736658624736 18446744073709551615 4239104 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/13827/statm: 2555 1975 104 186 0 2368 0
[pid=13828] ppid=13811 vsize=374932 CPUtime=0.15 cores=0-7
/proc/13828/stat : 13828 (python) S 13811 13811 12854 0 -1 4202560 807 0 0 0 8 7 0 0 15 0 1 0 529843752 383930368 5773 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 2800763979201216771 0 0 17 6 0 0 0
/proc/13828/statm: 93733 5773 371 364 0 29642 0
[pid=13830] ppid=13828 vsize=10220 CPUtime=46.86 cores=0-7
/proc/13830/stat : 13830 (march_hi) R 13828 13830 13830 0 -1 4202496 2214 0 0 0 4685 1 0 0 25 0 1 0 529843752 10465280 1948 33554432000 4194304 4955518 140736745965488 18446744073709551615 4209730 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/13830/statm: 2555 1948 104 186 0 2368 0
[pid=13831] ppid=13811 vsize=385188 CPUtime=0.13 cores=0-7
/proc/13831/stat : 13831 (python) S 13811 13811 12854 0 -1 4202560 816 0 0 0 8 5 0 0 15 0 1 0 529843756 394432512 5779 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 3600157315400958209 0 0 17 1 0 0 0
/proc/13831/statm: 96297 5779 371 364 0 32203 0
[pid=13833] ppid=13831 vsize=10220 CPUtime=46.78 cores=0-7
/proc/13833/stat : 13833 (march_hi) R 13831 13833 13833 0 -1 4202496 2226 0 0 0 4678 0 0 0 25 0 1 0 529843757 10465280 1954 33554432000 4194304 4955518 140733714810320 18446744073709551615 4209740 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/13833/statm: 2555 1954 104 186 0 2368 0
[pid=13838] ppid=13811 vsize=405700 CPUtime=0.13 cores=0-7
/proc/13838/stat : 13838 (python) S 13811 13811 12854 0 -1 4202560 837 0 0 0 7 6 0 0 15 0 1 0 529843764 415436800 5888 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 9407892082523078916 0 0 17 0 0 0 0
/proc/13838/statm: 101425 5888 371 364 0 37325 0
[pid=13840] ppid=13838 vsize=10220 CPUtime=46.64 cores=0-7
/proc/13840/stat : 13840 (march_hi) R 13838 13840 13840 0 -1 4202496 2235 0 0 0 4663 1 0 0 25 0 1 0 529843765 10465280 1961 33554432000 4194304 4955518 140734988190928 18446744073709551615 4209653 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/13840/statm: 2555 1961 104 186 0 2368 0
[pid=13841] ppid=13811 vsize=415956 CPUtime=0.12 cores=0-7
/proc/13841/stat : 13841 (python) S 13811 13811 12854 0 -1 4202560 872 0 0 0 7 5 0 0 15 0 1 0 529843768 425938944 5829 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 18105574415998615809 0 0 17 6 0 0 0
/proc/13841/statm: 103989 5829 377 364 0 39886 0
[pid=13843] ppid=13841 vsize=10220 CPUtime=46.73 cores=0-7
/proc/13843/stat : 13843 (march_hi) R 13841 13843 13843 0 -1 4202496 2236 0 0 0 4673 0 0 0 25 0 1 0 529843769 10465280 1959 33554432000 4194304 4955518 140737061776736 18446744073709551615 4290986 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/13843/statm: 2555 1959 104 186 0 2368 0
Current children cumulated CPU time (s) 282.79
Current children cumulated vsize (KiB) 2737172

[startup+48.7013 s]
/proc/loadavg: 8.03 8.88 9.02 8/355 13844
/proc/meminfo: memFree=17503772/32951124 swapFree=67111524/67111528
[pid=13811] ppid=13809 vsize=374980 CPUtime=0.79 cores=0-7
/proc/13811/stat : 13811 (python) S 13809 13811 12854 0 -1 4202496 15537 3913 1 0 64 10 4 1 17 0 1 0 529843666 383979520 7327 33554432000 4194304 5683644 140735921285008 18446744073709551615 264738235199 0 0 16781312 2 18446744071562232103 0 0 17 6 0 0 0
/proc/13811/statm: 93745 7327 1891 364 0 29642 0
[pid=13821] ppid=13811 vsize=0 CPUtime=47.73 cores=0-7
/proc/13821/stat : 13821 (python) Z 13811 13811 12854 0 -1 4202588 1079 2268 0 0 8 7 4758 0 15 0 1 0 529843745 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 6 0 0 0
/proc/13821/statm: 0 0 0 0 0 0 0
[pid=13825] ppid=13811 vsize=0 CPUtime=47.73 cores=0-7
/proc/13825/stat : 13825 (python) Z 13811 13811 12854 0 -1 4202588 1064 2271 0 0 10 6 4757 0 15 0 1 0 529843748 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 0 0 0 0
/proc/13825/statm: 0 0 0 0 0 0 0
[pid=13828] ppid=13811 vsize=374932 CPUtime=0.15 cores=0-7
/proc/13828/stat : 13828 (python) S 13811 13811 12854 0 -1 4202560 807 0 0 0 8 7 0 0 15 0 1 0 529843752 383930368 5773 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 3538492305268769027 0 0 17 6 0 0 0
/proc/13828/statm: 93733 5773 371 364 0 29642 0
[pid=13830] ppid=13828 vsize=10220 CPUtime=47.65 cores=0-7
/proc/13830/stat : 13830 (march_hi) R 13828 13830 13830 0 -1 4202496 2238 0 0 0 4764 1 0 0 25 0 1 0 529843752 10465280 1972 33554432000 4194304 4955518 140736745965488 18446744073709551615 4229723 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/13830/statm: 2555 1972 104 186 0 2368 0
[pid=13831] ppid=13811 vsize=385188 CPUtime=0.14 cores=0-7
/proc/13831/stat : 13831 (python) S 13811 13811 12854 0 -1 4202560 816 0 0 0 9 5 0 0 15 0 1 0 529843756 394432512 5779 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 2800763979201216769 0 0 17 1 0 0 0
/proc/13831/statm: 96297 5779 371 364 0 32203 0
[pid=13833] ppid=13831 vsize=10220 CPUtime=47.56 cores=0-7
/proc/13833/stat : 13833 (march_hi) R 13831 13833 13833 0 -1 4202496 2251 0 0 0 4756 0 0 0 25 0 1 0 529843757 10465280 1979 33554432000 4194304 4955518 140733714810320 18446744073709551615 4238422 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/13833/statm: 2555 1979 104 186 0 2368 0
[pid=13838] ppid=13811 vsize=405700 CPUtime=0.15 cores=0-7
/proc/13838/stat : 13838 (python) S 13811 13811 12854 0 -1 4202560 838 0 0 0 8 7 0 0 15 0 1 0 529843764 415436800 5888 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 2800763979201216772 0 0 17 0 0 0 0
/proc/13838/statm: 101425 5888 371 364 0 37325 0
[pid=13840] ppid=13838 vsize=10220 CPUtime=47.44 cores=0-7
/proc/13840/stat : 13840 (march_hi) R 13838 13840 13840 0 -1 4202496 2261 0 0 0 4743 1 0 0 25 0 1 0 529843765 10465280 1987 33554432000 4194304 4955518 140734988190928 18446744073709551615 4209664 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/13840/statm: 2555 1987 104 186 0 2368 0
[pid=13841] ppid=13811 vsize=415956 CPUtime=0.14 cores=0-7
/proc/13841/stat : 13841 (python) S 13811 13811 12854 0 -1 4202560 872 0 0 0 8 6 0 0 15 0 1 0 529843768 425938944 5829 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 2800763979201216769 0 0 17 6 0 0 0
/proc/13841/statm: 103989 5829 377 364 0 39886 0
[pid=13843] ppid=13841 vsize=10220 CPUtime=47.53 cores=0-7
/proc/13843/stat : 13843 (march_hi) R 13841 13843 13843 0 -1 4202496 2261 0 0 0 4753 0 0 0 25 0 1 0 529843769 10465280 1984 33554432000 4194304 4955518 140737061776736 18446744073709551615 4227664 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/13843/statm: 2555 1984 104 186 0 2368 0
Current children cumulated CPU time (s) 287.01
Current children cumulated vsize (KiB) 1997636

[startup+49.1013 s]
/proc/loadavg: 8.03 8.88 9.02 8/355 13844
/proc/meminfo: memFree=17503772/32951124 swapFree=67111524/67111528
[pid=13811] ppid=13809 vsize=374980 CPUtime=0.79 cores=0-7
/proc/13811/stat : 13811 (python) S 13809 13811 12854 0 -1 4202496 15537 3913 1 0 64 10 4 1 15 0 1 0 529843666 383979520 7327 33554432000 4194304 5683644 140735921285008 18446744073709551615 264738235199 0 0 16781312 2 18446744071562232103 0 0 17 6 0 0 0
/proc/13811/statm: 93745 7327 1891 364 0 29642 0
[pid=13821] ppid=13811 vsize=0 CPUtime=47.73 cores=0-7
/proc/13821/stat : 13821 (python) Z 13811 13811 12854 0 -1 4202588 1079 2268 0 0 8 7 4758 0 15 0 1 0 529843745 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 6 0 0 0
/proc/13821/statm: 0 0 0 0 0 0 0
[pid=13825] ppid=13811 vsize=0 CPUtime=47.73 cores=0-7
/proc/13825/stat : 13825 (python) Z 13811 13811 12854 0 -1 4202588 1064 2271 0 0 10 6 4757 0 15 0 1 0 529843748 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 0 0 0 0
/proc/13825/statm: 0 0 0 0 0 0 0
[pid=13828] ppid=13811 vsize=374932 CPUtime=0.15 cores=0-7
/proc/13828/stat : 13828 (python) S 13811 13811 12854 0 -1 4202560 807 0 0 0 8 7 0 0 15 0 1 0 529843752 383930368 5773 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 3538492305268769027 0 0 17 6 0 0 0
/proc/13828/statm: 93733 5773 371 364 0 29642 0
[pid=13830] ppid=13828 vsize=10220 CPUtime=48.05 cores=0-7
/proc/13830/stat : 13830 (march_hi) R 13828 13830 13830 0 -1 4202496 2251 0 0 0 4804 1 0 0 25 0 1 0 529843752 10465280 1985 33554432000 4194304 4955518 140736745965488 18446744073709551615 4224619 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/13830/statm: 2555 1985 104 186 0 2368 0
[pid=13831] ppid=13811 vsize=385188 CPUtime=0.14 cores=0-7
/proc/13831/stat : 13831 (python) S 13811 13811 12854 0 -1 4202560 816 0 0 0 9 5 0 0 15 0 1 0 529843756 394432512 5779 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 2800763979201216769 0 0 17 1 0 0 0
/proc/13831/statm: 96297 5779 371 364 0 32203 0
[pid=13833] ppid=13831 vsize=10220 CPUtime=47.96 cores=0-7
/proc/13833/stat : 13833 (march_hi) R 13831 13833 13833 0 -1 4202496 2264 0 0 0 4796 0 0 0 25 0 1 0 529843757 10465280 1992 33554432000 4194304 4955518 140733714810320 18446744073709551615 4209664 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/13833/statm: 2555 1992 104 186 0 2368 0
[pid=13838] ppid=13811 vsize=0 CPUtime=47.8 cores=0-7
/proc/13838/stat : 13838 (python) Z 13811 13811 12854 0 -1 4202588 1143 2277 0 0 8 7 4764 1 16 0 1 0 529843764 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 0 0 0 0
/proc/13838/statm: 0 0 0 0 0 0 0
[pid=13841] ppid=13811 vsize=0 CPUtime=47.94 cores=0-7
/proc/13841/stat : 13841 (python) Z 13811 13811 12854 0 -1 4202588 1141 2279 0 0 8 6 4780 0 15 0 1 0 529843768 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 6 0 0 0
/proc/13841/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 288.29
Current children cumulated vsize (KiB) 1155540

[startup+49.3002 s]
/proc/loadavg: 8.03 8.88 9.02 8/355 13844
/proc/meminfo: memFree=17503772/32951124 swapFree=67111524/67111528
[pid=13811] ppid=13809 vsize=374980 CPUtime=192.32 cores=0-7
/proc/13811/stat : 13811 (python) S 13809 13811 12854 0 -1 4202496 15540 17374 1 0 64 10 19125 33 15 0 1 0 529843666 383979520 7327 33554432000 4194304 5683644 140735921285008 18446744073709551615 264738235199 0 0 16781312 2 18446744071562232103 0 0 17 6 0 0 0
/proc/13811/statm: 93745 7327 1891 364 0 29642 0
[pid=13828] ppid=13811 vsize=374932 CPUtime=0.15 cores=0-7
/proc/13828/stat : 13828 (python) S 13811 13811 12854 0 -1 4202560 807 0 0 0 8 7 0 0 15 0 1 0 529843752 383930368 5773 33554432000 4194304 5683644 140735921285008 18446744073709551615 264726425158 0 0 16781312 514 3538492305268769027 0 0 17 6 0 0 0
/proc/13828/statm: 93733 5773 371 364 0 29642 0
[pid=13830] ppid=13828 vsize=10220 CPUtime=48.25 cores=0-7
/proc/13830/stat : 13830 (march_hi) R 13828 13830 13830 0 -1 4202496 2258 0 0 0 4824 1 0 0 25 0 1 0 529843752 10465280 1992 33554432000 4194304 4955518 140736745965488 18446744073709551615 4290543 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/13830/statm: 2555 1992 104 186 0 2368 0
[pid=13841] ppid=13811 vsize=0 CPUtime=47.94 cores=0-7
/proc/13841/stat : 13841 (python) Z 13811 13811 12854 0 -1 4202588 1141 2279 0 0 8 6 4780 0 15 0 1 0 529843768 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 6 0 0 0
/proc/13841/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 288.66
Current children cumulated vsize (KiB) 760132

Child status: 1
Real time (s): 49.3951
CPU time (s): 288.829
CPU user time (s): 288.209
CPU system time (s): 0.619905
CPU usage (%): 584.732
Max. virtual memory (cumulated for all children) (KiB): 2737172

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 288.209
system time used= 0.619905
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 43902
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 5193
involuntary context switches= 2776

runsolver used 0.156976 second user time and 0.379942 second system time

The end

Launcher Data 

Begin job on node141 at 2011-05-03 00:11:20
IDJOB=3322099
IDBENCH=82877
IDSOLVER=1712
FILE ID=node141/3322099-1304374280
RUNJOBID= node141-1304366465-12868
PBS_JOBID= 13181732
Free space on /tmp= 72852 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v400-c1704-S1050579435-065.SATISFIABLE.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3322099-1304374280/watcher-3322099-1304374280 -o /tmp/evaluation-result-3322099-1304374280/solver-3322099-1304374280 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3322099-1304374280.cnf -seed 350712018 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= a9aff6c7ce1658eb3edc84aa7f38478c
RANDOM SEED=350712018

node141.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5362.96
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      17588964 kB
Buffers:       1434032 kB
Cached:        9028340 kB
SwapCached:          4 kB
Active:        7769256 kB
Inactive:      6970164 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      17588964 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:           31892 kB
Writeback:           0 kB
AnonPages:     4277068 kB
Mapped:          25148 kB
Slab:           521176 kB
PageTables:      33976 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 77446932 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72820 MiB
End job on node141 at 2011-05-03 00:12:10