Trace number 3317245

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (TO) 4800.2 1203.06

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v500-c2130-S1762880211-075.UNKNOWN.cnf
MD5SUM57a2cff28dc53061839e3102dd9b3447
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables500
Number of clauses2130
Sum of the clauses size6390
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32130
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.33/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.33/0.67	c INFO: solving HOME/instance-3317245-1304268512.cnf
0.33/0.67	c INFO: computing task features
0.33/0.68	c INFO: computed variable-clause graph statistics
0.33/0.68	c INFO: computed clause balance statistics
0.41/0.71	c INFO: computed variable balance statistics
0.41/0.74	c INFO: computed small-clause counts
0.41/0.74	c INFO: computed Horn-clause counts
0.41/0.75	c INFO: computed variable graph statistics
0.41/0.75	c INFO: collected features for HOME/instance-3317245-1304268512.cnf in 0.08 s
0.41/0.78	c INFO: running gnovelty+2@0 for 97 with 4799 remaining (b = 0.57)
0.50/0.81	c INFO: running march_hi@0 for 388 with 4799 remaining (b = 0.34)
0.50/0.83	c INFO: running march_hi@0 for 485 with 4799 remaining (b = 0.34)
0.50/0.86	c INFO: running march_hi@0 for 97 with 4799 remaining (b = 0.10)
400.03/100.97	c INFO: running gnovelty+2@0 for 97 with 4699 remaining (b = 0.07)
411.61/103.87	c INFO: running march_hi@100 for 97 with 4696 remaining (b = 0.07)
796.09/200.17	c INFO: running march_hi@0 for 485 with 4600 remaining (b = 0.28)
805.26/202.49	c INFO: running march_hi@199 for 194 with 4598 remaining (b = 0.13)
1570.23/394.02	c INFO: running march_hi@389 for 97 with 4406 remaining (b = 0.07)
1570.23/394.06	c INFO: running march_hi@0 for 97 with 4406 remaining (b = 0.07)
1955.12/490.47	c INFO: running march_hi@95 for 97 with 4310 remaining (b = 0.06)
1956.31/490.70	c INFO: running march_hi@781 for 194 with 4309 remaining (b = 0.10)
1958.30/491.21	c INFO: running march_hi@0 for 291 with 4309 remaining (b = 0.19)
2345.23/588.17	c INFO: running march_hi@192 for 291 with 4212 remaining (b = 0.20)
2746.88/688.77	c INFO: running march_hi@1270 for 97 with 4111 remaining (b = 0.08)
2753.25/690.39	c INFO: running march_hi@97 for 97 with 4110 remaining (b = 0.07)
3124.96/783.44	c INFO: running march_hi@1758 for 388 with 4017 remaining (b = 0.28)
3128.58/784.39	c INFO: running march_hi@97 for 97 with 4016 remaining (b = 0.07)
3150.14/789.79	c INFO: running march_hi@192 for 291 with 4010 remaining (b = 0.21)
3515.83/881.30	c INFO: running march_hi@195 for 194 with 3919 remaining (b = 0.15)
3516.63/881.58	c INFO: running march_hi@389 for 97 with 3919 remaining (b = 0.08)
3890.35/975.18	c INFO: running march_hi@0 for 97 with 3825 remaining (b = 0.08)
4291.21/1075.59	c INFO: running march_hi@483 for 97 with 3725 remaining (b = 0.08)
4297.59/1077.12	c INFO: running march_hi@488 for 97 with 3723 remaining (b = 0.07)
4303.57/1078.63	c INFO: running march_hi@288 for 194 with 3722 remaining (b = 0.15)
4687.70/1174.87	c INFO: running march_hi@2050 for 97 with 3625 remaining (b = 0.22)
4689.66/1175.31	c INFO: running march_hi@192 for 291 with 3625 remaining (b = 0.18)
4690.86/1175.60	c INFO: running march_hi@0 for 97 with 3625 remaining (b = 0.07)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317245-1304268512/watcher-3317245-1304268512 -o /tmp/evaluation-result-3317245-1304268512/solver-3317245-1304268512 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317245-1304268512.cnf -seed 909190674 -budget 4800 -cores 4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.79 7.93 5.88 5/281 16196
/proc/meminfo: memFree=23709920/32950940 swapFree=67111364/67111528
[pid=16196] ppid=16194 vsize=15756 CPUtime=0 cores=0,2,4,6
/proc/16196/stat : 16196 (python) S 16194 16196 15742 0 -1 4202496 840 0 0 0 0 0 0 0 25 0 1 0 617082014 16134144 724 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988590592 0 2147483391 16781312 2 18446744071699199166 0 0 17 4 0 0 0
/proc/16196/statm: 3939 724 334 364 0 426 0

[startup+0.048687 s]
/proc/loadavg: 7.79 7.93 5.88 5/281 16196
/proc/meminfo: memFree=23709920/32950940 swapFree=67111364/67111528
[pid=16196] ppid=16194 vsize=18884 CPUtime=0.02 cores=0,2,4,6
/proc/16196/stat : 16196 (python) S 16194 16196 15742 0 -1 4202496 1288 0 0 0 2 0 0 0 25 0 1 0 617082014 19337216 1141 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988590592 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/16196/statm: 4721 1141 421 364 0 690 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 18884

[startup+0.101149 s]
/proc/loadavg: 7.79 7.93 5.88 5/281 16196
/proc/meminfo: memFree=23709920/32950940 swapFree=67111364/67111528
[pid=16196] ppid=16194 vsize=93960 CPUtime=0.05 cores=0,2,4,6
/proc/16196/stat : 16196 (python) S 16194 16196 15742 0 -1 4202496 2692 0 0 0 4 1 0 0 25 0 1 0 617082014 96215040 2407 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988588389 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/16196/statm: 23490 2407 764 364 0 1622 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 93960

[startup+0.301083 s]
/proc/loadavg: 7.79 7.93 5.88 5/281 16196
/proc/meminfo: memFree=23709920/32950940 swapFree=67111364/67111528
[pid=16196] ppid=16194 vsize=135752 CPUtime=0.17 cores=0,2,4,6
/proc/16196/stat : 16196 (python) R 16194 16196 15742 0 -1 4202496 5963 934 1 0 13 4 0 0 25 0 1 0 617082014 139010048 4152 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988588549 0 2147483391 16781312 2 0 0 0 17 6 0 0 0
/proc/16196/statm: 33938 4152 1032 364 0 3124 0
Current children cumulated CPU time (s) 0.17
Current children cumulated vsize (KiB) 135752

[startup+0.700973 s]
/proc/loadavg: 7.79 7.93 5.88 5/281 16196
/proc/meminfo: memFree=23709920/32950940 swapFree=67111364/67111528
[pid=16196] ppid=16194 vsize=278300 CPUtime=0.41 cores=0,2,4,6
/proc/16196/stat : 16196 (python) R 16194 16196 15742 0 -1 4202496 9320 934 1 0 29 12 0 0 25 0 1 0 617082014 284979200 6992 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988252083 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/16196/statm: 69575 6992 1826 364 0 5499 0
Current children cumulated CPU time (s) 0.41
Current children cumulated vsize (KiB) 278300

[startup+1.50083 s]
/proc/loadavg: 7.79 7.93 5.88 9/294 16213
/proc/meminfo: memFree=23669764/32950940 swapFree=67111364/67111528
[pid=16196] ppid=16194 vsize=320268 CPUtime=0.56 cores=0,2,4,6
/proc/16196/stat : 16196 (python) S 16194 16196 15742 0 -1 4202496 12360 934 1 0 44 12 0 0 25 0 5 0 617082014 327954432 7264 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000419275 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/16196/statm: 80067 7264 1881 364 0 15976 0
[pid=16202] ppid=16196 vsize=278516 CPUtime=0 cores=0,2,4,6
/proc/16202/stat : 16202 (python) S 16196 16196 15742 0 -1 4202560 741 0 0 0 0 0 0 0 25 0 1 0 617082092 285200384 5563 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988612623 0 0 16781312 514 4279027130471284735 0 0 17 2 0 0 0
/proc/16202/statm: 69629 5563 354 364 0 5547 0
[pid=16204] ppid=16202 vsize=1017800 CPUtime=0.71 cores=0,2,4,6
/proc/16204/stat : 16204 (gnovelty+2) R 16202 16204 16204 0 -1 4202496 440 0 0 0 71 0 0 0 25 0 1 0 617082093 1042227200 178 33554432000 134512640 135663704 4287947760 18446744073709551615 134525544 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/16204/statm: 254450 178 113 282 0 254166 0
[pid=16205] ppid=16196 vsize=289344 CPUtime=0 cores=0,2,4,6
/proc/16205/stat : 16205 (python) S 16196 16196 15742 0 -1 4202560 750 0 0 0 0 0 0 0 22 0 1 0 617082095 296288256 5694 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988612678 0 0 16781312 514 10591789028707631364 0 0 17 2 0 0 0
/proc/16205/statm: 72336 5694 348 364 0 8251 0
[pid=16207] ppid=16205 vsize=2352 CPUtime=0.68 cores=0,2,4,6
/proc/16207/stat : 16207 (march_hi) R 16205 16207 16207 0 -1 4202496 651 0 0 0 68 0 0 0 25 0 1 0 617082095 2408448 388 33554432000 4194304 4955518 140734504349296 18446744073709551615 4227664 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/16207/statm: 588 388 104 186 0 401 0
[pid=16208] ppid=16196 vsize=299228 CPUtime=0 cores=0,2,4,6
/proc/16208/stat : 16208 (python) S 16196 16196 15742 0 -1 4202560 759 0 0 0 0 0 0 0 23 0 1 0 617082098 306409472 5602 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988612678 0 0 16781312 514 9324707430812057858 0 0 17 0 0 0 0
/proc/16208/statm: 74807 5602 348 364 0 10719 0
[pid=16210] ppid=16208 vsize=2352 CPUtime=0.65 cores=0,2,4,6
/proc/16210/stat : 16210 (march_hi) R 16208 16210 16210 0 -1 4202496 653 0 0 0 65 0 0 0 25 0 1 0 617082098 2408448 387 33554432000 4194304 4955518 140734376444112 18446744073709551615 4209664 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/16210/statm: 588 387 104 186 0 401 0
[pid=16211] ppid=16196 vsize=310024 CPUtime=0 cores=0,2,4,6
/proc/16211/stat : 16211 (python) S 16196 16196 15742 0 -1 4202560 768 0 0 0 0 0 0 0 23 0 1 0 617082101 317464576 5727 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988612678 0 0 16781312 514 4359985820891840769 0 0 17 6 0 0 0
/proc/16211/statm: 77506 5727 348 364 0 13415 0
[pid=16213] ppid=16211 vsize=2352 CPUtime=0.61 cores=0,2,4,6
/proc/16213/stat : 16213 (march_hi) R 16211 16213 16213 0 -1 4202496 651 0 0 0 61 0 0 0 25 0 1 0 617082101 2408448 385 33554432000 4194304 4955518 140733206528640 18446744073709551615 4209628 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/16213/statm: 588 385 104 186 0 401 0
[pid=16196/tid=16203] ppid=16194 vsize=320268 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16203/stat : 16203 (python) S 16194 16196 15742 0 -1 4202560 27 934 0 0 0 0 0 0 25 0 5 0 617082093 327954432 7264 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 2 0 0 0
[pid=16196/tid=16206] ppid=16194 vsize=320268 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16206/stat : 16206 (python) S 16194 16196 15742 0 -1 4202560 22 934 0 0 0 0 0 0 25 0 5 0 617082095 327954432 7264 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 6 0 0 0
[pid=16196/tid=16209] ppid=16194 vsize=320268 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16209/stat : 16209 (python) S 16194 16196 15742 0 -1 4202560 24 934 0 0 0 0 0 0 25 0 5 0 617082098 327954432 7264 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 0 0 0 0
[pid=16196/tid=16212] ppid=16194 vsize=320268 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16212/stat : 16212 (python) S 16194 16196 15742 0 -1 4202560 22 934 0 0 0 0 0 0 25 0 5 0 617082101 327954432 7264 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3.21
Current children cumulated vsize (KiB) 2522236

[startup+3.10055 s]
/proc/loadavg: 7.79 7.93 5.88 9/294 16213
/proc/meminfo: memFree=23668268/32950940 swapFree=67111364/67111528
[pid=16196] ppid=16194 vsize=320268 CPUtime=0.56 cores=0,2,4,6
/proc/16196/stat : 16196 (python) S 16194 16196 15742 0 -1 4202496 12360 934 1 0 44 12 0 0 25 0 5 0 617082014 327954432 7264 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000419275 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/16196/statm: 80067 7264 1881 364 0 15976 0
[pid=16202] ppid=16196 vsize=278516 CPUtime=0 cores=0,2,4,6
/proc/16202/stat : 16202 (python) S 16196 16196 15742 0 -1 4202560 741 0 0 0 0 0 0 0 18 0 1 0 617082092 285200384 5563 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988612623 0 0 16781312 514 4279027130471252228 0 0 17 2 0 0 0
/proc/16202/statm: 69629 5563 354 364 0 5547 0
[pid=16204] ppid=16202 vsize=1017800 CPUtime=2.31 cores=0,2,4,6
/proc/16204/stat : 16204 (gnovelty+2) R 16202 16204 16204 0 -1 4202496 440 0 0 0 231 0 0 0 25 0 1 0 617082093 1042227200 178 33554432000 134512640 135663704 4287947760 18446744073709551615 134546460 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/16204/statm: 254450 178 113 282 0 254166 0
[pid=16205] ppid=16196 vsize=289344 CPUtime=0 cores=0,2,4,6
/proc/16205/stat : 16205 (python) S 16196 16196 15742 0 -1 4202560 755 0 0 0 0 0 0 0 15 0 1 0 617082095 296288256 5694 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988612678 0 0 16781312 514 10591789028707631364 0 0 17 2 0 0 0
/proc/16205/statm: 72336 5694 348 364 0 8251 0
[pid=16207] ppid=16205 vsize=2984 CPUtime=2.28 cores=0,2,4,6
/proc/16207/stat : 16207 (march_hi) R 16205 16207 16207 0 -1 4202496 775 0 0 0 228 0 0 0 25 0 1 0 617082095 3055616 512 33554432000 4194304 4955518 140734504349296 18446744073709551615 4224615 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/16207/statm: 746 512 104 186 0 559 0
[pid=16208] ppid=16196 vsize=299228 CPUtime=0 cores=0,2,4,6
/proc/16208/stat : 16208 (python) S 16196 16196 15742 0 -1 4202560 772 0 0 0 0 0 0 0 15 0 1 0 617082098 306409472 5602 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988612678 0 0 16781312 514 9324707430812057858 0 0 17 0 0 0 0
/proc/16208/statm: 74807 5602 348 364 0 10719 0
[pid=16210] ppid=16208 vsize=2984 CPUtime=2.25 cores=0,2,4,6
/proc/16210/stat : 16210 (march_hi) R 16208 16210 16210 0 -1 4202496 777 0 0 0 225 0 0 0 25 0 1 0 617082098 3055616 511 33554432000 4194304 4955518 140734376444112 18446744073709551615 4286676 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/16210/statm: 746 511 104 186 0 559 0
[pid=16211] ppid=16196 vsize=310024 CPUtime=0 cores=0,2,4,6
/proc/16211/stat : 16211 (python) S 16196 16196 15742 0 -1 4202560 773 0 0 0 0 0 0 0 15 0 1 0 617082101 317464576 5727 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988612678 0 0 16781312 514 3439685517472399617 0 0 17 6 0 0 0
/proc/16211/statm: 77506 5727 348 364 0 13415 0
[pid=16213] ppid=16211 vsize=2984 CPUtime=2.21 cores=0,2,4,6
/proc/16213/stat : 16213 (march_hi) R 16211 16213 16213 0 -1 4202496 775 0 0 0 221 0 0 0 25 0 1 0 617082101 3055616 509 33554432000 4194304 4955518 140733206528640 18446744073709551615 4238710 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/16213/statm: 746 509 104 186 0 559 0
[pid=16196/tid=16203] ppid=16194 vsize=320268 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16203/stat : 16203 (python) S 16194 16196 15742 0 -1 4202560 27 934 0 0 0 0 0 0 25 0 5 0 617082093 327954432 7264 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 2 0 0 0
[pid=16196/tid=16206] ppid=16194 vsize=320268 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16206/stat : 16206 (python) S 16194 16196 15742 0 -1 4202560 22 934 0 0 0 0 0 0 25 0 5 0 617082095 327954432 7264 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 6 0 0 0
[pid=16196/tid=16209] ppid=16194 vsize=320268 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16209/stat : 16209 (python) S 16194 16196 15742 0 -1 4202560 24 934 0 0 0 0 0 0 25 0 5 0 617082098 327954432 7264 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 0 0 0 0
[pid=16196/tid=16212] ppid=16194 vsize=320268 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16212/stat : 16212 (python) S 16194 16196 15742 0 -1 4202560 22 934 0 0 0 0 0 0 25 0 5 0 617082101 327954432 7264 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 9.61
Current children cumulated vsize (KiB) 2524132

[startup+6.30099 s]
/proc/loadavg: 7.81 7.93 5.89 9/294 16213
/proc/meminfo: memFree=23664532/32950940 swapFree=67111364/67111528
[pid=16196] ppid=16194 vsize=320268 CPUtime=0.56 cores=0,2,4,6
/proc/16196/stat : 16196 (python) S 16194 16196 15742 0 -1 4202496 12360 934 1 0 44 12 0 0 25 0 5 0 617082014 327954432 7264 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000419275 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/16196/statm: 80067 7264 1881 364 0 15976 0
[pid=16202] ppid=16196 vsize=278516 CPUtime=0 cores=0,2,4,6
/proc/16202/stat : 16202 (python) S 16196 16196 15742 0 -1 4202560 900 0 0 0 0 0 0 0 18 0 1 0 617082092 285200384 5586 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988612623 0 0 16781312 514 4279027130471252228 0 0 17 2 0 0 0
/proc/16202/statm: 69629 5586 377 364 0 5547 0
[pid=16204] ppid=16202 vsize=1017800 CPUtime=5.51 cores=0,2,4,6
/proc/16204/stat : 16204 (gnovelty+2) R 16202 16204 16204 0 -1 4202496 440 0 0 0 551 0 0 0 25 0 1 0 617082093 1042227200 178 33554432000 134512640 135663704 4287947760 18446744073709551615 134544240 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/16204/statm: 254450 178 113 282 0 254166 0
[pid=16205] ppid=16196 vsize=289344 CPUtime=0 cores=0,2,4,6
/proc/16205/stat : 16205 (python) S 16196 16196 15742 0 -1 4202560 912 0 0 0 0 0 0 0 15 0 1 0 617082095 296288256 5717 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988612678 0 0 16781312 514 4317676338577113348 0 0 17 2 0 0 0
/proc/16205/statm: 72336 5717 371 364 0 8251 0
[pid=16207] ppid=16205 vsize=3724 CPUtime=5.45 cores=0,2,4,6
/proc/16207/stat : 16207 (march_hi) R 16205 16207 16207 0 -1 4202496 936 0 0 0 545 0 0 0 25 0 1 0 617082095 3813376 673 33554432000 4194304 4955518 140734504349296 18446744073709551615 4227109 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/16207/statm: 931 673 104 186 0 744 0
[pid=16208] ppid=16196 vsize=299228 CPUtime=0 cores=0,2,4,6
/proc/16208/stat : 16208 (python) S 16196 16196 15742 0 -1 4202560 905 0 0 0 0 0 0 0 16 0 1 0 617082098 306409472 5625 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988612678 0 0 16781312 514 8976190007475667202 0 0 17 0 0 0 0
/proc/16208/statm: 74807 5625 371 364 0 10719 0
[pid=16210] ppid=16208 vsize=3724 CPUtime=5.44 cores=0,2,4,6
/proc/16210/stat : 16210 (march_hi) R 16208 16210 16210 0 -1 4202496 939 0 0 0 544 0 0 0 25 0 1 0 617082098 3813376 673 33554432000 4194304 4955518 140734376444112 18446744073709551615 4209695 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/16210/statm: 931 673 104 186 0 744 0
[pid=16211] ppid=16196 vsize=310024 CPUtime=0 cores=0,2,4,6
/proc/16211/stat : 16211 (python) S 16196 16196 15742 0 -1 4202560 909 0 0 0 0 0 0 0 15 0 1 0 617082101 317464576 5750 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988612678 0 0 16781312 514 3439685517472432127 0 0 17 6 0 0 0
/proc/16211/statm: 77506 5750 371 364 0 13415 0
[pid=16213] ppid=16211 vsize=3724 CPUtime=5.37 cores=0,2,4,6
/proc/16213/stat : 16213 (march_hi) R 16211 16213 16213 0 -1 4202496 936 0 0 0 537 0 0 0 25 0 1 0 617082101 3813376 670 33554432000 4194304 4955518 140733206528640 18446744073709551615 4245448 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/16213/statm: 931 670 104 186 0 744 0
[pid=16196/tid=16203] ppid=16194 vsize=320268 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16203/stat : 16203 (python) S 16194 16196 15742 0 -1 4202560 27 934 0 0 0 0 0 0 25 0 5 0 617082093 327954432 7264 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 2 0 0 0
[pid=16196/tid=16206] ppid=16194 vsize=320268 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16206/stat : 16206 (python) S 16194 16196 15742 0 -1 4202560 22 934 0 0 0 0 0 0 25 0 5 0 617082095 327954432 7264 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 6 0 0 0
[pid=16196/tid=16209] ppid=16194 vsize=320268 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16209/stat : 16209 (python) S 16194 16196 15742 0 -1 4202560 24 934 0 0 0 0 0 0 25 0 5 0 617082098 327954432 7264 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 0 0 0 0
[pid=16196/tid=16212] ppid=16194 vsize=320268 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16212/stat : 16212 (python) S 16194 16196 15742 0 -1 4202560 22 934 0 0 0 0 0 0 25 0 5 0 617082101 327954432 7264 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 22.33
Current children cumulated vsize (KiB) 2526352

[startup+12.7009 s]
/proc/loadavg: 7.82 7.93 5.90 9/294 16213
/proc/meminfo: memFree=23661356/32950940 swapFree=67111364/67111528
[pid=16196] ppid=16194 vsize=320268 CPUtime=0.56 cores=0,2,4,6
/proc/16196/stat : 16196 (python) S 16194 16196 15742 0 -1 4202496 12360 934 1 0 44 12 0 0 25 0 5 0 617082014 327954432 7264 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000419275 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/16196/statm: 80067 7264 1881 364 0 15976 0
[pid=16202] ppid=16196 vsize=278516 CPUtime=0.01 cores=0,2,4,6
/proc/16202/stat : 16202 (python) S 16196 16196 15742 0 -1 4202560 902 0 0 0 0 1 0 0 18 0 1 0 617082092 285200384 5586 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988612623 0 0 16781312 514 17666565807011758340 0 0 17 2 0 0 0
/proc/16202/statm: 69629 5586 377 364 0 5547 0
[pid=16204] ppid=16202 vsize=1017800 CPUtime=11.91 cores=0,2,4,6
/proc/16204/stat : 16204 (gnovelty+2) R 16202 16204 16204 0 -1 4202496 440 0 0 0 1191 0 0 0 25 0 1 0 617082093 1042227200 178 33554432000 134512640 135663704 4287947760 18446744073709551615 134525715 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/16204/statm: 254450 178 113 282 0 254166 0
[pid=16205] ppid=16196 vsize=289344 CPUtime=0.01 cores=0,2,4,6

################
# More data... #
################

/proc/16210/stat : 16210 (march_hi) T 16208 16210 16210 0 -1 4202496 34924 0 0 0 116826 11 0 0 25 0 1 0 617082098 199450624 34658 33554432000 4194304 4955518 140734376444112 18446744073709551615 4229652 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/16210/statm: 48694 34658 104 186 0 48507 0
[pid=16208/tid=16386] ppid=16196 vsize=299228 CPUtime=0 cores=0,2,4,6
/proc/16208/task/16386/stat : 16386 (python) S 16196 16196 15742 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 617131132 306409472 5656 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 514 18446604450707658048 0 0 -1 0 0 0 0
[pid=16211] ppid=16196 vsize=310024 CPUtime=2.1 cores=0,2,4,6
/proc/16211/stat : 16211 (python) S 16196 16196 15742 0 -1 4202560 1176 0 0 0 116 94 0 0 15 0 2 0 617082101 317464576 5780 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988612678 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/16211/statm: 77506 5780 400 364 0 13415 0
[pid=16213] ppid=16211 vsize=194776 CPUtime=1176.6 cores=0,2,4,6
/proc/16213/stat : 16213 (march_hi) R 16211 16213 16213 0 -1 4202496 35163 0 0 0 117652 8 0 0 25 0 1 0 617082101 199450624 34897 33554432000 4194304 4955518 140733206528640 18446744073709551615 4209688 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/16213/statm: 48694 34897 104 186 0 48507 0
[pid=16211/tid=16220] ppid=16196 vsize=310024 CPUtime=0 cores=0,2,4,6
/proc/16211/task/16220/stat : 16220 (python) S 16196 16196 15742 0 -1 4202560 49 0 0 0 0 0 0 0 15 0 2 0 617092398 317464576 5780 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 514 18446604453713557440 0 0 -1 6 0 0 0
[pid=16217] ppid=16196 vsize=320280 CPUtime=0.16 cores=0,2,4,6
/proc/16217/stat : 16217 (python) S 16196 16196 15742 0 -1 4202560 1105 0 0 0 9 7 0 0 19 0 2 0 617092112 327966720 5784 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000419275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/16217/statm: 80070 5784 400 364 0 15976 0
[pid=16219] ppid=16217 vsize=1017800 CPUtime=99.11 cores=0,2,4,6
/proc/16219/stat : 16219 (gnovelty+2) T 16217 16219 16219 0 -1 4202496 441 0 0 0 9911 0 0 0 25 0 1 0 617092112 1042227200 177 33554432000 134512640 135663704 4292451216 18446744073709551615 134545631 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/16219/statm: 254450 177 113 282 0 254166 0
[pid=16217/tid=16364] ppid=16196 vsize=320280 CPUtime=0 cores=0,2,4,6
/proc/16217/task/16364/stat : 16364 (python) S 16196 16196 15742 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 617102029 327966720 5784 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 514 18446604453906398208 0 0 -1 6 0 0 0
[pid=16365] ppid=16196 vsize=330724 CPUtime=1.76 cores=0,2,4,6
/proc/16365/stat : 16365 (python) S 16196 16196 15742 0 -1 4202560 1182 0 0 0 94 82 0 0 15 0 2 0 617102032 338661376 5836 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988612678 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/16365/statm: 82681 5836 401 364 0 18584 0
[pid=16367] ppid=16365 vsize=164992 CPUtime=976.83 cores=0,2,4,6
/proc/16367/stat : 16367 (march_hi) R 16365 16367 16367 0 -1 4202496 29959 0 0 0 97675 8 0 0 25 0 1 0 617102032 168951808 29695 33554432000 4194304 4955518 140734085616096 18446744073709551615 4224691 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/16367/statm: 41248 29695 104 186 0 41061 0
[pid=16365/tid=16393] ppid=16196 vsize=330724 CPUtime=0 cores=0,2,4,6
/proc/16365/task/16393/stat : 16393 (python) S 16196 16196 15742 0 -1 4202560 49 0 0 0 0 0 0 0 15 0 2 0 617151051 338661376 5836 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 514 18446604453985827904 0 0 -1 2 0 0 0
[pid=16419] ppid=16196 vsize=340892 CPUtime=0.22 cores=0,2,4,6
/proc/16419/stat : 16419 (python) S 16196 16196 15742 0 -1 4202560 1197 0 0 0 11 11 0 0 15 0 2 0 617179533 349073408 5844 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988612678 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/16419/statm: 85223 5844 406 364 0 21123 0
[pid=16421] ppid=16419 vsize=23656 CPUtime=108.42 cores=0,2,4,6
/proc/16421/stat : 16421 (march_hi) R 16419 16421 16421 0 -1 4202496 4556 0 0 0 10841 1 0 0 25 0 1 0 617179533 24223744 4291 33554432000 4194304 4955518 140733232657952 18446744073709551615 4284433 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/16421/statm: 5914 4291 104 186 0 5727 0
[pid=16419/tid=16424] ppid=16196 vsize=340892 CPUtime=0 cores=0,2,4,6
/proc/16419/task/16424/stat : 16424 (python) S 16196 16196 15742 0 -1 4202560 95 0 0 0 0 0 0 0 16 0 2 0 617189724 349073408 5844 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 514 18446604445720597632 0 0 -1 2 0 0 0
[pid=16428] ppid=16196 vsize=351276 CPUtime=0 cores=0,2,4,6
/proc/16428/stat : 16428 (python) S 16196 16196 15742 0 -1 4202560 978 0 0 0 0 0 0 0 15 0 1 0 617199575 359706624 5829 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988612678 0 0 16781312 514 17351793224460501249 0 0 17 2 0 0 0
/proc/16428/statm: 87819 5829 377 364 0 23716 0
[pid=16430] ppid=16428 vsize=3724 CPUtime=6.64 cores=0,2,4,6
/proc/16430/stat : 16430 (march_hi) R 16428 16430 16430 0 -1 4202496 989 0 0 0 664 0 0 0 25 0 1 0 617199575 3813376 720 33554432000 4194304 4955518 140736635804480 18446744073709551615 4238469 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/16430/statm: 931 720 104 186 0 744 0
[pid=16196/tid=16203] ppid=16194 vsize=361520 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16203/stat : 16203 (python) S 16194 16196 15742 0 -1 4202560 27 934 0 0 0 0 0 0 25 0 9 0 617082093 370196480 7346 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 2 0 0 0
[pid=16196/tid=16206] ppid=16194 vsize=361520 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16206/stat : 16206 (python) S 16194 16196 15742 0 -1 4202560 30 934 0 0 0 0 0 0 18 0 9 0 617082095 370196480 7346 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 6 0 0 0
[pid=16196/tid=16209] ppid=16194 vsize=361520 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16209/stat : 16209 (python) S 16194 16196 15742 0 -1 4202560 30 934 0 0 0 0 0 0 18 0 9 0 617082098 370196480 7346 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 0 0 0 0
[pid=16196/tid=16212] ppid=16194 vsize=361520 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16212/stat : 16212 (python) S 16194 16196 15742 0 -1 4202560 34 934 0 0 0 0 0 0 18 0 9 0 617082101 370196480 7346 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 6 0 0 0
[pid=16196/tid=16218] ppid=16194 vsize=361520 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16218/stat : 16218 (python) S 16194 16196 15742 0 -1 4202560 32 934 0 0 0 0 0 0 20 0 9 0 617092112 370196480 7346 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 6 0 0 0
[pid=16196/tid=16366] ppid=16194 vsize=361520 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16366/stat : 16366 (python) S 16194 16196 15742 0 -1 4202560 32 934 0 0 0 0 0 0 18 0 9 0 617102032 370196480 7346 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 6 0 0 0
[pid=16196/tid=16420] ppid=16194 vsize=361520 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16420/stat : 16420 (python) S 16194 16196 15742 0 -1 4202560 31 934 0 0 0 0 0 0 18 0 9 0 617179533 370196480 7346 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 2 0 0 0
[pid=16196/tid=16429] ppid=16194 vsize=361520 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16429/stat : 16429 (python) S 16194 16196 15742 0 -1 4202560 28 934 0 0 0 0 0 0 20 0 9 0 617199575 370196480 7346 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4717.57
Current children cumulated vsize (KiB) 5704348



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203 s]
/proc/loadavg: 8.13 8.12 7.48 9/283 16431
/proc/meminfo: memFree=22755264/32950940 swapFree=67111364/67111528
[pid=16196] ppid=16194 vsize=361520 CPUtime=1.23 cores=0,2,4,6
/proc/16196/stat : 16196 (python) S 16194 16196 15742 0 -1 4202496 17214 934 1 0 109 14 0 0 20 0 9 0 617082014 370196480 7346 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000419275 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/16196/statm: 90380 7346 1890 364 0 26277 0
[pid=16202] ppid=16196 vsize=288760 CPUtime=0.17 cores=0,2,4,6
/proc/16202/stat : 16202 (python) S 16196 16196 15742 0 -1 4202560 1088 0 0 0 9 8 0 0 19 0 2 0 617082092 295690240 5624 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000419275 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/16202/statm: 72190 5624 410 364 0 8108 0
[pid=16204] ppid=16202 vsize=1017800 CPUtime=100.15 cores=0,2,4,6
/proc/16204/stat : 16204 (gnovelty+2) T 16202 16204 16204 0 -1 4202496 440 0 0 0 10015 0 0 0 25 0 1 0 617082093 1042227200 178 33554432000 134512640 135663704 4287947760 18446744073709551615 134525271 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/16204/statm: 254450 178 113 282 0 254166 0
[pid=16202/tid=16216] ppid=16196 vsize=288760 CPUtime=0 cores=0,2,4,6
/proc/16202/task/16216/stat : 16216 (python) S 16196 16196 15742 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 617092109 295690240 5624 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 514 18446604453908981888 0 0 -1 2 0 0 0
[pid=16205] ppid=16196 vsize=289344 CPUtime=1.91 cores=0,2,4,6
/proc/16205/stat : 16205 (python) S 16196 16196 15742 0 -1 4202560 1150 0 0 0 105 86 0 0 16 0 2 0 617082095 296288256 5747 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000419275 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/16205/statm: 72336 5747 400 364 0 8251 0
[pid=16207] ppid=16205 vsize=194776 CPUtime=1071.81 cores=0,2,4,6
/proc/16207/stat : 16207 (march_hi) T 16205 16207 16207 0 -1 4202496 32496 0 0 0 107170 11 0 0 25 0 1 0 617082095 199450624 32233 33554432000 4194304 4955518 140734504349296 18446744073709551615 4209760 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/16207/statm: 48694 32233 104 186 0 48507 0
[pid=16205/tid=16382] ppid=16196 vsize=289344 CPUtime=0 cores=0,2,4,6
/proc/16205/task/16382/stat : 16382 (python) S 16196 16196 15742 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 617121418 296288256 5747 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 514 18446604453917980608 0 0 -1 2 0 0 0
[pid=16208] ppid=16196 vsize=299228 CPUtime=2.09 cores=0,2,4,6
/proc/16208/stat : 16208 (python) S 16196 16196 15742 0 -1 4202560 1164 0 0 0 115 94 0 0 15 0 2 0 617082098 306409472 5656 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000419275 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/16208/statm: 74807 5656 401 364 0 10719 0
[pid=16210] ppid=16208 vsize=194776 CPUtime=1168.37 cores=0,2,4,6
/proc/16210/stat : 16210 (march_hi) T 16208 16210 16210 0 -1 4202496 34924 0 0 0 116826 11 0 0 25 0 1 0 617082098 199450624 34658 33554432000 4194304 4955518 140734376444112 18446744073709551615 4229652 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/16210/statm: 48694 34658 104 186 0 48507 0
[pid=16208/tid=16386] ppid=16196 vsize=299228 CPUtime=0 cores=0,2,4,6
/proc/16208/task/16386/stat : 16386 (python) S 16196 16196 15742 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 617131132 306409472 5656 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 514 18446604450707658048 0 0 -1 0 0 0 0
[pid=16211] ppid=16196 vsize=310024 CPUtime=2.14 cores=0,2,4,6
/proc/16211/stat : 16211 (python) S 16196 16196 15742 0 -1 4202560 1176 0 0 0 118 96 0 0 15 0 2 0 617082101 317464576 5780 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988612678 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/16211/statm: 77506 5780 400 364 0 13415 0
[pid=16213] ppid=16211 vsize=194776 CPUtime=1197.3 cores=0,2,4,6
/proc/16213/stat : 16213 (march_hi) R 16211 16213 16213 0 -1 4202496 35693 0 0 0 119722 8 0 0 25 0 1 0 617082101 199450624 35427 33554432000 4194304 4955518 140733206528640 18446744073709551615 4285410 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/16213/statm: 48694 35427 104 186 0 48507 0
[pid=16211/tid=16220] ppid=16196 vsize=310024 CPUtime=0 cores=0,2,4,6
/proc/16211/task/16220/stat : 16220 (python) S 16196 16196 15742 0 -1 4202560 49 0 0 0 0 0 0 0 15 0 2 0 617092398 317464576 5780 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 514 18446604453713557440 0 0 -1 6 0 0 0
[pid=16217] ppid=16196 vsize=320280 CPUtime=0.16 cores=0,2,4,6
/proc/16217/stat : 16217 (python) S 16196 16196 15742 0 -1 4202560 1105 0 0 0 9 7 0 0 19 0 2 0 617092112 327966720 5784 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000419275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/16217/statm: 80070 5784 400 364 0 15976 0
[pid=16219] ppid=16217 vsize=1017800 CPUtime=99.11 cores=0,2,4,6
/proc/16219/stat : 16219 (gnovelty+2) T 16217 16219 16219 0 -1 4202496 441 0 0 0 9911 0 0 0 25 0 1 0 617092112 1042227200 177 33554432000 134512640 135663704 4292451216 18446744073709551615 134545631 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/16219/statm: 254450 177 113 282 0 254166 0
[pid=16217/tid=16364] ppid=16196 vsize=320280 CPUtime=0 cores=0,2,4,6
/proc/16217/task/16364/stat : 16364 (python) S 16196 16196 15742 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 617102029 327966720 5784 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 514 18446604453906398208 0 0 -1 6 0 0 0
[pid=16365] ppid=16196 vsize=330724 CPUtime=1.78 cores=0,2,4,6
/proc/16365/stat : 16365 (python) S 16196 16196 15742 0 -1 4202560 1182 0 0 0 95 83 0 0 15 0 2 0 617102032 338661376 5836 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988612678 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/16365/statm: 82681 5836 401 364 0 18584 0
[pid=16367] ppid=16365 vsize=164992 CPUtime=997.51 cores=0,2,4,6
/proc/16367/stat : 16367 (march_hi) R 16365 16367 16367 0 -1 4202496 30534 0 0 0 99743 8 0 0 25 0 1 0 617102032 168951808 30270 33554432000 4194304 4955518 140734085616096 18446744073709551615 4224687 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/16367/statm: 41248 30270 104 186 0 41061 0
[pid=16365/tid=16393] ppid=16196 vsize=330724 CPUtime=0 cores=0,2,4,6
/proc/16365/task/16393/stat : 16393 (python) S 16196 16196 15742 0 -1 4202560 49 0 0 0 0 0 0 0 15 0 2 0 617151051 338661376 5836 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 514 18446604453985827904 0 0 -1 2 0 0 0
[pid=16419] ppid=16196 vsize=340892 CPUtime=0.25 cores=0,2,4,6
/proc/16419/stat : 16419 (python) S 16196 16196 15742 0 -1 4202560 1203 0 0 0 12 13 0 0 15 0 2 0 617179533 349073408 5844 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988612678 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/16419/statm: 85223 5844 406 364 0 21123 0
[pid=16421] ppid=16419 vsize=27380 CPUtime=128.93 cores=0,2,4,6
/proc/16421/stat : 16421 (march_hi) R 16419 16421 16421 0 -1 4202496 5255 0 0 0 12892 1 0 0 25 0 1 0 617179533 28037120 4990 33554432000 4194304 4955518 140733232657952 18446744073709551615 4209740 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/16421/statm: 6845 4990 104 186 0 6658 0
[pid=16419/tid=16424] ppid=16196 vsize=340892 CPUtime=0 cores=0,2,4,6
/proc/16419/task/16424/stat : 16424 (python) S 16196 16196 15742 0 -1 4202560 95 0 0 0 0 0 0 0 16 0 2 0 617189724 349073408 5844 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 514 18446604445720597632 0 0 -1 2 0 0 0
[pid=16428] ppid=16196 vsize=351276 CPUtime=0.04 cores=0,2,4,6
/proc/16428/stat : 16428 (python) S 16196 16196 15742 0 -1 4202560 989 0 0 0 2 2 0 0 15 0 1 0 617199575 359706624 5829 33554432000 4194304 5683644 140735571308320 18446744073709551615 215988612678 0 0 16781312 514 17666565807011758337 0 0 17 2 0 0 0
/proc/16428/statm: 87819 5829 377 364 0 23716 0
[pid=16430] ppid=16428 vsize=8576 CPUtime=27.25 cores=0,2,4,6
/proc/16430/stat : 16430 (march_hi) R 16428 16430 16430 0 -1 4202496 1761 0 0 0 2725 0 0 0 25 0 1 0 617199575 8781824 1492 33554432000 4194304 4955518 140736635804480 18446744073709551615 4227651 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/16430/statm: 2144 1492 104 186 0 1957 0
[pid=16196/tid=16203] ppid=16194 vsize=361520 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16203/stat : 16203 (python) S 16194 16196 15742 0 -1 4202560 27 934 0 0 0 0 0 0 25 0 9 0 617082093 370196480 7346 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 2 0 0 0
[pid=16196/tid=16206] ppid=16194 vsize=361520 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16206/stat : 16206 (python) S 16194 16196 15742 0 -1 4202560 30 934 0 0 0 0 0 0 18 0 9 0 617082095 370196480 7346 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 6 0 0 0
[pid=16196/tid=16209] ppid=16194 vsize=361520 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16209/stat : 16209 (python) S 16194 16196 15742 0 -1 4202560 30 934 0 0 0 0 0 0 18 0 9 0 617082098 370196480 7346 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 0 0 0 0
[pid=16196/tid=16212] ppid=16194 vsize=361520 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16212/stat : 16212 (python) S 16194 16196 15742 0 -1 4202560 34 934 0 0 0 0 0 0 18 0 9 0 617082101 370196480 7346 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 6 0 0 0
[pid=16196/tid=16218] ppid=16194 vsize=361520 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16218/stat : 16218 (python) S 16194 16196 15742 0 -1 4202560 32 934 0 0 0 0 0 0 20 0 9 0 617092112 370196480 7346 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 6 0 0 0
[pid=16196/tid=16366] ppid=16194 vsize=361520 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16366/stat : 16366 (python) S 16194 16196 15742 0 -1 4202560 32 934 0 0 0 0 0 0 18 0 9 0 617102032 370196480 7346 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 6 0 0 0
[pid=16196/tid=16420] ppid=16194 vsize=361520 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16420/stat : 16420 (python) S 16194 16196 15742 0 -1 4202560 31 934 0 0 0 0 0 0 18 0 9 0 617179533 370196480 7346 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 2 0 0 0
[pid=16196/tid=16429] ppid=16194 vsize=361520 CPUtime=0 cores=0,2,4,6
/proc/16196/task/16429/stat : 16429 (python) S 16194 16196 15742 0 -1 4202560 28 934 0 0 0 0 0 0 20 0 9 0 617199575 370196480 7346 33554432000 4194304 5683644 140735571308320 18446744073709551615 216000416177 0 0 16781312 2 18446604453930827904 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.2
Current children cumulated vsize (KiB) 5712924

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 16196 and gives
#  childrusage.ru_utime.tv_sec=1
#  childrusage.ru_utime.tv_usec=106831
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=148977
# CPU time returned by wait4() is 1.25581
# while last known CPU time is 4800.2
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1203.06
CPU time (s): 4800.2
CPU user time (s): 4795.78
CPU system time (s): 4.42
CPU usage (%): 399
Max. virtual memory (cumulated for all children) (KiB): 5712924

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1.10683
system time used= 0.148977
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 18606
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2937
involuntary context switches= 73

runsolver used 2.78758 second user time and 6.01209 second system time

The end

Launcher Data 

Begin job on node117 at 2011-05-01 18:48:32
IDJOB=3317245
IDBENCH=82710
IDSOLVER=1705
FILE ID=node117/3317245-1304268512
RUNJOBID= node117-1304267285-15760
PBS_JOBID= 13173962
Free space on /tmp= 72592 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v500-c2130-S1762880211-075.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317245-1304268512/watcher-3317245-1304268512 -o /tmp/evaluation-result-3317245-1304268512/solver-3317245-1304268512 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317245-1304268512.cnf -seed 909190674 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 57a2cff28dc53061839e3102dd9b3447
RANDOM SEED=909190674

node117.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5267.44
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950940 kB
MemFree:      23710092 kB
Buffers:       1904516 kB
Cached:        5782068 kB
SwapCached:        160 kB
Active:        2138636 kB
Inactive:      6508048 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950940 kB
LowFree:      23710092 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:           31912 kB
Writeback:           0 kB
AnonPages:      960084 kB
Mapped:          23316 kB
Slab:           508016 kB
PageTables:      25612 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586996 kB
Committed_AS: 25141248 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72556 MiB
End job on node117 at 2011-05-01 19:08:37