Trace number 3317054

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (TO) 4800.09 1231.73

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v35000-c147000-S419010076-063-UNKNOWN.cnf
MD5SUM07407a86c8d0f1e46c34022f68908549
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark606.272
Satisfiable
(Un)Satisfiability was proved
Number of variables35000
Number of clauses147000
Sum of the clauses size441000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3147000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.61/21.22	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.61/21.22	c INFO: solving HOME/instance-3317054-1304267280.cnf
0.61/21.22	c INFO: computing task features
1.09/21.72	c INFO: computed variable-clause graph statistics
1.09/21.74	c INFO: computed clause balance statistics
3.69/24.32	c INFO: computed variable balance statistics
5.69/26.38	c INFO: computed small-clause counts
5.69/26.39	c INFO: computed Horn-clause counts
7.29/27.95	c INFO: computed variable graph statistics
7.29/27.95	c INFO: collected features for HOME/instance-3317054-1304267280.cnf in 6.71 s
7.29/27.98	c INFO: running hybridGM3@0 for 97 with 4793 remaining (b = 0.36)
7.38/28.01	c INFO: running hybridGM3@0 for 97 with 4793 remaining (b = 0.21)
7.38/28.04	c INFO: running hybridGM3@0 for 97 with 4793 remaining (b = 0.14)
7.38/28.07	c INFO: running hybridGM3@0 for 97 with 4793 remaining (b = 0.10)
407.41/128.22	c INFO: running hybridGM3@0 for 97 with 4693 remaining (b = 0.08)
407.41/128.25	c INFO: running hybridGM3@0 for 97 with 4692 remaining (b = 0.08)
407.81/128.31	c INFO: running hybridGM3@0 for 97 with 4692 remaining (b = 0.08)
407.81/128.34	c INFO: running hybridGM3@100 for 97 with 4692 remaining (b = 0.08)
787.21/223.36	c INFO: running hybridGM3@100 for 97 with 4597 remaining (b = 0.08)
787.21/223.39	c INFO: running hybridGM3@99 for 97 with 4597 remaining (b = 0.08)
787.59/223.46	c INFO: running TNM@0 for 97 with 4597 remaining (b = 0.10)
787.59/223.49	c INFO: running hybridGM3@100 for 97 with 4597 remaining (b = 0.08)
1187.68/323.63	c INFO: running hybridGM3@0 for 194 with 4497 remaining (b = 0.14)
1191.30/324.54	c INFO: running hybridGM3@0 for 194 with 4496 remaining (b = 0.10)
1191.30/324.58	c INFO: running hybridGM3@0 for 194 with 4496 remaining (b = 0.07)
1191.58/324.66	c INFO: running TNM@0 for 97 with 4496 remaining (b = 0.03)
1590.75/424.86	c INFO: running hybridGM3@0 for 194 with 4396 remaining (b = 0.05)
1966.38/519.12	c INFO: running hybridGM3@100 for 97 with 4302 remaining (b = 0.02)
1973.53/520.96	c INFO: running hybridGM3@101 for 97 with 4300 remaining (b = 0.02)
1973.91/521.02	c INFO: running hybridGM3@100 for 97 with 4300 remaining (b = 0.02)
2352.72/616.13	c INFO: running hybridGM3@295 for 97 with 4205 remaining (b = 0.02)
2369.46/620.31	c INFO: running hybridGM3@297 for 97 with 4200 remaining (b = 0.02)
2373.01/621.20	c INFO: running TNM@0 for 97 with 4200 remaining (b = 0.02)
2373.35/621.34	c INFO: running hybridGM3@0 for 194 with 4199 remaining (b = 0.04)
2748.17/715.49	c INFO: running hybridGM3@395 for 97 with 4105 remaining (b = 0.01)
2755.73/717.32	c INFO: running hybridGM3@396 for 97 with 4103 remaining (b = 0.01)
2772.05/721.41	c INFO: running TNM@0 for 97 with 4099 remaining (b = 0.02)
3134.84/812.51	c INFO: running TNM@0 for 97 with 4008 remaining (b = 0.01)
3147.18/815.69	c INFO: running TNM@0 for 97 with 4005 remaining (b = 0.01)
3155.89/817.84	c INFO: running hybridGM3@93 for 97 with 4003 remaining (b = 0.02)
3190.96/826.63	c INFO: running TNM@0 for 97 with 3994 remaining (b = 0.01)
3533.07/912.72	c INFO: running TNM@194 for 1843 with 3908 remaining (b = 0.21)
3534.30/913.03	c INFO: running hybridGM3@0 for 194 with 3908 remaining (b = 0.03)
3545.75/915.90	c INFO: running hybridGM3@0 for 194 with 3905 remaining (b = 0.02)
3589.04/926.86	c INFO: running hybridGM3@0 for 194 with 3894 remaining (b = 0.02)
4327.27/1112.60	c INFO: running hybridGM3@97 for 97 with 3708 remaining (b = 0.01)
4362.23/1121.44	c INFO: running hybridGM3@99 for 97 with 3699 remaining (b = 0.01)
4369.77/1123.36	c INFO: running adaptg2wsat2009++@0 for 97 with 3697 remaining (b = 0.01)
4705.14/1207.79	c INFO: running hybridGM3@296 for 97 with 3613 remaining (b = 0.01)
4736.51/1215.63	c INFO: running hybridGM3@296 for 97 with 3605 remaining (b = 0.01)
4787.77/1228.58	c INFO: running TNM@196 for 1358 with 3592 remaining (b = 0.11)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317054-1304267280/watcher-3317054-1304267280 -o /tmp/evaluation-result-3317054-1304267280/solver-3317054-1304267280 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317054-1304267280.cnf -seed 1260138252 -budget 4800 -cores 4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.00 0.00 0.00 2/174 29322
/proc/meminfo: memFree=29864420/32951124 swapFree=67111348/67111528
[pid=29322] ppid=29319 vsize=6240 CPUtime=0 cores=0,2,4,6
/proc/29322/stat : 29322 (python) S 29319 29322 29287 0 -1 4194304 97 0 2 0 0 0 0 0 20 0 1 0 519143572 6389760 51 33554432000 4194304 5683644 140735338205264 18446744073709551615 254256692871 0 2147483391 4096 0 18446744071699207358 0 0 17 0 0 0 0
/proc/29322/statm: 1560 51 40 364 0 42 0

[startup+0.041122 s]
/proc/loadavg: 0.00 0.00 0.00 2/174 29322
/proc/meminfo: memFree=29864420/32951124 swapFree=67111348/67111528
[pid=29322] ppid=29319 vsize=15080 CPUtime=0 cores=0,2,4,6
/proc/29322/stat : 29322 (python) D 29319 29322 29287 0 -1 4202496 611 0 6 0 0 0 0 0 17 0 1 0 519143572 15441920 513 33554432000 4194304 5683644 140735338205264 18446744073709551615 254261614592 0 0 16781312 2 0 0 0 17 4 0 0 1
/proc/29322/statm: 3770 513 300 364 0 257 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15080

[startup+0.101192 s]
/proc/loadavg: 0.00 0.00 0.00 2/174 29322
/proc/meminfo: memFree=29864420/32951124 swapFree=67111348/67111528
[pid=29322] ppid=29319 vsize=15384 CPUtime=0 cores=0,2,4,6
/proc/29322/stat : 29322 (python) S 29319 29322 29287 0 -1 4202496 815 0 6 0 0 0 0 0 17 0 1 0 519143572 15753216 705 33554432000 4194304 5683644 140735338205264 18446744073709551615 254261612389 0 2147483391 16781312 2 18446744071699207358 0 0 17 4 0 0 1
/proc/29322/statm: 3846 705 334 364 0 333 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15384

[startup+0.301122 s]
/proc/loadavg: 0.00 0.00 0.00 2/174 29322
/proc/meminfo: memFree=29864420/32951124 swapFree=67111348/67111528
[pid=29322] ppid=29319 vsize=16396 CPUtime=0.01 cores=0,2,4,6
/proc/29322/stat : 29322 (python) S 29319 29322 29287 0 -1 4202496 1110 0 6 0 1 0 0 0 15 0 1 0 519143572 16789504 984 33554432000 4194304 5683644 140735338205264 18446744073709551615 254261612389 0 2147483391 16781312 2 18446744071699207358 0 0 17 4 0 0 3
/proc/29322/statm: 4099 984 408 364 0 586 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16396

[startup+0.701062 s]
/proc/loadavg: 0.00 0.00 0.00 2/174 29322
/proc/meminfo: memFree=29864420/32951124 swapFree=67111348/67111528
[pid=29322] ppid=29319 vsize=16396 CPUtime=0.01 cores=0,2,4,6
/proc/29322/stat : 29322 (python) S 29319 29322 29287 0 -1 4202496 1112 0 6 0 1 0 0 0 15 0 1 0 519143572 16789504 985 33554432000 4194304 5683644 140735338205264 18446744073709551615 254261614592 0 2147483391 16781312 2 18446744071699207358 0 0 17 4 0 0 3
/proc/29322/statm: 4099 985 409 364 0 586 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16396

[startup+1.50094 s]
/proc/loadavg: 0.00 0.00 0.00 2/178 29326
/proc/meminfo: memFree=29856284/32951124 swapFree=67111348/67111528
[pid=29322] ppid=29319 vsize=18888 CPUtime=0.03 cores=0,2,4,6
/proc/29322/stat : 29322 (python) D 29319 29322 29287 0 -1 4202496 1232 0 7 0 2 1 0 0 15 0 1 0 519143572 19341312 1094 33554432000 4194304 5683644 140735338205264 18446744073709551615 254261616384 0 0 16781312 2 18446744071562234166 0 0 17 4 0 0 7
/proc/29322/statm: 4722 1094 421 364 0 691 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 18888

[startup+3.10055 s]
/proc/loadavg: 0.00 0.00 0.00 2/178 29326
/proc/meminfo: memFree=29850172/32951124 swapFree=67111348/67111528
[pid=29322] ppid=29319 vsize=36508 CPUtime=0.09 cores=0,2,4,6
/proc/29322/stat : 29322 (python) S 29319 29322 29287 0 -1 4202496 1597 0 14 0 6 3 0 0 15 0 1 0 519143572 37384192 1444 33554432000 4194304 5683644 140735338205264 18446744073709551615 254261614592 0 2147483391 16781312 2 18446744071699207358 0 0 17 4 0 0 12
/proc/29322/statm: 9127 1444 452 364 0 944 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 36508
heavy processes:

[startup+6.30106 s]
/proc/loadavg: 0.16 0.03 0.01 3/178 29326
/proc/meminfo: memFree=29847512/32951124 swapFree=67111348/67111528
[pid=29322] ppid=29319 vsize=84604 CPUtime=0.12 cores=0,2,4,6
/proc/29322/stat : 29322 (python) S 29319 29322 29287 0 -1 4202496 2387 0 22 0 8 4 0 0 15 0 1 0 519143572 86634496 2136 33554432000 4194304 5683644 140735338205264 18446744073709551615 254261614592 0 2147483391 16781312 2 18446744071699207358 0 0 17 4 0 0 60
/proc/29322/statm: 21151 2136 707 364 0 1368 0
Current children cumulated CPU time (s) 0.12
Current children cumulated vsize (KiB) 84604
heavy processes:

[startup+12.701 s]
/proc/loadavg: 0.23 0.05 0.02 2/178 29334
/proc/meminfo: memFree=29820684/32951124 swapFree=67111348/67111528
[pid=29322] ppid=29319 vsize=135752 CPUtime=0.32 cores=0,2,4,6
/proc/29322/stat : 29322 (python) D 29319 29322 29287 0 -1 4202496 5924 925 42 8 20 12 0 0 16 0 1 0 519143572 139010048 4152 33554432000 4194304 5683644 140735338205264 18446744073709551615 254261612549 0 0 16781312 2 0 0 0 17 4 0 0 154
/proc/29322/statm: 33938 4152 1032 364 0 3124 0
Current children cumulated CPU time (s) 0.32
Current children cumulated vsize (KiB) 135752
heavy processes:

[startup+25.5006 s]
/proc/loadavg: 0.72 0.16 0.05 6/190 29347
/proc/meminfo: memFree=29719668/32951124 swapFree=67111348/67111528
[pid=29322] ppid=29319 vsize=312976 CPUtime=4.89 cores=0,2,4,6
/proc/29322/stat : 29322 (python) R 29319 29322 29287 0 -1 4202496 21858 925 113 8 466 23 0 0 22 0 1 0 519143572 320487424 15630 33554432000 4194304 5683644 140735338205264 18446744073709551615 47117911249140 0 0 16781312 2 0 0 0 17 2 0 0 268
/proc/29322/statm: 78244 15630 1827 364 0 14168 0
Current children cumulated CPU time (s) 4.89
Current children cumulated vsize (KiB) 312976
heavy processes:

[startup+51.1011 s]
/proc/loadavg: 3.03 0.74 0.25 10/202 29359
/proc/meminfo: memFree=29577248/32951124 swapFree=67111348/67111528
[pid=29322] ppid=29319 vsize=351280 CPUtime=7.46 cores=0,2,4,6
/proc/29322/stat : 29322 (python) S 29319 29322 29287 0 -1 4202496 24828 925 113 8 722 24 0 0 25 0 5 0 519143572 359710720 15017 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273443275 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 268
/proc/29322/statm: 87820 15017 1881 364 0 23729 0
[pid=29348] ppid=29322 vsize=310268 CPUtime=0.02 cores=0,2,4,6
/proc/29348/stat : 29348 (python) S 29322 29322 29287 0 -1 4202560 902 0 0 0 1 1 0 0 15 0 1 0 519146371 317714432 13496 33554432000 4194304 5683644 140735338205264 18446744073709551615 254261636623 0 0 16781312 514 8886200199127466244 0 0 17 2 0 0 0
/proc/29348/statm: 77567 13496 376 364 0 13485 0
[pid=29350] ppid=29348 vsize=268424 CPUtime=23.1 cores=0,2,4,6
/proc/29350/stat : 29350 (hybridGM3) R 29348 29350 29350 0 -1 4202496 8086 0 0 0 2310 0 0 0 25 0 1 0 519146371 274866176 7824 33554432000 134512640 135204385 4293799904 18446744073709551615 134537483 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/29350/statm: 67106 7824 64 169 0 66936 0
[pid=29351] ppid=29322 vsize=320524 CPUtime=0.02 cores=0,2,4,6
/proc/29351/stat : 29351 (python) S 29322 29322 29287 0 -1 4202560 925 0 0 0 1 1 0 0 15 0 1 0 519146374 328216576 13494 33554432000 4194304 5683644 140735338205264 18446744073709551615 254261636678 0 0 16781312 514 8886200199127466244 0 0 17 6 0 0 0
/proc/29351/statm: 80131 13494 370 364 0 16046 0
[pid=29353] ppid=29351 vsize=268424 CPUtime=23.04 cores=0,2,4,6
/proc/29353/stat : 29353 (hybridGM3) R 29351 29353 29353 0 -1 4202496 8089 0 0 0 2303 1 0 0 25 0 1 0 519146374 274866176 7826 33554432000 134512640 135204385 4292175856 18446744073709551615 134536698 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/29353/statm: 67106 7826 64 169 0 66936 0
[pid=29354] ppid=29322 vsize=330780 CPUtime=0.03 cores=0,2,4,6
/proc/29354/stat : 29354 (python) S 29322 29322 29287 0 -1 4202560 930 0 0 0 2 1 0 0 15 0 1 0 519146377 338718720 13498 33554432000 4194304 5683644 140735338205264 18446744073709551615 254261636678 0 0 16781312 514 8886200199127466244 0 0 17 6 0 0 0
/proc/29354/statm: 82695 13498 370 364 0 18607 0
[pid=29356] ppid=29354 vsize=268424 CPUtime=22.82 cores=0,2,4,6
/proc/29356/stat : 29356 (hybridGM3) R 29354 29356 29356 0 -1 4202496 8088 0 0 0 2281 1 0 0 25 0 1 0 519146378 274866176 7824 33554432000 134512640 135204385 4293662400 18446744073709551615 134537063 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/29356/statm: 67106 7824 64 169 0 66936 0
[pid=29357] ppid=29322 vsize=341036 CPUtime=0.03 cores=0,2,4,6
/proc/29357/stat : 29357 (python) S 29322 29322 29287 0 -1 4202560 947 0 0 0 1 2 0 0 15 0 1 0 519146380 349220864 13502 33554432000 4194304 5683644 140735338205264 18446744073709551615 254261636678 0 0 16781312 514 8886200199127466241 0 0 17 0 0 0 0
/proc/29357/statm: 85259 13502 370 364 0 21168 0
[pid=29359] ppid=29357 vsize=268424 CPUtime=22.98 cores=0,2,4,6
/proc/29359/stat : 29359 (hybridGM3) R 29357 29359 29359 0 -1 4202496 8081 0 0 0 2297 1 0 0 25 0 1 0 519146380 274866176 7819 33554432000 134512640 135204385 4288654080 18446744073709551615 134545866 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/29359/statm: 67106 7819 64 169 0 66936 0
[pid=29322/tid=29349] ppid=29319 vsize=351280 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29349/stat : 29349 (python) S 29319 29322 29287 0 -1 4202560 26 925 0 8 0 0 0 0 25 0 5 0 519146371 359710720 15017 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 2 0 0 0
[pid=29322/tid=29352] ppid=29319 vsize=351280 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29352/stat : 29352 (python) S 29319 29322 29287 0 -1 4202560 23 925 0 8 0 0 0 0 18 0 5 0 519146374 359710720 15017 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 6 0 0 0
[pid=29322/tid=29355] ppid=29319 vsize=351280 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29355/stat : 29355 (python) S 29319 29322 29287 0 -1 4202560 21 925 0 8 0 0 0 0 25 0 5 0 519146377 359710720 15017 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 6 0 0 0
[pid=29322/tid=29358] ppid=29319 vsize=351280 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29358/stat : 29358 (python) S 29319 29322 29287 0 -1 4202560 29 925 0 8 0 0 0 0 25 0 5 0 519146380 359710720 15017 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 99.5
Current children cumulated vsize (KiB) 2727584

[startup+102.308 s]
/proc/loadavg: 5.93 1.89 0.66 10/202 29361
/proc/meminfo: memFree=29577356/32951124 swapFree=67111348/67111528
[pid=29322] ppid=29319 vsize=351280 CPUtime=7.46 cores=0,2,4,6
/proc/29322/stat : 29322 (python) S 29319 29322 29287 0 -1 4202496 24828 925 113 8 722 24 0 0 25 0 5 0 519143572 359710720 15017 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273443275 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 268
/proc/29322/statm: 87820 15017 1881 364 0 23729 0
[pid=29348] ppid=29322 vsize=310268 CPUtime=0.09 cores=0,2,4,6
/proc/29348/stat : 29348 (python) S 29322 29322 29287 0 -1 4202560 916 0 0 0 5 4 0 0 15 0 1 0 519146371 317714432 13496 33554432000 4194304 5683644 140735338205264 18446744073709551615 254261636623 0 0 16781312 514 8886200199127466244 0 0 17 2 0 0 0
/proc/29348/statm: 77567 13496 376 364 0 13485 0
[pid=29350] ppid=29348 vsize=268424 CPUtime=74.31 cores=0,2,4,6
/proc/29350/stat : 29350 (hybridGM3) R 29348 29350 29350 0 -1 4202496 8086 0 0 0 7431 0 0 0 25 0 1 0 519146371 274866176 7824 33554432000 134512640 135204385 4293799904 18446744073709551615 134536737 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/29350/statm: 67106 7824 64 169 0 66936 0
[pid=29351] ppid=29322 vsize=320524 CPUtime=0.09 cores=0,2,4,6
/proc/29351/stat : 29351 (python) S 29322 29322 29287 0 -1 4202560 941 0 0 0 5 4 0 0 15 0 1 0 519146374 328216576 13494 33554432000 4194304 5683644 140735338205264 18446744073709551615 254261636678 0 0 16781312 514 8886200199127466244 0 0 17 6 0 0 0
/proc/29351/statm: 80131 13494 370 364 0 16046 0
[pid=29353] ppid=29351 vsize=268424 CPUtime=74.18 cores=0,2,4,6
/proc/29353/stat : 29353 (hybridGM3) R 29351 29353 29353 0 -1 4202496 8089 0 0 0 7417 1 0 0 25 0 1 0 519146374 274866176 7826 33554432000 134512640 135204385 4292175856 18446744073709551615 134537383 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/29353/statm: 67106 7826 64 169 0 66936 0
[pid=29354] ppid=29322 vsize=330780 CPUtime=0.09 cores=0,2,4,6
/proc/29354/stat : 29354 (python) S 29322 29322 29287 0 -1 4202560 945 0 0 0 5 4 0 0 15 0 1 0 519146377 338718720 13498 33554432000 4194304 5683644 140735338205264 18446744073709551615 254261636678 0 0 16781312 514 8886200199127466244 0 0 17 6 0 0 0
/proc/29354/statm: 82695 13498 370 364 0 18607 0
[pid=29356] ppid=29354 vsize=268424 CPUtime=73.57 cores=0,2,4,6
/proc/29356/stat : 29356 (hybridGM3) R 29354 29356 29356 0 -1 4202496 8092 0 0 0 7356 1 0 0 25 0 1 0 519146378 274866176 7828 33554432000 134512640 135204385 4293662400 18446744073709551615 134536887 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/29356/statm: 67106 7828 64 169 0 66936 0
[pid=29357] ppid=29322 vsize=341036 CPUtime=0.09 cores=0,2,4,6
/proc/29357/stat : 29357 (python) S 29322 29322 29287 0 -1 4202560 963 0 0 0 3 6 0 0 15 0 1 0 519146380 349220864 13502 33554432000 4194304 5683644 140735338205264 18446744073709551615 254261636678 0 0 16781312 514 8886200199127466241 0 0 17 0 0 0 0
/proc/29357/statm: 85259 13502 370 364 0 21168 0
[pid=29359] ppid=29357 vsize=268424 CPUtime=74.12 cores=0,2,4,6
/proc/29359/stat : 29359 (hybridGM3) R 29357 29359 29359 0 -1 4202496 8082 0 0 0 7411 1 0 0 25 0 1 0 519146380 274866176 7820 33554432000 134512640 135204385 4288654080 18446744073709551615 134536698 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/29359/statm: 67106 7820 64 169 0 66936 0
[pid=29322/tid=29349] ppid=29319 vsize=351280 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29349/stat : 29349 (python) S 29319 29322 29287 0 -1 4202560 26 925 0 8 0 0 0 0 25 0 5 0 519146371 359710720 15017 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 2 0 0 0
[pid=29322/tid=29352] ppid=29319 vsize=351280 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29352/stat : 29352 (python) S 29319 29322 29287 0 -1 4202560 23 925 0 8 0 0 0 0 18 0 5 0 519146374 359710720 15017 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 6 0 0 0
[pid=29322/tid=29355] ppid=29319 vsize=351280 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29355/stat : 29355 (python) S 29319 29322 29287 0 -1 4202560 21 925 0 8 0 0 0 0 25 0 5 0 519146377 359710720 15017 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 6 0 0 0
[pid=29322/tid=29358] ppid=29319 vsize=351280 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29358/stat : 29358 (python) S 29319 29322 29287 0 -1 4202560 29 925 0 8 0 0 0 0 25 0 5 0 519146380 359710720 15017 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 304
Current children cumulated vsize (KiB) 2727584

[startup+162.301 s]
/proc/loadavg: 7.29 3.01 1.12 10/222 29382
/proc/meminfo: memFree=29538836/32951124 swapFree=67111348/67111528
[pid=29322] ppid=29319 vsize=351280 CPUtime=7.58 cores=0,2,4,6
/proc/29322/stat : 29322 (python) S 29319 29322 29287 0 -1 4202496 25538 925 113 8 733 25 0 0 18 0 5 0 519143572 359710720 15018 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273443275 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 268

################
# More data... #
################

[pid=29425] ppid=29423 vsize=268424 CPUtime=487.16 cores=0,2,4,6
/proc/29425/stat : 29425 (hybridGM3) T 29423 29425 29425 0 -1 4202496 8098 0 0 0 48715 1 0 0 25 0 1 0 519176028 274866176 7831 33554432000 134512640 135204385 4294814416 18446744073709551615 134537583 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/29425/statm: 67106 7831 64 169 0 66936 0
[pid=29423/tid=29445] ppid=29322 vsize=371804 CPUtime=0 cores=0,2,4,6
/proc/29423/task/29445/stat : 29445 (python) S 29322 29322 29287 0 -1 4202560 49 0 0 0 0 0 0 0 15 0 2 0 519195666 380727296 13550 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 514 18446604453968837312 0 0 -1 4 0 0 0
[pid=29426] ppid=29322 vsize=382060 CPUtime=0.45 cores=0,2,4,6
/proc/29426/stat : 29426 (python) S 29322 29322 29287 0 -1 4202560 1230 0 0 0 24 21 0 0 15 0 2 0 519176031 391229440 13554 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/29426/statm: 95515 13554 405 364 0 31412 0
[pid=29428] ppid=29426 vsize=268420 CPUtime=292.38 cores=0,2,4,6
/proc/29428/stat : 29428 (hybridGM3) T 29426 29428 29428 0 -1 4202496 8093 0 0 0 29236 2 0 0 25 0 1 0 519176031 274862080 7823 33554432000 134512640 135204385 4292230096 18446744073709551615 134536986 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/29428/statm: 67105 7823 64 169 0 66935 0
[pid=29426/tid=29446] ppid=29322 vsize=382060 CPUtime=0 cores=0,2,4,6
/proc/29426/task/29446/stat : 29446 (python) S 29322 29322 29287 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 519195671 391229440 13554 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 514 18446604440278878208 0 0 -1 6 0 0 0
[pid=29429] ppid=29322 vsize=392316 CPUtime=0.15 cores=0,2,4,6
/proc/29429/stat : 29429 (python) S 29322 29322 29287 0 -1 4202560 1161 0 0 0 7 8 0 0 19 0 2 0 519176039 401731584 13552 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/29429/statm: 98079 13552 399 364 0 33973 0
[pid=29431] ppid=29429 vsize=465044 CPUtime=100.13 cores=0,2,4,6
/proc/29431/stat : 29431 (TNM) T 29429 29431 29431 0 -1 4202496 6018 0 0 0 10012 1 0 0 25 0 1 0 519176040 476205056 5750 33554432000 134512640 135008748 4294139968 18446744073709551615 134527693 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/29431/statm: 116261 5750 46 122 0 115161 0
[pid=29429/tid=29435] ppid=29322 vsize=392316 CPUtime=0 cores=0,2,4,6
/proc/29429/task/29435/stat : 29435 (python) S 29322 29322 29287 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 519186054 401731584 13552 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 514 18446604453942576384 0 0 -1 6 0 0 0
[pid=29436] ppid=29322 vsize=402572 CPUtime=0.3 cores=0,2,4,6
/proc/29436/stat : 29436 (python) S 29322 29322 29287 0 -1 4202560 1194 0 0 0 17 13 0 0 16 0 2 0 519186059 412233728 13556 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/29436/statm: 100643 13556 399 364 0 36534 0
[pid=29438] ppid=29436 vsize=268420 CPUtime=194.46 cores=0,2,4,6
/proc/29438/stat : 29438 (hybridGM3) T 29436 29438 29438 0 -1 4202496 8097 0 0 0 19445 1 0 0 25 0 1 0 519186059 274862080 7826 33554432000 134512640 135204385 4289040736 18446744073709551615 134537292 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/29438/statm: 67105 7826 64 169 0 66935 0
[pid=29436/tid=29459] ppid=29322 vsize=402572 CPUtime=0 cores=0,2,4,6
/proc/29436/task/29459/stat : 29459 (python) S 29322 29322 29287 0 -1 4202560 48 0 0 0 0 0 0 0 17 0 2 0 519205703 412233728 13556 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 514 18446604453979851712 0 0 -1 6 0 0 0
[pid=29456] ppid=29322 vsize=412828 CPUtime=0.71 cores=0,2,4,6
/proc/29456/stat : 29456 (python) S 29322 29322 29287 0 -1 4202560 1208 0 0 0 42 29 0 0 18 0 2 0 519205693 422735872 13560 33554432000 4194304 5683644 140735338205264 18446744073709551615 254261636678 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/29456/statm: 103207 13560 399 364 0 39095 0
[pid=29458] ppid=29456 vsize=465044 CPUtime=419.03 cores=0,2,4,6
/proc/29458/stat : 29458 (TNM) R 29456 29458 29458 0 -1 4202496 6021 0 0 0 41899 4 0 0 25 0 1 0 519205693 476205056 5750 33554432000 134512640 135008748 4287484448 18446744073709551615 134528940 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/29458/statm: 116261 5750 46 122 0 115161 0
[pid=29456/tid=29467] ppid=29322 vsize=412828 CPUtime=0 cores=0,2,4,6
/proc/29456/task/29467/stat : 29467 (python) S 29322 29322 29287 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 519215710 422735872 13560 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 514 18446604453956684736 0 0 -1 2 0 0 0
[pid=29460] ppid=29322 vsize=423084 CPUtime=0.49 cores=0,2,4,6
/proc/29460/stat : 29460 (python) S 29322 29322 29287 0 -1 4202560 1213 0 0 0 27 22 0 0 15 0 2 0 519205707 433238016 13564 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/29460/statm: 105771 13564 399 364 0 41656 0
[pid=29462] ppid=29460 vsize=268424 CPUtime=291.31 cores=0,2,4,6
/proc/29462/stat : 29462 (hybridGM3) T 29460 29462 29462 0 -1 4202496 8099 0 0 0 29129 2 0 0 25 0 1 0 519205708 274866176 7826 33554432000 134512640 135204385 4290748080 18446744073709551615 134536698 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/29462/statm: 67106 7826 64 169 0 66936 0
[pid=29460/tid=29520] ppid=29322 vsize=423084 CPUtime=0 cores=0,2,4,6
/proc/29460/task/29520/stat : 29520 (python) S 29322 29322 29287 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 519225352 433238016 13564 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 514 18446604453956682944 0 0 -1 6 0 0 0
[pid=29468] ppid=29322 vsize=433340 CPUtime=0.18 cores=0,2,4,6
/proc/29468/stat : 29468 (python) S 29322 29322 29287 0 -1 4202560 1175 0 0 0 9 9 0 0 19 0 2 0 519215714 443740160 13568 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/29468/statm: 108335 13568 399 364 0 44217 0
[pid=29470] ppid=29468 vsize=465044 CPUtime=103.04 cores=0,2,4,6
/proc/29470/stat : 29470 (TNM) T 29468 29470 29470 0 -1 4202496 6020 0 0 0 10301 3 0 0 25 0 1 0 519215714 476205056 5749 33554432000 134512640 135008748 4291677664 18446744073709551615 134524765 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/29470/statm: 116261 5749 46 122 0 115161 0
[pid=29468/tid=29577] ppid=29322 vsize=433340 CPUtime=0 cores=0,2,4,6
/proc/29468/task/29577/stat : 29577 (python) S 29322 29322 29287 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 519226232 443740160 13568 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 514 18446604453985337408 0 0 -1 4 0 0 0
[pid=29474] ppid=29322 vsize=443596 CPUtime=0.17 cores=0,2,4,6
/proc/29474/stat : 29474 (python) S 29322 29322 29287 0 -1 4202560 1172 0 0 0 10 7 0 0 19 0 2 0 519224824 454242304 13572 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/29474/statm: 110899 13572 399 364 0 46778 0
[pid=29476] ppid=29474 vsize=465044 CPUtime=99.98 cores=0,2,4,6
/proc/29476/stat : 29476 (TNM) T 29474 29476 29476 0 -1 4202496 6021 0 0 0 9993 5 0 0 25 0 1 0 519224824 476205056 5749 33554432000 134512640 135008748 4294567888 18446744073709551615 134527436 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/29476/statm: 116261 5749 46 122 0 115161 0
[pid=29474/tid=29583] ppid=29322 vsize=443596 CPUtime=0 cores=0,2,4,6
/proc/29474/task/29583/stat : 29583 (python) S 29322 29322 29287 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 519234840 454242304 13572 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 514 18446604453899396288 0 0 -1 2 0 0 0
[pid=29486] ppid=29322 vsize=453852 CPUtime=0.18 cores=0,2,4,6
/proc/29486/stat : 29486 (python) S 29322 29322 29287 0 -1 4202560 1234 0 0 0 9 9 0 0 18 0 2 0 519225142 464744448 13584 33554432000 4194304 5683644 140735338205264 18446744073709551615 254261636678 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/29486/statm: 113463 13584 407 364 0 49339 0
[pid=29488] ppid=29486 vsize=465044 CPUtime=100.46 cores=0,2,4,6
/proc/29488/stat : 29488 (TNM) R 29486 29488 29488 0 -1 4202496 6024 0 0 0 10044 2 0 0 25 0 1 0 519225142 476205056 5749 33554432000 134512640 135008748 4289563264 18446744073709551615 134527477 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/29488/statm: 116261 5749 46 122 0 115161 0
[pid=29486/tid=29587] ppid=29322 vsize=453852 CPUtime=0 cores=0,2,4,6
/proc/29486/task/29587/stat : 29587 (python) S 29322 29322 29287 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 519235158 464744448 13584 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 514 18446604453930568576 0 0 -1 4 0 0 0
[pid=29578] ppid=29322 vsize=464108 CPUtime=0.16 cores=0,2,4,6
/proc/29578/stat : 29578 (python) S 29322 29322 29287 0 -1 4202560 1215 0 0 0 9 7 0 0 19 0 2 0 519226236 475246592 13588 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/29578/statm: 116027 13588 407 364 0 51900 0
[pid=29580] ppid=29578 vsize=465044 CPUtime=99.78 cores=0,2,4,6
/proc/29580/stat : 29580 (TNM) T 29578 29580 29580 0 -1 4202496 6027 0 0 0 9977 1 0 0 25 0 1 0 519226237 476205056 5749 33554432000 134512640 135008748 4291590624 18446744073709551615 134528974 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/29580/statm: 116261 5749 46 122 0 115161 0
[pid=29578/tid=29600] ppid=29322 vsize=464108 CPUtime=0 cores=0,2,4,6
/proc/29578/task/29600/stat : 29600 (python) S 29322 29322 29287 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 519236254 475246592 13588 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 514 18446604453895372992 0 0 -1 4 0 0 0
[pid=29584] ppid=29322 vsize=474364 CPUtime=0.53 cores=0,2,4,6
/proc/29584/stat : 29584 (python) S 29322 29322 29287 0 -1 4202560 1223 0 0 0 31 22 0 0 15 0 2 0 519234876 485748736 13586 33554432000 4194304 5683644 140735338205264 18446744073709551615 254261636678 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/29584/statm: 118591 13586 401 364 0 54461 0
[pid=29586] ppid=29584 vsize=268424 CPUtime=317.82 cores=0,2,4,6
/proc/29586/stat : 29586 (hybridGM3) R 29584 29586 29586 0 -1 4202496 8106 0 0 0 31781 1 0 0 25 0 1 0 519234877 274866176 7826 33554432000 134512640 135204385 4294165344 18446744073709551615 134536646 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/29586/statm: 67106 7826 64 169 0 66936 0
[pid=29584/tid=29651] ppid=29322 vsize=474364 CPUtime=0 cores=0,2,4,6
/proc/29584/task/29651/stat : 29651 (python) S 29322 29322 29287 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 519254828 485748736 13586 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 514 18446604453951335296 0 0 -1 6 0 0 0
[pid=29588] ppid=29322 vsize=484620 CPUtime=0.52 cores=0,2,4,6
/proc/29588/stat : 29588 (python) S 29322 29322 29287 0 -1 4202560 1232 0 0 0 29 23 0 0 15 0 2 0 519235163 496250880 13590 33554432000 4194304 5683644 140735338205264 18446744073709551615 254261636678 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/29588/statm: 121155 13590 401 364 0 57022 0
[pid=29590] ppid=29588 vsize=268424 CPUtime=308.75 cores=0,2,4,6
/proc/29590/stat : 29590 (hybridGM3) R 29588 29590 29590 0 -1 4202496 8100 0 0 0 30873 2 0 0 25 0 1 0 519235164 274866176 7828 33554432000 134512640 135204385 4289816128 18446744073709551615 134537435 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/29590/statm: 67106 7828 64 169 0 66936 0
[pid=29588/tid=29664] ppid=29322 vsize=484620 CPUtime=0 cores=0,2,4,6
/proc/29588/task/29664/stat : 29664 (python) S 29322 29322 29287 0 -1 4202560 45 0 0 0 0 0 0 0 15 0 2 0 519255712 496250880 13590 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 514 18446604453963798656 0 0 -1 6 0 0 0
[pid=29601] ppid=29322 vsize=494876 CPUtime=0.33 cores=0,2,4,6
/proc/29601/stat : 29601 (python) S 29322 29322 29287 0 -1 4202560 1229 0 0 0 17 16 0 0 16 0 2 0 519236258 506753024 13593 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/29601/statm: 123719 13593 400 364 0 59583 0
[pid=29603] ppid=29601 vsize=268424 CPUtime=196.44 cores=0,2,4,6
/proc/29603/stat : 29603 (hybridGM3) T 29601 29603 29603 0 -1 4202496 8099 0 0 0 19642 2 0 0 25 0 1 0 519236259 274866176 7822 33554432000 134512640 135204385 4294790704 18446744073709551615 134537383 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/29603/statm: 67106 7822 64 169 0 66936 0
[pid=29601/tid=29665] ppid=29322 vsize=494876 CPUtime=0 cores=0,2,4,6
/proc/29601/task/29665/stat : 29665 (python) S 29322 29322 29287 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 519255904 506753024 13593 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 514 18446604453928072192 0 0 -1 6 0 0 0
[pid=29666] ppid=29322 vsize=505132 CPUtime=0.17 cores=0,2,4,6
/proc/29666/stat : 29666 (python) S 29322 29322 29287 0 -1 4202560 1240 0 0 0 11 6 0 0 19 0 2 0 519255909 517255168 13598 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/29666/statm: 126283 13598 401 364 0 62144 0
[pid=29668] ppid=29666 vsize=431704 CPUtime=102.35 cores=0,2,4,6
/proc/29668/stat : 29668 (adaptg2wsat2009) T 29666 29668 29668 0 -1 4202496 6262 0 0 0 10234 1 0 0 25 0 1 0 519255909 442064896 5983 33554432000 134512640 135107411 4289958512 18446744073709551615 134531435 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/29668/statm: 107926 5983 59 146 0 106802 0
[pid=29666/tid=29676] ppid=29322 vsize=505132 CPUtime=0 cores=0,2,4,6
/proc/29666/task/29676/stat : 29676 (python) S 29322 29322 29287 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 519266427 517255168 13598 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 514 18446604453985338304 0 0 -1 6 0 0 0
[pid=29322/tid=29349] ppid=29319 vsize=515376 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29349/stat : 29349 (python) S 29319 29322 29287 0 -1 4202560 30 925 0 8 0 0 0 0 15 0 21 0 519146371 527745024 15114 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 2 0 0 0
[pid=29322/tid=29352] ppid=29319 vsize=515376 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29352/stat : 29352 (python) S 29319 29322 29287 0 -1 4202560 27 925 0 8 0 0 0 0 18 0 21 0 519146374 527745024 15114 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 6 0 0 0
[pid=29322/tid=29355] ppid=29319 vsize=515376 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29355/stat : 29355 (python) S 29319 29322 29287 0 -1 4202560 26 925 0 8 0 0 0 0 16 0 21 0 519146377 527745024 15114 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 0 0 0 0
[pid=29322/tid=29358] ppid=29319 vsize=515376 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29358/stat : 29358 (python) S 29319 29322 29287 0 -1 4202560 29 925 0 8 0 0 0 0 18 0 21 0 519146380 527745024 15114 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 0 0 0 0
[pid=29322/tid=29402] ppid=29319 vsize=515376 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29402/stat : 29402 (python) S 29319 29322 29287 0 -1 4202560 24 925 0 8 0 0 0 0 20 0 21 0 519165919 527745024 15114 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 0 0 0 0
[pid=29322/tid=29421] ppid=29319 vsize=515376 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29421/stat : 29421 (python) S 29319 29322 29287 0 -1 4202560 36 925 0 8 0 0 0 0 18 0 21 0 519175937 527745024 15114 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 6 0 0 0
[pid=29322/tid=29424] ppid=29319 vsize=515376 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29424/stat : 29424 (python) S 29319 29322 29287 0 -1 4202560 31 925 0 8 0 0 0 0 18 0 21 0 519176027 527745024 15114 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 4 0 0 0
[pid=29322/tid=29427] ppid=29319 vsize=515376 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29427/stat : 29427 (python) S 29319 29322 29287 0 -1 4202560 94 925 0 8 0 0 0 0 18 0 21 0 519176031 527745024 15114 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 6 0 0 0
[pid=29322/tid=29430] ppid=29319 vsize=515376 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29430/stat : 29430 (python) S 29319 29322 29287 0 -1 4202560 29 925 0 8 0 0 0 0 21 0 21 0 519176040 527745024 15114 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 6 0 0 0
[pid=29322/tid=29437] ppid=29319 vsize=515376 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29437/stat : 29437 (python) S 29319 29322 29287 0 -1 4202560 26 925 0 8 0 0 0 0 20 0 21 0 519186059 527745024 15114 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 6 0 0 0
[pid=29322/tid=29457] ppid=29319 vsize=515376 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29457/stat : 29457 (python) S 29319 29322 29287 0 -1 4202560 32 925 0 8 0 0 0 0 18 0 21 0 519205693 527745024 15114 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 6 0 0 0
[pid=29322/tid=29461] ppid=29319 vsize=515376 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29461/stat : 29461 (python) S 29319 29322 29287 0 -1 4202560 34 925 0 8 0 0 0 0 18 0 21 0 519205708 527745024 15114 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 6 0 0 0
[pid=29322/tid=29469] ppid=29319 vsize=515376 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29469/stat : 29469 (python) S 29319 29322 29287 0 -1 4202560 29 925 0 8 0 0 0 0 20 0 21 0 519215714 527745024 15114 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 6 0 0 0
[pid=29322/tid=29475] ppid=29319 vsize=515376 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29475/stat : 29475 (python) S 29319 29322 29287 0 -1 4202560 29 925 0 8 0 0 0 0 20 0 21 0 519224824 527745024 15114 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 6 0 0 0
[pid=29322/tid=29487] ppid=29319 vsize=515376 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29487/stat : 29487 (python) S 29319 29322 29287 0 -1 4202560 33 925 0 8 0 0 0 0 18 0 21 0 519225142 527745024 15114 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 6 0 0 0
[pid=29322/tid=29579] ppid=29319 vsize=515376 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29579/stat : 29579 (python) S 29319 29322 29287 0 -1 4202560 30 925 0 8 0 0 0 0 20 0 21 0 519226237 527745024 15114 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 6 0 0 0
[pid=29322/tid=29585] ppid=29319 vsize=515376 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29585/stat : 29585 (python) S 29319 29322 29287 0 -1 4202560 37 925 0 8 0 0 0 0 18 0 21 0 519234876 527745024 15114 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 6 0 0 0
[pid=29322/tid=29589] ppid=29319 vsize=515376 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29589/stat : 29589 (python) S 29319 29322 29287 0 -1 4202560 33 925 0 8 0 0 0 0 18 0 21 0 519235164 527745024 15114 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 6 0 0 0
[pid=29322/tid=29602] ppid=29319 vsize=515376 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29602/stat : 29602 (python) S 29319 29322 29287 0 -1 4202560 30 925 0 8 0 0 0 0 20 0 21 0 519236259 527745024 15114 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 6 0 0 0
[pid=29322/tid=29667] ppid=29319 vsize=515376 CPUtime=0 cores=0,2,4,6
/proc/29322/task/29667/stat : 29667 (python) S 29319 29322 29287 0 -1 4202560 32 925 0 8 0 0 0 0 20 0 21 0 519255909 527745024 15114 33554432000 4194304 5683644 140735338205264 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682048 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4800.09
Current children cumulated vsize (KiB) 15587712

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 29322 and gives
#  childrusage.ru_utime.tv_sec=8
#  childrusage.ru_utime.tv_usec=645685
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=328949
# CPU time returned by wait4() is 8.97463
# while last known CPU time is 4800.09
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1231.73
CPU time (s): 4800.09
CPU user time (s): 4796.01
CPU system time (s): 4.08
CPU usage (%): 389.704
Max. virtual memory (cumulated for all children) (KiB): 15587716

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 8.64569
system time used= 0.328949
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 43747
page faults= 121
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 19239
involuntary context switches= 134

runsolver used 5.9381 second user time and 15.3377 second system time

The end

Launcher Data 

Begin job on node140 at 2011-05-01 18:28:01
IDJOB=3317054
IDBENCH=82519
IDSOLVER=1705
FILE ID=node140/3317054-1304267280
RUNJOBID= node140-1304267280-29305
PBS_JOBID= 13173939
Free space on /tmp= 73328 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v35000-c147000-S419010076-063-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317054-1304267280/watcher-3317054-1304267280 -o /tmp/evaluation-result-3317054-1304267280/solver-3317054-1304267280 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317054-1304267280.cnf -seed 1260138252 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 07407a86c8d0f1e46c34022f68908549
RANDOM SEED=1260138252

node140.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.14
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      29865096 kB
Buffers:        980652 kB
Cached:        1548276 kB
SwapCached:        176 kB
Active:         211120 kB
Inactive:      2380116 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      29865096 kB
SwapTotal:    67111528 kB
SwapFree:     67111348 kB
Dirty:           67308 kB
Writeback:           0 kB
AnonPages:       62404 kB
Mapped:          14540 kB
Slab:           431488 kB
PageTables:       4636 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   184576 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73256 MiB
End job on node140 at 2011-05-01 18:48:36