Trace number 3262310

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.59 1202.32

General information on the benchmark

Namecrafted/kullmann/VanDerWaerden/VanderWaerden_pd_3k/
VanDerWaerden_pd_2-3-20_390.cnf
MD5SUM9f9afdb81b528e7852015fad23de6353
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark193.51
Satisfiable
(Un)Satisfiability was proved
Number of variables195
Number of clauses20607
Sum of the clauses size92222
Maximum clause length20
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 265
Number of clauses of size 318744
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 51798

Solver Data 

0.00/0.02	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262310-1304092115.cnf 
0.00/0.02	c Using SATTIMEOUT of 4800
0.00/0.02	c Using SATRAM of 15500
0.00/0.02	c Size of subset node = 32
0.00/0.02	c Variables = 195
0.00/0.02	c Clauses = 20607
0.00/0.02	c Literals = 92222
0.00/0.02	c Reduced size:
0.00/0.02	c Variables = 195
0.00/0.02	c Clauses = 20607
0.00/0.02	c Literals = 92222
0.00/0.02	c a2
0.09/0.12	c a1
0.19/0.28	c Independent sub-problems = 1
0.19/0.28	c Sub-problem with 195 variables and 20607 clauses
0.19/0.28	c using permutation method 4
0.19/0.28	c count = 1
0.19/0.28	c Ordering finished 0.270 seconds
0.19/0.28	c mapped 0.280 seconds
0.19/0.28	c permuted 0.280 seconds
0.19/0.28	c about to sort
0.19/0.29	c sorted
0.29/0.35	c clauses now 20607
0.29/0.35	c init finished 0.280 seconds
0.29/0.35	c tree built 0.350 seconds
0.29/0.35	c 
0.29/0.35	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.29/0.35	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.29/0.35	c ................................ | ..........                    |                                  0.3         0.0             
0.29/0.35	c ................................ | ..........                  1 |                                  0.3         2.9             
0.29/0.35	c ................................ | ..........                  2 |                                  0.3         5.7             
0.29/0.35	c ................................ | ..........                  3 |                                  0.3         8.6             
0.29/0.35	c ................................ | ..........                  4 |                                  0.3        11.4             
0.29/0.35	c ................................ | ..........                  5 |                                  0.3        14.3             
0.29/0.38	c ................................ | ..........               1000 | 35           492                 0.5      2222.2        66857
95.98/24.30	c ................................ | ..........             333000 | 31178        120039             96.0      3468.8     22765882
192.27/48.44	c ................................ | ..........             537000 | 52571        192867            192.4      2790.5     36525607
288.53/72.53	c ................................ | ..........             685000 | 70562        255317            288.6      2373.3     48827986
384.78/96.63	c ................................ | ..........             807000 | 84532        318703            384.9      2096.6     59484610
480.26/120.58	c ................................ | *.........             910000 | 94960        375444            480.6      1893.5     68012870
576.12/144.52	c ................................ | *.........             996000 | 103983       421982            576.2      1728.6     75526318
672.00/168.54	c ................................ | *.........            1075000 | 112474       463025            672.2      1599.3     83405298
767.88/192.54	c ................................ | *.........            1161000 | 122172       500028            768.0      1511.6     89921649
864.93/216.84	c ................................ | *.........            1236000 | 132102       536930            865.1      1428.7     96094840
960.82/240.87	c ................................ | **........            1311000 | 139824       571784            961.1      1364.0    101962717
1056.67/264.81	c ................................ | **........            1382000 | 147196       601865           1056.7      1307.8    108172129
1153.34/289.07	c ................................ | **........            1445000 | 154461       627597           1153.7      1252.5    114066711
1249.21/313.05	c ................................ | **........            1510000 | 161929       653647           1249.4      1208.6    119455620
1343.88/336.77	c ................................ | **........            1568000 | 169149       681891           1344.2      1166.5    125327391
1440.95/361.00	c ................................ | ***.......            1629000 | 175853       708095           1441.0      1130.5    130648947
1537.22/385.15	c ................................ | ***.......            1684000 | 181925       733191           1537.4      1095.3    135790321
1632.29/408.98	c ................................ | ***.......            1734000 | 188251       758315           1632.6      1062.1    140182130
1728.16/432.93	c ................................ | ***.......            1783000 | 193972       780652           1728.3      1031.6    144445993
1824.83/457.18	c ................................ | ***.......            1832000 | 199972       802972           1825.2      1003.7    148959824
1920.30/481.09	c ................................ | ****......            1878000 | 206081       822912           1920.7       977.8    153240137
2016.97/505.27	c ................................ | ****......            1922000 | 212378       845515           2017.3       952.8    157678198
2113.63/529.41	c ................................ | ****......            1960000 | 217026       864974           2113.7       927.3    162211972
2209.12/553.32	c ................................ | ****......            1999000 | 223241       884123           2209.2       904.9    166465611
2305.39/577.44	c ................................ | ****......            2043000 | 229040       904409           2305.5       886.1    170671724
2401.64/601.54	c ................................ | *****.....            2097000 | 236019       922895           2401.8       873.1    175139407
2496.72/625.33	c ................................ | *****.....            2141000 | 242099       940253           2496.8       857.5    178247898
2592.98/649.42	c ................................ | *****.....            2187000 | 247884       956521           2593.1       843.4    181881845
2688.45/673.31	c ................................ | *****.....            2234000 | 253624       972627           2688.5       830.9    185263334
2785.04/697.53	c ................................ | *****.....            2275000 | 259557       988626           2785.2       816.8    186517693
2880.50/721.42	c ................................ | ******....            2333000 | 267048       1008267          2880.6       809.9    186517693
2976.77/745.52	c ................................ | ******....            2390000 | 274686       1027309          2976.9       802.9    186517693
3073.00/769.66	c ................................ | ******....            2447000 | 281477       1048817          3073.3       796.2    187047775
3167.67/793.39	c ................................ | ******....            2496000 | 288293       1067154          3168.0       787.9    188850813
3264.34/817.52	c ................................ | ******....            2538000 | 292720       1088353          3264.4       777.5    190641574
3360.61/841.67	c ................................ | *******...            2586000 | 297958       1105865          3360.9       769.4    192930941
3457.27/865.89	c ................................ | *******...            2633000 | 303631       1124873          3457.6       761.5    195156566
3553.17/889.89	c ................................ | *******...            2682000 | 309484       1144743          3553.5       754.7    197657475
3648.22/913.60	c ................................ | *******...            2734000 | 315187       1162840          3648.2       749.4    199846735
3744.09/937.69	c ................................ | *******...            2787000 | 321666       1181247          3744.4       744.3    202756240
3841.56/962.06	c ................................ | ********..            2834000 | 327907       1200066          3841.8       737.7    205292652
3936.63/985.86	c ................................ | ********..            2879000 | 333794       1218828          3936.9       731.3    208845068
4032.52/1009.87	c ................................ | ********..            2924000 | 339709       1235053          4032.8       725.1    212901525
4128.37/1033.85	c ................................ | ********..            2977000 | 346148       1250394          4128.6       721.1    217038665
4224.24/1057.84	c ................................ | ********..            3029000 | 352519       1264405          4224.4       717.0    220906050
4320.51/1081.94	c ................................ | *********.            3079000 | 358337       1279785          4320.7       712.6    224382259
4417.57/1106.26	c ................................ | *********.            3126000 | 363707       1295275          4417.8       707.6    227632293
4512.65/1130.05	c ................................ | *********.            3174000 | 369151       1310744          4512.9       703.3    230763621
4608.92/1154.16	c ................................ | *********.            3219000 | 374912       1326927          4609.2       698.4    233670445
4706.79/1178.65	c ................................ | *********.            3267000 | 380652       1341979          4707.0       694.1    236989058

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3262310-1304092115/watcher-3262310-1304092115 -o /tmp/evaluation-result-3262310-1304092115/solver-3262310-1304092115 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262310-1304092115.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.68 4.92 6.53 3/173 21191
/proc/meminfo: memFree=30889152/32951124 swapFree=67111464/67111528
[pid=21191] ppid=21188 vsize=9540 CPUtime=0 cores=0,2,4,6
/proc/21191/stat : 21191 (ttsth-5-4) R 21188 21191 20765 0 -1 4202496 438 0 0 0 0 0 0 0 19 0 1 0 599442066 9768960 381 33554432000 4194304 4338100 140734180660368 18446744073709551615 260020056773 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/21191/statm: 2385 381 146 36 0 290 0

[startup+0.091175 s]
/proc/loadavg: 3.68 4.92 6.53 3/173 21191
/proc/meminfo: memFree=30889152/32951124 swapFree=67111464/67111528
[pid=21191] ppid=21188 vsize=23112 CPUtime=0.08 cores=0,2,4,6
/proc/21191/stat : 21191 (ttsth-5-4) R 21188 21191 20765 0 -1 4202496 3238 0 0 0 8 0 0 0 19 0 1 0 599442066 23666688 3113 33554432000 4194304 4338100 140734180660368 18446744073709551615 260019791581 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/21191/statm: 5778 3113 341 36 0 3683 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 23112

[startup+0.10116 s]
/proc/loadavg: 3.68 4.92 6.53 3/173 21191
/proc/meminfo: memFree=30889152/32951124 swapFree=67111464/67111528
[pid=21191] ppid=21188 vsize=23112 CPUtime=0.09 cores=0,2,4,6
/proc/21191/stat : 21191 (ttsth-5-4) R 21188 21191 20765 0 -1 4202496 3238 0 0 0 9 0 0 0 19 0 1 0 599442066 23666688 3113 33554432000 4194304 4338100 140734180660368 18446744073709551615 260019791050 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/21191/statm: 5778 3113 341 36 0 3683 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 23112

[startup+0.301136 s]
/proc/loadavg: 3.68 4.92 6.53 3/173 21191
/proc/meminfo: memFree=30889152/32951124 swapFree=67111464/67111528
[pid=21191] ppid=21188 vsize=17724 CPUtime=0.29 cores=0,2,4,6
/proc/21191/stat : 21191 (ttsth-5-4) R 21188 21191 20765 0 -1 4202496 4695 0 0 0 29 0 0 0 20 0 1 0 599442066 18149376 2432 33554432000 4194304 4338100 140734180660368 18446744073709551615 260020056773 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/21191/statm: 4431 2438 164 36 0 2336 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 17724

[startup+0.700988 s]
/proc/loadavg: 3.68 4.92 6.53 3/173 21191
/proc/meminfo: memFree=30889152/32951124 swapFree=67111464/67111528
[pid=21191] ppid=21188 vsize=171900 CPUtime=1.71 cores=0,2,4,6
/proc/21191/stat : 21191 (ttsth-5-4) S 21188 21191 20765 0 -1 4202496 14908 0 0 0 170 1 0 0 20 0 5 0 599442066 176025600 12604 33554432000 4194304 4338100 140734180660368 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21191/statm: 42975 12604 176 36 0 40880 0
Current children cumulated CPU time (s) 1.71
Current children cumulated vsize (KiB) 171900

[startup+1.50084 s]
/proc/loadavg: 4.03 4.97 6.53 9/186 21207
/proc/meminfo: memFree=30761832/32951124 swapFree=67111464/67111528
[pid=21191] ppid=21188 vsize=225472 CPUtime=4.91 cores=0,2,4,6
/proc/21191/stat : 21191 (ttsth-5-4) S 21188 21191 20765 0 -1 4202496 28115 0 0 0 488 3 0 0 20 0 5 0 599442066 230883328 25811 33554432000 4194304 4338100 140734180660368 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21191/statm: 56368 25811 176 36 0 54273 0
[pid=21191/tid=21200] ppid=21188 vsize=225472 CPUtime=1.13 cores=0,2,4,6
/proc/21191/task/21200/stat : 21200 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 4637 0 0 0 112 1 0 0 25 0 5 0 599442101 230883328 25811 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21191/tid=21201] ppid=21188 vsize=225472 CPUtime=1.13 cores=0,2,4,6
/proc/21191/task/21201/stat : 21201 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 5880 0 0 0 113 0 0 0 19 0 5 0 599442101 230883328 25811 33554432000 4194304 4338100 140734180660368 18446744073709551615 260020044884 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21191/tid=21202] ppid=21188 vsize=225472 CPUtime=1.13 cores=0,2,4,6
/proc/21191/task/21202/stat : 21202 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 5037 0 0 0 113 0 0 0 19 0 5 0 599442101 230883328 25811 33554432000 4194304 4338100 140734180660368 18446744073709551615 4256032 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21191/tid=21203] ppid=21188 vsize=225472 CPUtime=1.13 cores=0,2,4,6
/proc/21191/task/21203/stat : 21203 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 4970 0 0 0 113 0 0 0 25 0 5 0 599442101 230883328 25811 33554432000 4194304 4338100 140734180660368 18446744073709551615 4240550 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4.91
Current children cumulated vsize (KiB) 225472

[startup+3.10055 s]
/proc/loadavg: 4.03 4.97 6.53 9/186 21207
/proc/meminfo: memFree=30642416/32951124 swapFree=67111464/67111528
[pid=21191] ppid=21188 vsize=310588 CPUtime=11.3 cores=0,2,4,6
/proc/21191/stat : 21191 (ttsth-5-4) S 21188 21191 20765 0 -1 4202496 50073 0 0 0 1124 6 0 0 20 0 5 0 599442066 318042112 47769 33554432000 4194304 4338100 140734180660368 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21191/statm: 77647 47769 176 36 0 75552 0
[pid=21191/tid=21200] ppid=21188 vsize=310588 CPUtime=2.73 cores=0,2,4,6
/proc/21191/task/21200/stat : 21200 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 9567 0 0 0 271 2 0 0 25 0 5 0 599442101 318042112 47769 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254291 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21191/tid=21201] ppid=21188 vsize=310588 CPUtime=2.74 cores=0,2,4,6
/proc/21191/task/21201/stat : 21201 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 11444 0 0 0 273 1 0 0 23 0 5 0 599442101 318042112 47769 33554432000 4194304 4338100 140734180660368 18446744073709551615 4285383 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21191/tid=21202] ppid=21188 vsize=310588 CPUtime=2.73 cores=0,2,4,6
/proc/21191/task/21202/stat : 21202 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 10101 0 0 0 273 0 0 0 23 0 5 0 599442101 318042112 47769 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21191/tid=21203] ppid=21188 vsize=310588 CPUtime=2.72 cores=0,2,4,6
/proc/21191/task/21203/stat : 21203 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 11370 0 0 0 271 1 0 0 25 0 5 0 599442101 318042112 47769 33554432000 4194304 4338100 140734180660368 18446744073709551615 4286370 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 11.3
Current children cumulated vsize (KiB) 310588

[startup+6.30096 s]
/proc/loadavg: 4.03 4.97 6.53 9/186 21207
/proc/meminfo: memFree=30361308/32951124 swapFree=67111464/67111528
[pid=21191] ppid=21188 vsize=439364 CPUtime=24.08 cores=0,2,4,6
/proc/21191/stat : 21191 (ttsth-5-4) S 21188 21191 20765 0 -1 4202496 81523 0 0 0 2396 12 0 0 20 0 5 0 599442066 449908736 79219 33554432000 4194304 4338100 140734180660368 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21191/statm: 109841 79219 176 36 0 107746 0
[pid=21191/tid=21200] ppid=21188 vsize=439364 CPUtime=5.92 cores=0,2,4,6
/proc/21191/task/21200/stat : 21200 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 15954 0 0 0 589 3 0 0 25 0 5 0 599442101 449908736 79219 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21191/tid=21201] ppid=21188 vsize=439364 CPUtime=5.94 cores=0,2,4,6
/proc/21191/task/21201/stat : 21201 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 20194 0 0 0 592 2 0 0 25 0 5 0 599442101 449908736 79219 33554432000 4194304 4338100 140734180660368 18446744073709551615 260032208678 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21191/tid=21202] ppid=21188 vsize=439364 CPUtime=5.93 cores=0,2,4,6
/proc/21191/task/21202/stat : 21202 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 16946 0 0 0 591 2 0 0 25 0 5 0 599442101 449908736 79219 33554432000 4194304 4338100 140734180660368 18446744073709551615 4256139 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21191/tid=21203] ppid=21188 vsize=439364 CPUtime=5.9 cores=0,2,4,6
/proc/21191/task/21203/stat : 21203 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 20838 0 0 0 588 2 0 0 25 0 5 0 599442101 449908736 79219 33554432000 4194304 4338100 140734180660368 18446744073709551615 4256139 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 24.08
Current children cumulated vsize (KiB) 439364

[startup+12.7008 s]
/proc/loadavg: 4.64 5.07 6.55 9/186 21207
/proc/meminfo: memFree=29979656/32951124 swapFree=67111464/67111528
[pid=21191] ppid=21188 vsize=605808 CPUtime=49.65 cores=0,2,4,6
/proc/21191/stat : 21191 (ttsth-5-4) S 21188 21191 20765 0 -1 4202496 124626 0 0 0 4945 20 0 0 20 0 5 0 599442066 620347392 122322 33554432000 4194304 4338100 140734180660368 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21191/statm: 151452 122322 176 36 0 149357 0
[pid=21191/tid=21200] ppid=21188 vsize=605808 CPUtime=12.32 cores=0,2,4,6
/proc/21191/task/21200/stat : 21200 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 24358 0 0 0 1227 5 0 0 25 0 5 0 599442101 620347392 122322 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21191/tid=21201] ppid=21188 vsize=605808 CPUtime=12.34 cores=0,2,4,6
/proc/21191/task/21201/stat : 21201 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 31511 0 0 0 1230 4 0 0 25 0 5 0 599442101 620347392 122322 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21191/tid=21202] ppid=21188 vsize=605808 CPUtime=12.33 cores=0,2,4,6
/proc/21191/task/21202/stat : 21202 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 27330 0 0 0 1229 4 0 0 25 0 5 0 599442101 620347392 122322 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254304 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21191/tid=21203] ppid=21188 vsize=605808 CPUtime=12.27 cores=0,2,4,6
/proc/21191/task/21203/stat : 21203 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 33836 0 0 0 1223 4 0 0 25 0 5 0 599442101 620347392 122322 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 49.65
Current children cumulated vsize (KiB) 605808

[startup+25.5004 s]
/proc/loadavg: 5.16 5.17 6.56 9/186 21207
/proc/meminfo: memFree=29396484/32951124 swapFree=67111464/67111528
[pid=21191] ppid=21188 vsize=888252 CPUtime=100.77 cores=0,2,4,6
/proc/21191/stat : 21191 (ttsth-5-4) S 21188 21191 20765 0 -1 4202496 195099 0 0 0 10042 35 0 0 20 0 5 0 599442066 909570048 192795 33554432000 4194304 4338100 140734180660368 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21191/statm: 222063 192795 176 36 0 219968 0
[pid=21191/tid=21200] ppid=21188 vsize=888252 CPUtime=25.13 cores=0,2,4,6
/proc/21191/task/21200/stat : 21200 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 37769 0 0 0 2505 8 0 0 25 0 5 0 599442101 909570048 192795 33554432000 4194304 4338100 140734180660368 18446744073709551615 260019791091 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21191/tid=21201] ppid=21188 vsize=888252 CPUtime=25.13 cores=0,2,4,6
/proc/21191/task/21201/stat : 21201 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 49955 0 0 0 2505 8 0 0 25 0 5 0 599442101 909570048 192795 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21191/tid=21202] ppid=21188 vsize=888252 CPUtime=25.13 cores=0,2,4,6
/proc/21191/task/21202/stat : 21202 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 44084 0 0 0 2505 8 0 0 25 0 5 0 599442101 909570048 192795 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21191/tid=21203] ppid=21188 vsize=888252 CPUtime=25 cores=0,2,4,6
/proc/21191/task/21203/stat : 21203 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 55700 0 0 0 2492 8 0 0 25 0 5 0 599442101 909570048 192795 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 100.77
Current children cumulated vsize (KiB) 888252

[startup+51.1007 s]
/proc/loadavg: 6.19 5.41 6.61 9/186 21208
/proc/meminfo: memFree=28557672/32951124 swapFree=67111464/67111528
[pid=21191] ppid=21188 vsize=1336876 CPUtime=203.03 cores=0,2,4,6
/proc/21191/stat : 21191 (ttsth-5-4) S 21188 21191 20765 0 -1 4202496 306590 0 0 0 20246 57 0 0 20 0 5 0 599442066 1368961024 304286 33554432000 4194304 4338100 140734180660368 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21191/statm: 334219 304286 176 36 0 332124 0
[pid=21191/tid=21200] ppid=21188 vsize=1336876 CPUtime=50.72 cores=0,2,4,6
/proc/21191/task/21200/stat : 21200 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 60290 0 0 0 5060 12 0 0 25 0 5 0 599442101 1368961024 304286 33554432000 4194304 4338100 140734180660368 18446744073709551615 260032208590 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21191/tid=21201] ppid=21188 vsize=1336876 CPUtime=50.73 cores=0,2,4,6
/proc/21191/task/21201/stat : 21201 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 86332 0 0 0 5057 16 0 0 25 0 5 0 599442101 1368961024 304286 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21191/tid=21202] ppid=21188 vsize=1336876 CPUtime=50.63 cores=0,2,4,6
/proc/21191/task/21202/stat : 21202 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 63334 0 0 0 5051 12 0 0 25 0 5 0 599442101 1368961024 304286 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21191/tid=21203] ppid=21188 vsize=1336876 CPUtime=50.55 cores=0,2,4,6
/proc/21191/task/21203/stat : 21203 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 89043 0 0 0 5041 14 0 0 25 0 5 0 599442101 1368961024 304286 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 203.03
Current children cumulated vsize (KiB) 1336876

[startup+102.306 s]
/proc/loadavg: 7.34 5.86 6.69 9/186 21209
/proc/meminfo: memFree=27118988/32951124 swapFree=67111464/67111528
[pid=21191] ppid=21188 vsize=2083716 CPUtime=407.57 cores=0,2,4,6
/proc/21191/stat : 21191 (ttsth-5-4) S 21188 21191 20765 0 -1 4202496 493977 0 0 0 40666 91 0 0 20 0 5 0 599442066 2133725184 491673 33554432000 4194304 4338100 140734180660368 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21191/statm: 520929 491673 176 36 0 518834 0
[pid=21191/tid=21200] ppid=21188 vsize=2083716 CPUtime=101.92 cores=0,2,4,6
/proc/21191/task/21200/stat : 21200 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 86422 0 0 0 10175 17 0 0 25 0 5 0 599442101 2133725184 491673 33554432000 4194304 4338100 140734180660368 18446744073709551615 4257010 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21191/tid=21201] ppid=21188 vsize=2083716 CPUtime=101.94 cores=0,2,4,6
/proc/21191/task/21201/stat : 21201 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 145632 0 0 0 10168 26 0 0 25 0 5 0 599442101 2133725184 491673 33554432000 4194304 4338100 140734180660368 18446744073709551615 4255963 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21191/tid=21202] ppid=21188 vsize=2083716 CPUtime=101.57 cores=0,2,4,6
/proc/21191/task/21202/stat : 21202 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 98148 0 0 0 10138 19 0 0 25 0 5 0 599442101 2133725184 491673 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21191/tid=21203] ppid=21188 vsize=2083716 CPUtime=101.76 cores=0,2,4,6
/proc/21191/task/21203/stat : 21203 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 156184 0 0 0 10149 27 0 0 25 0 5 0 599442101 2133725184 491673 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 407.57
Current children cumulated vsize (KiB) 2083716

[startup+162.301 s]
/proc/loadavg: 7.82 6.26 6.77 9/186 21211
/proc/meminfo: memFree=25872604/32951124 swapFree=67111464/67111528
[pid=21191] ppid=21188 vsize=2700080 CPUtime=647.23 cores=0,2,4,6
/proc/21191/stat : 21191 (ttsth-5-4) S 21188 21191 20765 0 -1 4202496 648636 0 0 0 64598 125 0 0 20 0 5 0 599442066 2764881920 646332 33554432000 4194304 4338100 140734180660368 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21191/statm: 675020 646332 176 36 0 672925 0
[pid=21191/tid=21200] ppid=21188 vsize=2700080 CPUtime=161.91 cores=0,2,4,6
/proc/21191/task/21200/stat : 21200 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 111617 0 0 0 16169 22 0 0 25 0 5 0 599442101 2764881920 646332 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21191/tid=21201] ppid=21188 vsize=2700080 CPUtime=161.94 cores=0,2,4,6
/proc/21191/task/21201/stat : 21201 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 195367 0 0 0 16157 37 0 0 25 0 5 0 599442101 2764881920 646332 33554432000 4194304 4338100 140734180660368 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21191/tid=21202] ppid=21188 vsize=2700080 CPUtime=161.24 cores=0,2,4,6
/proc/21191/task/21202/stat : 21202 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 132784 0 0 0 16097 27 0 0 25 0 5 0 599442101 2764881920 646332 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21191/tid=21203] ppid=21188 vsize=2700080 CPUtime=161.75 cores=0,2,4,6
/proc/21191/task/21203/stat : 21203 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 201277 0 0 0 16139 36 0 0 25 0 5 0 599442101 2764881920 646332 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 647.23
Current children cumulated vsize (KiB) 2700080


################
# More data... #
################

/proc/21191/task/21203/stat : 21203 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 464055 0 0 0 76083 94 0 0 25 0 5 0 599442101 6442221568 1521193 33554432000 4194304 4338100 140734180660368 18446744073709551615 4286370 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3043.86
Current children cumulated vsize (KiB) 6291232

[startup+822.301 s]
/proc/loadavg: 8.13 7.89 7.43 9/186 21237
/proc/meminfo: memFree=18952704/32951124 swapFree=67111464/67111528
[pid=21191] ppid=21188 vsize=6438908 CPUtime=3283.51 cores=0,2,4,6
/proc/21191/stat : 21191 (ttsth-5-4) S 21188 21191 20765 0 -1 4202496 1560213 0 0 0 328027 324 0 0 20 0 5 0 599442066 6593441792 1557909 33554432000 4194304 4338100 140734180660368 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21191/statm: 1609727 1557909 176 36 0 1607632 0
[pid=21191/tid=21200] ppid=21188 vsize=6438908 CPUtime=821.19 cores=0,2,4,6
/proc/21191/task/21200/stat : 21200 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 289546 0 0 0 82056 63 0 0 25 0 5 0 599442101 6593441792 1557909 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21191/tid=21201] ppid=21188 vsize=6438908 CPUtime=821.94 cores=0,2,4,6
/proc/21191/task/21201/stat : 21201 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 479874 0 0 0 82101 93 0 0 25 0 5 0 599442101 6593441792 1557909 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21191/tid=21202] ppid=21188 vsize=6438908 CPUtime=818.23 cores=0,2,4,6
/proc/21191/task/21202/stat : 21202 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 312503 0 0 0 81754 69 0 0 25 0 5 0 599442101 6593441792 1557909 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21191/tid=21203] ppid=21188 vsize=6438908 CPUtime=821.76 cores=0,2,4,6
/proc/21191/task/21203/stat : 21203 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 470699 0 0 0 82080 96 0 0 25 0 5 0 599442101 6593441792 1557909 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3283.51
Current children cumulated vsize (KiB) 6438908

[startup+882.301 s]
/proc/loadavg: 8.12 7.92 7.47 9/186 21239
/proc/meminfo: memFree=18546688/32951124 swapFree=67111464/67111528
[pid=21191] ppid=21188 vsize=6617864 CPUtime=3523.18 cores=0,2,4,6
/proc/21191/stat : 21191 (ttsth-5-4) S 21188 21191 20765 0 -1 4202496 1605171 0 0 0 351984 334 0 0 20 0 5 0 599442066 6776692736 1602867 33554432000 4194304 4338100 140734180660368 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21191/statm: 1654466 1602867 176 36 0 1652371 0
[pid=21191/tid=21200] ppid=21188 vsize=6617864 CPUtime=880.86 cores=0,2,4,6
/proc/21191/task/21200/stat : 21200 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 289605 0 0 0 88023 63 0 0 25 0 5 0 599442101 6776692736 1602867 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21191/tid=21201] ppid=21188 vsize=6617864 CPUtime=881.95 cores=0,2,4,6
/proc/21191/task/21201/stat : 21201 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 500115 0 0 0 88098 97 0 0 25 0 5 0 599442101 6776692736 1602867 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21191/tid=21202] ppid=21188 vsize=6617864 CPUtime=878.22 cores=0,2,4,6
/proc/21191/task/21202/stat : 21202 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 312588 0 0 0 87753 69 0 0 25 0 5 0 599442101 6776692736 1602867 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21191/tid=21203] ppid=21188 vsize=6617864 CPUtime=881.77 cores=0,2,4,6
/proc/21191/task/21203/stat : 21203 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 495272 0 0 0 88075 102 0 0 25 0 5 0 599442101 6776692736 1602867 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3523.18
Current children cumulated vsize (KiB) 6617864

[startup+942.3 s]
/proc/loadavg: 8.11 7.95 7.50 9/186 21240
/proc/meminfo: memFree=18150456/32951124 swapFree=67111464/67111528
[pid=21191] ppid=21188 vsize=6821712 CPUtime=3762.87 cores=0,2,4,6
/proc/21191/stat : 21191 (ttsth-5-4) S 21188 21191 20765 0 -1 4202496 1656327 0 0 0 375941 346 0 0 20 0 5 0 599442066 6985433088 1654023 33554432000 4194304 4338100 140734180660368 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21191/statm: 1705428 1654023 176 36 0 1703333 0
[pid=21191/tid=21200] ppid=21188 vsize=6821712 CPUtime=940.54 cores=0,2,4,6
/proc/21191/task/21200/stat : 21200 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 289659 0 0 0 93990 64 0 0 25 0 5 0 599442101 6985433088 1654023 33554432000 4194304 4338100 140734180660368 18446744073709551615 4239520 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21191/tid=21201] ppid=21188 vsize=6821712 CPUtime=941.95 cores=0,2,4,6
/proc/21191/task/21201/stat : 21201 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 522621 0 0 0 94095 100 0 0 25 0 5 0 599442101 6985433088 1654023 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21191/tid=21202] ppid=21188 vsize=6821712 CPUtime=938.23 cores=0,2,4,6
/proc/21191/task/21202/stat : 21202 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 320045 0 0 0 93752 71 0 0 25 0 5 0 599442101 6985433088 1654023 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21191/tid=21203] ppid=21188 vsize=6821712 CPUtime=941.76 cores=0,2,4,6
/proc/21191/task/21203/stat : 21203 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 516411 0 0 0 94068 108 0 0 25 0 5 0 599442101 6985433088 1654023 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3762.87
Current children cumulated vsize (KiB) 6821712

[startup+1002.3 s]
/proc/loadavg: 8.10 7.97 7.54 9/186 21242
/proc/meminfo: memFree=17808412/32951124 swapFree=67111464/67111528
[pid=21191] ppid=21188 vsize=7078868 CPUtime=4002.53 cores=0,2,4,6
/proc/21191/stat : 21191 (ttsth-5-4) S 21188 21191 20765 0 -1 4202496 1720495 0 0 0 399892 361 0 0 20 0 5 0 599442066 7248760832 1718191 33554432000 4194304 4338100 140734180660368 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21191/statm: 1769717 1718192 176 36 0 1767622 0
[pid=21191/tid=21200] ppid=21188 vsize=7078868 CPUtime=1000.22 cores=0,2,4,6
/proc/21191/task/21200/stat : 21200 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 297470 0 0 0 99956 66 0 0 25 0 5 0 599442101 7248760832 1718192 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21191/tid=21201] ppid=21188 vsize=7078868 CPUtime=1001.95 cores=0,2,4,6
/proc/21191/task/21201/stat : 21201 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 546936 0 0 0 100089 106 0 0 25 0 5 0 599442101 7248760832 1718192 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21191/tid=21202] ppid=21188 vsize=7078868 CPUtime=998.21 cores=0,2,4,6
/proc/21191/task/21202/stat : 21202 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 331134 0 0 0 99748 73 0 0 25 0 5 0 599442101 7248760832 1718192 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254294 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21191/tid=21203] ppid=21188 vsize=7078868 CPUtime=1001.77 cores=0,2,4,6
/proc/21191/task/21203/stat : 21203 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 537365 0 0 0 100063 114 0 0 25 0 5 0 599442101 7248760832 1718192 33554432000 4194304 4338100 140734180660368 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4002.53
Current children cumulated vsize (KiB) 7078868

[startup+1062.3 s]
/proc/loadavg: 8.15 8.00 7.57 9/186 21244
/proc/meminfo: memFree=17419492/32951124 swapFree=67111464/67111528
[pid=21191] ppid=21188 vsize=7392196 CPUtime=4242.22 cores=0,2,4,6
/proc/21191/stat : 21191 (ttsth-5-4) S 21188 21191 20765 0 -1 4202496 1799409 0 0 0 423844 378 0 0 20 0 5 0 599442066 7569608704 1797105 33554432000 4194304 4338100 140734180660368 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21191/statm: 1848049 1797105 176 36 0 1845954 0
[pid=21191/tid=21200] ppid=21188 vsize=7392196 CPUtime=1059.9 cores=0,2,4,6
/proc/21191/task/21200/stat : 21200 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 314269 0 0 0 105920 70 0 0 25 0 5 0 599442101 7569608704 1797105 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21191/tid=21201] ppid=21188 vsize=7392196 CPUtime=1061.96 cores=0,2,4,6
/proc/21191/task/21201/stat : 21201 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 568780 0 0 0 106086 110 0 0 25 0 5 0 599442101 7569608704 1797105 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21191/tid=21202] ppid=21188 vsize=7392196 CPUtime=1058.21 cores=0,2,4,6
/proc/21191/task/21202/stat : 21202 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 344448 0 0 0 105745 76 0 0 25 0 5 0 599442101 7569608704 1797105 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21191/tid=21203] ppid=21188 vsize=7392196 CPUtime=1061.77 cores=0,2,4,6
/proc/21191/task/21203/stat : 21203 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 564321 0 0 0 106058 119 0 0 25 0 5 0 599442101 7569608704 1797105 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4242.22
Current children cumulated vsize (KiB) 7392196

[startup+1122.3 s]
/proc/loadavg: 8.11 8.01 7.60 9/186 21245
/proc/meminfo: memFree=17016672/32951124 swapFree=67111464/67111528
[pid=21191] ppid=21188 vsize=7646092 CPUtime=4481.89 cores=0,2,4,6
/proc/21191/stat : 21191 (ttsth-5-4) S 21188 21191 20765 0 -1 4202496 1862851 0 0 0 447798 391 0 0 20 0 5 0 599442066 7829598208 1860547 33554432000 4194304 4338100 140734180660368 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21191/statm: 1911523 1860547 176 36 0 1909428 0
[pid=21191/tid=21200] ppid=21188 vsize=7646092 CPUtime=1119.57 cores=0,2,4,6
/proc/21191/task/21200/stat : 21200 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 330709 0 0 0 111884 73 0 0 25 0 5 0 599442101 7829598208 1860547 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21191/tid=21201] ppid=21188 vsize=7646092 CPUtime=1121.95 cores=0,2,4,6
/proc/21191/task/21201/stat : 21201 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 578932 0 0 0 112083 112 0 0 25 0 5 0 599442101 7829598208 1860547 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21191/tid=21202] ppid=21188 vsize=7646092 CPUtime=1118.21 cores=0,2,4,6
/proc/21191/task/21202/stat : 21202 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 358281 0 0 0 111742 79 0 0 25 0 5 0 599442101 7829598208 1860547 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21191/tid=21203] ppid=21188 vsize=7646092 CPUtime=1121.78 cores=0,2,4,6
/proc/21191/task/21203/stat : 21203 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 587338 0 0 0 112054 124 0 0 25 0 5 0 599442101 7829598208 1860547 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4481.89
Current children cumulated vsize (KiB) 7646092

[startup+1182.3 s]
/proc/loadavg: 8.09 8.02 7.63 9/186 21247
/proc/meminfo: memFree=16627244/32951124 swapFree=67111464/67111528
[pid=21191] ppid=21188 vsize=7890472 CPUtime=4721.57 cores=0,2,4,6
/proc/21191/stat : 21191 (ttsth-5-4) S 21188 21191 20765 0 -1 4202496 1924125 0 0 0 471752 405 0 0 20 0 5 0 599442066 8079843328 1921821 33554432000 4194304 4338100 140734180660368 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21191/statm: 1972618 1921821 176 36 0 1970523 0
[pid=21191/tid=21200] ppid=21188 vsize=7890472 CPUtime=1179.25 cores=0,2,4,6
/proc/21191/task/21200/stat : 21200 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 346395 0 0 0 117849 76 0 0 25 0 5 0 599442101 8079843328 1921821 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21191/tid=21201] ppid=21188 vsize=7890472 CPUtime=1181.95 cores=0,2,4,6
/proc/21191/task/21201/stat : 21201 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 587773 0 0 0 118081 114 0 0 25 0 5 0 599442101 8079843328 1921821 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21191/tid=21202] ppid=21188 vsize=7890472 CPUtime=1178.21 cores=0,2,4,6
/proc/21191/task/21202/stat : 21202 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 375665 0 0 0 117736 85 0 0 25 0 5 0 599442101 8079843328 1921821 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21191/tid=21203] ppid=21188 vsize=7890472 CPUtime=1181.77 cores=0,2,4,6
/proc/21191/task/21203/stat : 21203 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 606701 0 0 0 118050 127 0 0 25 0 5 0 599442101 8079843328 1921821 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4721.57
Current children cumulated vsize (KiB) 7890472



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202 s]
/proc/loadavg: 8.06 8.02 7.64 9/186 21248
/proc/meminfo: memFree=16507428/32951124 swapFree=67111464/67111528
[pid=21191] ppid=21188 vsize=7953164 CPUtime=4800.25 cores=0,2,4,6
/proc/21191/stat : 21191 (ttsth-5-4) S 21188 21191 20765 0 -1 4202496 1940422 0 0 0 479616 409 0 0 20 0 5 0 599442066 8144039936 1938118 33554432000 4194304 4338100 140734180660368 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21191/statm: 1988291 1938118 176 36 0 1986196 0
[pid=21191/tid=21200] ppid=21188 vsize=7953164 CPUtime=1198.81 cores=0,2,4,6
/proc/21191/task/21200/stat : 21200 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 351014 0 0 0 119804 77 0 0 25 0 5 0 599442101 8144039936 1938118 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254259 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21191/tid=21201] ppid=21188 vsize=7953164 CPUtime=1201.66 cores=0,2,4,6
/proc/21191/task/21201/stat : 21201 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 590180 0 0 0 120050 116 0 0 25 0 5 0 599442101 8144039936 1938118 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21191/tid=21202] ppid=21188 vsize=7953164 CPUtime=1197.91 cores=0,2,4,6
/proc/21191/task/21202/stat : 21202 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 379251 0 0 0 119706 85 0 0 25 0 5 0 599442101 8144039936 1938118 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21191/tid=21203] ppid=21188 vsize=7953164 CPUtime=1201.49 cores=0,2,4,6
/proc/21191/task/21203/stat : 21203 (ttsth-5-4) R 21188 21191 20765 0 -1 4202560 612386 0 0 0 120020 129 0 0 25 0 5 0 599442101 8144039936 1938118 33554432000 4194304 4338100 140734180660368 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.25
Current children cumulated vsize (KiB) 7953164

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.2 s]
/proc/loadavg: 8.06 8.02 7.64 3/182 21250
/proc/meminfo: memFree=25386196/32951124 swapFree=67111464/67111528
[pid=21191] ppid=21188 vsize=0 CPUtime=4800.47 cores=0,2,4,6
/proc/21191/stat : 21191 (ttsth-5-4) Z 21188 21191 20765 0 -1 4203548 1940425 0 0 0 479618 429 0 0 18 0 2 0 599442066 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21191/statm: 0 0 0 0 0 0 0
[pid=21191/tid=21200] ppid=21188 vsize=0 CPUtime=1199.01 cores=0,2,4,6
/proc/21191/task/21200/stat : 21200 (ttsth-5-4) R 21188 21191 20765 0 -1 4203588 351016 0 0 0 119805 96 0 0 25 0 2 0 599442101 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.47
Current children cumulated vsize (KiB) 0

[startup+1202.31 s]
/proc/loadavg: 8.06 8.02 7.64 3/182 21250
/proc/meminfo: memFree=25386196/32951124 swapFree=67111464/67111528
[pid=21191] ppid=21188 vsize=0 CPUtime=4800.57 cores=0,2,4,6
/proc/21191/stat : 21191 (ttsth-5-4) Z 21188 21191 20765 0 -1 4203548 1940425 0 0 0 479618 439 0 0 18 0 2 0 599442066 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21191/statm: 0 0 0 0 0 0 0
[pid=21191/tid=21200] ppid=21188 vsize=0 CPUtime=1199.12 cores=0,2,4,6
/proc/21191/task/21200/stat : 21200 (ttsth-5-4) R 21188 21191 20765 0 -1 4203588 351016 0 0 0 119805 107 0 0 25 0 2 0 599442101 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.57
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.32
CPU time (s): 4800.59
CPU user time (s): 4796.19
CPU system time (s): 4.40533
CPU usage (%): 399.277
Max. virtual memory (cumulated for all children) (KiB): 7953164

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4796.18
system time used= 4.40533
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1940425
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 662
involuntary context switches= 20027

runsolver used 2.08468 second user time and 4.47532 second system time

The end

Launcher Data 

Begin job on node110 at 2011-04-29 17:48:35
IDJOB=3262310
IDBENCH=83292
IDSOLVER=1654
FILE ID=node110/3262310-1304092115
RUNJOBID= node110-1304092115-21178
PBS_JOBID= 13170568
Free space on /tmp= 72840 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/kullmann/VanDerWaerden/VanderWaerden_pd_3k/VanDerWaerden_pd_2-3-20_390.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3262310-1304092115/watcher-3262310-1304092115 -o /tmp/evaluation-result-3262310-1304092115/solver-3262310-1304092115 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262310-1304092115.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 9f9afdb81b528e7852015fad23de6353
RANDOM SEED=5293113

node110.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      30889448 kB
Buffers:        898888 kB
Cached:         692596 kB
SwapCached:         64 kB
Active:         117312 kB
Inactive:      1538520 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      30889448 kB
SwapTotal:    67111528 kB
SwapFree:     67111464 kB
Dirty:             452 kB
Writeback:           0 kB
AnonPages:       64008 kB
Mapped:          14488 kB
Slab:           341836 kB
PageTables:       4720 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   188784 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72840 MiB
End job on node110 at 2011-04-29 18:08:39