Trace number 3262486

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.52 1203.34

General information on the benchmark

Namecrafted/kullmann/VanDerWaerden/VanderWaerden_pd_3k/
VanDerWaerden_pd_2-3-23_505.cnf
MD5SUM8e7b614904543aa28491ea45c761cbd5
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark97.5682
Satisfiable
(Un)Satisfiability was proved
Number of variables253
Number of clauses34014
Sum of the clauses size150994
Maximum clause length23
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2336
Number of clauses of size 331196
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 52482

Solver Data 

0.04/0.04	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262486-1304092668.cnf 
0.04/0.04	c Using SATTIMEOUT of 4800
0.04/0.04	c Using SATRAM of 15500
0.04/0.04	c Size of subset node = 32
0.04/0.04	c Variables = 253
0.04/0.04	c Clauses = 34014
0.04/0.04	c Literals = 150994
0.04/0.04	c Reduced size:
0.04/0.04	c Variables = 253
0.04/0.04	c Clauses = 34014
0.04/0.04	c Literals = 150994
0.04/0.04	c a2
0.19/0.24	c a1
0.59/0.61	c Independent sub-problems = 1
0.59/0.61	c Sub-problem with 253 variables and 34014 clauses
0.59/0.61	c using permutation method 4
0.59/0.61	c count = 1
0.59/0.61	c Ordering finished 0.590 seconds
0.59/0.61	c mapped 0.600 seconds
0.59/0.61	c permuted 0.600 seconds
0.59/0.61	c about to sort
0.59/0.62	c sorted
0.69/0.74	c clauses now 34014
0.69/0.74	c init finished 0.610 seconds
0.69/0.74	c tree built 0.730 seconds
0.69/0.74	c 
0.69/0.74	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.69/0.74	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.69/0.74	c ................................ | ..........                    |                                  0.7         0.0             
0.69/0.74	c ................................ | ..........                  1 |                                  0.7         1.4             
0.69/0.74	c ................................ | ..........                  2 |                                  0.7         2.7             
0.69/0.74	c ................................ | ..........                  3 |                                  0.7         4.1             
0.69/0.74	c ................................ | ..........                  4 |                                  0.7         5.5             
0.69/0.74	c ................................ | ..........                  5 |                                  0.7         6.8             
0.69/0.77	c ................................ | ..........               1000 | 54           305                 0.8      1204.8        43779
95.85/24.75	c ................................ | ..........             417000 | 38651        195905             96.1      4340.6     23145832
191.69/48.79	c ................................ | ..........             647000 | 64943        298335            192.1      3368.7     38031577
287.96/72.88	c ................................ | ..........             828000 | 88352        379952            288.3      2872.1     49689537
383.82/96.86	c ................................ | ..........             992000 | 103347       447587            384.1      2582.9     58588662
480.07/120.96	c ................................ | *.........            1128000 | 114570       508367            480.3      2348.6     66799275
576.29/145.00	c ................................ | *.........            1257000 | 123800       560204            576.3      2181.2     73576254
671.73/168.97	c ................................ | *.........            1377000 | 132968       606604            672.0      2049.0     80552689
768.33/193.16	c ................................ | *.........            1495000 | 144901       651695            768.6      1945.1     86952368
864.55/217.22	c ................................ | *.........            1600000 | 151927       696860            864.6      1850.5     91330668
959.99/241.12	c ................................ | **........            1723000 | 161565       738241            960.1      1794.7     93351159
1055.86/265.14	c ................................ | **........            1849000 | 174491       776495           1056.0      1750.9     99962535
1152.50/289.34	c ................................ | **........            1971000 | 188858       813897           1152.7      1710.0    106321434
1248.33/313.38	c ................................ | **........            2091000 | 203849       845165           1248.7      1674.6    112106258
1343.80/337.28	c ................................ | **........            2200000 | 216259       878148           1344.1      1636.7    117083685
1440.43/361.40	c ................................ | ***.......            2310000 | 228663       907809           1440.4      1603.7    120289568
1536.28/385.41	c ................................ | ***.......            2407000 | 241091       931979           1536.3      1566.7    124556953
1632.11/409.44	c ................................ | ***.......            2506000 | 254282       957706           1632.3      1535.3    129695874
1727.99/433.47	c ................................ | ***.......            2602000 | 267993       985899           1728.3      1505.6    133776693
1823.83/457.45	c ................................ | ***.......            2695000 | 281626       1011932          1824.0      1477.5    135903970
1920.46/481.63	c ................................ | ****......            2792000 | 295908       1034662          1920.6      1453.7    136171546
2016.36/505.85	c ................................ | ****......            2887000 | 310584       1057756          2016.6      1431.6    136764607
2112.39/530.05	c ................................ | ****......            2985000 | 326292       1082909          2112.6      1412.9    138882568
2208.23/554.04	c ................................ | ****......            3087000 | 343065       1104387          2208.4      1397.8    141863614
2304.08/578.01	c ................................ | ****......            3182000 | 357961       1126829          2304.2      1381.0    142533032
2400.34/602.17	c ................................ | *****.....            3273000 | 373668       1151012          2400.7      1363.4    144929876
2496.18/626.16	c ................................ | *****.....            3365000 | 388842       1171155          2496.4      1347.9    148526174
2592.42/650.27	c ................................ | *****.....            3468000 | 404576       1192659          2592.7      1337.6    153457397
2688.29/674.24	c ................................ | *****.....            3559000 | 416153       1215619          2688.4      1323.8    158262292
2784.95/698.41	c ................................ | *****.....            3642000 | 425152       1239063          2785.0      1307.7    160178956
2881.17/722.59	c ................................ | ******....            3727000 | 435741       1259163          2881.5      1293.4    161295808
2977.03/746.56	c ................................ | ******....            3798000 | 444151       1277357          2977.3      1275.7    163753126
3072.47/770.45	c ................................ | ******....            3868000 | 453304       1296042          3072.7      1258.8    167559619
3168.73/794.50	c ................................ | ******....            3939000 | 462337       1313990          3168.8      1243.1    169907503
3264.58/818.58	c ................................ | ******....            4008000 | 471937       1331245          3264.9      1227.6    170086746
3360.03/842.46	c ................................ | *******...            4078000 | 480546       1346966          3360.3      1213.6    172427673
3456.69/866.69	c ................................ | *******...            4150000 | 490111       1364736          3457.1      1200.4    176506638
3552.53/890.69	c ................................ | *******...            4216000 | 499301       1383623          3552.9      1186.6    180873797
3648.79/914.78	c ................................ | *******...            4288000 | 509099       1403416          3649.1      1175.1    185680855
3745.03/938.82	c ................................ | *******...            4356000 | 518737       1418169          3745.1      1163.1    188440081
3840.88/962.86	c ................................ | ********..            4423000 | 528205       1434606          3841.1      1151.5    191072088
3936.35/986.79	c ................................ | ********..            4487000 | 538105       1455593          3936.7      1139.8    193600215
4032.59/1010.81	c ................................ | ********..            4554000 | 547124       1472131          4032.6      1129.3    198771990
4128.81/1034.93	c ................................ | ********..            4617000 | 556615       1488511          4128.9      1118.2    204002739
4223.87/1058.75	c ................................ | ********..            4678000 | 565981       1505610          4224.1      1107.5    208369649
4320.90/1083.01	c ................................ | *********.            4739000 | 575267       1521850          4320.9      1096.8    213649848
4415.97/1106.89	c ................................ | *********.            4799000 | 585057       1539293          4416.3      1086.7    217290233
4513.01/1131.10	c ................................ | *********.            4866000 | 594449       1555868          4513.0      1078.2    221160497
4608.49/1155.07	c ................................ | *********.            4930000 | 603815       1572276          4608.8      1069.7    222561785
4703.92/1178.93	c ................................ | *********.            4991000 | 613597       1586707          4704.1      1061.0    227081548

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3262486-1304092668/watcher-3262486-1304092668 -o /tmp/evaluation-result-3262486-1304092668/solver-3262486-1304092668 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262486-1304092668.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.16 7.89 7.96 3/175 25222
/proc/meminfo: memFree=30898528/32951132 swapFree=67111356/67111528
[pid=25222] ppid=25220 vsize=9408 CPUtime=0 cores=0,2,4,6
/proc/25222/stat : 25222 (ttsth-5-4) R 25220 25222 24497 0 -1 4202496 406 0 0 0 0 0 0 0 21 0 1 0 520866459 9633792 348 33554432000 4194304 4338100 140734618900624 18446744073709551615 274664469189 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/25222/statm: 2352 349 146 36 0 257 0

[startup+0.04459 s]
/proc/loadavg: 7.16 7.89 7.96 3/175 25222
/proc/meminfo: memFree=30898528/32951132 swapFree=67111356/67111528
[pid=25222] ppid=25220 vsize=13116 CPUtime=0.04 cores=0,2,4,6
/proc/25222/stat : 25222 (ttsth-5-4) R 25220 25222 24497 0 -1 4202496 1338 0 0 0 4 0 0 0 21 0 1 0 520866459 13430784 1280 33554432000 4194304 4338100 140734618900624 18446744073709551615 4221603 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/25222/statm: 3279 1280 150 36 0 1184 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 13116

[startup+0.100585 s]
/proc/loadavg: 7.16 7.89 7.96 3/175 25222
/proc/meminfo: memFree=30898528/32951132 swapFree=67111356/67111528
[pid=25222] ppid=25220 vsize=36448 CPUtime=0.09 cores=0,2,4,6
/proc/25222/stat : 25222 (ttsth-5-4) R 25220 25222 24497 0 -1 4202496 5700 0 0 0 9 0 0 0 21 0 1 0 520866459 37322752 5481 33554432000 4194304 4338100 140734618900624 18446744073709551615 274664203316 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/25222/statm: 9112 5481 577 36 0 7017 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 36448

[startup+0.300543 s]
/proc/loadavg: 7.16 7.89 7.96 3/175 25222
/proc/meminfo: memFree=30898528/32951132 swapFree=67111356/67111528
[pid=25222] ppid=25220 vsize=15920 CPUtime=0.29 cores=0,2,4,6
/proc/25222/stat : 25222 (ttsth-5-4) R 25220 25222 24497 0 -1 4202496 6409 0 0 0 29 0 0 0 22 0 1 0 520866459 16302080 1976 33554432000 4194304 4338100 140734618900624 18446744073709551615 4245205 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/25222/statm: 3980 1977 155 36 0 1885 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 15920

[startup+0.700393 s]
/proc/loadavg: 7.16 7.89 7.96 3/175 25222
/proc/meminfo: memFree=30898528/32951132 swapFree=67111356/67111528
[pid=25222] ppid=25220 vsize=35584 CPUtime=0.69 cores=0,2,4,6
/proc/25222/stat : 25222 (ttsth-5-4) R 25220 25222 24497 0 -1 4202496 11419 0 0 0 68 1 0 0 24 0 1 0 520866459 36438016 6919 33554432000 4194304 4338100 140734618900624 18446744073709551615 4278360 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/25222/statm: 8896 6919 164 36 0 6801 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 35584

[startup+1.50022 s]
/proc/loadavg: 7.16 7.89 7.96 10/184 25231
/proc/meminfo: memFree=30791108/32951132 swapFree=67111356/67111528
[pid=25222] ppid=25220 vsize=196316 CPUtime=3.75 cores=0,2,4,6
/proc/25222/stat : 25222 (ttsth-5-4) S 25220 25222 24497 0 -1 4202496 23037 0 0 0 372 3 0 0 24 0 5 0 520866459 201027584 18537 33554432000 4194304 4338100 140734618900624 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/25222/statm: 49079 18539 176 36 0 46984 0
[pid=25222/tid=25228] ppid=25220 vsize=196316 CPUtime=0.75 cores=0,2,4,6
/proc/25222/task/25228/stat : 25228 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 2823 0 0 0 75 0 0 0 18 0 5 0 520866534 201027584 18541 33554432000 4194304 4338100 140734618900624 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25222/tid=25229] ppid=25220 vsize=196316 CPUtime=0.74 cores=0,2,4,6
/proc/25222/task/25229/stat : 25229 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 1952 0 0 0 74 0 0 0 18 0 5 0 520866534 201027584 18542 33554432000 4194304 4338100 140734618900624 18446744073709551615 4239476 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25222/tid=25230] ppid=25220 vsize=196316 CPUtime=0.74 cores=0,2,4,6
/proc/25222/task/25230/stat : 25230 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 2326 0 0 0 74 0 0 0 25 0 5 0 520866534 201027584 18543 33554432000 4194304 4338100 140734618900624 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25222/tid=25231] ppid=25220 vsize=196316 CPUtime=0.74 cores=0,2,4,6
/proc/25222/task/25231/stat : 25231 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 2662 0 0 0 74 0 0 0 19 0 5 0 520866534 201027584 18543 33554432000 4194304 4338100 140734618900624 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3.75
Current children cumulated vsize (KiB) 196316

[startup+3.10094 s]
/proc/loadavg: 7.31 7.91 7.96 10/184 25231
/proc/meminfo: memFree=30664432/32951132 swapFree=67111356/67111528
[pid=25222] ppid=25220 vsize=266060 CPUtime=10.14 cores=0,2,4,6
/proc/25222/stat : 25222 (ttsth-5-4) S 25220 25222 24497 0 -1 4202496 40674 0 0 0 1008 6 0 0 24 0 5 0 520866459 272445440 36174 33554432000 4194304 4338100 140734618900624 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/25222/statm: 66515 36174 176 36 0 64420 0
[pid=25222/tid=25228] ppid=25220 vsize=266060 CPUtime=2.34 cores=0,2,4,6
/proc/25222/task/25228/stat : 25228 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 7808 0 0 0 234 0 0 0 22 0 5 0 520866534 272445440 36174 33554432000 4194304 4338100 140734618900624 18446744073709551615 4256131 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25222/tid=25229] ppid=25220 vsize=266060 CPUtime=2.33 cores=0,2,4,6
/proc/25222/task/25229/stat : 25229 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 5032 0 0 0 233 0 0 0 22 0 5 0 520866534 272445440 36174 33554432000 4194304 4338100 140734618900624 18446744073709551615 4206915 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25222/tid=25230] ppid=25220 vsize=266060 CPUtime=2.34 cores=0,2,4,6
/proc/25222/task/25230/stat : 25230 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 6597 0 0 0 233 1 0 0 25 0 5 0 520866534 272445440 36174 33554432000 4194304 4338100 140734618900624 18446744073709551615 4256993 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25222/tid=25231] ppid=25220 vsize=266060 CPUtime=2.34 cores=0,2,4,6
/proc/25222/task/25231/stat : 25231 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 7958 0 0 0 233 1 0 0 22 0 5 0 520866534 272445440 36175 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 10.14
Current children cumulated vsize (KiB) 266060

[startup+6.30039 s]
/proc/loadavg: 7.31 7.91 7.96 10/184 25231
/proc/meminfo: memFree=30348564/32951132 swapFree=67111356/67111528
[pid=25222] ppid=25220 vsize=392104 CPUtime=22.36 cores=0,2,4,6
/proc/25222/stat : 25222 (ttsth-5-4) S 25220 25222 24497 0 -1 4202496 72042 0 0 0 2221 15 0 0 24 0 5 0 520866459 401514496 67542 33554432000 4194304 4338100 140734618900624 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/25222/statm: 98026 67543 176 36 0 95931 0
[pid=25222/tid=25228] ppid=25220 vsize=392104 CPUtime=5.55 cores=0,2,4,6
/proc/25222/task/25228/stat : 25228 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 16957 0 0 0 553 2 0 0 25 0 5 0 520866534 401514496 67543 33554432000 4194304 4338100 140734618900624 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25222/tid=25229] ppid=25220 vsize=392104 CPUtime=5.28 cores=0,2,4,6
/proc/25222/task/25229/stat : 25229 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 10739 0 0 0 527 1 0 0 25 0 5 0 520866534 401514496 67544 33554432000 4194304 4338100 140734618900624 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25222/tid=25230] ppid=25220 vsize=392104 CPUtime=5.25 cores=0,2,4,6
/proc/25222/task/25230/stat : 25230 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 13011 0 0 0 519 6 0 0 25 0 5 0 520866534 401514496 67544 33554432000 4194304 4338100 140734618900624 18446744073709551615 274676622491 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25222/tid=25231] ppid=25220 vsize=392104 CPUtime=5.52 cores=0,2,4,6
/proc/25222/task/25231/stat : 25231 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 18057 0 0 0 549 3 0 0 25 0 5 0 520866534 401514496 67544 33554432000 4194304 4338100 140734618900624 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 22.36
Current children cumulated vsize (KiB) 392104

[startup+12.7003 s]
/proc/loadavg: 7.42 7.91 7.96 10/184 25232
/proc/meminfo: memFree=29853748/32951132 swapFree=67111356/67111528
[pid=25222] ppid=25220 vsize=587096 CPUtime=47.92 cores=0,2,4,6
/proc/25222/stat : 25222 (ttsth-5-4) S 25220 25222 24497 0 -1 4202496 122065 0 0 0 4767 25 0 0 24 0 5 0 520866459 601186304 117565 33554432000 4194304 4338100 140734618900624 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/25222/statm: 146774 117565 176 36 0 144679 0
[pid=25222/tid=25228] ppid=25220 vsize=587096 CPUtime=11.94 cores=0,2,4,6
/proc/25222/task/25228/stat : 25228 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 31655 0 0 0 1189 5 0 0 25 0 5 0 520866534 601186304 117565 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25222/tid=25229] ppid=25220 vsize=587096 CPUtime=11.67 cores=0,2,4,6
/proc/25222/task/25229/stat : 25229 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 19768 0 0 0 1164 3 0 0 25 0 5 0 520866534 601186304 117565 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25222/tid=25230] ppid=25220 vsize=587096 CPUtime=11.64 cores=0,2,4,6
/proc/25222/task/25230/stat : 25230 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 24491 0 0 0 1157 7 0 0 25 0 5 0 520866534 601186304 117565 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25222/tid=25231] ppid=25220 vsize=587096 CPUtime=11.88 cores=0,2,4,6
/proc/25222/task/25231/stat : 25231 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 32872 0 0 0 1182 6 0 0 25 0 5 0 520866534 601186304 117565 33554432000 4194304 4338100 140734618900624 18446744073709551615 4286315 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 47.92
Current children cumulated vsize (KiB) 587096

[startup+25.5011 s]
/proc/loadavg: 7.51 7.91 7.96 10/184 25232
/proc/meminfo: memFree=29151280/32951132 swapFree=67111356/67111528
[pid=25222] ppid=25220 vsize=907736 CPUtime=99.04 cores=0,2,4,6
/proc/25222/stat : 25222 (ttsth-5-4) S 25220 25222 24497 0 -1 4202496 201625 0 0 0 9864 40 0 0 24 0 5 0 520866459 929521664 197125 33554432000 4194304 4338100 140734618900624 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/25222/statm: 226934 197125 176 36 0 224839 0
[pid=25222/tid=25228] ppid=25220 vsize=907736 CPUtime=24.74 cores=0,2,4,6
/proc/25222/task/25228/stat : 25228 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 53524 0 0 0 2465 9 0 0 25 0 5 0 520866534 929521664 197125 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25222/tid=25229] ppid=25220 vsize=907736 CPUtime=24.47 cores=0,2,4,6
/proc/25222/task/25229/stat : 25229 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 37589 0 0 0 2440 7 0 0 25 0 5 0 520866534 929521664 197125 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25222/tid=25230] ppid=25220 vsize=907736 CPUtime=24.44 cores=0,2,4,6
/proc/25222/task/25230/stat : 25230 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 39891 0 0 0 2434 10 0 0 25 0 5 0 520866534 929521664 197125 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25222/tid=25231] ppid=25220 vsize=907736 CPUtime=24.61 cores=0,2,4,6
/proc/25222/task/25231/stat : 25231 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 57342 0 0 0 2450 11 0 0 25 0 5 0 520866534 929521664 197125 33554432000 4194304 4338100 140734618900624 18446744073709551615 4234336 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 99.04
Current children cumulated vsize (KiB) 907736

[startup+51.1006 s]
/proc/loadavg: 7.73 7.94 7.97 10/184 25232
/proc/meminfo: memFree=28157380/32951132 swapFree=67111356/67111528
[pid=25222] ppid=25220 vsize=1397420 CPUtime=201.28 cores=0,2,4,6
/proc/25222/stat : 25222 (ttsth-5-4) S 25220 25222 24497 0 -1 4202496 323603 0 0 0 20064 64 0 0 24 0 5 0 520866459 1430958080 319103 33554432000 4194304 4338100 140734618900624 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/25222/statm: 349355 319104 176 36 0 347260 0
[pid=25222/tid=25228] ppid=25220 vsize=1397420 CPUtime=50.35 cores=0,2,4,6
/proc/25222/task/25228/stat : 25228 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 82084 0 0 0 5020 15 0 0 25 0 5 0 520866534 1430958080 319104 33554432000 4194304 4338100 140734618900624 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25222/tid=25229] ppid=25220 vsize=1397420 CPUtime=50.06 cores=0,2,4,6
/proc/25222/task/25229/stat : 25229 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 72107 0 0 0 4992 14 0 0 25 0 5 0 520866534 1430958080 319104 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25222/tid=25230] ppid=25220 vsize=1397420 CPUtime=50.03 cores=0,2,4,6
/proc/25222/task/25230/stat : 25230 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 63857 0 0 0 4989 14 0 0 25 0 5 0 520866534 1430958080 319104 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25222/tid=25231] ppid=25220 vsize=1397420 CPUtime=50.07 cores=0,2,4,6
/proc/25222/task/25231/stat : 25231 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 92277 0 0 0 4990 17 0 0 25 0 5 0 520866534 1430958080 319104 33554432000 4194304 4338100 140734618900624 18446744073709551615 4285872 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 201.28
Current children cumulated vsize (KiB) 1397420

[startup+102.307 s]
/proc/loadavg: 8.01 7.98 7.98 9/184 25234
/proc/meminfo: memFree=26777860/32951132 swapFree=67111356/67111528
[pid=25222] ppid=25220 vsize=2073724 CPUtime=405.79 cores=0,2,4,6
/proc/25222/stat : 25222 (ttsth-5-4) S 25220 25222 24497 0 -1 4202496 493480 0 0 0 40481 98 0 0 24 0 5 0 520866459 2123493376 488980 33554432000 4194304 4338100 140734618900624 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/25222/statm: 518431 488980 176 36 0 516336 0
[pid=25222/tid=25228] ppid=25220 vsize=2073724 CPUtime=101.55 cores=0,2,4,6
/proc/25222/task/25228/stat : 25228 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 128477 0 0 0 10131 24 0 0 25 0 5 0 520866534 2123493376 488980 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25222/tid=25229] ppid=25220 vsize=2073724 CPUtime=101.26 cores=0,2,4,6
/proc/25222/task/25229/stat : 25229 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 111650 0 0 0 10103 23 0 0 25 0 5 0 520866534 2123493376 488980 33554432000 4194304 4338100 140734618900624 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25222/tid=25230] ppid=25220 vsize=2073724 CPUtime=101.21 cores=0,2,4,6
/proc/25222/task/25230/stat : 25230 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 100118 0 0 0 10100 21 0 0 25 0 5 0 520866534 2123493376 488980 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25222/tid=25231] ppid=25220 vsize=2073724 CPUtime=100.98 cores=0,2,4,6
/proc/25222/task/25231/stat : 25231 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 139956 0 0 0 10072 26 0 0 25 0 5 0 520866534 2123493376 488980 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 405.79
Current children cumulated vsize (KiB) 2073724

[startup+162.3 s]
/proc/loadavg: 8.11 8.01 7.99 10/184 25236
/proc/meminfo: memFree=25666284/32951132 swapFree=67111356/67111528
[pid=25222] ppid=25220 vsize=2641116 CPUtime=645.39 cores=0,2,4,6
/proc/25222/stat : 25222 (ttsth-5-4) S 25220 25222 24497 0 -1 4202496 636079 0 0 0 64405 134 0 0 24 0 5 0 520866459 2704502784 631579 33554432000 4194304 4338100 140734618900624 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/25222/statm: 660279 631579 176 36 0 658184 0
[pid=25222/tid=25228] ppid=25220 vsize=2641116 CPUtime=161.55 cores=0,2,4,6
/proc/25222/task/25228/stat : 25228 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 161940 0 0 0 16122 33 0 0 25 0 5 0 520866534 2704502784 631579 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25222/tid=25229] ppid=25220 vsize=2641116 CPUtime=161.24 cores=0,2,4,6
/proc/25222/task/25229/stat : 25229 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 141653 0 0 0 16093 31 0 0 25 0 5 0 520866534 2704502784 631579 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25222/tid=25230] ppid=25220 vsize=2641116 CPUtime=161.18 cores=0,2,4,6
/proc/25222/task/25230/stat : 25230 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 130580 0 0 0 16088 30 0 0 25 0 5 0 520866534 2704502784 631579 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25222/tid=25231] ppid=25220 vsize=2641116 CPUtime=160.66 cores=0,2,4,6
/proc/25222/task/25231/stat : 25231 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 188627 0 0 0 16028 38 0 0 25 0 5 0 520866534 2704502784 631579 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 645.39
Current children cumulated vsize (KiB) 2641116


################
# More data... #
################

/proc/25222/task/25231/stat : 25231 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 399930 0 0 0 75845 100 0 0 25 0 5 0 520866534 5687123968 1351918 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3040.15
Current children cumulated vsize (KiB) 5553832

[startup+822.301 s]
/proc/loadavg: 8.18 8.11 8.04 10/184 25265
/proc/meminfo: memFree=18718596/32951132 swapFree=67111356/67111528
[pid=25222] ppid=25220 vsize=5676352 CPUtime=3279.76 cores=0,2,4,6
/proc/25222/stat : 25222 (ttsth-5-4) S 25220 25222 24497 0 -1 4202496 1387557 0 0 0 327649 327 0 0 24 0 5 0 520866459 5812584448 1382928 33554432000 4194304 4338100 140734618900624 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/25222/statm: 1419088 1382928 176 36 0 1416993 0
[pid=25222/tid=25228] ppid=25220 vsize=5676352 CPUtime=821.29 cores=0,2,4,6
/proc/25222/task/25228/stat : 25228 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 440092 0 0 0 82032 97 0 0 25 0 5 0 520866534 5812584448 1382928 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25222/tid=25229] ppid=25220 vsize=5676352 CPUtime=818.05 cores=0,2,4,6
/proc/25222/task/25229/stat : 25229 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 273364 0 0 0 81741 64 0 0 25 0 5 0 520866534 5812584448 1382928 33554432000 4194304 4338100 140734618900624 18446744073709551615 4284177 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25222/tid=25230] ppid=25220 vsize=5676352 CPUtime=820.2 cores=0,2,4,6
/proc/25222/task/25230/stat : 25230 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 244551 0 0 0 81961 59 0 0 25 0 5 0 520866534 5812584448 1382928 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25222/tid=25231] ppid=25220 vsize=5676352 CPUtime=819.45 cores=0,2,4,6
/proc/25222/task/25231/stat : 25231 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 416271 0 0 0 81841 104 0 0 25 0 5 0 520866534 5812584448 1382928 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3279.76
Current children cumulated vsize (KiB) 5676352

[startup+882.301 s]
/proc/loadavg: 8.18 8.12 8.04 10/184 25267
/proc/meminfo: memFree=18168756/32951132 swapFree=67111356/67111528
[pid=25222] ppid=25220 vsize=5955404 CPUtime=3519.38 cores=0,2,4,6
/proc/25222/stat : 25222 (ttsth-5-4) S 25220 25222 24497 0 -1 4202496 1458081 0 0 0 351597 341 0 0 24 0 5 0 520866459 6098333696 1453452 33554432000 4194304 4338100 140734618900624 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/25222/statm: 1488851 1453452 176 36 0 1486756 0
[pid=25222/tid=25228] ppid=25220 vsize=5955404 CPUtime=881.28 cores=0,2,4,6
/proc/25222/task/25228/stat : 25228 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 463483 0 0 0 88026 102 0 0 25 0 5 0 520866534 6098333696 1453452 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25222/tid=25229] ppid=25220 vsize=5955404 CPUtime=877.71 cores=0,2,4,6
/proc/25222/task/25229/stat : 25229 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 286119 0 0 0 87704 67 0 0 25 0 5 0 520866534 6098333696 1453452 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25222/tid=25230] ppid=25220 vsize=5955404 CPUtime=880.18 cores=0,2,4,6
/proc/25222/task/25230/stat : 25230 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 252128 0 0 0 87957 61 0 0 25 0 5 0 520866534 6098333696 1453452 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25222/tid=25231] ppid=25220 vsize=5955404 CPUtime=879.44 cores=0,2,4,6
/proc/25222/task/25231/stat : 25231 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 443072 0 0 0 87836 108 0 0 25 0 5 0 520866534 6098333696 1453452 33554432000 4194304 4338100 140734618900624 18446744073709551615 4256139 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3519.38
Current children cumulated vsize (KiB) 5955404

[startup+942.3 s]
/proc/loadavg: 8.23 8.14 8.05 10/184 25268
/proc/meminfo: memFree=17585928/32951132 swapFree=67111356/67111528
[pid=25222] ppid=25220 vsize=6268732 CPUtime=3759.01 cores=0,2,4,6
/proc/25222/stat : 25222 (ttsth-5-4) S 25220 25222 24497 0 -1 4202496 1536187 0 0 0 375542 359 0 0 24 0 5 0 520866459 6419181568 1531558 33554432000 4194304 4338100 140734618900624 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/25222/statm: 1567183 1531558 176 36 0 1565088 0
[pid=25222/tid=25228] ppid=25220 vsize=6268732 CPUtime=941.25 cores=0,2,4,6
/proc/25222/task/25228/stat : 25228 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 480077 0 0 0 94020 105 0 0 25 0 5 0 520866534 6419181568 1531558 33554432000 4194304 4338100 140734618900624 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25222/tid=25229] ppid=25220 vsize=6268732 CPUtime=937.36 cores=0,2,4,6
/proc/25222/task/25229/stat : 25229 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 300899 0 0 0 93665 71 0 0 25 0 5 0 520866534 6419181568 1531558 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25222/tid=25230] ppid=25220 vsize=6268732 CPUtime=940.17 cores=0,2,4,6
/proc/25222/task/25230/stat : 25230 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 273729 0 0 0 93952 65 0 0 25 0 5 0 520866534 6419181568 1531558 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25222/tid=25231] ppid=25220 vsize=6268732 CPUtime=939.45 cores=0,2,4,6
/proc/25222/task/25231/stat : 25231 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 468203 0 0 0 93831 114 0 0 25 0 5 0 520866534 6419181568 1531558 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3759.01
Current children cumulated vsize (KiB) 6268732

[startup+1002.3 s]
/proc/loadavg: 8.13 8.13 8.05 10/184 25270
/proc/meminfo: memFree=17099812/32951132 swapFree=67111356/67111528
[pid=25222] ppid=25220 vsize=6513376 CPUtime=3998.64 cores=0,2,4,6
/proc/25222/stat : 25222 (ttsth-5-4) S 25220 25222 24497 0 -1 4202496 1596582 0 0 0 399492 372 0 0 24 0 5 0 520866459 6669697024 1591953 33554432000 4194304 4338100 140734618900624 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/25222/statm: 1628344 1591953 176 36 0 1626249 0
[pid=25222/tid=25228] ppid=25220 vsize=6513376 CPUtime=1001.25 cores=0,2,4,6
/proc/25222/task/25228/stat : 25228 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 488570 0 0 0 100018 107 0 0 25 0 5 0 520866534 6669697024 1591953 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25222/tid=25229] ppid=25220 vsize=6513376 CPUtime=997.02 cores=0,2,4,6
/proc/25222/task/25229/stat : 25229 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 312269 0 0 0 99628 74 0 0 25 0 5 0 520866534 6669697024 1591953 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25222/tid=25230] ppid=25220 vsize=6513376 CPUtime=1000.16 cores=0,2,4,6
/proc/25222/task/25230/stat : 25230 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 286927 0 0 0 99948 68 0 0 25 0 5 0 520866534 6669697024 1591953 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25222/tid=25231] ppid=25220 vsize=6513376 CPUtime=999.44 cores=0,2,4,6
/proc/25222/task/25231/stat : 25231 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 495537 0 0 0 99824 120 0 0 25 0 5 0 520866534 6669697024 1591953 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3998.64
Current children cumulated vsize (KiB) 6513376

[startup+1062.3 s]
/proc/loadavg: 8.09 8.12 8.05 9/184 25271
/proc/meminfo: memFree=16537188/32951132 swapFree=67111356/67111528
[pid=25222] ppid=25220 vsize=6892524 CPUtime=4238.25 cores=0,2,4,6
/proc/25222/stat : 25222 (ttsth-5-4) S 25220 25222 24497 0 -1 4202496 1692145 0 0 0 423433 392 0 0 24 0 5 0 520866459 7057944576 1687516 33554432000 4194304 4338100 140734618900624 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/25222/statm: 1723131 1687516 176 36 0 1721036 0
[pid=25222/tid=25228] ppid=25220 vsize=6892524 CPUtime=1061.25 cores=0,2,4,6
/proc/25222/task/25228/stat : 25228 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 513482 0 0 0 106012 113 0 0 25 0 5 0 520866534 7057944576 1687516 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25222/tid=25229] ppid=25220 vsize=6892524 CPUtime=1056.65 cores=0,2,4,6
/proc/25222/task/25229/stat : 25229 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 330871 0 0 0 105586 79 0 0 25 0 5 0 520866534 7057944576 1687516 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25222/tid=25230] ppid=25220 vsize=6892524 CPUtime=1060.13 cores=0,2,4,6
/proc/25222/task/25230/stat : 25230 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 300802 0 0 0 105943 70 0 0 25 0 5 0 520866534 7057944576 1687516 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25222/tid=25231] ppid=25220 vsize=6892524 CPUtime=1059.44 cores=0,2,4,6
/proc/25222/task/25231/stat : 25231 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 533711 0 0 0 105817 127 0 0 25 0 5 0 520866534 7057944576 1687516 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4238.25
Current children cumulated vsize (KiB) 6892524

[startup+1122.3 s]
/proc/loadavg: 8.12 8.12 8.05 10/184 25273
/proc/meminfo: memFree=16062216/32951132 swapFree=67111356/67111528
[pid=25222] ppid=25220 vsize=7234136 CPUtime=4477.87 cores=0,2,4,6
/proc/25222/stat : 25222 (ttsth-5-4) S 25220 25222 24497 0 -1 4202496 1777624 0 0 0 447379 408 0 0 24 0 5 0 520866459 7407755264 1772995 33554432000 4194304 4338100 140734618900624 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/25222/statm: 1808534 1772995 176 36 0 1806439 0
[pid=25222/tid=25228] ppid=25220 vsize=7234136 CPUtime=1121.25 cores=0,2,4,6
/proc/25222/task/25228/stat : 25228 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 534100 0 0 0 112008 117 0 0 25 0 5 0 520866534 7407755264 1772995 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25222/tid=25229] ppid=25220 vsize=7234136 CPUtime=1116.28 cores=0,2,4,6
/proc/25222/task/25229/stat : 25229 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 347224 0 0 0 111547 81 0 0 25 0 5 0 520866534 7407755264 1772995 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25222/tid=25230] ppid=25220 vsize=7234136 CPUtime=1120.12 cores=0,2,4,6
/proc/25222/task/25230/stat : 25230 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 312882 0 0 0 111939 73 0 0 25 0 5 0 520866534 7407755264 1772995 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25222/tid=25231] ppid=25220 vsize=7234136 CPUtime=1119.44 cores=0,2,4,6
/proc/25222/task/25231/stat : 25231 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 570139 0 0 0 111811 133 0 0 25 0 5 0 520866534 7407755264 1772995 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4477.87
Current children cumulated vsize (KiB) 7234136

[startup+1182.3 s]
/proc/loadavg: 8.09 8.11 8.05 10/184 25275
/proc/meminfo: memFree=15596688/32951132 swapFree=67111356/67111528
[pid=25222] ppid=25220 vsize=7478648 CPUtime=4717.5 cores=0,2,4,6
/proc/25222/stat : 25222 (ttsth-5-4) S 25220 25222 24497 0 -1 4202496 1838584 0 0 0 471331 419 0 0 24 0 5 0 520866459 7658135552 1833955 33554432000 4194304 4338100 140734618900624 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/25222/statm: 1869662 1833955 176 36 0 1867567 0
[pid=25222/tid=25228] ppid=25220 vsize=7478648 CPUtime=1181.25 cores=0,2,4,6
/proc/25222/task/25228/stat : 25228 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 552626 0 0 0 118004 121 0 0 25 0 5 0 520866534 7658135552 1833955 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25222/tid=25229] ppid=25220 vsize=7478648 CPUtime=1175.93 cores=0,2,4,6
/proc/25222/task/25229/stat : 25229 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 356668 0 0 0 117509 84 0 0 25 0 5 0 520866534 7658135552 1833955 33554432000 4194304 4338100 140734618900624 18446744073709551615 4206899 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25222/tid=25230] ppid=25220 vsize=7478648 CPUtime=1180.1 cores=0,2,4,6
/proc/25222/task/25230/stat : 25230 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 320647 0 0 0 117936 74 0 0 25 0 5 0 520866534 7658135552 1833955 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25222/tid=25231] ppid=25220 vsize=7478648 CPUtime=1179.45 cores=0,2,4,6
/proc/25222/task/25231/stat : 25231 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 595364 0 0 0 117808 137 0 0 25 0 5 0 520866534 7658135552 1833955 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4717.5
Current children cumulated vsize (KiB) 7478648



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203 s]
/proc/loadavg: 8.06 8.10 8.05 5/178 25276
/proc/meminfo: memFree=23425840/32951132 swapFree=67111356/67111528
[pid=25222] ppid=25220 vsize=7603900 CPUtime=4800.16 cores=0,2,4,6
/proc/25222/stat : 25222 (ttsth-5-4) S 25220 25222 24497 0 -1 4202496 1869712 0 0 0 479590 426 0 0 24 0 5 0 520866459 7786393600 1865083 33554432000 4194304 4338100 140734618900624 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/25222/statm: 1900975 1865083 176 36 0 1898880 0
[pid=25222/tid=25228] ppid=25220 vsize=7603900 CPUtime=1201.96 cores=0,2,4,6
/proc/25222/task/25228/stat : 25228 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 560949 0 0 0 120073 123 0 0 25 0 5 0 520866534 7786393600 1865083 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25222/tid=25229] ppid=25220 vsize=7603900 CPUtime=1196.5 cores=0,2,4,6
/proc/25222/task/25229/stat : 25229 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 360414 0 0 0 119564 86 0 0 25 0 5 0 520866534 7786393600 1865083 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25222/tid=25230] ppid=25220 vsize=7603900 CPUtime=1200.8 cores=0,2,4,6
/proc/25222/task/25230/stat : 25230 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 328235 0 0 0 120004 76 0 0 25 0 5 0 520866534 7786393600 1865083 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254413 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25222/tid=25231] ppid=25220 vsize=7603900 CPUtime=1200.14 cores=0,2,4,6
/proc/25222/task/25231/stat : 25231 (ttsth-5-4) R 25220 25222 24497 0 -1 4202560 606835 0 0 0 119875 139 0 0 25 0 5 0 520866534 7786393600 1865083 33554432000 4194304 4338100 140734618900624 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.16
Current children cumulated vsize (KiB) 7603900

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.2 s]
/proc/loadavg: 8.06 8.10 8.05 5/178 25276
/proc/meminfo: memFree=23425840/32951132 swapFree=67111356/67111528
[pid=25222] ppid=25220 vsize=0 CPUtime=4800.37 cores=0,2,4,6
/proc/25222/stat : 25222 (ttsth-5-4) Z 25220 25222 24497 0 -1 4203548 1869723 0 0 0 479592 445 0 0 18 0 2 0 520866459 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/25222/statm: 0 0 0 0 0 0 0
[pid=25222/tid=25229] ppid=25220 vsize=0 CPUtime=1196.69 cores=0,2,4,6
/proc/25222/task/25229/stat : 25229 (ttsth-5-4) R 25220 25222 24497 0 -1 4203588 360415 0 0 0 119564 105 0 0 25 0 2 0 520866534 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.37
Current children cumulated vsize (KiB) 0

[startup+1203.3 s]
/proc/loadavg: 8.06 8.10 8.05 2/176 25277
/proc/meminfo: memFree=29927756/32951132 swapFree=67111356/67111528
[pid=25222] ppid=25220 vsize=0 CPUtime=4800.48 cores=0,2,4,6
/proc/25222/stat : 25222 (ttsth-5-4) Z 25220 25222 24497 0 -1 4203548 1869723 0 0 0 479592 456 0 0 18 0 2 0 520866459 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/25222/statm: 0 0 0 0 0 0 0
[pid=25222/tid=25229] ppid=25220 vsize=0 CPUtime=1196.79 cores=0,2,4,6
/proc/25222/task/25229/stat : 25229 (ttsth-5-4) R 25220 25222 24497 0 -1 4203588 360415 0 0 0 119564 115 0 0 25 0 2 0 520866534 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.48
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1203.34
CPU time (s): 4800.52
CPU user time (s): 4795.93
CPU system time (s): 4.5933
CPU usage (%): 398.933
Max. virtual memory (cumulated for all children) (KiB): 7603900

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4795.93
system time used= 4.5933
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1869723
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1190
involuntary context switches= 20799

runsolver used 1.81572 second user time and 4.75028 second system time

The end

Launcher Data 

Begin job on node131 at 2011-04-29 17:57:48
IDJOB=3262486
IDBENCH=83313
IDSOLVER=1654
FILE ID=node131/3262486-1304092668
RUNJOBID= node131-1304092668-25202
PBS_JOBID= 13170548
Free space on /tmp= 71384 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/kullmann/VanDerWaerden/VanderWaerden_pd_3k/VanDerWaerden_pd_2-3-23_505.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3262486-1304092668/watcher-3262486-1304092668 -o /tmp/evaluation-result-3262486-1304092668/solver-3262486-1304092668 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262486-1304092668.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 8e7b614904543aa28491ea45c761cbd5
RANDOM SEED=410564026

node131.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      30898808 kB
Buffers:        190456 kB
Cached:        1484624 kB
SwapCached:        168 kB
Active:         135096 kB
Inactive:      1610372 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      30898808 kB
SwapTotal:    67111528 kB
SwapFree:     67111356 kB
Dirty:            1364 kB
Writeback:           0 kB
AnonPages:       69528 kB
Mapped:          15528 kB
Slab:           243004 kB
PageTables:       4848 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS:   214508 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71384 MiB
End job on node131 at 2011-04-29 18:17:53