Trace number 3259978

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.28 1221.29

General information on the benchmark

Namecrafted/mosoi/
289/289-sat-4x35.cnf
MD5SUM3a0c9ef65cb2e498876675e6a31f16d6
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.007998
Satisfiable
(Un)Satisfiability was proved
Number of variables560
Number of clauses15260
Sum of the clauses size59360
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2840
Number of clauses of size 30
Number of clauses of size 414420
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.01	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259978-1304061417.cnf 
0.00/0.01	c Using SATTIMEOUT of 4800
0.00/0.01	c Using SATRAM of 15500
0.00/0.01	c Size of subset node = 32
0.00/0.01	c Variables = 560
0.00/0.01	c Clauses = 15260
0.00/0.01	c Literals = 59360
0.00/0.01	c Reduced size:
0.00/0.01	c Variables = 560
0.00/0.01	c Clauses = 15260
0.00/0.01	c Literals = 59360
0.00/0.01	c a2
0.01/0.04	c a1
0.79/0.88	c Independent sub-problems = 1
0.79/0.88	c Sub-problem with 560 variables and 15260 clauses
0.79/0.88	c using permutation method 4
0.79/0.88	c count = 2
0.79/0.88	c Ordering finished 0.870 seconds
0.79/0.88	c mapped 0.870 seconds
0.79/0.88	c permuted 0.870 seconds
0.79/0.88	c about to sort
0.79/0.88	c sorted
0.89/0.91	c clauses now 15260
0.89/0.91	c init finished 0.880 seconds
0.89/0.91	c tree built 0.900 seconds
0.89/0.91	c 
0.89/0.91	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.89/0.91	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.89/0.91	c ................................ | ..........                    |                                  0.9         0.0             
0.89/0.91	c ................................ | ..........                  1 |                                  0.9         1.1             
0.89/0.91	c ................................ | ..........                  2 |                                  0.9         2.2             
0.89/0.91	c ................................ | ..........                  3 |                                  0.9         3.3             
0.89/0.91	c ................................ | ..........                  4 |                                  0.9         4.4             
0.89/0.91	c ................................ | ..........                  5 |                                  0.9         5.6             
0.89/0.91	c ................................ | ..........               1000 | 14           1219                0.9      1098.9         7086
95.79/24.86	c ................................ | ..........            1424000 | 326163       2525720            96.0     14828.7     20754779
191.79/49.07	c ................................ | ..........            2293000 | 560946       3731639           192.1     11937.7     32350605
287.61/73.39	c ................................ | ..........            3624000 | 909568       5591733           288.0     12583.3     43518184
383.83/97.64	c ................................ | ..........            4323000 | 1101674      6503088           384.0     11257.5     48353532
479.81/121.97	c ................................ | *.........            5141000 | 1345862      7358643           480.1     10708.4     51075117
575.89/146.44	c ................................ | *.........            6254000 | 1660959      8653113           576.1     10856.5     59159219
671.70/170.88	c ................................ | *.........            7444000 | 1988022      10078729          672.0     11077.1     63071872
767.83/195.45	c ................................ | *.........            8877000 | 2380008      11699712          768.0     11558.1     68897332
863.95/219.66	c ................................ | *.........            9369000 | 2509155      12400767          864.2     10841.0     71424545
960.02/243.84	c ................................ | **........            9766000 | 2630414      12838502          960.2     10170.7     74764239
1055.95/268.04	c ................................ | **........           10216000 | 2766219      13309377         1056.1      9673.0     78228022
1151.88/292.34	c ................................ | **........           10802000 | 2944721      13890656         1152.0      9376.3     82115964
1247.83/316.77	c ................................ | **........           11657000 | 3196272      14762794         1248.1      9339.6     85272951
1343.83/341.16	c ................................ | **........           12122000 | 3344186      15180005         1344.1      9018.8     87681085
1439.97/365.49	c ................................ | ***.......           12706000 | 3525401      15694793         1440.3      8821.5     88634731
1536.05/390.01	c ................................ | ***.......           13638000 | 3802411      16595733         1536.1      8878.3     89325363
1631.98/414.57	c ................................ | ***.......           14710000 | 4102355      17812177         1632.2      9012.2     89529291
1727.66/439.08	c ................................ | ***.......           15790000 | 4398913      19040481         1728.0      9137.7     91371780
1824.18/463.73	c ................................ | ***.......           16838000 | 4675724      20344781         1824.3      9229.7     93028359
1920.04/488.21	c ................................ | ****......           17474000 | 4882547      20766938         1920.1      9100.7     93430112
2015.84/512.83	c ................................ | ****......           18256000 | 5136026      21292664         2016.0      9055.6     95102813
2111.90/537.93	c ................................ | ****......           19346000 | 5462922      22182881         2112.0      9160.0     98261640
2208.00/562.70	c ................................ | ****......           20372000 | 5769607      23014823         2208.0      9226.4     99441641
2303.68/587.58	c ................................ | ****......           21491000 | 6085706      24114956         2304.0      9327.6    100643752
2401.91/612.82	c ................................ | *****.....           22788000 | 6428996      25543489         2402.0      9487.1    102139218
2496.63/636.62	c ................................ | *****.....           22935000 | 6473755      25719895         2496.7      9186.0    102345463
2592.47/660.79	c ................................ | *****.....           23107000 | 6525617      25929402         2592.8      8911.8    103237920
2687.98/684.85	c ................................ | *****.....           23392000 | 6611515      26244471         2688.2      8701.8    105677585
2784.78/709.16	c ................................ | *****.....           23640000 | 6685942      26509437         2785.0      8488.2    107135399
2879.97/733.14	c ................................ | ******....           23894000 | 6764448      26773733         2880.2      8296.1    108344015
2975.97/757.59	c ................................ | ******....           24323000 | 6895456      27192339         2976.3      8172.1    109012772
3072.00/782.02	c ................................ | ******....           24883000 | 7067410      27685904         3072.1      8099.7    111005520
3167.90/806.24	c ................................ | ******....           25146000 | 7145853      27953748         3168.1      7937.3    113499631
3263.66/830.39	c ................................ | ******....           25398000 | 7224180      28214725         3264.0      7781.2    116432853
3359.70/854.59	c ................................ | *******...           25826000 | 7353628      28624892         3360.1      7686.1    121449425
3456.05/878.94	c ................................ | *******...           26160000 | 7457197      28937578         3456.2      7569.1    124399307
3551.87/903.33	c ................................ | *******...           26734000 | 7630643      29446860         3552.0      7526.4    127767902
3648.04/927.91	c ................................ | *******...           27459000 | 7845880      30120239         3648.1      7527.0    128640618
3743.77/952.46	c ................................ | *******...           28054000 | 8022999      30673112         3744.0      7493.0    130236977
3840.01/977.22	c ................................ | ********..           28967000 | 8291412      31466573         3840.1      7543.3    132954710
3935.88/1001.67	c ................................ | ********..           29461000 | 8429262      32026056         3936.2      7484.6    135460694
4031.97/1025.94	c ................................ | ********..           29678000 | 8499335      32232751         4032.2      7360.3    135500414
4128.20/1050.22	c ................................ | ********..           30004000 | 8601178      32529887         4128.3      7267.9    135500414
4223.97/1074.45	c ................................ | ********..           30289000 | 8691552      32792115         4224.2      7170.4    136405960
4319.71/1098.89	c ................................ | *********.           30852000 | 8862252      33285586         4320.1      7141.5    138660161
4416.18/1123.40	c ................................ | *********.           31438000 | 9039900      33787886         4416.2      7118.8    141444839
4511.85/1147.63	c ................................ | *********.           31715000 | 9127953      34037003         4512.0      7029.0    142838188
4607.98/1172.14	c ................................ | *********.           32187000 | 9271948      34445345         4608.1      6984.8    144162124
4704.37/1196.50	c ................................ | *********.           32576000 | 9390929      34788079         4704.4      6924.6    145469652
4799.78/1221.07	c CPU limit (4800) reached
4799.78/1221.07	c Time = 4800.06 seconds
4799.78/1221.07	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3259978-1304061417/watcher-3259978-1304061417 -o /tmp/evaluation-result-3259978-1304061417/solver-3259978-1304061417 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259978-1304061417.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.08 8.09 8.02 5/181 29944
/proc/meminfo: memFree=26585372/32951124 swapFree=67111524/67111528
[pid=29944] ppid=29942 vsize=9408 CPUtime=0 cores=0,2,4,6
/proc/29944/stat : 29944 (ttsth-5-4) R 29942 29944 29500 0 -1 4202496 403 0 0 0 0 0 0 0 25 0 1 0 498557169 9633792 345 33554432000 4194304 4338100 140734615176816 18446744073709551615 4223897 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29944/statm: 2352 345 147 36 0 257 0

[startup+0.0255551 s]
/proc/loadavg: 8.08 8.09 8.02 5/181 29944
/proc/meminfo: memFree=26585372/32951124 swapFree=67111524/67111528
[pid=29944] ppid=29942 vsize=15244 CPUtime=0.01 cores=0,2,4,6
/proc/29944/stat : 29944 (ttsth-5-4) R 29942 29944 29500 0 -1 4202496 1894 0 0 0 1 0 0 0 25 0 1 0 498557169 15609856 1835 33554432000 4194304 4338100 140734615176816 18446744073709551615 228094846168 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29944/statm: 3811 1835 152 36 0 1716 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 15244

[startup+0.100541 s]
/proc/loadavg: 8.08 8.09 8.02 5/181 29944
/proc/meminfo: memFree=26585372/32951124 swapFree=67111524/67111528
[pid=29944] ppid=29942 vsize=15324 CPUtime=0.09 cores=0,2,4,6
/proc/29944/stat : 29944 (ttsth-5-4) R 29942 29944 29500 0 -1 4202496 1957 0 0 0 9 0 0 0 25 0 1 0 498557169 15691776 1868 33554432000 4194304 4338100 140734615176816 18446744073709551615 4295667 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29944/statm: 3831 1868 155 36 0 1736 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 15324

[startup+0.300524 s]
/proc/loadavg: 8.08 8.09 8.02 5/181 29944
/proc/meminfo: memFree=26585372/32951124 swapFree=67111524/67111528
[pid=29944] ppid=29942 vsize=15324 CPUtime=0.29 cores=0,2,4,6
/proc/29944/stat : 29944 (ttsth-5-4) R 29942 29944 29500 0 -1 4202496 1957 0 0 0 29 0 0 0 25 0 1 0 498557169 15691776 1868 33554432000 4194304 4338100 140734615176816 18446744073709551615 4295986 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29944/statm: 3831 1868 155 36 0 1736 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 15324

[startup+0.70043 s]
/proc/loadavg: 8.08 8.09 8.02 5/181 29944
/proc/meminfo: memFree=26585372/32951124 swapFree=67111524/67111528
[pid=29944] ppid=29942 vsize=15324 CPUtime=0.69 cores=0,2,4,6
/proc/29944/stat : 29944 (ttsth-5-4) R 29942 29944 29500 0 -1 4202496 1957 0 0 0 69 0 0 0 25 0 1 0 498557169 15691776 1868 33554432000 4194304 4338100 140734615176816 18446744073709551615 4295904 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29944/statm: 3831 1868 155 36 0 1736 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 15324

[startup+1.50025 s]
/proc/loadavg: 7.60 7.98 7.99 10/186 29980
/proc/meminfo: memFree=26540332/32951124 swapFree=67111524/67111528
[pid=29944] ppid=29942 vsize=300028 CPUtime=3.24 cores=0,2,4,6
/proc/29944/stat : 29944 (ttsth-5-4) S 29942 29944 29500 0 -1 4202496 12731 0 0 0 322 2 0 0 25 0 5 0 498557169 307228672 12004 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29944/statm: 75007 12004 175 36 0 72912 0
[pid=29944/tid=29977] ppid=29942 vsize=300028 CPUtime=0.58 cores=0,2,4,6
/proc/29944/task/29977/stat : 29977 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 2533 0 0 0 58 0 0 0 18 0 5 0 498557260 307228672 12004 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29944/tid=29978] ppid=29942 vsize=300028 CPUtime=0.58 cores=0,2,4,6
/proc/29944/task/29978/stat : 29978 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 1766 0 0 0 58 0 0 0 25 0 5 0 498557260 307228672 12004 33554432000 4194304 4338100 140734615176816 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29944/tid=29979] ppid=29942 vsize=300028 CPUtime=0.58 cores=0,2,4,6
/proc/29944/task/29979/stat : 29979 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 2304 0 0 0 58 0 0 0 25 0 5 0 498557260 307228672 12004 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29944/tid=29980] ppid=29942 vsize=300028 CPUtime=0.57 cores=0,2,4,6
/proc/29944/task/29980/stat : 29980 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 2781 0 0 0 57 0 0 0 25 0 5 0 498557260 307228672 12004 33554432000 4194304 4338100 140734615176816 18446744073709551615 4229545 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3.24
Current children cumulated vsize (KiB) 300028

[startup+3.10095 s]
/proc/loadavg: 7.60 7.98 7.99 10/186 29980
/proc/meminfo: memFree=26464196/32951124 swapFree=67111524/67111528
[pid=29944] ppid=29942 vsize=366904 CPUtime=9.63 cores=0,2,4,6
/proc/29944/stat : 29944 (ttsth-5-4) S 29942 29944 29500 0 -1 4202496 30765 0 0 0 956 7 0 0 25 0 5 0 498557169 375709696 30038 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29944/statm: 91726 30038 175 36 0 89631 0
[pid=29944/tid=29977] ppid=29942 vsize=366904 CPUtime=2.18 cores=0,2,4,6
/proc/29944/task/29977/stat : 29977 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 7266 0 0 0 216 2 0 0 21 0 5 0 498557260 375709696 30038 33554432000 4194304 4338100 140734615176816 18446744073709551615 4256087 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29944/tid=29978] ppid=29942 vsize=366904 CPUtime=2.18 cores=0,2,4,6
/proc/29944/task/29978/stat : 29978 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 4405 0 0 0 217 1 0 0 25 0 5 0 498557260 375709696 30038 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29944/tid=29979] ppid=29942 vsize=366904 CPUtime=2.17 cores=0,2,4,6
/proc/29944/task/29979/stat : 29979 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 6954 0 0 0 216 1 0 0 19 0 5 0 498557260 375709696 30038 33554432000 4194304 4338100 140734615176816 18446744073709551615 4255947 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29944/tid=29980] ppid=29942 vsize=366904 CPUtime=2.16 cores=0,2,4,6
/proc/29944/task/29980/stat : 29980 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 8793 0 0 0 215 1 0 0 25 0 5 0 498557260 375709696 30039 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 9.63
Current children cumulated vsize (KiB) 366904

[startup+6.30036 s]
/proc/loadavg: 7.60 7.98 7.99 9/186 29980
/proc/meminfo: memFree=26280972/32951124 swapFree=67111524/67111528
[pid=29944] ppid=29942 vsize=490848 CPUtime=22.35 cores=0,2,4,6
/proc/29944/stat : 29944 (ttsth-5-4) S 29942 29944 29500 0 -1 4202496 62274 0 0 0 2219 16 0 0 25 0 5 0 498557169 502628352 61547 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29944/statm: 122712 61547 175 36 0 120617 0
[pid=29944/tid=29977] ppid=29942 vsize=490848 CPUtime=5.36 cores=0,2,4,6
/proc/29944/task/29977/stat : 29977 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 14857 0 0 0 532 4 0 0 25 0 5 0 498557260 502628352 61547 33554432000 4194304 4338100 140734615176816 18446744073709551615 228094846394 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29944/tid=29978] ppid=29942 vsize=490848 CPUtime=5.35 cores=0,2,4,6
/proc/29944/task/29978/stat : 29978 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 9786 0 0 0 532 3 0 0 18 0 5 0 498557260 502628352 61547 33554432000 4194304 4338100 140734615176816 18446744073709551615 4256384 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29944/tid=29979] ppid=29942 vsize=490848 CPUtime=5.36 cores=0,2,4,6
/proc/29944/task/29979/stat : 29979 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 15795 0 0 0 532 4 0 0 20 0 5 0 498557260 502628352 61547 33554432000 4194304 4338100 140734615176816 18446744073709551615 4258020 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29944/tid=29980] ppid=29942 vsize=490848 CPUtime=5.34 cores=0,2,4,6
/proc/29944/task/29980/stat : 29980 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 18489 0 0 0 531 3 0 0 23 0 5 0 498557260 502628352 61548 33554432000 4194304 4338100 140734615176816 18446744073709551615 4240540 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 22.35
Current children cumulated vsize (KiB) 490848

[startup+12.7012 s]
/proc/loadavg: 7.66 7.98 7.99 9/186 29980
/proc/meminfo: memFree=26000432/32951124 swapFree=67111524/67111528
[pid=29944] ppid=29942 vsize=662500 CPUtime=47.86 cores=0,2,4,6
/proc/29944/stat : 29944 (ttsth-5-4) S 29942 29944 29500 0 -1 4202496 105747 0 0 0 4757 29 0 0 25 0 5 0 498557169 678400000 105020 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29944/statm: 165625 105021 176 36 0 163530 0
[pid=29944/tid=29977] ppid=29942 vsize=662500 CPUtime=11.74 cores=0,2,4,6
/proc/29944/task/29977/stat : 29977 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 24920 0 0 0 1168 6 0 0 25 0 5 0 498557260 678400000 105021 33554432000 4194304 4338100 140734615176816 18446744073709551615 228094845973 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29944/tid=29978] ppid=29942 vsize=662500 CPUtime=11.73 cores=0,2,4,6
/proc/29944/task/29978/stat : 29978 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 17530 0 0 0 1167 6 0 0 19 0 5 0 498557260 678400000 105021 33554432000 4194304 4338100 140734615176816 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29944/tid=29979] ppid=29942 vsize=662500 CPUtime=11.74 cores=0,2,4,6
/proc/29944/task/29979/stat : 29979 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 27722 0 0 0 1166 8 0 0 25 0 5 0 498557260 678400000 105021 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254304 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29944/tid=29980] ppid=29942 vsize=662500 CPUtime=11.7 cores=0,2,4,6
/proc/29944/task/29980/stat : 29980 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 32229 0 0 0 1164 6 0 0 25 0 5 0 498557260 678400000 105021 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254400 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 47.86
Current children cumulated vsize (KiB) 662500

[startup+25.5008 s]
/proc/loadavg: 7.71 7.98 7.99 10/186 29980
/proc/meminfo: memFree=25527880/32951124 swapFree=67111524/67111528
[pid=29944] ppid=29942 vsize=939512 CPUtime=98.55 cores=0,2,4,6
/proc/29944/stat : 29944 (ttsth-5-4) S 29942 29944 29500 0 -1 4202496 177186 0 0 0 9795 60 0 0 25 0 5 0 498557169 962060288 176459 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29944/statm: 234878 176459 176 36 0 232783 0
[pid=29944/tid=29977] ppid=29942 vsize=939512 CPUtime=24.37 cores=0,2,4,6
/proc/29944/task/29977/stat : 29977 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 43368 0 0 0 2423 14 0 0 19 0 5 0 498557260 962060288 176459 33554432000 4194304 4338100 140734615176816 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29944/tid=29978] ppid=29942 vsize=939512 CPUtime=24.4 cores=0,2,4,6
/proc/29944/task/29978/stat : 29978 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 30080 0 0 0 2427 13 0 0 24 0 5 0 498557260 962060288 176459 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29944/tid=29979] ppid=29942 vsize=939512 CPUtime=24.47 cores=0,2,4,6
/proc/29944/task/29979/stat : 29979 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 46793 0 0 0 2431 16 0 0 25 0 5 0 498557260 962060288 176459 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29944/tid=29980] ppid=29942 vsize=939512 CPUtime=24.37 cores=0,2,4,6
/proc/29944/task/29980/stat : 29980 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 53598 0 0 0 2423 14 0 0 18 0 5 0 498557260 962060288 176459 33554432000 4194304 4338100 140734615176816 18446744073709551615 228095147758 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 98.55
Current children cumulated vsize (KiB) 939512

[startup+51.1011 s]
/proc/loadavg: 7.69 7.95 7.98 9/186 29981
/proc/meminfo: memFree=25019320/32951124 swapFree=67111524/67111528
[pid=29944] ppid=29942 vsize=1339420 CPUtime=200.04 cores=0,2,4,6
/proc/29944/stat : 29944 (ttsth-5-4) S 29942 29944 29500 0 -1 4202496 278565 0 0 0 19890 114 0 0 25 0 5 0 498557169 1371566080 277838 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29944/statm: 334855 277838 176 36 0 332760 0
[pid=29944/tid=29977] ppid=29942 vsize=1339420 CPUtime=49.68 cores=0,2,4,6
/proc/29944/task/29977/stat : 29977 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 71952 0 0 0 4938 30 0 0 18 0 5 0 498557260 1371566080 277839 33554432000 4194304 4338100 140734615176816 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29944/tid=29978] ppid=29942 vsize=1339420 CPUtime=49.81 cores=0,2,4,6
/proc/29944/task/29978/stat : 29978 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 46048 0 0 0 4955 26 0 0 25 0 5 0 498557260 1371566080 277840 33554432000 4194304 4338100 140734615176816 18446744073709551615 228094845973 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29944/tid=29979] ppid=29942 vsize=1339420 CPUtime=49.92 cores=0,2,4,6
/proc/29944/task/29979/stat : 29979 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 73156 0 0 0 4964 28 0 0 25 0 5 0 498557260 1371566080 277841 33554432000 4194304 4338100 140734615176816 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29944/tid=29980] ppid=29942 vsize=1339420 CPUtime=49.69 cores=0,2,4,6
/proc/29944/task/29980/stat : 29980 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 84064 0 0 0 4941 28 0 0 18 0 5 0 498557260 1371566080 277841 33554432000 4194304 4338100 140734615176816 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 200.04
Current children cumulated vsize (KiB) 1339420

[startup+102.307 s]
/proc/loadavg: 7.88 7.96 7.98 9/186 29982
/proc/meminfo: memFree=24376596/32951124 swapFree=67111524/67111528
[pid=29944] ppid=29942 vsize=1841176 CPUtime=402.55 cores=0,2,4,6
/proc/29944/stat : 29944 (ttsth-5-4) S 29942 29944 29500 0 -1 4202496 405252 0 0 0 40040 215 0 0 25 0 5 0 498557169 1885364224 404525 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29944/statm: 460294 404525 176 36 0 458199 0
[pid=29944/tid=29977] ppid=29942 vsize=1841176 CPUtime=100.16 cores=0,2,4,6
/proc/29944/task/29977/stat : 29977 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 108185 0 0 0 9955 61 0 0 25 0 5 0 498557260 1885364224 404525 33554432000 4194304 4338100 140734615176816 18446744073709551615 228094846368 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29944/tid=29978] ppid=29942 vsize=1841176 CPUtime=100.42 cores=0,2,4,6
/proc/29944/task/29978/stat : 29978 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 62782 0 0 0 9990 52 0 0 20 0 5 0 498557260 1885364224 404525 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29944/tid=29979] ppid=29942 vsize=1841176 CPUtime=100.63 cores=0,2,4,6
/proc/29944/task/29979/stat : 29979 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 108366 0 0 0 10012 51 0 0 25 0 5 0 498557260 1885364224 404525 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29944/tid=29980] ppid=29942 vsize=1841176 CPUtime=100.39 cores=0,2,4,6
/proc/29944/task/29980/stat : 29980 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 122572 0 0 0 9991 48 0 0 22 0 5 0 498557260 1885364224 404525 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 402.55
Current children cumulated vsize (KiB) 1841176

[startup+162.301 s]
/proc/loadavg: 7.95 7.96 7.98 9/186 29984
/proc/meminfo: memFree=23796340/32951124 swapFree=67111524/67111528
[pid=29944] ppid=29942 vsize=2269724 CPUtime=638.22 cores=0,2,4,6
/proc/29944/stat : 29944 (ttsth-5-4) S 29942 29944 29500 0 -1 4202496 515085 0 0 0 63489 333 0 0 25 0 5 0 498557169 2324197376 514358 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29944/statm: 567431 514358 176 36 0 565336 0
[pid=29944/tid=29977] ppid=29942 vsize=2269724 CPUtime=158.77 cores=0,2,4,6
/proc/29944/task/29977/stat : 29977 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 150186 0 0 0 15786 91 0 0 25 0 5 0 498557260 2324197376 514358 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29944/tid=29978] ppid=29942 vsize=2269724 CPUtime=159.34 cores=0,2,4,6
/proc/29944/task/29978/stat : 29978 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 74230 0 0 0 15851 83 0 0 18 0 5 0 498557260 2324197376 514358 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29944/tid=29979] ppid=29942 vsize=2269724 CPUtime=159.81 cores=0,2,4,6
/proc/29944/task/29979/stat : 29979 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 133065 0 0 0 15903 78 0 0 25 0 5 0 498557260 2324197376 514358 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29944/tid=29980] ppid=29942 vsize=2269724 CPUtime=159.35 cores=0,2,4,6
/proc/29944/task/29980/stat : 29980 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 154257 0 0 0 15857 78 0 0 25 0 5 0 498557260 2324197376 514358 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 638.22
Current children cumulated vsize (KiB) 2269724


################
# More data... #
################

/proc/29944/task/29980/stat : 29980 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 255023 0 0 0 69025 230 0 0 25 0 5 0 498557260 3972222976 924330 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 2757.73
Current children cumulated vsize (KiB) 3879124

[startup+762.307 s]
/proc/loadavg: 8.15 8.12 8.04 9/186 30003
/proc/meminfo: memFree=18608820/32951124 swapFree=67111524/67111528
[pid=29944] ppid=29942 vsize=3958908 CPUtime=2994.79 cores=0,2,4,6
/proc/29944/stat : 29944 (ttsth-5-4) S 29942 29944 29500 0 -1 4202496 948444 0 0 0 298148 1331 0 0 25 0 5 0 498557169 4053921792 947717 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29944/statm: 989727 947717 176 36 0 987632 0
[pid=29944/tid=29977] ppid=29942 vsize=3958908 CPUtime=743.01 cores=0,2,4,6
/proc/29944/task/29977/stat : 29977 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 384313 0 0 0 73874 427 0 0 25 0 5 0 498557260 4053921792 947717 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29944/tid=29978] ppid=29942 vsize=3958908 CPUtime=747.71 cores=0,2,4,6
/proc/29944/task/29978/stat : 29978 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 101046 0 0 0 74367 404 0 0 18 0 5 0 498557260 4053921792 947717 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29944/tid=29979] ppid=29942 vsize=3958908 CPUtime=751.25 cores=0,2,4,6
/proc/29944/task/29979/stat : 29979 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 198031 0 0 0 74873 252 0 0 19 0 5 0 498557260 4053921792 947717 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29944/tid=29980] ppid=29942 vsize=3958908 CPUtime=751.88 cores=0,2,4,6
/proc/29944/task/29980/stat : 29980 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 261707 0 0 0 74942 246 0 0 25 0 5 0 498557260 4053921792 947717 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2994.79
Current children cumulated vsize (KiB) 3958908

[startup+822.3 s]
/proc/loadavg: 8.14 8.13 8.05 9/186 30005
/proc/meminfo: memFree=18265796/32951124 swapFree=67111524/67111528
[pid=29944] ppid=29942 vsize=4141784 CPUtime=3232.11 cores=0,2,4,6
/proc/29944/stat : 29944 (ttsth-5-4) S 29942 29944 29500 0 -1 4202496 995341 0 0 0 321802 1409 0 0 25 0 5 0 498557169 4241186816 994614 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29944/statm: 1035446 994614 176 36 0 1033351 0
[pid=29944/tid=29977] ppid=29942 vsize=4141784 CPUtime=802.19 cores=0,2,4,6
/proc/29944/task/29977/stat : 29977 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 412868 0 0 0 79770 449 0 0 25 0 5 0 498557260 4241186816 994614 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29944/tid=29978] ppid=29942 vsize=4141784 CPUtime=807.14 cores=0,2,4,6
/proc/29944/task/29978/stat : 29978 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 101408 0 0 0 80290 424 0 0 18 0 5 0 498557260 4241186816 994614 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29944/tid=29979] ppid=29942 vsize=4141784 CPUtime=810.73 cores=0,2,4,6
/proc/29944/task/29979/stat : 29979 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 198395 0 0 0 80806 267 0 0 25 0 5 0 498557260 4241186816 994614 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29944/tid=29980] ppid=29942 vsize=4141784 CPUtime=811.12 cores=0,2,4,6
/proc/29944/task/29980/stat : 29980 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 279323 0 0 0 80845 267 0 0 25 0 5 0 498557260 4241186816 994614 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3232.11
Current children cumulated vsize (KiB) 4141784

[startup+882.3 s]
/proc/loadavg: 8.16 8.14 8.06 9/186 30006
/proc/meminfo: memFree=17767900/32951124 swapFree=67111524/67111528
[pid=29944] ppid=29942 vsize=4446652 CPUtime=3469.46 cores=0,2,4,6
/proc/29944/stat : 29944 (ttsth-5-4) S 29942 29944 29500 0 -1 4202496 1072173 0 0 0 345426 1520 0 0 25 0 5 0 498557169 4553371648 1071446 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29944/statm: 1111663 1071446 176 36 0 1109568 0
[pid=29944/tid=29977] ppid=29942 vsize=4446652 CPUtime=861.34 cores=0,2,4,6
/proc/29944/task/29977/stat : 29977 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 440771 0 0 0 85661 473 0 0 25 0 5 0 498557260 4553371648 1071446 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29944/tid=29978] ppid=29942 vsize=4446652 CPUtime=866.44 cores=0,2,4,6
/proc/29944/task/29978/stat : 29978 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 101773 0 0 0 86193 451 0 0 20 0 5 0 498557260 4553371648 1071446 33554432000 4194304 4338100 140734615176816 18446744073709551615 4229573 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29944/tid=29979] ppid=29942 vsize=4446652 CPUtime=870.31 cores=0,2,4,6
/proc/29944/task/29979/stat : 29979 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 219815 0 0 0 86737 294 0 0 25 0 5 0 498557260 4553371648 1071446 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29944/tid=29980] ppid=29942 vsize=4446652 CPUtime=870.42 cores=0,2,4,6
/proc/29944/task/29980/stat : 29980 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 306467 0 0 0 86742 300 0 0 25 0 5 0 498557260 4553371648 1071446 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3469.46
Current children cumulated vsize (KiB) 4446652

[startup+942.3 s]
/proc/loadavg: 8.17 8.15 8.06 9/186 30008
/proc/meminfo: memFree=17398964/32951124 swapFree=67111524/67111528
[pid=29944] ppid=29942 vsize=4599040 CPUtime=3704.11 cores=0,2,4,6
/proc/29944/stat : 29944 (ttsth-5-4) S 29942 29944 29500 0 -1 4202496 1112297 0 0 0 368771 1640 0 0 25 0 5 0 498557169 4709416960 1111570 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29944/statm: 1149760 1111570 176 36 0 1147665 0
[pid=29944/tid=29977] ppid=29942 vsize=4599040 CPUtime=919.29 cores=0,2,4,6
/proc/29944/task/29977/stat : 29977 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 458298 0 0 0 91419 510 0 0 25 0 5 0 498557260 4709416960 1111570 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29944/tid=29978] ppid=29942 vsize=4599040 CPUtime=925.32 cores=0,2,4,6
/proc/29944/task/29978/stat : 29978 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 102329 0 0 0 92043 489 0 0 18 0 5 0 498557260 4709416960 1111570 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29944/tid=29979] ppid=29942 vsize=4599040 CPUtime=929.11 cores=0,2,4,6
/proc/29944/task/29979/stat : 29979 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 237037 0 0 0 92592 319 0 0 18 0 5 0 498557260 4709416960 1111570 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29944/tid=29980] ppid=29942 vsize=4599040 CPUtime=929.45 cores=0,2,4,6
/proc/29944/task/29980/stat : 29980 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 311286 0 0 0 92625 320 0 0 25 0 5 0 498557260 4709416960 1111570 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3704.11
Current children cumulated vsize (KiB) 4599040

[startup+1002.3 s]
/proc/loadavg: 8.18 8.15 8.07 9/186 30009
/proc/meminfo: memFree=16951024/32951124 swapFree=67111524/67111528
[pid=29944] ppid=29942 vsize=4789360 CPUtime=3938.68 cores=0,2,4,6
/proc/29944/stat : 29944 (ttsth-5-4) S 29942 29944 29500 0 -1 4202496 1161977 0 0 0 392131 1737 0 0 25 0 5 0 498557169 4904304640 1161250 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29944/statm: 1197340 1161250 176 36 0 1195245 0
[pid=29944/tid=29977] ppid=29942 vsize=4789360 CPUtime=977.08 cores=0,2,4,6
/proc/29944/task/29977/stat : 29977 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 493291 0 0 0 97165 543 0 0 25 0 5 0 498557260 4904304640 1161250 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29944/tid=29978] ppid=29942 vsize=4789360 CPUtime=984.03 cores=0,2,4,6
/proc/29944/task/29978/stat : 29978 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 102937 0 0 0 97883 520 0 0 25 0 5 0 498557260 4904304640 1161250 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29944/tid=29979] ppid=29942 vsize=4789360 CPUtime=988.13 cores=0,2,4,6
/proc/29944/task/29979/stat : 29979 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 237672 0 0 0 98481 332 0 0 20 0 5 0 498557260 4904304640 1161250 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29944/tid=29980] ppid=29942 vsize=4789360 CPUtime=988.49 cores=0,2,4,6
/proc/29944/task/29980/stat : 29980 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 324730 0 0 0 98510 339 0 0 21 0 5 0 498557260 4904304640 1161250 33554432000 4194304 4338100 140734615176816 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3938.68
Current children cumulated vsize (KiB) 4789360

[startup+1062.3 s]
/proc/loadavg: 8.06 8.12 8.06 9/186 30011
/proc/meminfo: memFree=16915908/32951124 swapFree=67111524/67111528
[pid=29944] ppid=29942 vsize=4881720 CPUtime=4176.13 cores=0,2,4,6
/proc/29944/stat : 29944 (ttsth-5-4) S 29942 29944 29500 0 -1 4202496 1168016 0 0 0 415797 1816 0 0 25 0 5 0 498557169 4998881280 1167289 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29944/statm: 1220430 1167289 176 36 0 1218335 0
[pid=29944/tid=29977] ppid=29942 vsize=4881720 CPUtime=1036.04 cores=0,2,4,6
/proc/29944/task/29977/stat : 29977 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 496115 0 0 0 103037 567 0 0 25 0 5 0 498557260 4998881280 1167289 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29944/tid=29978] ppid=29942 vsize=4881720 CPUtime=1043.29 cores=0,2,4,6
/proc/29944/task/29978/stat : 29978 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 105467 0 0 0 103779 550 0 0 25 0 5 0 498557260 4998881280 1167289 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29944/tid=29979] ppid=29942 vsize=4881720 CPUtime=1047.72 cores=0,2,4,6
/proc/29944/task/29979/stat : 29979 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 237952 0 0 0 104427 345 0 0 19 0 5 0 498557260 4998881280 1167289 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29944/tid=29980] ppid=29942 vsize=4881720 CPUtime=1048.14 cores=0,2,4,6
/proc/29944/task/29980/stat : 29980 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 325135 0 0 0 104463 351 0 0 25 0 5 0 498557260 4998881280 1167289 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4176.13
Current children cumulated vsize (KiB) 4881720

[startup+1122.3 s]
/proc/loadavg: 7.67 8.00 8.02 9/186 30013
/proc/meminfo: memFree=16732356/32951124 swapFree=67111524/67111528
[pid=29944] ppid=29942 vsize=5058604 CPUtime=4411.79 cores=0,2,4,6
/proc/29944/stat : 29944 (ttsth-5-4) S 29942 29944 29500 0 -1 4202496 1213056 0 0 0 439249 1930 0 0 25 0 5 0 498557169 5180010496 1212329 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29944/statm: 1264651 1212329 176 36 0 1262556 0
[pid=29944/tid=29977] ppid=29942 vsize=5058604 CPUtime=1094.51 cores=0,2,4,6
/proc/29944/task/29977/stat : 29977 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 528745 0 0 0 108853 598 0 0 25 0 5 0 498557260 5180010496 1212329 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29944/tid=29978] ppid=29942 vsize=5058604 CPUtime=1102.17 cores=0,2,4,6
/proc/29944/task/29978/stat : 29978 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 105872 0 0 0 109635 582 0 0 23 0 5 0 498557260 5180010496 1212329 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29944/tid=29979] ppid=29942 vsize=5058604 CPUtime=1106.81 cores=0,2,4,6
/proc/29944/task/29979/stat : 29979 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 238366 0 0 0 110314 367 0 0 21 0 5 0 498557260 5180010496 1212329 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29944/tid=29980] ppid=29942 vsize=5058604 CPUtime=1107.34 cores=0,2,4,6
/proc/29944/task/29980/stat : 29980 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 336726 0 0 0 110354 380 0 0 18 0 5 0 498557260 5180010496 1212329 33554432000 4194304 4338100 140734615176816 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4411.79
Current children cumulated vsize (KiB) 5058604

[startup+1182.3 s]
/proc/loadavg: 7.91 8.01 8.02 9/186 30014
/proc/meminfo: memFree=16607708/32951124 swapFree=67111524/67111528
[pid=29944] ppid=29942 vsize=5156760 CPUtime=4648.52 cores=0,2,4,6
/proc/29944/stat : 29944 (ttsth-5-4) S 29942 29944 29500 0 -1 4202496 1238945 0 0 0 462825 2027 0 0 25 0 5 0 498557169 5280522240 1238218 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29944/statm: 1289190 1238218 176 36 0 1287095 0
[pid=29944/tid=29977] ppid=29942 vsize=5156760 CPUtime=1153.34 cores=0,2,4,6
/proc/29944/task/29977/stat : 29977 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 542097 0 0 0 114709 625 0 0 25 0 5 0 498557260 5280522240 1238218 33554432000 4194304 4338100 140734615176816 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29944/tid=29978] ppid=29942 vsize=5156760 CPUtime=1161.47 cores=0,2,4,6
/proc/29944/task/29978/stat : 29978 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 106197 0 0 0 115533 614 0 0 18 0 5 0 498557260 5280522240 1238218 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29944/tid=29979] ppid=29942 vsize=5156760 CPUtime=1166.01 cores=0,2,4,6
/proc/29944/task/29979/stat : 29979 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 238692 0 0 0 116218 383 0 0 25 0 5 0 498557260 5280522240 1238218 33554432000 4194304 4338100 140734615176816 18446744073709551615 228094846394 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29944/tid=29980] ppid=29942 vsize=5156760 CPUtime=1166.76 cores=0,2,4,6
/proc/29944/task/29980/stat : 29980 (ttsth-5-4) R 29942 29944 29500 0 -1 4202560 348612 0 0 0 116273 403 0 0 25 0 5 0 498557260 5280522240 1238218 33554432000 4194304 4338100 140734615176816 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4648.52
Current children cumulated vsize (KiB) 5156760



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1221.1 s]
/proc/loadavg: 6.86 7.77 7.94 2/173 30033
/proc/meminfo: memFree=22093720/32951124 swapFree=67111524/67111528
[pid=29944] ppid=29942 vsize=0 CPUtime=4800.09 cores=0,2,4,6
/proc/29944/stat : 29944 (ttsth-5-4) Z 29942 29944 29500 0 -1 4203548 1259664 0 0 0 477914 2095 0 0 18 0 2 0 498557169 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29944/statm: 0 0 0 0 0 0 0
[pid=29944/tid=29980] ppid=29942 vsize=0 CPUtime=1204.93 cores=0,2,4,6
/proc/29944/task/29980/stat : 29980 (ttsth-5-4) R 29942 29944 29500 0 -1 4203588 348877 0 0 0 120073 420 0 0 20 0 2 0 498557260 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.09
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1221.21 s]
/proc/loadavg: 6.86 7.77 7.94 2/173 30033
/proc/meminfo: memFree=22093720/32951124 swapFree=67111524/67111528
[pid=29944] ppid=29942 vsize=0 CPUtime=4800.2 cores=0,2,4,6
/proc/29944/stat : 29944 (ttsth-5-4) Z 29942 29944 29500 0 -1 4203548 1259664 0 0 0 477914 2106 0 0 18 0 2 0 498557169 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29944/statm: 0 0 0 0 0 0 0
[pid=29944/tid=29980] ppid=29942 vsize=0 CPUtime=1205.03 cores=0,2,4,6
/proc/29944/task/29980/stat : 29980 (ttsth-5-4) R 29942 29944 29500 0 -1 4203588 348877 0 0 0 120073 430 0 0 20 0 2 0 498557260 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.2
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1221.29
CPU time (s): 4800.28
CPU user time (s): 4779.14
CPU system time (s): 21.1398
CPU usage (%): 393.051
Max. virtual memory (cumulated for all children) (KiB): 5236016

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4779.14
system time used= 21.1398
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1259664
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 136011
involuntary context switches= 23031

runsolver used 1.80672 second user time and 3.97839 second system time

The end

Launcher Data 

Begin job on node142 at 2011-04-29 09:16:57
IDJOB=3259978
IDBENCH=83162
IDSOLVER=1654
FILE ID=node142/3259978-1304061417
RUNJOBID= node142-1304060210-29815
PBS_JOBID= 13170434
Free space on /tmp= 71316 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/mosoi/289/289-sat-4x35.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3259978-1304061417/watcher-3259978-1304061417 -o /tmp/evaluation-result-3259978-1304061417/solver-3259978-1304061417 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259978-1304061417.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 3a0c9ef65cb2e498876675e6a31f16d6
RANDOM SEED=173842860

node142.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5131.95
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      26585900 kB
Buffers:       1653600 kB
Cached:        4102020 kB
SwapCached:          4 kB
Active:         691452 kB
Inactive:      5135584 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      26585900 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:             432 kB
Writeback:           0 kB
AnonPages:       71360 kB
Mapped:          15064 kB
Slab:           475300 kB
PageTables:       4288 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   261508 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71312 MiB
End job on node142 at 2011-04-29 09:37:20