Trace number 3266314

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.58 1202.28

General information on the benchmark

NameSAT02/handmade/pehoushek/plainoldcnf/
5cnf_3900_3900_220.shuffled.cnf
MD5SUMf0b7e0675915478b15cab3695fd0fd62
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.035994
Satisfiable
(Un)Satisfiability was proved
Number of variables220
Number of clauses3432
Sum of the clauses size17000
Maximum clause length5
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32
Number of clauses of size 4156
Number of clauses of size 53274
Number of clauses of size over 50

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3266314-1304104641.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 220
0.00/0.00	c Clauses = 3432
0.00/0.00	c Literals = 17000
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 220
0.00/0.00	c Clauses = 3432
0.00/0.00	c Literals = 17000
0.00/0.00	c a2
0.00/0.01	c a1
0.19/0.28	c Independent sub-problems = 1
0.19/0.28	c Sub-problem with 220 variables and 3432 clauses
0.19/0.28	c using permutation method 4
0.19/0.28	c count = 2
0.19/0.28	c Ordering finished 0.280 seconds
0.19/0.28	c mapped 0.280 seconds
0.19/0.28	c permuted 0.280 seconds
0.19/0.28	c about to sort
0.19/0.28	c sorted
0.29/0.30	c clauses now 3432
0.29/0.30	c init finished 0.280 seconds
0.29/0.30	c tree built 0.290 seconds
0.29/0.30	c 
0.29/0.30	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.29/0.30	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.29/0.30	c ................................ | ..........                    |                                  0.3         0.0             
0.29/0.30	c ................................ | ..........                  1 |                                  0.3         3.4             
0.29/0.30	c ................................ | ..........                  2 |                                  0.3         6.9             
0.29/0.30	c ................................ | ..........                  3 |                                  0.3        10.3             
0.29/0.30	c ................................ | ..........                  4 |                                  0.3        13.8             
0.29/0.30	c ................................ | ..........                  5 |                                  0.3        17.2             
0.29/0.31	c ................................ | ..........               1000 | 59           1412                0.3      3125.0        12867
96.14/24.30	c ................................ | ..........             553000 | 118418       686011             96.2      5751.4     21772073
192.04/48.37	c ................................ | ..........             855000 | 193009       1000227           192.3      4445.7     34882969
287.89/72.35	c ................................ | ..........            1033000 | 232237       1236936           288.1      3585.3     43991629
384.16/96.45	c ................................ | ..........            1157000 | 265254       1398523           384.4      3010.0     51497405
480.03/120.44	c ................................ | *.........            1321000 | 307936       1590486           480.2      2751.0     59523693
576.69/144.61	c ................................ | *.........            1460000 | 345690       1748223           576.7      2531.5     66174977
672.56/168.60	c ................................ | *.........            1577000 | 379838       1882918           672.6      2344.7     72270456
768.43/192.65	c ................................ | *.........            1724000 | 416480       2008598           768.6      2243.0     79442664
863.89/216.59	c ................................ | *.........            1949000 | 474948       2161705           864.3      2255.1     86529770
959.76/240.56	c ................................ | **........            2138000 | 525099       2300025           960.0      2227.0     93253634
1056.03/264.68	c ................................ | **........            2287000 | 563982       2420685          1056.4      2165.0     99843342
1151.90/288.66	c ................................ | **........            2413000 | 601011       2517549          1152.1      2094.4    105868686
1248.55/312.85	c ................................ | **........            2534000 | 633851       2619454          1248.8      2029.2    111570748
1344.02/336.72	c ................................ | **........            2668000 | 673718       2730832          1344.1      1984.9    116293734
1440.31/360.82	c ................................ | ***.......            2791000 | 710119       2833908          1440.4      1937.7    119571068
1536.57/384.95	c ................................ | ***.......            2862000 | 728464       2920562          1536.8      1862.4    122514090
1632.42/408.91	c ................................ | ***.......            2928000 | 744947       3004824          1632.5      1793.6    126185107
1727.87/432.89	c ................................ | ***.......            2993000 | 761439       3089488          1728.2      1731.8    129066550
1823.75/456.89	c ................................ | ***.......            3071000 | 780678       3180062          1824.1      1683.6    133294922
1920.42/481.05	c ................................ | ****......            3149000 | 801446       3257319          1920.6      1639.6    136149795
2016.67/505.12	c ................................ | ****......            3219000 | 820775       3329705          2016.8      1596.1    140308374
2112.16/529.01	c ................................ | ****......            3304000 | 841887       3400248          2112.2      1564.3    144031550
2208.36/553.15	c ................................ | ****......            3359000 | 855130       3470094          2208.6      1520.9    146376633
2306.62/577.72	c ................................ | ****......            3399000 | 865471       3532814          2306.7      1473.5    147571192
2402.12/601.61	c ................................ | *****.....            3441000 | 876448       3597878          2402.1      1432.5    149612656
2495.97/625.12	c ................................ | *****.....            3494000 | 888987       3678856          2496.1      1399.8    152004191
2592.23/649.22	c ................................ | *****.....            3537000 | 901330       3739042          2592.3      1364.4    154636282
2690.48/673.88	c ................................ | *****.....            3596000 | 917104       3805430          2690.8      1336.4    158129997
2785.94/697.79	c ................................ | *****.....            3647000 | 930918       3866998          2786.3      1308.9    161263120
2879.82/721.26	c ................................ | ******....            3695000 | 945224       3923634          2880.1      1283.0    164546895
2976.89/745.51	c ................................ | ******....            3754000 | 960915       3979205          2976.9      1261.0    167661606
3072.74/769.60	c ................................ | ******....            3808000 | 976148       4038929          3073.1      1239.1    170721297
3168.21/793.49	c ................................ | ******....            3858000 | 990856       4108480          3168.6      1217.6    174254107
3264.07/817.45	c ................................ | ******....            3938000 | 1010533      4191797          3264.3      1206.4    177349775
3360.36/841.57	c ................................ | *******...            4023000 | 1031193      4269174          3360.6      1197.1    180042210
3456.20/865.57	c ................................ | *******...            4118000 | 1054424      4342623          3456.5      1191.4    183946551
3552.86/889.70	c ................................ | *******...            4208000 | 1078523      4413728          3552.9      1184.4    187938464
3647.92/913.57	c ................................ | *******...            4304000 | 1103525      4491965          3648.2      1179.8    191006977
3744.59/937.72	c ................................ | *******...            4398000 | 1130872      4565398          3744.7      1174.5    194476377
3840.46/961.72	c ................................ | ********..            4487000 | 1153117      4639711          3840.6      1168.3    198349083
3937.09/985.97	c ................................ | ********..            4557000 | 1171571      4710438          3937.4      1157.4    200413689
4032.18/1009.70	c ................................ | ********..            4621000 | 1188108      4773217          4032.2      1146.0    202115938
4129.62/1034.17	c ................................ | ********..            4701000 | 1210848      4836277          4129.9      1138.3    205820663
4223.89/1057.76	c ................................ | ********..            4769000 | 1231887      4895266          4224.1      1129.0    208892252
4320.15/1081.80	c ................................ | *********.            4849000 | 1254976      4959212          4320.2      1122.4    211211543
4416.41/1105.95	c ................................ | *********.            4919000 | 1272267      5023516          4416.6      1113.7    214094056
4513.48/1130.24	c ................................ | *********.            4969000 | 1286199      5082091          4513.6      1100.9    217120641
4608.93/1154.15	c ................................ | *********.            5016000 | 1300665      5132479          4609.2      1088.3    219611353
4704.00/1177.95	c ................................ | *********.            5073000 | 1317588      5188178          4704.2      1078.4    223002167

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3266314-1304104641/watcher-3266314-1304104641 -o /tmp/evaluation-result-3266314-1304104641/solver-3266314-1304104641 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3266314-1304104641.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.16 7.91 7.99 2/176 3794
/proc/meminfo: memFree=19216800/32950928 swapFree=67111528/67111528
[pid=3794] ppid=3790 vsize=9508 CPUtime=0 cores=1,3,5,7
/proc/3794/stat : 3794 (ttsth-5-4) R 3790 3794 3320 0 -1 4202496 465 0 0 0 0 0 0 0 25 0 1 0 522063763 9736192 407 33554432000 4194304 4338100 140734408438528 18446744073709551615 274473628357 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/3794/statm: 2410 410 150 36 0 315 0

[startup+0.0505341 s]
/proc/loadavg: 7.16 7.91 7.99 2/176 3794
/proc/meminfo: memFree=19216800/32950928 swapFree=67111528/67111528
[pid=3794] ppid=3790 vsize=10036 CPUtime=0.04 cores=1,3,5,7
/proc/3794/stat : 3794 (ttsth-5-4) R 3790 3794 3320 0 -1 4202496 602 0 0 0 4 0 0 0 25 0 1 0 522063763 10276864 543 33554432000 4194304 4338100 140734408438528 18446744073709551615 4295900 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/3794/statm: 2509 543 155 36 0 414 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 10036

[startup+0.100525 s]
/proc/loadavg: 7.16 7.91 7.99 2/176 3794
/proc/meminfo: memFree=19216800/32950928 swapFree=67111528/67111528
[pid=3794] ppid=3790 vsize=10036 CPUtime=0.09 cores=1,3,5,7
/proc/3794/stat : 3794 (ttsth-5-4) R 3790 3794 3320 0 -1 4202496 602 0 0 0 9 0 0 0 25 0 1 0 522063763 10276864 543 33554432000 4194304 4338100 140734408438528 18446744073709551615 4295986 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/3794/statm: 2509 543 155 36 0 414 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 10036

[startup+0.300493 s]
/proc/loadavg: 7.16 7.91 7.99 2/176 3794
/proc/meminfo: memFree=19216800/32950928 swapFree=67111528/67111528
[pid=3794] ppid=3790 vsize=15880 CPUtime=0.29 cores=1,3,5,7
/proc/3794/stat : 3794 (ttsth-5-4) R 3790 3794 3320 0 -1 4202496 2077 0 0 0 29 0 0 0 25 0 1 0 522063763 16261120 1989 33554432000 4194304 4338100 140734408438528 18446744073709551615 274473628357 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/3794/statm: 3970 1992 163 36 0 1875 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 15880

[startup+0.700424 s]
/proc/loadavg: 7.16 7.91 7.99 2/176 3794
/proc/meminfo: memFree=19216800/32950928 swapFree=67111528/67111528
[pid=3794] ppid=3790 vsize=227676 CPUtime=1.88 cores=1,3,5,7
/proc/3794/stat : 3794 (ttsth-5-4) S 3790 3794 3320 0 -1 4202496 9916 0 0 0 187 1 0 0 25 0 5 0 522063763 233140224 9826 33554432000 4194304 4338100 140734408438528 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/3794/statm: 56919 9827 175 36 0 54824 0
Current children cumulated CPU time (s) 1.88
Current children cumulated vsize (KiB) 227676

[startup+1.50028 s]
/proc/loadavg: 7.16 7.91 7.99 10/186 3804
/proc/meminfo: memFree=19097696/32950928 swapFree=67111528/67111528
[pid=3794] ppid=3790 vsize=275368 CPUtime=5.07 cores=1,3,5,7
/proc/3794/stat : 3794 (ttsth-5-4) S 3790 3794 3320 0 -1 4202496 22779 0 0 0 504 3 0 0 25 0 5 0 522063763 281976832 22689 33554432000 4194304 4338100 140734408438528 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/3794/statm: 68842 22689 175 36 0 66747 0
[pid=3794/tid=3801] ppid=3790 vsize=275368 CPUtime=1.18 cores=1,3,5,7
/proc/3794/task/3801/stat : 3801 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 5572 0 0 0 117 1 0 0 25 0 5 0 522063793 281976832 22689 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3794/tid=3802] ppid=3790 vsize=275368 CPUtime=1.19 cores=1,3,5,7
/proc/3794/task/3802/stat : 3802 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 4290 0 0 0 119 0 0 0 19 0 5 0 522063793 281976832 22689 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3794/tid=3803] ppid=3790 vsize=275368 CPUtime=1.18 cores=1,3,5,7
/proc/3794/task/3803/stat : 3803 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 5644 0 0 0 118 0 0 0 19 0 5 0 522063793 281976832 22689 33554432000 4194304 4338100 140734408438528 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3794/tid=3804] ppid=3790 vsize=275368 CPUtime=1.18 cores=1,3,5,7
/proc/3794/task/3804/stat : 3804 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 5030 0 0 0 118 0 0 0 25 0 5 0 522063793 281976832 22689 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 5.07
Current children cumulated vsize (KiB) 275368

[startup+3.101 s]
/proc/loadavg: 7.16 7.91 7.99 10/186 3804
/proc/meminfo: memFree=18982756/32950928 swapFree=67111528/67111528
[pid=3794] ppid=3790 vsize=351592 CPUtime=11.46 cores=1,3,5,7
/proc/3794/stat : 3794 (ttsth-5-4) S 3790 3794 3320 0 -1 4202496 42048 0 0 0 1139 7 0 0 25 0 5 0 522063763 360030208 41958 33554432000 4194304 4338100 140734408438528 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/3794/statm: 87898 41958 175 36 0 85803 0
[pid=3794/tid=3801] ppid=3790 vsize=351592 CPUtime=2.77 cores=1,3,5,7
/proc/3794/task/3801/stat : 3801 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 11725 0 0 0 275 2 0 0 25 0 5 0 522063793 360030208 41958 33554432000 4194304 4338100 140734408438528 18446744073709551615 4256770 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3794/tid=3802] ppid=3790 vsize=351592 CPUtime=2.79 cores=1,3,5,7
/proc/3794/task/3802/stat : 3802 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 8296 0 0 0 278 1 0 0 24 0 5 0 522063793 360030208 41958 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3794/tid=3803] ppid=3790 vsize=351592 CPUtime=2.79 cores=1,3,5,7
/proc/3794/task/3803/stat : 3803 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 10903 0 0 0 278 1 0 0 23 0 5 0 522063793 360030208 41958 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3794/tid=3804] ppid=3790 vsize=351592 CPUtime=2.79 cores=1,3,5,7
/proc/3794/task/3804/stat : 3804 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 8881 0 0 0 278 1 0 0 25 0 5 0 522063793 360030208 41958 33554432000 4194304 4338100 140734408438528 18446744073709551615 274473362808 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 11.46
Current children cumulated vsize (KiB) 351592

[startup+6.30044 s]
/proc/loadavg: 7.23 7.91 7.99 10/186 3804
/proc/meminfo: memFree=18738220/32950928 swapFree=67111528/67111528
[pid=3794] ppid=3790 vsize=468744 CPUtime=24.25 cores=1,3,5,7
/proc/3794/stat : 3794 (ttsth-5-4) S 3790 3794 3320 0 -1 4202496 71536 0 0 0 2411 14 0 0 25 0 5 0 522063763 479993856 71446 33554432000 4194304 4338100 140734408438528 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/3794/statm: 117186 71446 175 36 0 115091 0
[pid=3794/tid=3801] ppid=3790 vsize=468744 CPUtime=5.96 cores=1,3,5,7
/proc/3794/task/3801/stat : 3801 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 21016 0 0 0 591 5 0 0 25 0 5 0 522063793 479993856 71446 33554432000 4194304 4338100 140734408438528 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3794/tid=3802] ppid=3790 vsize=468744 CPUtime=5.99 cores=1,3,5,7
/proc/3794/task/3802/stat : 3802 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 14269 0 0 0 596 3 0 0 25 0 5 0 522063793 479993856 71446 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3794/tid=3803] ppid=3790 vsize=468744 CPUtime=5.99 cores=1,3,5,7
/proc/3794/task/3803/stat : 3803 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 18764 0 0 0 596 3 0 0 25 0 5 0 522063793 479993856 71447 33554432000 4194304 4338100 140734408438528 18446744073709551615 274485780262 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3794/tid=3804] ppid=3790 vsize=468744 CPUtime=5.98 cores=1,3,5,7
/proc/3794/task/3804/stat : 3804 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 15245 0 0 0 596 2 0 0 25 0 5 0 522063793 479993856 71447 33554432000 4194304 4338100 140734408438528 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 24.25
Current children cumulated vsize (KiB) 468744

[startup+12.7003 s]
/proc/loadavg: 7.29 7.91 7.99 10/186 3804
/proc/meminfo: memFree=18382196/32950928 swapFree=67111528/67111528
[pid=3794] ppid=3790 vsize=636068 CPUtime=49.81 cores=1,3,5,7
/proc/3794/stat : 3794 (ttsth-5-4) S 3790 3794 3320 0 -1 4202496 115002 0 0 0 4958 23 0 0 25 0 5 0 522063763 651333632 114912 33554432000 4194304 4338100 140734408438528 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/3794/statm: 159017 114912 175 36 0 156922 0
[pid=3794/tid=3801] ppid=3790 vsize=636068 CPUtime=12.32 cores=1,3,5,7
/proc/3794/task/3801/stat : 3801 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 35897 0 0 0 1224 8 0 0 25 0 5 0 522063793 651333632 114912 33554432000 4194304 4338100 140734408438528 18446744073709551615 274473664112 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3794/tid=3802] ppid=3790 vsize=636068 CPUtime=12.39 cores=1,3,5,7
/proc/3794/task/3802/stat : 3802 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 21720 0 0 0 1234 5 0 0 25 0 5 0 522063793 651333632 114912 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3794/tid=3803] ppid=3790 vsize=636068 CPUtime=12.39 cores=1,3,5,7
/proc/3794/task/3803/stat : 3803 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 29478 0 0 0 1234 5 0 0 25 0 5 0 522063793 651333632 114913 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3794/tid=3804] ppid=3790 vsize=636068 CPUtime=12.38 cores=1,3,5,7
/proc/3794/task/3804/stat : 3804 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 25664 0 0 0 1234 4 0 0 25 0 5 0 522063793 651333632 114913 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 49.81
Current children cumulated vsize (KiB) 636068

[startup+25.5011 s]
/proc/loadavg: 7.51 7.93 7.99 10/186 3804
/proc/meminfo: memFree=17872672/32950928 swapFree=67111528/67111528
[pid=3794] ppid=3790 vsize=907188 CPUtime=100.94 cores=1,3,5,7
/proc/3794/stat : 3794 (ttsth-5-4) S 3790 3794 3320 0 -1 4202496 183521 0 0 0 10057 37 0 0 25 0 5 0 522063763 928960512 183431 33554432000 4194304 4338100 140734408438528 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/3794/statm: 226797 183431 176 36 0 224702 0
[pid=3794/tid=3801] ppid=3790 vsize=907188 CPUtime=25.05 cores=1,3,5,7
/proc/3794/task/3801/stat : 3801 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 60749 0 0 0 2492 13 0 0 25 0 5 0 522063793 928960512 183431 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3794/tid=3802] ppid=3790 vsize=907188 CPUtime=25.19 cores=1,3,5,7
/proc/3794/task/3802/stat : 3802 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 33373 0 0 0 2511 8 0 0 25 0 5 0 522063793 928960512 183431 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3794/tid=3803] ppid=3790 vsize=907188 CPUtime=25.19 cores=1,3,5,7
/proc/3794/task/3803/stat : 3803 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 46094 0 0 0 2511 8 0 0 25 0 5 0 522063793 928960512 183432 33554432000 4194304 4338100 140734408438528 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3794/tid=3804] ppid=3790 vsize=907188 CPUtime=25.19 cores=1,3,5,7
/proc/3794/task/3804/stat : 3804 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 41063 0 0 0 2512 7 0 0 25 0 5 0 522063793 928960512 183432 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 100.94
Current children cumulated vsize (KiB) 907188

[startup+51.1006 s]
/proc/loadavg: 7.68 7.93 7.99 10/186 3805
/proc/meminfo: memFree=17124944/32950928 swapFree=67111528/67111528
[pid=3794] ppid=3790 vsize=1339088 CPUtime=203.2 cores=1,3,5,7
/proc/3794/stat : 3794 (ttsth-5-4) S 3790 3794 3320 0 -1 4202496 291937 0 0 0 20260 60 0 0 25 0 5 0 522063763 1371226112 291847 33554432000 4194304 4338100 140734408438528 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/3794/statm: 334772 291847 176 36 0 332677 0
[pid=3794/tid=3801] ppid=3790 vsize=1339088 CPUtime=50.51 cores=1,3,5,7
/proc/3794/task/3801/stat : 3801 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 98374 0 0 0 5031 20 0 0 25 0 5 0 522063793 1371226112 291847 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3794/tid=3802] ppid=3790 vsize=1339088 CPUtime=50.78 cores=1,3,5,7
/proc/3794/task/3802/stat : 3802 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 53167 0 0 0 5066 12 0 0 25 0 5 0 522063793 1371226112 291847 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3794/tid=3803] ppid=3790 vsize=1339088 CPUtime=50.79 cores=1,3,5,7
/proc/3794/task/3803/stat : 3803 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 73269 0 0 0 5065 14 0 0 25 0 5 0 522063793 1371226112 291847 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3794/tid=3804] ppid=3790 vsize=1339088 CPUtime=50.79 cores=1,3,5,7
/proc/3794/task/3804/stat : 3804 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 64884 0 0 0 5067 12 0 0 25 0 5 0 522063793 1371226112 291847 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 203.2
Current children cumulated vsize (KiB) 1339088

[startup+102.307 s]
/proc/loadavg: 7.96 7.97 8.00 10/186 3806
/proc/meminfo: memFree=16068200/32950928 swapFree=67111528/67111528
[pid=3794] ppid=3790 vsize=1881912 CPUtime=407.75 cores=1,3,5,7
/proc/3794/stat : 3794 (ttsth-5-4) S 3790 3794 3320 0 -1 4202496 430149 0 0 0 40677 98 0 0 25 0 5 0 522063763 1927077888 430059 33554432000 4194304 4338100 140734408438528 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/3794/statm: 470478 430060 176 36 0 468383 0
[pid=3794/tid=3801] ppid=3790 vsize=1881912 CPUtime=101.43 cores=1,3,5,7
/proc/3794/task/3801/stat : 3801 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 135847 0 0 0 10111 32 0 0 25 0 5 0 522063793 1927077888 430060 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3794/tid=3802] ppid=3790 vsize=1881912 CPUtime=101.99 cores=1,3,5,7
/proc/3794/task/3802/stat : 3802 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 82166 0 0 0 10180 19 0 0 25 0 5 0 522063793 1927077888 430060 33554432000 4194304 4338100 140734408438528 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3794/tid=3803] ppid=3790 vsize=1881912 CPUtime=101.99 cores=1,3,5,7
/proc/3794/task/3803/stat : 3803 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 110148 0 0 0 10175 24 0 0 25 0 5 0 522063793 1927077888 430060 33554432000 4194304 4338100 140734408438528 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3794/tid=3804] ppid=3790 vsize=1881912 CPUtime=101.99 cores=1,3,5,7
/proc/3794/task/3804/stat : 3804 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 99746 0 0 0 10179 20 0 0 25 0 5 0 522063793 1927077888 430060 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 407.75
Current children cumulated vsize (KiB) 1881912

[startup+162.301 s]
/proc/loadavg: 8.07 8.00 8.01 10/186 3808
/proc/meminfo: memFree=15085108/32950928 swapFree=67111528/67111528
[pid=3794] ppid=3790 vsize=2444940 CPUtime=647.39 cores=1,3,5,7
/proc/3794/stat : 3794 (ttsth-5-4) S 3790 3794 3320 0 -1 4202496 570610 0 0 0 64606 133 0 0 25 0 5 0 522063763 2503618560 570520 33554432000 4194304 4338100 140734408438528 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/3794/statm: 611235 570520 176 36 0 609140 0
[pid=3794/tid=3801] ppid=3790 vsize=2444940 CPUtime=161.11 cores=1,3,5,7
/proc/3794/task/3801/stat : 3801 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 184251 0 0 0 16067 44 0 0 25 0 5 0 522063793 2503618560 570520 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3794/tid=3802] ppid=3790 vsize=2444940 CPUtime=161.99 cores=1,3,5,7
/proc/3794/task/3802/stat : 3802 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 108774 0 0 0 16173 26 0 0 25 0 5 0 522063793 2503618560 570520 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3794/tid=3803] ppid=3790 vsize=2444940 CPUtime=161.99 cores=1,3,5,7
/proc/3794/task/3803/stat : 3803 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 145443 0 0 0 16166 33 0 0 25 0 5 0 522063793 2503618560 570520 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3794/tid=3804] ppid=3790 vsize=2444940 CPUtime=161.97 cores=1,3,5,7
/proc/3794/task/3804/stat : 3804 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 129899 0 0 0 16169 28 0 0 25 0 5 0 522063793 2503618560 570520 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 647.39
Current children cumulated vsize (KiB) 2444940


################
# More data... #
################

/proc/meminfo: memFree=9173200/32950928 swapFree=67111528/67111528
[pid=3794] ppid=3790 vsize=5659380 CPUtime=3044.02 cores=1,3,5,7
/proc/3794/stat : 3794 (ttsth-5-4) S 3790 3794 3320 0 -1 4202496 1375158 0 0 0 304037 365 0 0 25 0 5 0 522063763 5795205120 1375068 33554432000 4194304 4338100 140734408438528 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/3794/statm: 1414845 1375068 176 36 0 1412750 0
[pid=3794/tid=3801] ppid=3790 vsize=5659380 CPUtime=758.38 cores=1,3,5,7
/proc/3794/task/3801/stat : 3801 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 468647 0 0 0 75711 127 0 0 25 0 5 0 522063793 5795205120 1375068 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3794/tid=3802] ppid=3790 vsize=5659380 CPUtime=761.94 cores=1,3,5,7
/proc/3794/task/3802/stat : 3802 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 272666 0 0 0 76122 72 0 0 25 0 5 0 522063793 5795205120 1375068 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3794/tid=3803] ppid=3790 vsize=5659380 CPUtime=761.45 cores=1,3,5,7
/proc/3794/task/3803/stat : 3803 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 344704 0 0 0 76056 89 0 0 25 0 5 0 522063793 5795205120 1375068 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254317 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3794/tid=3804] ppid=3790 vsize=5659380 CPUtime=761.93 cores=1,3,5,7
/proc/3794/task/3804/stat : 3804 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 286898 0 0 0 76117 76 0 0 25 0 5 0 522063793 5795205120 1375068 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3044.02
Current children cumulated vsize (KiB) 5659380

[startup+822.301 s]
/proc/loadavg: 8.15 8.08 8.01 10/186 3825
/proc/meminfo: memFree=8706572/32950928 swapFree=67111528/67111528
[pid=3794] ppid=3790 vsize=5902032 CPUtime=3283.64 cores=1,3,5,7
/proc/3794/stat : 3794 (ttsth-5-4) S 3790 3794 3320 0 -1 4202496 1437134 0 0 0 327980 384 0 0 25 0 5 0 522063763 6043680768 1437044 33554432000 4194304 4338100 140734408438528 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/3794/statm: 1475508 1437044 176 36 0 1473413 0
[pid=3794/tid=3801] ppid=3790 vsize=5902032 CPUtime=818.36 cores=1,3,5,7
/proc/3794/task/3801/stat : 3801 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 489426 0 0 0 81703 133 0 0 25 0 5 0 522063793 6043680768 1437044 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3794/tid=3802] ppid=3790 vsize=5902032 CPUtime=821.92 cores=1,3,5,7
/proc/3794/task/3802/stat : 3802 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 282578 0 0 0 82118 74 0 0 25 0 5 0 522063793 6043680768 1437044 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3794/tid=3803] ppid=3790 vsize=5902032 CPUtime=821.12 cores=1,3,5,7
/proc/3794/task/3803/stat : 3803 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 361413 0 0 0 82017 95 0 0 25 0 5 0 522063793 6043680768 1437044 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3794/tid=3804] ppid=3790 vsize=5902032 CPUtime=821.91 cores=1,3,5,7
/proc/3794/task/3804/stat : 3804 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 301474 0 0 0 82110 81 0 0 25 0 5 0 522063793 6043680768 1437044 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3283.64
Current children cumulated vsize (KiB) 5902032

[startup+882.3 s]
/proc/loadavg: 8.15 8.09 8.02 10/186 3827
/proc/meminfo: memFree=8290904/32950928 swapFree=67111528/67111528
[pid=3794] ppid=3790 vsize=6178960 CPUtime=3523.29 cores=1,3,5,7
/proc/3794/stat : 3794 (ttsth-5-4) S 3790 3794 3320 0 -1 4202496 1507298 0 0 0 351925 404 0 0 25 0 5 0 522063763 6327255040 1507208 33554432000 4194304 4338100 140734408438528 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/3794/statm: 1544740 1507208 176 36 0 1542645 0
[pid=3794/tid=3801] ppid=3790 vsize=6178960 CPUtime=878.36 cores=1,3,5,7
/proc/3794/task/3801/stat : 3801 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 518792 0 0 0 87695 141 0 0 25 0 5 0 522063793 6327255040 1507208 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3794/tid=3802] ppid=3790 vsize=6178960 CPUtime=881.92 cores=1,3,5,7
/proc/3794/task/3802/stat : 3802 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 291264 0 0 0 88115 77 0 0 25 0 5 0 522063793 6327255040 1507208 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3794/tid=3803] ppid=3790 vsize=6178960 CPUtime=880.79 cores=1,3,5,7
/proc/3794/task/3803/stat : 3803 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 380234 0 0 0 87979 100 0 0 25 0 5 0 522063793 6327255040 1507208 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3794/tid=3804] ppid=3790 vsize=6178960 CPUtime=881.89 cores=1,3,5,7
/proc/3794/task/3804/stat : 3804 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 314765 0 0 0 88105 84 0 0 25 0 5 0 522063793 6327255040 1507208 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3523.29
Current children cumulated vsize (KiB) 6178960

[startup+942.301 s]
/proc/loadavg: 8.05 8.07 8.01 10/186 3829
/proc/meminfo: memFree=7832968/32950928 swapFree=67111528/67111528
[pid=3794] ppid=3790 vsize=6446504 CPUtime=3762.97 cores=1,3,5,7
/proc/3794/stat : 3794 (ttsth-5-4) S 3790 3794 3320 0 -1 4202496 1575511 0 0 0 375871 426 0 0 25 0 5 0 522063763 6601220096 1575421 33554432000 4194304 4338100 140734408438528 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/3794/statm: 1611626 1575421 176 36 0 1609531 0
[pid=3794/tid=3801] ppid=3790 vsize=6446504 CPUtime=938.35 cores=1,3,5,7
/proc/3794/task/3801/stat : 3801 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 550578 0 0 0 93684 151 0 0 25 0 5 0 522063793 6601220096 1575421 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3794/tid=3802] ppid=3790 vsize=6446504 CPUtime=941.92 cores=1,3,5,7
/proc/3794/task/3802/stat : 3802 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 300038 0 0 0 94112 80 0 0 25 0 5 0 522063793 6601220096 1575421 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3794/tid=3803] ppid=3790 vsize=6446504 CPUtime=940.46 cores=1,3,5,7
/proc/3794/task/3803/stat : 3803 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 398086 0 0 0 93941 105 0 0 25 0 5 0 522063793 6601220096 1575421 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3794/tid=3804] ppid=3790 vsize=6446504 CPUtime=941.89 cores=1,3,5,7
/proc/3794/task/3804/stat : 3804 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 324566 0 0 0 94101 88 0 0 25 0 5 0 522063793 6601220096 1575421 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3762.97
Current children cumulated vsize (KiB) 6446504

[startup+1002.3 s]
/proc/loadavg: 8.02 8.06 8.01 10/186 3830
/proc/meminfo: memFree=7437248/32950928 swapFree=67111528/67111528
[pid=3794] ppid=3790 vsize=6641980 CPUtime=4002.6 cores=1,3,5,7
/proc/3794/stat : 3794 (ttsth-5-4) S 3790 3794 3320 0 -1 4202496 1624621 0 0 0 399820 440 0 0 25 0 5 0 522063763 6801387520 1624531 33554432000 4194304 4338100 140734408438528 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/3794/statm: 1660495 1624531 176 36 0 1658400 0
[pid=3794/tid=3801] ppid=3790 vsize=6641980 CPUtime=998.36 cores=1,3,5,7
/proc/3794/task/3801/stat : 3801 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 565989 0 0 0 99681 155 0 0 25 0 5 0 522063793 6801387520 1624531 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3794/tid=3802] ppid=3790 vsize=6641980 CPUtime=1001.88 cores=1,3,5,7
/proc/3794/task/3802/stat : 3802 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 308802 0 0 0 100106 82 0 0 25 0 5 0 522063793 6801387520 1624531 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3794/tid=3803] ppid=3790 vsize=6641980 CPUtime=1000.13 cores=1,3,5,7
/proc/3794/task/3803/stat : 3803 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 409487 0 0 0 99905 108 0 0 25 0 5 0 522063793 6801387520 1624531 33554432000 4194304 4338100 140734408438528 18446744073709551615 274485781653 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3794/tid=3804] ppid=3790 vsize=6641980 CPUtime=1001.89 cores=1,3,5,7
/proc/3794/task/3804/stat : 3804 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 338100 0 0 0 100096 93 0 0 25 0 5 0 522063793 6801387520 1624531 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4002.6
Current children cumulated vsize (KiB) 6641980

[startup+1062.3 s]
/proc/loadavg: 8.05 8.06 8.01 10/186 3832
/proc/meminfo: memFree=6985252/32950928 swapFree=67111528/67111528
[pid=3794] ppid=3790 vsize=6890628 CPUtime=4242.26 cores=1,3,5,7
/proc/3794/stat : 3794 (ttsth-5-4) S 3790 3794 3320 0 -1 4202496 1687538 0 0 0 423768 458 0 0 25 0 5 0 522063763 7056003072 1687448 33554432000 4194304 4338100 140734408438528 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/3794/statm: 1722657 1687448 176 36 0 1720562 0
[pid=3794/tid=3801] ppid=3790 vsize=6890628 CPUtime=1058.35 cores=1,3,5,7
/proc/3794/task/3801/stat : 3801 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 586775 0 0 0 105674 161 0 0 25 0 5 0 522063793 7056003072 1687448 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3794/tid=3802] ppid=3790 vsize=6890628 CPUtime=1061.88 cores=1,3,5,7
/proc/3794/task/3802/stat : 3802 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 322853 0 0 0 106102 86 0 0 25 0 5 0 522063793 7056003072 1687448 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3794/tid=3803] ppid=3790 vsize=6890628 CPUtime=1059.81 cores=1,3,5,7
/proc/3794/task/3803/stat : 3803 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 426012 0 0 0 105868 113 0 0 25 0 5 0 522063793 7056003072 1687448 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3794/tid=3804] ppid=3790 vsize=6890628 CPUtime=1061.88 cores=1,3,5,7
/proc/3794/task/3804/stat : 3804 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 349655 0 0 0 106092 96 0 0 25 0 5 0 522063793 7056003072 1687448 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4242.26
Current children cumulated vsize (KiB) 6890628

[startup+1122.3 s]
/proc/loadavg: 8.10 8.07 8.01 10/186 3834
/proc/meminfo: memFree=6586820/32950928 swapFree=67111528/67111528
[pid=3794] ppid=3790 vsize=7101612 CPUtime=4481.92 cores=1,3,5,7
/proc/3794/stat : 3794 (ttsth-5-4) S 3790 3794 3320 0 -1 4202496 1740629 0 0 0 447719 473 0 0 25 0 5 0 522063763 7272050688 1740539 33554432000 4194304 4338100 140734408438528 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/3794/statm: 1775403 1740539 176 36 0 1773308 0
[pid=3794/tid=3801] ppid=3790 vsize=7101612 CPUtime=1118.36 cores=1,3,5,7
/proc/3794/task/3801/stat : 3801 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 609295 0 0 0 111668 168 0 0 25 0 5 0 522063793 7272050688 1740539 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3794/tid=3802] ppid=3790 vsize=7101612 CPUtime=1121.88 cores=1,3,5,7
/proc/3794/task/3802/stat : 3802 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 330079 0 0 0 112100 88 0 0 25 0 5 0 522063793 7272050688 1740539 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3794/tid=3803] ppid=3790 vsize=7101612 CPUtime=1119.48 cores=1,3,5,7
/proc/3794/task/3803/stat : 3803 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 439249 0 0 0 111831 117 0 0 25 0 5 0 522063793 7272050688 1740539 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3794/tid=3804] ppid=3790 vsize=7101612 CPUtime=1121.87 cores=1,3,5,7
/proc/3794/task/3804/stat : 3804 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 359763 0 0 0 112088 99 0 0 25 0 5 0 522063793 7272050688 1740539 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4481.92
Current children cumulated vsize (KiB) 7101612

[startup+1182.3 s]
/proc/loadavg: 8.11 8.09 8.02 10/186 3835
/proc/meminfo: memFree=6145352/32950928 swapFree=67111528/67111528
[pid=3794] ppid=3790 vsize=7340620 CPUtime=4721.58 cores=1,3,5,7
/proc/3794/stat : 3794 (ttsth-5-4) S 3790 3794 3320 0 -1 4202496 1801497 0 0 0 471667 491 0 0 25 0 5 0 522063763 7516794880 1801407 33554432000 4194304 4338100 140734408438528 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/3794/statm: 1835155 1801407 176 36 0 1833060 0
[pid=3794/tid=3801] ppid=3790 vsize=7340620 CPUtime=1178.36 cores=1,3,5,7
/proc/3794/task/3801/stat : 3801 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 630476 0 0 0 117662 174 0 0 25 0 5 0 522063793 7516794880 1801407 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254291 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3794/tid=3802] ppid=3790 vsize=7340620 CPUtime=1181.85 cores=1,3,5,7
/proc/3794/task/3802/stat : 3802 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 341861 0 0 0 118095 90 0 0 25 0 5 0 522063793 7516794880 1801407 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3794/tid=3803] ppid=3790 vsize=7340620 CPUtime=1179.15 cores=1,3,5,7
/proc/3794/task/3803/stat : 3803 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 454142 0 0 0 117795 120 0 0 25 0 5 0 522063793 7516794880 1801407 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3794/tid=3804] ppid=3790 vsize=7340620 CPUtime=1181.88 cores=1,3,5,7
/proc/3794/task/3804/stat : 3804 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 372775 0 0 0 118084 104 0 0 25 0 5 0 522063793 7516794880 1801407 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4721.58
Current children cumulated vsize (KiB) 7340620



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202 s]
/proc/loadavg: 8.21 8.11 8.02 10/186 3836
/proc/meminfo: memFree=6027660/32950928 swapFree=67111528/67111528
[pid=3794] ppid=3790 vsize=7400564 CPUtime=4800.28 cores=1,3,5,7
/proc/3794/stat : 3794 (ttsth-5-4) S 3790 3794 3320 0 -1 4202496 1816398 0 0 0 479533 495 0 0 25 0 5 0 522063763 7578177536 1816308 33554432000 4194304 4338100 140734408438528 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/3794/statm: 1850141 1816308 176 36 0 1848046 0
[pid=3794/tid=3801] ppid=3790 vsize=7400564 CPUtime=1198.07 cores=1,3,5,7
/proc/3794/task/3801/stat : 3801 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 636011 0 0 0 119631 176 0 0 25 0 5 0 522063793 7578177536 1816308 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3794/tid=3802] ppid=3790 vsize=7400564 CPUtime=1201.55 cores=1,3,5,7
/proc/3794/task/3802/stat : 3802 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 344456 0 0 0 120064 91 0 0 25 0 5 0 522063793 7578177536 1816308 33554432000 4194304 4338100 140734408438528 18446744073709551615 4242360 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3794/tid=3803] ppid=3790 vsize=7400564 CPUtime=1198.74 cores=1,3,5,7
/proc/3794/task/3803/stat : 3803 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 457845 0 0 0 119753 121 0 0 25 0 5 0 522063793 7578177536 1816308 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254352 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3794/tid=3804] ppid=3790 vsize=7400564 CPUtime=1201.58 cores=1,3,5,7
/proc/3794/task/3804/stat : 3804 (ttsth-5-4) R 3790 3794 3320 0 -1 4202560 375843 0 0 0 120053 105 0 0 25 0 5 0 522063793 7578177536 1816308 33554432000 4194304 4338100 140734408438528 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.28
Current children cumulated vsize (KiB) 7400564

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.2 s]
/proc/loadavg: 8.21 8.11 8.02 4/182 3838
/proc/meminfo: memFree=15551264/32950928 swapFree=67111528/67111528
[pid=3794] ppid=3790 vsize=0 CPUtime=4800.5 cores=1,3,5,7
/proc/3794/stat : 3794 (ttsth-5-4) Z 3790 3794 3320 0 -1 4203548 1816398 0 0 0 479535 515 0 0 18 0 2 0 522063763 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/3794/statm: 0 0 0 0 0 0 0
[pid=3794/tid=3803] ppid=3790 vsize=0 CPUtime=1198.94 cores=1,3,5,7
/proc/3794/task/3803/stat : 3803 (ttsth-5-4) R 3790 3794 3320 0 -1 4203588 457845 0 0 0 119753 141 0 0 25 0 2 0 522063793 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4800.5
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.28
CPU time (s): 4800.58
CPU user time (s): 4795.35
CPU system time (s): 5.22421
CPU usage (%): 399.29
Max. virtual memory (cumulated for all children) (KiB): 7400564

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4795.35
system time used= 5.2242
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1816398
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1927
involuntary context switches= 46495

runsolver used 1.83872 second user time and 4.72928 second system time

The end

Launcher Data 

Begin job on node125 at 2011-04-29 21:17:21
IDJOB=3266314
IDBENCH=87453
IDSOLVER=1654
FILE ID=node125/3266314-1304104641
RUNJOBID= node125-1304104641-3776
PBS_JOBID= 13170627
Free space on /tmp= 72764 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT02/handmade/pehoushek/plainoldcnf/5cnf_3900_3900_220.shuffled.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3266314-1304104641/watcher-3266314-1304104641 -o /tmp/evaluation-result-3266314-1304104641/solver-3266314-1304104641 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3266314-1304104641.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= f0b7e0675915478b15cab3695fd0fd62
RANDOM SEED=1217696561

node125.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5266.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      19217096 kB
Buffers:       2218808 kB
Cached:       10918988 kB
SwapCached:          0 kB
Active:        8428992 kB
Inactive:      4773004 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      19217096 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:             156 kB
Writeback:           0 kB
AnonPages:       63172 kB
Mapped:          14400 kB
Slab:           467500 kB
PageTables:       4556 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   188008 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72764 MiB
End job on node125 at 2011-04-29 21:37:25