All geek questions in one place

Solarium and Java highlighting required

Background:

We have a vendor-supplied Java application that has a slightly large bunch of Java. Without going into too much information, the application is a black box for us, but we feel that we need to take on the task to try to set up the work and fix the problems.

The 64-bit SunOS 10 machine has 16 GB of RAM, and the only non-system application that runs is the JVM for this application. The 64-bit JVM runs in JBoss, which, in my opinion, is not relevant to this discussion, and the maximum heap size is 8 GB, which in my opinion matters.

Recently, the problem is that we get various errors in memory. The heap is not full when these errors occur, and the error asks “Outside the swap space?”. The provider wants us to simply increase the swap from 2 GB to 4 GB. It is on a system with 16 GB, and the application is only 8 GB. We think this would be a bad idea for performance.

My question is:

So, we found that file caching uses all the remaining free memory to improve performance. This is usually not a problem, but it seems to fragment memory. Because the JVM Hotspot requires continuous memory space, we realized that this fragmentation of memory leads to the use of swap space, which is not fragmented.

However, I'm not sure how much I understand the relationship between fragmentation and the requirement for continuous memory. Of course, fragmentation refers only to the fragmentation of the physical ram. With virtual memory, it is entirely possible to allocate a continuous piece of RAM without supporting its continuous part. In other words, a continuous piece of physical memory will be displayed in the current process as a continuous piece of virtual memory.

So, I think there was not a single question in this question, but does anyone know more about this and can call back? Any links citing this issue with contiguous memory on 64-bit systems?

What I have found so far:

So far, every link I found in the "continuous memory" problem has been more related to how the virtual address space is laid out on 32-bit address systems. Since we use a 64-bit system (with 48-bit addressing), there is a large amount of virtual address space to allocate large contiguous fragments.

I searched all the information on the Internet, but still could not find the information I am looking for.

Updates:

  • To be clear, I did not try to get an answer to the question of why I get OOM errors, but I try to understand the relationship between the possible fragmented system RAM and the adjacent piece of virtual memory needed by java.
  • prstat -Z
ZONEID NPROC SWAP RSS MEMORY TIME CPU ZONE 0 75 4270M 3855M 24% 92:24:03 0.3% global
  • echo ":: memstat" | mdb -k
 Page Summary Pages MB %Tot ------------ ---------------- ---------------- ---- Kernel 326177 2548 16% ZFS File Data 980558 7660 48% Anon 561287 4385 27% Exec and libs 12196 95 1% Page cache 17849 139 1% Free (cachelist) 4023 31 0% Free (freelist) 156064 1219 8% Total 2058154 16079 Physical 2042090 15953

  • Where I used to think that the ZFS file data is freely available memory, since then I found out that this is not so and it can very well be the cause of errors.

  • vmstat 5 5

 kthr memory page disk faults cpu rbw swap free re mf pi po fr de sr vc vc vc -- in sy cs us sy id 0 0 0 2161320 2831768 12 55 0 0 0 0 0 3 4 -0 0 1089 1320 1048 1 1 98 0 0 0 819720 1505856 0 14 0 0 0 0 0 4 0 0 0 1189 748 1307 1 0 99 0 0 0 819456 1505648 0 1 0 0 0 0 0 0 0 0 0 1024 729 1108 0 0 99 0 0 0 819456 1505648 0 1 0 0 0 0 0 0 0 0 0 879 648 899 0 0 99 0 0 0 819416 1505608 0 1 0 0 0 0 0 0 3 0 0 1000 688 1055 0 0 99

  • These command exits were executed when the application was running in a healthy state. Now we track all of the above and register it if we again see paging space errors.

  • The following shows how the JVM grew to 8 GB and then was restarted. The effect of this is that ZFS ARC declined (to 26% RAM) until it increased again. How does it look now?

  • vmstat 5 5

 kthr memory page disk faults cpu rbw swap free re mf pi po fr de sr vc vc -- -- in sy cs us sy id 0 0 0 1372568 2749528 11 41 0 0 0 0 0 2 3 0 0 713 418 539 0 0 99 0 0 0 3836576 4648888 140 228 0 0 0 0 0 0 0 0 0 1178 5344 1117 3 2 95 0 0 0 3840448 4653744 16 45 0 0 0 0 0 0 0 0 0 1070 1013 952 1 3 96 0 0 0 3839168 4652720 6 53 0 0 0 0 0 0 0 0 0 564 575 313 0 6 93 0 0 0 3840208 4653752 7 68 0 0 0 0 0 3 0 0 0 1284 1014 1264 1 1 98
  • swap -s

total: 4341344k bytes allocated + 675384k reserved = 5016728k used, available 3840880k

+4
source share
2 answers

When an error message suggests that the swap space may not be large enough, I usually trust it and significantly increase the swap size.

I would suggest that you do this first, up to 4 GB or even 8 GB, and see what happens. Swap expansion does not affect performance. This is a common misconception. What affects performance is lack of RAM, and not too much swap space.

Only if the problem remains after the change, I try to explore alternative tracks, for example, possibly memory fragmentation.

Edit:

The output from memstat, prstat, and vmstat shows that your system is out of virtual memory. There is absolutely no need to investigate other unusual causes, such as memory fragmentation. You have more free memory (~ 1.5G) than free virtual memory (~ 800 MB). This means that there are many unused (so far) memory backups. Again, just add some swap space to fix this. This will not have any impact on performance since you have enough RAM.

Edit: (part 2)

Now we know that you are using ZFS, since your application can use up to 8 GB (and even more if you take into account the heap without the heap), you should reduce the maximum ARC size to ensure that these 8 GB are immediately available for the JVM, instead to rely on proprietary OS adjustments that may currently be confused by a short swap. For more information on how to do this, see Limiting the Chapter of the ARC Cache in the ZFS Evil Configuration Guide .

+1
source

You may get this error if you run out of main swap space. While the 2GB shift these days is pretty small, if you need to use the swap space, you have a performance problem, as this means that you force applications to swap to disk.

Increasing the maximum heap in this situation will not help, because the problem is the lack of memory. It may even make it more likely.

The solution may be to use less memory (by reducing the number of other applications running on the system) or increasing the main memory (these days 16 GB is not much for the server, my home PC has 24;)

EDIT: Another thing I've seen causing this problem is the heavy use of direct memory with heap memory. for example, by default, your maximum direct memory matches the maximum heap size. This means that your application can use almost 8 GB of direct memory before using the whole bunch. An application may find that there is not enough space in the OS for the main + swap to allocate memory in a heap and may die.

I try to use direct memory as much as possible, but this is done to avoid using a lot of heap .;)

0
source

Source: https://habr.com/ru/post/1399019/

More articles:


All Articles