Stable Verified Tencent Cloud Account Tencent Cloud TKE Node Memory Leak and OOM Eviction Troubleshooting

If your TKE nodes keep getting MemoryPressure, pods are being evicted, or containers are dying with OOMKilled, the real problem is usually not “memory is low” in the abstract. In practice, you need to answer three questions fast:

1. Is it a pod-level memory issue or a node-level memory leak?
2. Stable Verified Tencent Cloud Account Can you restore service immediately without making the bill explode?
3. If you need to buy more capacity now, will your account, KYC, and payment method block the operation?

I’ll focus on the questions that usually matter during a real incident: what to check first, how to tell whether the node is leaking memory, what to do before the next eviction happens, and what account/payment problems can stop you from expanding the cluster in time.

First: don’t guess. Check whether the node is really leaking memory

Stable Verified Tencent Cloud Account When TKE starts evicting pods, many teams immediately increase node specs. That sometimes works, but in support cases I’ve seen, about half of the incidents were not fixed by “buying bigger nodes.” They were caused by resource settings, a sidecar leak, log pressure, or a hidden system process consuming memory.

Start with the node and pod view together:

kubectl top nodes
kubectl top pods -A --sort-by=memory
kubectl describe node <node-name>
kubectl get events -A --sort-by='.lastTimestamp'

Stable Verified Tencent Cloud Account Then check whether the node is under genuine pressure:

• Node allocatable memory vs. actual workload sum
• Eviction threshold messages from kubelet
• Pod memory limit and request settings
• System processes like logging agents, CNI, kube-proxy, CSI, or container runtime

A common trap: the node looks like it still has “free” memory because Linux cache is counted as reclaimable, but kubelet still sees low available memory and starts evicting pods under pressure. So don’t rely only on top-level “free” memory numbers.

The most common reasons TKE nodes hit OOM or eviction

In real production clusters, the cause is usually one of these:

• Application memory leak: memory keeps climbing after traffic stabilizes.
• Container memory limit too low: the app is healthy, but the limit is unrealistic for peak traffic.
• Requests/limits mismatch: scheduling packed too many pods onto one node.
• Sidecar overhead: service mesh, log collector, and metrics agents consume more than expected.
• Log burst or file cache pressure: noisy logging pushes the node over the edge.
• Node reserved memory too small: kubelet, container runtime, and system services fight with workloads.
• Traffic spike: connection buffers, queue depth, or JVM heap grow too fast.

If the same node evicts different pods repeatedly, the problem is more likely node-side pressure. If one workload keeps dying while the node remains healthy, it is probably a pod memory limit or application leak.

How to tell if it’s a real node memory leak

A node leak means memory usage on the host keeps rising even after the pod-level usage does not. That can happen with the kernel page cache, log agent buffers, conntrack tables, or a system process leak.

Useful checks:

free -h
cat /proc/meminfo
ps aux --sort=-%mem | head
dmesg | egrep -i 'oom|memory|killed process'
journalctl -u kubelet --since "1 hour ago"

What you are looking for:

• Memory climbs across the whole node, not just one container.
• Kubelet eviction messages appear before app errors.
• One daemonset or runtime process keeps growing.
• Node pressure remains after restarting the app.

One case I handled had repeated evictions on a TKE node running a Java service. The team assumed the app itself was leaking. After checking host memory, the actual culprit was a log collection daemonset and a JVM heap setting that allowed the process to use almost all container memory. The fix was not just “bigger node”; it was a combination of log rate control, JVM tuning, and a more realistic memory request.

What to do in the first 30 minutes

If production is already evicting pods, your first job is to stop the bleeding. Don’t start with a rewrite or a long root-cause discussion.

1. Identify the hottest pod:

   kubectl top pods -A --sort-by=memory

2. Check whether the pod is hitting its limit:

   kubectl describe pod <pod>

   Look for OOMKilled, Evicted, or Exceeded memory limit.
3. Scale out if the workload can tolerate it:
   • Increase replica count for stateless services
   • Shift traffic away from the bad node
   • Cordon and drain a node if it is obviously unhealthy
4. Restart only the leaking pod if you need immediate relief and the service is stateless.
5. Buy time before permanent fixes: raise pod limit or move to a larger node only if the account is ready to purchase and your billing method can pass risk checks.

If your cluster uses subscription nodes and renewal is due soon, check that first. I’ve seen “node abnormal” reports that were actually caused by an expired or about-to-expire instance, not a pure memory incident. For production nodes, auto-renew or a longer billing buffer is safer than waiting for manual renewal during an outage.

When the fix is in the app, not the node

In my experience, the most cost-effective fix is often tuning the workload instead of buying more memory.

Typical patterns that cause repeated OOMs:

• JVM services: heap too close to container limit, GC spikes, or metaspace growth.
• Go services: caches and buffers grow, but the team only monitors average RSS.
• Python services: worker processes fork too many times or keep large in-memory objects.
• Stable Verified Tencent Cloud Account Sidecar-heavy pods: the app is fine, but the log agent and mesh proxy push total memory over the limit.

A practical rule: if the memory graph climbs continuously for hours under steady traffic, treat it as a leak until proven otherwise. If it spikes only during traffic bursts and falls back after load drops, it is more likely a sizing issue.

If you need to buy more capacity, account setup can become the real blocker

During incidents, many teams discover that scaling the cluster is not just a technical problem. The purchase can fail because the cloud account is not ready. This is especially common on newer Tencent Cloud International accounts.

Before production traffic depends on the cluster, make sure the following are already in place:

• Identity verification (KYC) completed for the account type you use
• Billing profile filled with matching name and country/region information
• Primary payment method tested successfully before a real outage
• Quota and region limits confirmed for the instance family you want to buy
• Renewal policy configured if you use subscription nodes

Common reasons purchase or renewal fails

• Stable Verified Tencent Cloud Account KYC still pending: the account can browse the console but not create enough resources.
• Card verification fails: billing address mismatch, 3D Secure issue, bank fraud filter, or insufficient authorization.
• Risk control review: new account, unusual login location, repeated failed payments, or sudden high-value purchases.
• Region or product restriction: the account is not allowed to buy a certain region or instance type yet.
• Quota limit: the account is approved, but the specific resource cap is too low for the new node count.

One thing to avoid: repeated payment retries in a short period. That often makes risk control stricter, not looser. If a purchase is failing, stop and check whether the issue is KYC, billing name mismatch, or card approval instead of blindly clicking retry.

Payment methods: what works best in real operations

For international cloud accounts, the payment method matters as much as the budget. In practice:

• Credit/debit cards are the fastest for urgent pay-as-you-go expansion.
• Enterprise billing arrangements may support invoicing or bank transfer in some cases, but they are slower to activate.
• Prepaid or wallet-style funding can help control spend, but it may delay emergency purchases if the balance is not maintained.

For a production TKE environment, my recommendation is simple: keep a working card on file for emergency scaling, and use a separate approval process for larger enterprise commitments. That way, a memory incident does not wait on procurement.

Cost comparison: fixing the leak is usually cheaper than buying bigger nodes

Many teams ask whether it’s cheaper to increase node memory or to spend engineering time on the leak. The honest answer depends on how often the problem repeats.

Here is the usual pattern:

• One-time incident: buying a larger node may be the fastest way to restore service.
• Weekly or daily OOMs: increasing node size just hides the problem and raises spend every month.
• Unstable sidecar or log agent: tuning or removing it is usually cheaper than scaling the whole node pool.

On pay-as-you-go billing, moving up one memory size often increases the bill noticeably, but the exact delta depends on the region and instance family. In the incidents I’ve reviewed, a jump to a larger memory tier is usually a short-term fix, not a final answer. If the workload leaks memory at a fixed rate, the node will eventually hit pressure again no matter how much memory you buy.

If you are deciding between three options, use this order:

1. Adjust limits/requests and restart the leaking pod
2. Scale out stateless replicas
3. Scale up the node only if the first two cannot stabilize service

Real-world scenarios that show what usually matters

Stable Verified Tencent Cloud Account Scenario 1: Java service keeps getting OOMKilled after traffic spikes

Stable Verified Tencent Cloud Account The team increased the node size twice, but the pod still died during peak hour. The actual problem was that the JVM heap and container limit were too close, leaving no room for metaspace, native memory, and burst activity. The fix was to lower heap size relative to the limit, raise the pod limit slightly, and add more replicas. Result: fewer incidents and lower cost than continuing to buy larger nodes.

Scenario 2: Node-level eviction after log traffic surged

The app itself did not leak. A logging agent and oversized buffer settings caused memory growth on the host, while the workload was packed too tightly onto one node. The team rebuilt the node pool with better reserved memory settings, reduced log verbosity, and added alerting on node allocatable memory. Result: no more surprise evictions, and the cluster stopped oscillating between “fine” and “on fire.”

FAQ: the questions people usually ask right before they buy or renew

1. Why does the pod show OOMKilled but the node still has memory left?

Because the container hit its own memory limit first. That is a pod-level issue, not necessarily a node-level shortage.

2. Why does the node evict pods even when usage looks normal?

Because kubelet evicts based on allocatable memory, system reserve, and real-time pressure, not on a simple “used vs free” snapshot.

3. Should I just buy larger nodes?

Only as a temporary stabilizer. If the leak is in the app or sidecar, bigger nodes only delay the next failure.

4. What if I need to scale now but the Tencent Cloud purchase fails?

Check KYC status, billing name, card approval, and quota limits first. If the account is new, risk control may block large or repeated purchases until verification is complete.

5. Is pay-as-you-go better for incident response?

Usually yes. It is faster to expand than waiting for a subscription purchase or renewal flow, especially during an outage.

6. Do I need enterprise verification to run production TKE?

Not always, but enterprise verification helps when you need larger quotas, invoicing, or more stable procurement approval. If your cluster is production-critical, it is safer to complete verification before you need emergency capacity.

7. Can a payment card issue block renewals too?

Yes. Renewal failures are often just as disruptive as fresh purchases, especially if your nodes are subscription-based. Test the payment method before the renewal date.

8. How do I avoid this in the next incident?

Set memory alerts on node allocatable memory, container RSS, and eviction events. Keep one tested payment method on file, pre-complete KYC, and avoid running production nodes close to expiry.

What I would do before the next traffic spike

• Audit every workload’s memory request and limit.
• Check whether any sidecar or log agent has grown silently.
• Verify node reserved memory and eviction thresholds.
• Make sure the cloud account can purchase or renew without manual back-and-forth.
• Keep a working payment method and completed KYC before peak season.
• Prefer pay-as-you-go capacity for emergency expansion, and use subscription only where lifecycle is predictable.

If you handle the account side only after the outage starts, you are already late. In practice, the fastest way to keep a TKE memory incident under control is to combine technical tuning with billing readiness: fix the leak, reserve enough memory, and make sure procurement will not block your scale-out when it matters.