一次抗DDoS攻击实践

留言

2024.3.11号晚上6点多,taskon无法正常访问,但是在pod里ps -aux,以及看日志是正常的

rongyi老哥看cf和腾讯云load balance的流量,发现被DDoS攻击了

同时junjie也打电话,说攻击者在discord私聊说在发动攻击,要求索要6000-8000美刀


我们目前的架构是, 最前面是cloudflare,然后到腾讯云的load balance,再到pod

攻击者的ip主要来自东南亚,印尼,越南,孟加拉国,泰国,以及美国和德国

其实cf可以防一些,但我们没有充分设置, 可以临时禁止某些地区的ip等

后面找腾讯云接入高防,差不多5000刀每个月,非常贵

发动攻击者需要事前通过木马感染很多肉鸡,然后统一发动攻击,其实成本也非常高

基本是钱抗钱


后面我和毛博加了cf的一些规则,免费版可以设置两条,我们设置了10s内单个ip,只能请求全部taskon相关的地址1000次.

但其实因为前端打包的问题,1000次经常被限制…

https://www.sky350.com/1237.html

https://help.aliyun.com/zh/anti-ddos/product-overview/best-practices-for-mitigating-ddos-attacks?userCode=okjhlpr5


其实之前几分钟,已经收到了prom的告警:

17:54:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
[FIRING:2] NodeHighNumberConntrackEntriesUsed (node-exporter http-metrics node-exporter monitoring monitoring/prometheus-kube-prometheus-prometheus prometheus-prometheus-node-exporter warning)
100% of conntrack entries are used. https://runbooks.prometheus-operator.dev/runbooks/node/nodehighnumberconntrackentriesused Number of conntrack are getting close to the limit.
Alerts Firing:
Labels:
 - alertname = NodeHighNumberConntrackEntriesUsed
 - container = node-exporter
 - endpoint = http-metrics
 - instance = 172.22.0.27:9100
 - job = node-exporter
 - namespace = monitoring
 - pod = prometheus-prometheus-node-exporter-lzgbd
 - prometheus = monitoring/prometheus-kube-prometheus-prometheus
 - service = prometheus-prometheus-node-exporter
 - severity = warning
Annotations:
 - description = 100% of conntrack entries are used.
 - runbook_url = https://runbooks.prometheus-operator.dev/runbooks/node/nodehighnumberconntrackentriesused
 - summary = Number of conntrack are getting close to the limit.
Source: http://prometheus-kube-prometheus-prometheus.monitoring:9090/graph?g0.expr=%28node_nf_conntrack_entries+%2F+node_nf_conntrack_entries_limit%29+%3E+0.75&g0.tab=1
Labels:
 - alertname = NodeHighNumberConntrackEntriesUsed
 - container = node-exporter
 - endpoint = http-metrics
 - instance = 172.22.0.41:9100
 - job = node-exporter
 - namespace = monitoring
 - pod = prometheus-prometheus-node-exporter-l8tlr
 - prometheus = monitoring/prometheus-kube-prometheus-prometheus
 - service = prometheus-prometheus-node-exporter
 - severity = warning
Annotations:
 - description = 100% of conntrack entries are used.
 - runbook_url = https://runbooks.prometheus-operator.dev/runbooks/node/nodehighnumberconntrackentriesused
 - summary = Number of conntrack are getting close to the limit.
Source: http://prometheus-kube-prometheus-prometheus.monitoring:9090/graph?g0.expr=%28node_nf_conntrack_entries+%2F+node_nf_conntrack_entries_limit%29+%3E+0.75&g0.tab=1


AlertmanagerUrl:
http://prometheus-kube-prometheus-alertmanager.monitoring:9093/#/al

18:02:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[FIRING:1] NodeHighNumberConntrackEntriesUsed (node-exporter http-metrics 172.22.0.27:9100 node-exporter monitoring prometheus-prometheus-node-exporter-lzgbd monitoring/prometheus-kube-prometheus-prometheus prometheus-prometheus-node-exporter warning)
76.74% of conntrack entries are used. https://runbooks.prometheus-operator.dev/runbooks/node/nodehighnumberconntrackentriesused Number of conntrack are getting close to the limit.
Alerts Firing:
Labels:
 - alertname = NodeHighNumberConntrackEntriesUsed
 - container = node-exporter
 - endpoint = http-metrics
 - instance = 172.22.0.27:9100
 - job = node-exporter
 - namespace = monitoring
 - pod = prometheus-prometheus-node-exporter-lzgbd
 - prometheus = monitoring/prometheus-kube-prometheus-prometheus
 - service = prometheus-prometheus-node-exporter
 - severity = warning
Annotations:
 - description = 76.74% of conntrack entries are used.
 - runbook_url = https://runbooks.prometheus-operator.dev/runbooks/node/nodehighnumberconntrackentriesused
 - summary = Number of conntrack are getting close to the limit.
Source: http://prometheus-kube-prometheus-prometheus.monitoring:9090/graph?g0.expr=%28node_nf_conntrack_entries+%2F+node_nf_conntrack_entries_limit%29+%3E+0.75&g0.tab=1


AlertmanagerUrl:
http://prometheus-kube-prometheus-alertmanager.monitoring:9093/#/alerts?receiver=wechat

关于 NodeHighNumberConntrackEntriesUsed

10 个常用监控 Kubernetes 性能的 Prometheus Operator 指标

prometheus SQL汇总

报警规则说明

文章目录