linux的統(tǒng)計實現(xiàn)
2021-08-10 13:26:46

場景：

將下面的數(shù)據(jù)里category里的分類統(tǒng)計計數(shù)

數(shù)據(jù)源

es_ip10000.json

{"_index":"order","_type":"service","_id":"107.151.83.180:22","_score":1,"_source":{"ip":"107.151.83.180","parent_category":["支撐系統(tǒng)"],"category":["其他支撐系統(tǒng)"]}}
{"_index":"order","_type":"service","_id":"107.151.84.167:22","_score":1,"_source":{"ip":"107.151.84.167","parent_category":["支撐系統(tǒng)"],"category":["其他支撐系統(tǒng)"]}}
{"_index":"order","_type":"service","_id":"107.151.84.177:22","_score":1,"_source":{"ip":"107.151.84.177","parent_category":["支撐系統(tǒng)"],"category":["其他支撐系統(tǒng)"]}}
{"_index":"order","_type":"service","_id":"107.152.188.252:1723","_score":1,"_source":{"ip":"107.152.188.252","parent_category":["網(wǎng)絡(luò)產(chǎn)品"],"category":["路由器"]}}
{"_index":"order","_type":"service","_id":"107.151.89.125:1025","_score":1,"_source":{"ip":"107.151.89.125"}}
{"_index":"order","_type":"service","_id":"107.152.58.217:22","_score":1,"_source":{"ip":"107.152.58.217","parent_category":["支撐系統(tǒng)"],"category":["服務(wù)"]}}
{"_index":"order","_type":"subdomain","_id":"107.15.221.83:443","_score":1,"_source":{"ip":"107.15.221.83","parent_category":["辦公外設(shè)","系統(tǒng)軟件"],"category":["打印機(jī)","操作系統(tǒng)"]}}

取_source下的category字段

cat es_ip10000.json | jq ._source.category > category.txt

輸出結(jié)果

[
  "其他支撐系統(tǒng)"
]
[
  "其他支撐系統(tǒng)"
]
[
  "其他支撐系統(tǒng)"
]
[
  "路由器"
]
null
[
  "服務(wù)"
]
[
  "打印機(jī)",
  "操作系統(tǒng)"
]

用編輯器，去除 , [ 和 ]

處理后的結(jié)果

  "其他支撐系統(tǒng)"


  "其他支撐系統(tǒng)"


  "其他支撐系統(tǒng)"


  "路由器"

null

  "服務(wù)"


  "打印機(jī)"
  "操作系統(tǒng)"

排序 -->去重->統(tǒng)計->再排序

cat category.txt | sort | uniq -c | sort -n >category_count.txt

說明：

uniq -c #去重并統(tǒng)計

sort -n # 正序排序

輸出結(jié)果：

      1 null
      1   "操作系統(tǒng)"
      1   "打印機(jī)"
      1   "服務(wù)"
      1   "路由器"
      3   "其他支撐系統(tǒng)"
     12