So I followed this AWS blog and this documentation to launch a tiny t2 elasticsearch cluster to visualize VPC flow logs. Those links have instructions that guide you along setting up flow logs to flow into ES in a few different ways. I ended up following the documentation link and then downloading some kibana3 dashboards until I found one I liked.
Over time however, the little t2 ES cluster could not keep up, and I ran out of storage space and CPU credits. So I wanted to automate the deletion of indices / indexes so that the cluster would free up storage space and not churn through CPU. With more RAM available the cluster uses less CPU, so I had to limit how much data the single node ES cluster is storing. There is plenty of documentation online on how to use curl to delete elasticsearch indexes but I’m on windows most of the time so I decided to write a quick a powershell script to do it.
To use this script just update the esdomain variable to point to your ES cluster name. Also this filter will only work if the lambda script is creating cwl- indexes. Tweak it if your indexes are different. Run it and it will keep the last 2 weeks of indexes and delete anything older.
|
1 2 3 4 5 6 7 8 9 10 11 |
$webclient = New-Object system.net.webclient $esdomain = "https://search-YOURCLUSTERNAMEGOESHERE.us-east-1.es.amazonaws.com" $daystokeep = 14 $indexes = $webclient.DownloadString("$($esdomain)/_aliases?pretty=1") $indexeslist=($indexes.split('"',[System.StringSplitOptions]::RemoveEmptyEntries) | Select-String -AllMatches 'cwl-') | Sort-Object -Descending $indexeslistcount = ($indexeslist | Measure-Object).Count if ($indexeslistcount -gt $daystokeep) { $indexeslist | Select-Object -Skip $daystokeep | ForEach-Object { $webclient.UploadString("$($esdomain)/$($_)","DELETE",0) } } |
0 Comments.