{"id":61,"date":"2016-09-05T01:30:11","date_gmt":"2016-09-05T05:30:11","guid":{"rendered":"http:\/\/packetlost.com\/blog\/?p=61"},"modified":"2017-03-17T23:07:43","modified_gmt":"2017-03-18T03:07:43","slug":"delete-elasticsearch-indexes-with-powershell","status":"publish","type":"post","link":"https:\/\/packetlost.com\/blog\/2016\/09\/05\/delete-elasticsearch-indexes-with-powershell\/","title":{"rendered":"Delete ElasticSearch indexes with powershell"},"content":{"rendered":"<p>So I followed<a href=\"https:\/\/aws.amazon.com\/blogs\/aws\/cloudwatch-logs-subscription-consumer-elasticsearch-kibana-dashboards\/\"> this AWS blog<\/a> and this <a href=\"http:\/\/docs.aws.amazon.com\/AmazonCloudWatch\/latest\/logs\/CWL_ES_Stream.html\">documentation<\/a> to launch a tiny t2 elasticsearch cluster to visualize VPC flow logs. Those links have instructions that guide you along setting up flow logs to flow into ES in a few different ways. I ended up following the documentation link and then downloading some kibana3 dashboards until I found one I liked.<\/p>\n<p>Over time however, the little t2 ES cluster could not keep up, and I ran out of storage space and CPU credits. So I wanted to automate the deletion of indices \/ indexes so that the cluster would free up storage space and not churn through CPU. With more RAM available the cluster uses less CPU, so I had to limit how much data the single node ES cluster is storing. There is plenty of documentation online on how to use curl to delete elasticsearch indexes but I&#8217;m on windows most of the time so I decided to write a quick a powershell script to do it.<\/p>\n<p>To use this script just update the esdomain variable to point to your ES cluster name. Also this filter will only work if the lambda script is creating cwl- indexes. Tweak it if your indexes are different. Run it and it will keep the last 2 weeks of indexes and delete anything older.<\/p>\n<p>&nbsp;<\/p>\n<pre class=\"lang:ps decode:true \" title=\"Delete-ESIndexes\">$webclient = New-Object system.net.webclient\r\n$esdomain = \"https:\/\/search-YOURCLUSTERNAMEGOESHERE.us-east-1.es.amazonaws.com\"\r\n$daystokeep = 14\r\n\r\n$indexes = $webclient.DownloadString(\"$($esdomain)\/_aliases?pretty=1\")\r\n$indexeslist=($indexes.split('\"',[System.StringSplitOptions]::RemoveEmptyEntries) | Select-String -AllMatches 'cwl-') | Sort-Object -Descending\r\n$indexeslistcount = ($indexeslist | Measure-Object).Count\r\n\r\nif ($indexeslistcount -gt $daystokeep) {\r\n$indexeslist | Select-Object -Skip $daystokeep | ForEach-Object { $webclient.UploadString(\"$($esdomain)\/$($_)\",\"DELETE\",0) }\r\n}<\/pre>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>So I followed this AWS blog and this documentation to launch a tiny t2 elasticsearch cluster to visualize VPC flow logs. Those links have instructions that guide you along setting up flow logs to flow into ES in a few &hellip;<\/p>\n<p class=\"read-more\"><a href=\"https:\/\/packetlost.com\/blog\/2016\/09\/05\/delete-elasticsearch-indexes-with-powershell\/\">Read more &raquo;<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,2],"tags":[33,35,3],"class_list":["post-61","post","type-post","status-publish","format-standard","hentry","category-amazon-web-services","category-powershell","tag-elasticsearch","tag-indices","tag-powershell"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/packetlost.com\/blog\/wp-json\/wp\/v2\/posts\/61","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/packetlost.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/packetlost.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/packetlost.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/packetlost.com\/blog\/wp-json\/wp\/v2\/comments?post=61"}],"version-history":[{"count":6,"href":"https:\/\/packetlost.com\/blog\/wp-json\/wp\/v2\/posts\/61\/revisions"}],"predecessor-version":[{"id":67,"href":"https:\/\/packetlost.com\/blog\/wp-json\/wp\/v2\/posts\/61\/revisions\/67"}],"wp:attachment":[{"href":"https:\/\/packetlost.com\/blog\/wp-json\/wp\/v2\/media?parent=61"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/packetlost.com\/blog\/wp-json\/wp\/v2\/categories?post=61"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/packetlost.com\/blog\/wp-json\/wp\/v2\/tags?post=61"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}