linux guru

Monday, July 6, 2015

sqlr-listener error:

ERROR:

sqlr-listener error:
Unable to create a shared memory segment. This is usally because an
sqlr-listener is already running for the rovi instance.

If it is not running, something may have crashed and left an old segment
lying around. Use the ipcs command to inspect existing shared memory
segments and the ipcrm command to remove the shared memory segment with
id -1.

Error was: File exists

sqlr-connection error:
The pid file /usr/local/var/sqlrelay/tmp/pids/sqlr-listener-rovi was not found.
This usually means that the sqlr-listener
is not running.
The sqlr-listener must be running for the sqlr-connection to start.

FIX:

Step 1: get shared memory info:
root@devtest:~# ipcs

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x0101520c 385941504 ubuntu 640 679968 7
0x01015379 385974273 nobody 640 679968 3

------ Semaphore Arrays --------
key semid owner perms nsems
0x0101520c 9437184 ubuntu 600 13
0x01015379 9469953 nobody 600 13

------ Message Queues --------
key msqid owner perms used-bytes messages

Step 2: remove shared memory
ipcrm -m 385941504

ipcrm -m 385974273

Step 3:
start sqlr:
sqlr-start -id abc
sqlr-start -id efg

Friday, July 3, 2015

install ssdeep in ubuntu

sudo apt-get install build-essential libffi-dev python python-dev python-pip automake autoconf libtool

sudo BUILD_LIB=1 pip install ssdeep

root@SBGF:~/Downloads# ssdeep VirusShare_94e75b0bc1398e516e2f4d525cd9e2d1
ssdeep,1.1--blocksize:hash:hash,filename
393216:wfk5JDNLkgFEXwj38789F+GzKIVSVer6hiM4hjaf8IZa6:9r2nwjZ9FzKIkRYWPw6,"/home/chang/Downloads/VirusShare_94e75b0bc1398e516e2f4d525cd9e2d1"
root@SBGF:~/Downloads# ssdeep VirusShare_a985e7236a60d3b6027ee33124cfbd2c
ssdeep,1.1--blocksize:hash:hash,filename
49152:zUZWIDo67rsgVd7XWsPru4kMjc94tj8GskIt7qZi9qOu7aa:OV/sgVdbWsrkMw94tj8bkqqOu7,"/home/chang/Downloads/VirusShare_a985e7236a60d3b6027ee33124cfbd2c"

chang@SBGF:~/Downloads/9233160510$ ssdeep VirusShare_94e75b0bc1398e516e2f4d525cd9e2d1

ssdeep,1.1--blocksize:hash:hash,filename

393216:wfk5JDNLkgFEXwj38789F+GzKIVSVer6hiM4hjaf8IZa6:9r2nwjZ9FzKIkRYWPw6,"/home/chang/Downloads/9233160510/VirusShare_94e75b0bc1398e516e2f4d525cd9e2d1"

chang@SBGF:~/Downloads/9233160510$ ssdeep VirusShare_a985e7236a60d3b6027ee33124cfbd2c

ssdeep,1.1--blocksize:hash:hash,filename

49152:zUZWIDo67rsgVd7XWsPru4kMjc94tj8GskIt7qZi9qOu7aa:OV/sgVdbWsrkMw94tj8bkqqOu7,"/home/chang/Downloads/9233160510/VirusShare_a985e7236a60d3b6027ee33124cfbd2c"

chang@SBGF:~/Downloads/9233160510$ python

Python 2.7.11+ (default, Apr 17 2016, 14:00:29)

[GCC 5.3.1 20160413] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> iport ssdeep

File "<stdin>", line 1

iport ssdeep

SyntaxError: invalid syntax

>>> import ssdeep

>>> ssdeep.compare('393216:wfk5JDNLkgFEXwj38789F+GzKIVSVer6hiM4hjaf8IZa6:9r2nwjZ9FzKIkRYWPw6', '49152:zUZWIDo67rsgVd7XWsPru4kMjc94tj8GskIt7qZi9qOu7aa:OV/sgVdbWsrkMw94tj8bkqqOu7')

Thursday, April 9, 2015

1: root@es-master1z:~# curl -XGET 'http://localhost:9200/gold*/_mapping?pretty'

curl -XPUT 'http://localhost:9200/soci*/tweets/_mapping' -d '{"_ttl": {"enabled": true, "default": 5000}}'

2: curl -XPUT 'http://localhost:9200/smarttvsysinfo/'

3:

curl -XPUT 'http://localhost:9200/_template/template_1' -d '
{
"template" : "gold*",
"settings" : {
"analysis" : {
"filter" : {
"tweet_filter" : {
"type" : "word_delimiter",
"split_on_case_change":"false",
"split_on_numerics":"false",
"type_table": ["# => ALPHA", "@ => ALPHA", "_ => ALPHA"]
}
},
"analyzer" : {
"tweet_analyzer" : {
"type" : "custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase", "tweet_filter"]
}
}
}
},
"mappings" : {
"_default_" : {
"_ttl" : { "enabled" : true, "default": 180002 } ,
"properties" : {
"text" : {
"type" : "string",
"analyzer" : "tweet_analyzer"
},
"retweeted_status" : {
"properties" : {
"text" : {
"type" : "string",
"analyzer" : "tweet_analyzer"
}
}
},
"user" : {
"properties" : {
"screen_name" : {
"type" : "string",
"analyzer" : "tweet_analyzer"
}
}
},
"in_reply_to_screen_name" : {
"type" : "string",
"analyzer" : "tweet_analyzer"
}
}
}

}
}
'

Friday, March 27, 2015

elasticsearch create index and add documents

1, Create Index
curl -XPUT 'http://localhost:9200/twitter/'

2, add docs:

curl -XPUT 'http://localhost:9200/twitter/tweet/22' -d '{

"user" : "kimchy22",

"post_date" : "2015-03-27T12:12:12",

"message" : "trying out Elasticsearch"

curl -XPUT 'http://localhost:9200/twitter/tweet/21' -d '{

"user" : "kimchy21",

"post_date" : "2015-03-27T11:12:12",

"message" : "trying out Elasticsearch"

Thursday, March 26, 2015

curl search elasticsearch

curl -XPOST 'http://10.0.1.152:9200/logstash-*/_search?pretty' -d '
{
"query": {
"filtered": {
"query": {
"query_string": {
"query": "*listing*"
}
}
}
},
"fields": [
"code",
"time_taken",
"uri_path"
], "from": 0,
"size": 1,
"sort": {
"_score": {
"order": "asc"
}
},
"explain": true
}
'

Monday, March 16, 2015

ubuntu 14 64 bits install ELK - March 2015.

1, install jdk
apt-get update

apt-get upgrade

apt-get install openjdk-7-jdk

2, install elasticsearch-1.4.4

wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.4.deb

dpkg -i elasticsearch-1.4.4.deb

3, install logstash_1.4.2

dpkg -i logstash_1.4.2-1-2c0f5a1_all.deb

Create conf file:

vim /etc/logstash/conf.d/test.conf

input {
file {
type => "tomcat_access"
path => [ "/home/ubuntu/elk/samplelogs/*.log" ]
start_position => "beginning"
}
}
filter {
if [type] == "tomcat_access" {
grok{
patterns_dir => "./patterns"
match => [ "message", '%{IPORHOST:client} %{DATA} %{DATA:user} \[%{DATA:timestamp} %{ISO8601_TIMEZONE:timezone}\] \"%{WORD:method} %{URIPATH:uri_path}(%{URIPARAM:params}|) %{DATA:protocol}\" %{NUMBER:code} (%{NUMBER:bytes}|%{DATA}) %{NUMBER:time_taken}' ]
overwrite => [ "message" ]
}
date {
locale => "en"
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss" ]
timezone => "UTC"
target => "logTimestamp"
}
mutate {
convert => [ "code", "integer" ]
convert => [ "time_taken", "integer" ]
}
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
host => "localhost"
protocol => "http"
port => "9200"
node_name => "elk-test"
}
}

4, install kibana-4.0.1
wget https://download.elasticsearch.org/kibana/kibana/kibana-4.0.1-linux-x64.tar.gz

tar zxvf kibana-4.0.1-linux-x64.tar.gz

mv kibana-4.0.1-linux-x64 /opt/

5, Add kibana into startup service

https://github.com/akabdog/scripts/blob/master/kibana4_init

6, run services:

/etc/init.d/elasticsearch start

/etc/init.d/logstash start

/etc/init.d/kibana4 start

7, access kibana

http://x.x.x.x:5601/

If you cannot open this page, try to open the port 5601 from your firewall.

Friday, March 6, 2015

delete elasticsearch logstash index automatically

1, Wrote a shell script to delete two days old index.

dave@cm-log-manager:~/tools$ cat delete_ES_index.sh

#!/bin/bash

yesterday=`/bin/date --date="-1 days" +%Y.%m.%d`
echo $yesterday

twodayago=`/bin/date --date="-2 days" +%Y.%m.%d`
echo $twodayago
curl -XDELETE "http://localhost:9200/logstash-$twodayago"

threedayago=`/bin/date --date="-3 days" +%Y.%m.%d`
echo $threedayago
curl -XDELETE "http://localhost:9200/logstash-$threedayago"

fourdayago=`/bin/date --date="-4 days" +%Y.%m.%d`
echo $fourdayago
curl -XDELETE "http://localhost:9200/logstash-$fourdayago"

fivedayago=`/bin/date --date="-5 days" +%Y.%m.%d`
echo $fivedayago
curl -XDELETE "http://localhost:9200/logstash-$fivedayago"

2, Add this script to cronjob

1 23 * * * /home/dave/tools/delete_ES_index.sh

3, now only two days index left