Static hosting with AWS S3 super quick micro howto :)

  • Create an s3 bucket using your fully qualified domain name (FQDN) as the bucket name
  • Upload your content to the s3 bucket.
  • If you have s3cmd configured you can use this command to upload in the bucket otherwise use some other S3 clients
s3cmd put /yourpath/yourdir s3://yourbucketname.yourdomain.com/ --recursive
  • Give Read permissions for every file you want to be public
s3cmd setacl --acl-public --recursive s3://yourbucketname.yourdomain.com/

or to lock down setting it private (in case of need)

s3cmd setacl --acl-private --recursive s3://yourbucketname.yourdomain.com/
  • Click your bucket, go on properties and select “Static Website Hosting”, then “enable website hosting”, complete the fields for the index.html and 404.html files (must have these files in your website tree if you want for example 404 to work properly)
  • Take a note about you url Endpoint, you’ll need this later
  • Go on AWS Route53
  • Create a record for yoursite.yourdomain.com
  • select “Alias” = yes
  • your S3 static site will appear in the list of possible selections, choose the one with the bucket name above

Wait a couple of minutes (sometimes one minute is enough!) and …

You’re done!

Enjoy you new S3 static website :)

Cheers

Fabio

Ruby loops in Chef recipes

Multiple commands in “list” style:

%w{ ubuntu debian redhat centos }.each do |os|
  supports os
end

[ "foo", "bar", "baz", "uploads" ].each do |dir|
  directory "/var/www/#{dir}"  do
    owner "www-data"
    group "www-data"
    mode "1755"
    action :create
  end
end

Multiple commands with “dictionary style” referencing:

{
  "comment1" => "command1",
  "comment2" => "command2",
  "comment3" => "command3"
}.each do |comment, cmd|
  execute "#{comment}" do
    command "#{cmd}"
  end
end

Multiple commands with multidimensional list of dictionaries:

[
  {:name => "live", :url => "gituser@myurl.com:/myrepo", :branch => "master"},
  {:name => "staging", :url => "gituser@myurl.com:/myrepo", :branch => "staging"},
  {:name => "mobile", :url => "gituser@myurl.com:/myrepo/mobile", :branch => "master"}
].each do |repo|
  git "git clone/fetch for #{repo[:name]}" do
    repository "#{repo[:url]}"
    depth 10
    revision "#{repo[:branch]}"
    destination "/var/www/#{repo[:name]}/"
    user "www-data"
    action :sync
    not_if "[ -d /vagrant/#{repo[:name]}/ ]"
  end
end

Quick Markdown to Html in Bash

Get Markdown perl script (this is last version at the moment of writing)

cd /usr/local/src/
wget http://daringfireball.net/projects/downloads/Markdown_1.0.1.zip
unzip Markdown_1.0.1.zip

Go back to your working dir

cd -
mkdir html_output

For a single file:

perl /usr/local/src/Markdown_1.0.1/Markdown.pl --html4tags myfilename.md > html_output/myfilename.html

For multiple files:

for i in ./*.md; do perl /usr/local/src/Markdown_1.0.1/Markdown.pl --html4tags $i > html_output/${i%.*}.html; done

Markdown Cheat Sheet here:

https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet

Run chef-solo in 5 minutes : )

Get Chef and install it:

curl -sSL https://www.opscode.com/chef/install.sh | sudo bash

Create ~/solo.rb

cat <<EOF > ~/solo.rb
root = File.absolute_path(File.dirname(__FILE__))

file_cache_path root
cookbook_path root + '/chef-cookbooks'
EOF

Create ~/solo.json

cat <<EOF > ~/solo.json
{
    "run_list": [ "recipe[your-recipe-here::default]" ]
}
EOF

Checkout your recipe(s) in ~/chef-cookbooks

mkdir ~/chef-cookbooks && cd ~/chef-cookbooks && git clone user@yourserver:yourrepo/yourrecipe

Run chef-solo:

chef-solo -c solo.rb -j solo.json

Git: add empty dirs on repo

Git doesn’t add empty dirs to the repo by default, so for example when creating a new cookbook with:

knife cookbook create mycookbookname

it’s nice to have all the skeleton updated in the repo as well

The quickest and neat way I found so far is to populate all the empty dirs with a .gitignore file (but ignoring the .git directories in the process :) )
This can be done easily executing this command on the root of your repo:

find mycookbookname -name .git -prune -o -type d -empty -exec touch {}/.gitignore \;

This will produce a situation like this: (in the example below mycookbookname is android-phonegap)

fabio@zen:~/vagrant/chef-cookbooks/android-phonegap$ tree -a
.
├── attributes
│   └── .gitignore
├── CHANGELOG.md
├── definitions
│   └── .gitignore
├── files
│   └── default
│       └── .gitignore
├── libraries
│   └── .gitignore
├── metadata.rb
├── providers
│   └── .gitignore
├── README.md
├── recipes
│   ├── default.android-sdk.rb
│   ├── default.android-tools-adb.rb
│   ├── default.cordova.rb
│   ├── default.jdk.rb
│   ├── default.npm.rb
│   ├── default.rb
├── resources
│   └── .gitignore
└── templates
    └── default
        └── .gitignore

More info on this specific subject here:
http://stackoverflow.com/questions/115983/how-do-i-add-an-empty-directory-to-a-git-repository

(Kudos to Steffen :) )

Have a great day!

Fabio

Openvpn Debian to Debian in 10 minutes

We have 2 hosts that we would like to connect together with a software VPN.

The ‘server’ will be the one of the two on which is easier to have port 1194 udp published on the Internet.

On server and client hosts:

apt-get install openvpn
cp -av /usr/share/doc/openvpn/examples/easy-rsa/2.0 /etc/openvpn/easy-rsa-2.0
cd /etc/openvpn/easy-rsa-2.0/

On Server:

############## optional #############
show the content of the ‘vars’ files without comments:

/etc/openvpn/easy-rsa-2.0# egrep -v '^$|^#' vars

edit your details in vars file if you want to change defaults
############### end optional part ####

run the following commands:

cd /etc/openvpn/easy-rsa-2.0/
. ./vars 
./clean-all
./build-ca

answer the questions i.e.:
Country Name (2 letter code) [US]:UK
State or Province Name (full name) [CA]:UK
Locality Name (eg, city) [SanFrancisco]:London
Organization Name (eg, company) [Fort-Funston]:My Company
Organizational Unit Name (eg, section) []:DevOps
Common Name (eg, your name or your server’s hostname) [Fort-Funston CA]: myhostname
Name []: OpenVPN-CA
Email Address [me@myhost.mydomain]:myemail

run:

./build-key-server server

fill in same as above, the only difference is
Name []:server
challenge password (if needed or enter),
then y,y

(still on server host)

same procedure for client’s key:

./build-key myclientname

then generate Diffie Hellman parameter

./build-dh

Now we will find our newly-generated keys and certificates in the keys subdirectory.

Copy client key files (they are in the ‘key’ directory) on the ‘client’ host, with the client name.* you’ve chosen

Configuration

on server:

/usr/share/doc/openvpn/examples/sample-config-files# cp -av server.conf.gz /etc/openvpn/
cd /etc/openvpn
gunzip server.conf.gz

to look at the options in the server conf file (egrep will strip notes and comments):

egrep -v '^$|^#|^;' server.conf

then update the config file with the correct keys path:

sed -i /ca/s@ca.crt@easy-rsa-2.0/keys/ca.crt@ server.conf
sed -i /cert/s@server.crt@easy-rsa-2.0/keys/server.crt@ server.conf
sed -i /key/s@server.key@easy-rsa-2.0/keys/server.key@ server.conf
sed -i /dh/s@dh1024.pem@easy-rsa-2.0/keys/dh1024.pem@ server.conf

check the config is updated:

egrep -v '^$|^#|^;' server.conf

edit the server IP and netmask

On the Client:
In the following sed commands, ‘@’ is used as a field separator because we are using ‘/’s in the paths

/usr/share/doc/openvpn/examples/sample-config-files# cp -av client.conf /etc/openvpn/
cd /etc/openvpn/
sed -i /ca/s@ca.crt@easy-rsa-2.0/keys/ca.crt@ client.conf
sed -i /cert/s@client.crt@easy-rsa-2.0/keys/client.crt@ client.conf
sed -i /key/s@client.key@easy-rsa-2.0/keys/client.key@ client.conf

edit ‘remote’ on client.conf to reflect your server hostname or IP address

then add to client.conf:

cert yourpath/client1.crt
key yourpath/client1.key
#auth-user-pass       

Uncomment auth-user-pass to enable two factors auth (needs backend setup though)

Open firewall for UDP 1194 on Server with something like:

iptables -A INPUT  -s your_client_ip/32  -p udp --dport 1194  -m comment --comment "VPC OpenVPN" -j ACCEPT

Authorize traffic from your VPN clients towards the server (the following opens everything)

 
iptables -A INPUT -s 10.8.0.0/24  -j ACCEPT

Start openvpn on the server from the command line, to check things:

cd /etc/openvpn
openvpn server.conf

(Ctrl+c to stop)

Start openvpn on the client from the command line, to check things:

cd /etc/openvpn
openvpn client.conf

(Ctrl+c to stop)

Try to ping the hosts private ip’s and if all is ok, Ctrl+c to stop the daemons running in foreground, then start the service openvpn on both hosts:

service openvpn start

Munin not showing graphs

I’ll refer to hosts in this way (according to the official Munin documentation):

Master Server – The central Munin system where you connect to see your graphs

Node(s) – The hosts to be monitored (running munin-node, and obviously munin-node can be run on the graph server too to monitor itself locally)

    1. Open firewall on port 4949 target node with something like:
      iptables -A INPUT  -s <graph server IP>/32  -p tcp --dport 4949  -m comment --comment "Munin master server access"  -j ACCEPT
    2. Authorize the Master server IP on the Node(s), escaping the dots (assuming for instance your master ip is 12.34.56.78):
      echo 'allow ^12\.34\.56\.78$' >> /etc/munin/munin-node.conf

      OR use cidr_allow without escapes:

      echo 'cidr_allow ^12.34.56.78$' >> /etc/munin/munin-node.conf
    3. Test the connection with telnet
      telnet <Node ip> 4949

      Type:

       help

      then try for example:

      list

      and:

      fetch cpu
    4. Be sure the hostname on the Node (as given by the output of the ‘hostname’ command) is EQUAL to the hostname in  /etc/munin/munin.conf in the configuration of the Master
    5. Restart the node daemon on node(s) after making change (i.e. the hostname)
service munin-node restart
    1. If you don’t want to get annoyed waiting the cron 5 minutes for every change you make trigger it manually on the Master:
su - munin --shell=/bin/bash
/usr/share/munin/munin-update

As far as I understand, the above script has to run twice before actually producing graphs but I’m not sure about this last statement :)

htmls are in /var/cache/munin/www
rrds are in /var/lib/munin
logs in /var/log/munin/
(Debian Squeeze Defaults)