Pages

Friday 17 October 2014

Set up Chef on Ubuntu Machine

Ubuntu + Chef + Errors

On  chef server(new ubuntu machine)

cd ~
Install the chef server or install from web site
 => http://www.getchef.com/chef/install/
or
cli
=> wget https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/12.04/x86_64/chef-server_11.0.10-1.ubuntu.12.04_amd64.deb

Install the package using terminal
=> sudo dpkg -i chef-server*

Run this command to confugre chef server on the machine
=> sudo chef-server-ctl reconfigure

Open browser (write https and ip of server)
https://server_domain_or_IP

Add default username and password.


On workstation(diiferent ubuntu machine)

update the system
=> sudo apt-get update

Install Git
=> sudo apt-get install git

Install Curl
=> sudo apt-get install curl

You can download the chef client from wesite and manually install this.
http://www.getchef.com/chef/install/
otherwise run
=> curl -L https://www.opscode.com/chef/install.sh | sudo bash

Get the chef-repo from git
=> git clone https://github.com/opscode/chef-repo.git

create .chef repository
=> mkdir ~/chef-repo/.chef

Copy the keys:

Now we will put the keys into .chef folder so open in browser put chef server url
https://server_domain_or_IP

Login and Tap on the Clients > edit(chef-validator) > chek the checkbox > save
It will generate a key so just copy the private key and paste in

create a chef-validator file in .chef folder
=> vim chef-validator.pem
paste the key

There should be no extra space
Now same for the Users > admin > edit > check checkbox  for regenerate the private key > save
It will generate a key so just copy the private key and paste in

create a chef-validator file in .chef folder
=> vim admin.pem
paste the key

run knife command
=> knife configure --initial

PASTE THE PATH ASKED BY OPETIONS LIKE FOR admin key path should be /home/vagrant/chef-repo/.chef/admnin.pem
same for the all


No need to commit the .chef folder so
=> vim ~/chef-repo/.gitignore

Add .chef in the end of the file

.rake_test_cache

###
# Ignore Chef key files and secrets
###
.chef/*.pem
.chef/encrypted_data_bag_secret
.chef

run knife command to check if knife is configured or not
=> knife user list

if it gives error reconfigure the knife configuration

Client Node (another ubuntu instance) ip address(22.2.22.1)

now bootstrap to the clint using the following command
=> knife bootstrap 22.2.22.1 -x username -P password -N name_for_node --sudo

If bootstraps success than you will find a node in chef-repo/nodes

Now chek client list
=> knife client list


Errors faced by me

While creating node
Error: RuntimeError: Please set EDITOR environment variable
Sol:
export EDITOR=$(which vi)

ERROR: Failed to upload /home/vagrant/chef-repo/cookbooks/apt/recipes/default.rb
Sol:
Chef-server redirects api clients between its components (in this case erchef and bookshelf). These redirects are based on the FQDN of the chef-server. If the server doesn't have a valid FQDN at install time, this can result in redirects to https://localhost:443.

One solution is to fix the FQDN on chef-server and run chef-server-ctl reconfigure.

Another solution is to set a few options in /etc/chef-server/chef-server.rb (I had to create this file) and then run chef-server-ctl reconfigure.

server_name = "192.168.114.11"
api_fqdn server_name

nginx['url'] = "https://#{server_name}"
nginx['server_name'] = server_name
lb['fqdn'] = server_name
bookshelf['vip'] = server_name

Swap Memory in Ubuntu

Swap Memory + Ubuntu

I found these two blogs for swap memory which provide me a good understanding about swap memory and help me to increase swap memory of my Ubuntu system

https://www.digitalocean.com/community/tutorials/how-to-configure-virtual-memory-swap-file-on-a-vps

http://www.garron.me/en/bits/create-add-swap-file-ubuntu.html

Vagrant (Virtual Machine Manager on ubuntu)

Virtual Machine Manager + Vagrant + Generate Multiple VM's + Windows +
 Common Errors + VMware

Steps to starts

. Create a folder anywhere and give entry in the folder
   mkdir tarun

 . cd tarun

 . vagrant init   #it will generate a file named "vagrant" with cinfigurations.

 . cd vagrant

 . Add a box into the ".vagrant.d\boxes\" in C drive.

 . run command  :-
   vagrant box add precise64 "C:\Users\admin\.vagrant.d\boxes\precise64.box"

 . When you will enter into the vagrant folder you will find a vagrant configuration file. In vagrant configuration file create a global configuration for virtual box.
 
   config.vm.box = "precise64"

 . Create the three layers.
   vagrant up

 . Define 3 enviornments with same virtual box.

    config.vm.define "web" do |web|
      #web.vm.box = "web"
      #web.vm.hostname = "web"
      web.vm.network :private_network, ip: "192.168.33.10"
      web.vm.network :forwarded_port, guest: 80, host: 8080
    end

    config.vm.define "db" do |db|
      #db.vm.box = "db"
      #db.vm.hostname = "mysql"
      db.vm.network "private_network", ip: "192.168.10.22"
      db.vm.network :forwarded_port, guest: 3306, host: 8888
    end

    config.vm.define "master" do |master|
      #master.vm.box = "master"
      #master.vm.hostname = "master"
      master.vm.network "private_network", ip: "192.168.10.23"
      master.vm.network :forwarded_port, guest: 80, host: 8080
    end

  . Define networks to communicate.
    master.vm.network "private_network", ip: "192.168.10.23"

  . Define port  
    master.vm.network :forwarded_port, guest: 80, host: 8080

  . After changing run command
    vagrant reload   #helps to configure the changed configuration

  . After generate the three layers, check the virtual box.

  . To enter in the layers, use command
     vergant ssh web 


 ERRORS:

  Error: Failed to create the host-only adapter :-
  Sol:   Reinstall the virtualbox and run it as administration.
    Or update the virtual box

  Error: Vagrant cannot forward the specified ports on this VM, since they
      would collide with some other application that is already listening
      on these ports. The forwarded port to 8080 is already in use
      on the host machine.
  Sol: write    :-   , auto_correct: true
       else :- Change the host, if already defined.

Rails and Mysql Set up on ubuntu Server

Ruby + Rails + Mysql + RVM + Gemset + Apache

Steps to set up server

On Ubuntu Image

Reference Link

https://www.digitalocean.com/community/articles/how-to-install-ruby-on-rails-on-ubuntu-12-04-lts-precise-pangolin-with-rvm

1. Install curl first

sudo apt-get install curl

2. Install rvm

\curl -L https://get.rvm.io | bash -s stable

3. Path to load rvm (Given in comments after installation of )

source ~/.rvm/scripts/rvm
or
source ~/.profile

4. Requirements related to rvm

rvm requirements


5. Check version of rubies present on cloud

rvm list known

6. Install specific version

rvm install 1.9.3

7. Create gemset for particular rvm

rvm gemset create demo

8. Use gemset

rvm gemset use demo 

9. If you want to build a application from scratch than install gem rails.

gem install rails -v 3.2.14

10. Otherwise enter into the application and run

rvm use 1.9.3-p374@demo
bundle

--------------------------------------------------------------------------------------------------------------------------
Installing Mysql steps

Make sure your package management tools are up-to-date. Also make sure you install all the latest software available.

sudo apt-get update

sudo apt-get dist-upgrade

Install the MySQL server and client packages:

sudo apt-get install mysql-server mysql-client

You need to set a root password

sudo mysqladmin -u root -h localhost password 'mypassword'

sudo mysqladmin -u root -h myhostname password 'mypassword'

For Ruby dependencies install

sudo apt-get install libmysql-ruby

You can now access your MySQL server like this:

mysql -u root -p
--------------------------------------------------------------------------------------------------------------------------

Installing Apache has never been easier using apt-get.

First, I updated the repositories.

sudo apt-get update

Then fixed the locales.

sudo locale-gen en_US en_US.UTF-8 en_CA.UTF-8

sudo dpkg-reconfigure locales

And finally installed all the basic packages I thought I'd need.


sudo apt-get install apache2 curl git build-essential zlibc zlib1g-dev zlib1g libcurl4-openssl-dev libssl-dev libopenssl-ruby apache2-prefork-dev libapr1-dev libaprutil1-dev libreadline6 libreadline6-dev


Set Up PostgreSql On Ubuntu

sudo apt-get install postgresql postgresql-contrib

sudo apt-get install libpq-dev

sudo -u postgres createuser --superuser tarun

sudo -u postgres psql

postgres=# \password tarun

password: tarun

enter again password: tarun

\q

Client programs, by default, connect to the local host using your Ubuntu login name and expect to find a database with that name too. So to make things REALLY easy, use your new superuser privileges granted above to create a database with the same name as your login name:

 sudo -u postgres createdb tarun

Connecting to your own database to try out some SQL should now be as easy as:

 psql

Creating additional database is just as easy, so for example, after running this:

 create database mmagap;

 If you get the below error after restart the server

psql: FATAL: Peer authentication failed for user "postgres"


Switch to postgres user


$ sudo su - postgres   #it will ask for password and by default password is also postgres


After switch the user to postgres, open psql console


$ psql


so check the version of postgres if multiple versions are available


psql=# select VERSION();


PostgreSQL 9.1.13 on x86_64-unk....         # so version is 9.1


Now Open From postgres user


vim /etc/postgresql/9.1/main/pg_hba.conf   #  9.1 is version return form upper command


and replace


local   all             postgres                                peer


to


local   all             postgres                                md5


 
 





Wowza Server Setup with Ubuntu and Also Integration In Rails

Wowza + Flowplayer + Video Upload + Rails + Video Streaming + Stop video after some time + java setup on ubuntu

1. First set up java on ubuntu (should be 64 bit system) :- in browser

$ sudo sh -c 'echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" >> /etc/apt/sources.list'
$ sudo sh -c 'echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" >> /etc/apt/sources.list'
$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886
$ sudo apt-get update
$ sudo apt-get install oracle-java8-installer


2. Install wowza using (should have account on wowza) :- in browser
http://www.wowza.com/pricing/installer

Copy the secret key of wowza

3. Let it download in directory /home/garg/Downloads/ :- in terminal
cd /home/garg/Downloads/

4. Install wowza
sudo chmod +x WowzaStreamingEngine-4.0.4.deb.bin
sudo ./WowzaStreamingEngine-4.0.4.deb.bin

5. start wowza service :- in terminal
sudo service WowzaStreamingEngine start
// sudo service WowzaStreamingEngine stop

6. On you machine, check wowza is running :- in browser
http://127.0.0.1:1935/
// http://[wowza-ip-address]:1935

7. Start engine manager :- in terminal
sudo service WowzaStreamingEngineManager start

8. In browser
http://127.0.0.1:8088/enginemanager

Read and continue steps and run wowza
Login with user name and password you put in console on wowza installation

Keep flowplayer.content-3.2.9.swf' in assets

--------------------------------------------------------------------------------------------------------------------------
9. Set up amazon  S3 on wowza media server

- Click the "Server" tab at the top of the manager, and then select "Media Cache" in the left bar pane.

- Select "Sources" and "Add Media Cache Source"

- Selcect
Source Name = S3 Stream
Source Type = AmazonS3
Aws Key id = "amazon key"
AWS secret = "amazon secret"

Minimum Time To Live:
- select days to keep cache.

- Select "Applications" on top bar of the wowza server.

- Select "VOD Edge" named as "mediacache".

- select "Media Cache Sources", to your amazon "S3 Stream" => created media cache source name of amazon s3.

Add the following settings in code

<script type="text/javascript" src="http://releases.flowplayer.org/js/flowplayer-3.2.13.js"></script>

<%= form_for @taped, :url => upload_taped_video_videos_path, :html => { :multipart => true } do |form| %>
  <%= form.file_field :taped_video %>
  <%= form.submit "Upload" %>
<% end %><br/>

<%@taped_videos.each_with_index do |tape, i|%>
  <div id="videoSection_<%= i %>" style="width:300px;height:300px;"></div>
  <div class="cost"><%=tape.cost.to_f%></div>
  <div id="replaceLink_<%= i %>" style="display:none">
    <% if current_user %>
      <%= link_to "Purchase to watch", purchase_taped_video_video_path(tape.id)%>
    <% else %>
      <!-- <%#= link_to "Already Member?", new_user_session_path%> | <%#= link_to "Register", new_registration_path(resource_name) %> -->
      <%= link_to "Already Member?", purchase_taped_video_video_path(tape.id)%> | <%= link_to "Register", "/users/sign_up" %>
    <% end %>
  </div>
  <script type="text/javascript">
    $(document).ready(function(){
      $("#show-text").hide()
    });

    flowplayer("videoSection_<%= i %>", "http://releases.flowplayer.org/swf/flowplayer-3.2.18.swf",
      {
        clip: {
          url: "mp4:amazons3/vmh(bucket name on amazon)/file_path/file.mp4",
          #url: "<%=File.extname(tape.taped_video_file_name).split('.')[1]%>:amazons3/vmh<%=tape.taped_video.path%>",
          autoPlay: false,
          autoBuffering: true,
          provider: 'rtmp',
          onCuepoint: [6000, function(clip, point) {
            this.stop()
            var plugin = this.getPlugin("content");
            plugin.show(10000).setHtml($("#replaceLink_<%= i %>").html());
          }]
        },
        plugins: {
          content: {
            url: 'flowplayer.content-3.2.9.swf',
            display: 'none'
          },
          rtmp: {
          url: '<%= asset_path "flowplayer/flowplayer.rtmp-3.2.13.swf"%>',
          netConnectionUrl: 'rtmp://127.0.0.1/mediacache'
          }
        }
      }
    );

    var player = $f()

    player.onLoad(function()  {
        // this- variable points to the player's API
        // this.getScreen().animate({width:300, height:200});
    });
  </script>
  <br/><br/>
<%end%>

Tuesday 27 May 2014

Configuration Unicorn over nginx



Nginx Using Rails
Introduction
Nginx is an open source reverse proxy server for HTTP, HTTPS, SMTP, POP3, and IMAP protocols, as well as a load balancer, HTTP cache, and a web server.

How it works
Nginx works on event driven architecture. it means that notifications or signals are used to mark the initiation or completion of a process. Thus, the resources can be used by other process until a process initiation event is triggered and resource can be allocated and released dynamically. This leads to the optimized use of memory and CPU. Nginx doesn’t create a new process for a new request.
Nginx has one master process and several worker processes. The main purpose of the master process is to read and evaluate configuration, and maintain worker processes. Worker processes do actual processing of requests. nginx has focused on high performance, high concurrency and low memory usage.

Configuration
Unicorn
Write the following code on your local machine config/unicorn.rb
listen "127.0.0.1:8080"
worker_processes 4
preload_app true
user 'vagrant'
root = "/home/vagrant/app/sample_app/current"
working_directory root
pid "#{root}/tmp/pids/unicorn.pid"
stderr_path "#{root}/log/unicorn.log"
stdout_path "#{root}/log/unicorn.log"

# listen "/tmp/unicorn.sample_app.sock"
timeout 30
user: set user name of the server.
working_directory: path to the application.
listen: Url which will communicate with nginx.

Nginx Configuration:
On server machine, open `vi /etc/nginx/nginx.conf` and paste the below code.
worker_processes 1;
user vagrant;
pid /tmp/nginx.pid;
error_log /home/vagrant/app/sample_app/shared/log/nginx.error.log;

events {
  worker_connections 1024; # increase if you have lots of clients
  accept_mutex off; # "on" if nginx worker_processes > 1
}

http {
  include mime.types;
  default_type application/octet-stream;
  access_log  /home/vagrant/app/sample_app/shared/log/nginx.access.log combined;
  sendfile on;
  tcp_nopush on; # off may be better for *some* Comet/long-poll stuff
  tcp_nodelay off; # on may be better for some Comet/long-poll stuff

  upstream app_server {
    server 127.0.0.1:8080;
  }

  server {
    client_max_body_size 4G;
    server_name localhost;
    keepalive_timeout 600s;
    root /home/vagrant/app/sample_app/current;
    try_files $uri/index.html $uri.html $uri @app;

    location @app {
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Host $http_host;
      proxy_redirect off;
      proxy_pass http://app_server;
    }

    # Rails error pages
    error_page 500 502 503 504 /500.html;
    location = /500.html {
      root /home/vagrant/app/sample_app/current;
    }
  }
}
   `upstream` should have the IP or path that is given in unicorn `listen`.

Monday 26 May 2014

Unicorn Vs Phusion Passenger

Unicorn vs Phusion Passenger


Unicorn :- Unicorn is a Rack HTTP server that uses forked processes to handle multiple incoming requests concurrently. Unicorn is an HTTP server for Rack applications designed to only serve fast clients on low-latency, high-bandwidth connections and take advantage of features in Unix/Unix-like kernels. Slow clients should only be served by placing a reverse proxy capable of fully buffering both the the request and response in between Unicorn and slow clients.

Phusion Passenger :- Phusion Passenger is an application server which can directly integrate into Apache. It is designed to be easy to use, fast, stable and reliable. Phusion Passenger is a so-called polyglot application server because it supports applications written in multiple programming languages. At this time, Ruby and Python are supported.

Difference
Note: Neither Unicorn nor Phusion Passenger support Windows. Both of them require a Unix OS.
Unicorn and Phusion Passenger are both application servers that support Ruby. Although they share similar basic features, there are large differences in how they approach usage, and large differences in technical decisions.

Phusion Passenger is a polyglot, multi-application server. It supports  PythonNode.js  and  Meteor.
Unicorn is Ruby-only.
Unicorn does not support JRuby and Rubinius, while Phusion Passenger does.

Ease of use
Phusion Passenger requires less system administration knowledge, is easier to setup and requires less human management.
Unicorn acts more like a specialized component that you have to integrate in a larger system. Using Unicorn requires more system administration knowledge and skills.

 Scalability
Both Unicorn and Phusion Passenger can be scaled easily, through the use of HTTP load balancing tools and reverse proxies such as HAProxy and Nginx. Both of them support both inter-server scalability (scaling to multiple servers) as well as intra-server scalability (scaling within a server, on the process level) .

Concurrency
Both Unicorn, as well as the open source variant of Phusion Passenger, are multi-process single-threaded. The Enterprise variant can be configured to be either single-threaded or multithreaded.
Multithreading allows less memory usage and provides higher concurrency than multi-process single-threading. Multithreading is especially suitable for applications that require high I/O concurrency, e.g. applications that perform a lot of HTTP API calls or otherwise block on I/O, or applications which serve WebSockets.
Phusion Passenger Enterprise can be hybrid multi-process multi-threaded. That is, running multiple multithreaded processes. Hybrid mode allows Ruby and Python, which despite having a Global Interpreter Lock, to fully utilize all CPU cores.

Performance
Performance characteristics depends on the workload, so this should be explained in two parts.

CPU-bound, fast requests
For CPU-bound, fast requests that don't involve blocking I/O, Unicorn and Phusion Passenger (both the open source and Enterprise variant) perform similarly in production, but differently in microbenchmarks.
In microbenchmarks Unicorn is faster because in Phusion Passenger, all data goes through an additional process, the PassengerHelperAgent, which sanitizes request headers, coordinates process spawning, collects statistics, etc. The overhead is not big, approximately a little more than an extra read()/write() call to the kernel. The difference is almost unnoticable when benchmarking over the network. But in local machine microbenchmarks where you are benchmarking how quickly the app can do nothing, Phusion Passenger will appear to be twice as slow because of the extra proxy layer. On the other hand, that extra proxy layer is what allows us to provide accurate statistics and to implement robust process coordination, so it's not there for nothing. We do have some ideas on how to address even this in the future.

I/O-bound, slow requests
For slow requests that are bound by blocking I/O, Unicorn and the open source version of Phusion Passenger perform similarly, thanks to their identical I/O concurrency models.
Phusion Passenger Enterprise achieves higher concurrency than both Unicorn and the open source version of Phusion Passenger, thanks to support for multithreading.

Security
Phusion Passenger has a builtin security sandboxing feature.
Sandboxing allows one to run different applications in different sandboxes, so that if one application has a security vulnerability, its damage has a lower chance of spreading to other applications on the same system. This implemented by using operating system user account privilege separation features.
Phusion Passenger has I/O safety features built in, and does not require extra integration with a buffering reverse proxy.
Phusion Passenger in its Standalone mode can also be directly exposed to the Internet, so that it can be used with minimal setup time.
Unicorn does not provide any builtin sandbox features. It is possible to run Unicorn in a sandbox, but that is something the system administrator has to manually setup.
Unicorn also lacks certain I/O safety features.
Unicorn cannot be safely exposed to "slow clients" or internet, and must therefore be installed behind a buffering reverse proxy, e.g. Nginx. This is why Unicorn is only ever used together with Nginx, and cannot be directly exposed to the Internet by itself

Multitenancy
Phusion Passenger is designed for multitenant (multi-app) deployment by default. This shows in both usage and the management tools. With a single Phusion Passenger install, you can easily deploy multiple apps. With a single set of management tools, you can manage all your apps.
With Unicorn, you have to manage each app individually.

Management tools
Unicorn provides some management tools which allow you to stop or restart Unicorn and to query its status. The tooling is minimalistic, and provides limited information, though the information that is available is very useful.
Phusion Passenger provides management tools that provide much more insight. Phusion Passenger allows you to stop, restart and to query its status through command line tools like passenger-statuspassenger-configpassenger-memory-stats. These tools are regular command line tools.
These tools display everything Unicorn's tools display, plus the exact requests that are currently running, how long they've been running, the application's CPU and memory usage, etc.

Debugging and inspection
The open source version provides tools for debugging stuck applications by displaying all threads' backtraces, while Unicorn does not appear to have such functionality. Phusion Passenger Enterprise provides a live IRB console that you can attach to any live, running process for inspection. It also provides ruby-debug integration that you can use even in multi-process mode.