Crontab format

format of /etc/crontab:

# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name  command to be executed

Disabling weak ciphers in Tomcat

In order to disable weak ciphers, please modify your SSL Connector, container attribute inside server.xml with the following information:


ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"

How to create zip on ubuntu?

To zip a directory try:

zip -r zipfile.zip directory

How to get Unix Timestamp in Bash?

Simply use: 

date +%s

sed command to Insert a line before the last line of the file.

sudo sed -ie '$ i tomcat7 hard nofile 32768' /etc/security/limits.conf

This command will insert the entry "tomcat7 hard nofile 32768" before the last line of the file limits.conf. 

tar command to preserve file permissions?

Creating Tar: 

tar -pcvzf xxx.tar.gz ABC

Extracting: 

tar -pxvzf xxx.tar.gz ABC


Where, 
p == preserve permissions
c == create archive
v == verbose (print names while making tar)
z == gzip
f == tar file name

How do I log out another logged in user in ubuntu?

Use command: 

sudo pkill -KILL -u username

Bash command to convert string in variable to uppercase

Suppose you have a variable MY_VAR with value: 

MY_VAR="this is my test"

To convert it to uppercase simply use the ^^. For e.g.,  

echo ${MY_VAR^^}

This will output: THIS IS MY TEST

How can i determine what a soft link points to?

Given a file of arbitrary name, here is how you can determine what it links to, when the link destination can also have arbitrary contents: 

readlink -f <linkname>

How to filter only the first Row of awk Output?

For e.g., I need the ping output of where a particular dns resolves to. The problem is I get 2 lines of same text to where this DNS is getting resolved. I need to filter this so that I get just a single line. 

`ping -t 1 abc.mydomain.com | grep "elb." | awk ' { print $2 } '`

The above command outputs 2 lines of same text.. 

mylb.elb.amazonaws.com
mylb.elb.amazonaws.com

To overcome this problem, i modified this to: 

`ping -t 1 abc.mydomain.com | grep "elb." | awk ' { print $2; exit(0); } '`

Problem with MySql Replication: Last_Errno: 1580 Last_Error: Error 'You cannot 'ALTER' a log table if logging is enabled' on query. Default database: 'mysql'.q

Still don't know the root cause.. but here is how to fix it:

STOP SLAVE;
SET GLOBAL slow_query_log = 'OFF'; SET GLOBAL sql_log_bin = 0;
START SLAVE;

ERROR 1201 (HY000): Could not initialize master info structure

Getting errors on the slave with mysql replication: 

ERROR 1201 (HY000): Could not initialize master info structure

Couple of things to try and fix this issue: 

1.  Try "reset slave" and again try the Change command which should be like: 

CHANGE MASTER TO MASTER_HOST='10.x.x.x', MASTER_USER='repl', MASTER_PASSWORD='slave_password', MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=0;
mysql> START SLAVE;

2. Check if /etc/mysql/my.cnf have the server-id properly set for both master and slave machines. For both master and slave's server-id should be different.

3. If you are setting up multiple slaves, make sure that all the slaves have different server-id's. 

Erros with mysql. stop: Rejected send message, 1 matched rules; type="method_call",

Getting this error when restarting tomcat with following command: 

sudo service mysql restart

Tried below instead and that worked: 

sudo /etc/init.d/mysql restart

elb-describe-lbs: Malformed input-Malformed service URL: Reason: Connection refused -

Reason: 

Because your machine is not allowed to make a connection/request to 
https://elasticloadbalancing.amazonaws.com

Or there is a temporary network problem. 

How to reload variables from /etc/environment without re-logging in?

source /etc/environment in bash works for program run through terminal, but running programs through shortcuts at GUI does not work. For that you will have to relaunch those programs from terminal after having done source /etc/environment

Can’t locate DBI.pm in @INC Perl Script Error

Install the DBI.pm perl module by issue following in command line: 

perl -MCPAN -e 'install DBI'

That fixed it for me. 

What is MapReduce?

Its a framework to write programs that allow to process massive amount of unstructured data in parallel across a distributed cluster of computers. It achieves this using 2 functions: 

1.  Map: This function routes chunk of processing job across various nodes in the cluster. 
2.  Reduce: This function collates the work from various nodes, and resolves the results into a single value. 

This framework is said to be fault tolerant. How it achieves it, is by listening to various nodes to reply from time to time.. If a certain node does not reply in some amount of time, the node is thought to be dead. The work that was assigned to that node, is then reassigned to a different node. This way the detection and repair of any failures on the nodes is done at the application side. 

MapReduce allows for distributed processing of the map and reduction operations. Provided each mapping operation is independent of the others, all maps can be performed in parallel – though in practice it is limited by the number of independent data sources and/or the number of CPUs near each source. MapReduce is important because it allows ordinary developers to use MapReduce library routines to create parallel programs without having to worry about programming for intra-cluster communication, task monitoring or failure handling. It is useful for tasks such as data mining, log file analysis, financial analysis and scientific simulations.

There are various implementations of MapReduce in market today. Hadoop being one of the leading MapReduce implementation in market. The Hadoop project provides end to end Big Data Services.

What is Apache Hadoop?

Its an open-source software/framework for reliable, scalable, distributed computing.

This framework basically provides for distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-avaiability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.

Hadoop's own distributed file system (HDFS) allows for rapid data transfer among nodes and allows the system to continue operating uninterrupted in case of a node failure. This approach lowers the risk of system failure, even if a significant number of nodes become inoperative.

Hadoop was inspired by Google's MapReduce which is also a programming framework where application is broken down into small parts so that it can run on different nodes individually (Map) and then the results are collected and compiled as one (Reduce). 

What’s next after Cloud Computing – Big Data?

Yes.. Big Data is no longer just a Buzz word. Its a reality now where companies are understanding the real need for it.

Gartner defines Big Data as high volume, velocity and variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making. Organizations are discovering that important predictions can be made by sorting through and analyzing Big Data. But analyzing this data is not straightforward since a lot of data is unstructured and there are computational boundaries. Here is where Cloud technology will come into play.

Unix File Permissions

The Unix files access is controlled. There are three types of access (permissions):

• read
• write
• execute

Each file belongs to a specific user and group (ownership).

Access to the files is controlled by user, group, and what is called other/everyone permission bits and is usually set using a numerical value. For example, 644 as permission bit will result in:

Owner / User      Group    Other/ Everyone
         6                   4                 4

Each number represents the access level and it can be from 0 to 7. The access level, depending on the number is as follows:

0 - no access to the file whatsoever
1 - execute permissions only
2 - write permissions only 
3 - write and execute permissions
4 - read permissions only
5 - read and execute permissions
6 - read and write permissions
7 - read, write and execute permissions (full permissions)

Thus the above 644 permissions example will look like this:

Owner / User - Read and Write 
Group - Read and Write 
Other/ Everyone - Read only

To allow a script to be executed and read by everyone but the only one who can write in it is your user, you would need to set 755 as permissions.

Resolving mysql error 111

MySql Error 111 means connection refused, which may be coming because of several reasons. Some of which are: 

1.  Your MySQL server is configured that it listes to only connections from localhost. So check if you are getting this error only from some other machine or even from localhost where your DB Server is installed. 

If its only from other machines, check your my.cnf file for below items, comment them and restart tomcat. 

skip-networking
bind-address = 127.0.0.1

2.  If you could login as root to mysql then you should add user privileges. Try command below: 

GRANT ALL PRIVILEGES ON * . * TO  'username'@'IP_ADDRESS' IDENTIFIED BY  '*44612AC693E3B8F7AEA36B50168860122FE106A8'

flush privileges;

The string "*44612AC693E3B8F7AEA36B50168860122FE106A8" is actually the hash password as generated by mysql. Use below command to generate these hashes: 

mysql> select password('test123!');

+-------------------------------------------+

| password('test123!')                      |

+-------------------------------------------+

| *44612AC693E3B8F7AEA36B50168860122FE106A8 | 

+-------------------------------------------+

1 row in set (0.02 sec)

3.  Check firewalls to make sure that they are not blocking the connection. 

cURL Error (7): couldn't connect to host

"cURL Error (7): couldn't connect to host"

This error is self explanatory by itself.. It means that the connection to the host was not possible. This can be due to: 

1.   Your server has not started correctly. 

2.   The URL of the server you have entered is not correct. 

3.   There are firewall or other network settings which are preventing to browse the URL. 

The best way to troubleshoot such problem is first to test if we can browse the URL via a browser. 

DHCP configuration on a Ubuntu machine

-:DHCP configuration:-

·         Find out if the machine is detecting the network interface with command:-

Ifconfig –a | grep eth

·         It will tell you the network interface address for example eth0 eth1……. eth 6.

·         You can control the logical name for mac address by doing changes  in the below configuration file

vi /etc/udev/rules.d/70-persistent-net.rules.

for reference:-

 

·      Configure network configuration file .i.e. interface at location  vi /etc/network/interfaces as:-

 

·  

       Sudo ifup eth0 (will use dhclient for obtaining the dynamic IP/... if it does not restart the machine).

svn: Failed to add file : object of the same name is already scheduled for addition

This error is generally seen when you have made changes to the main svn area directly. It will come up everytime you will try to do svn up. 

To resolve revert back using following commands and then checkout/update from repository. 

svn revert .
svn cleanup
svn update

Change java Keystore Password and Private Key Password

Change Java Keystore Password

keytool -storepasswd -new new_storepass -keystore keystore.jks -storepass password

Change Private Key Password 

keytool -keypasswd -alias client -keypass old_password -new new_password -keystore client.jks -storepass password

What does cfengine Bootstrapping mean?

Cfengine bootstrapping means/servers the purpose of: 

1.  Starting/configuring cf-agent so that it runs automatically in every few minutes
2.  The agent connects to the defined policy server to get the updated policy files. 

Here is how you bootstrap cf-agent. 

cf-agent --bootstrap --policy-server=<ip_of_your_policy_distribution_server>

The above command when executed will start cf-agent to run in every 5 minutes to get the policy files from the policy server that you have given above and implement those policies. 

If you are configuring a machine to be used as a policy distribution server then all you need to do is use its own IP and it will give you proper messaging like "R: This host assumes the role of policy distribution host". 

Once the bootstrapping is over, you will notice that cf-execd and cf-serverd are running. 

See for example output: 


ps axw | grep "cf-"

  951 ?        Ss     0:00 /var/cfengine/bin/cf-execd
  957 ?        Ss     0:00 /var/cfengine/bin/cf-serverd

The agent machines will copy the policy files from your policy distribution servers /var/cfengine/masterfiles/ directory to /var/cfengine/inputs directory. 

sed command to replace first occurrence of a pattern in file

sed command to replace first occurrence of a pattern in file : 

sed '0,/pattern/s/pattern/replacement/' filename




That's simple and sweet!

How to recover deleted files in Linux

A file in linux file system is just a link to an inode. When you delete the file, actually the link is deleted and not the actual inode until all references to that inode are deleted. This gives me an impression that there has to be some way by which we can recover accidentally deleted files and directories in Linux. 

Unfortunately, I could not find any good and straight forward solution for this. There are couple of tools available today like Magic Rescue, photorec. You need to see if they work for you.. I did not find them worth for the files and directories that I deleted.

A look at cfengine binaries and what they do

cf-promises

This compiles and verifies the promises that have been defined. Basically used to pre-check configurations before you can actually deploy those.  

cf-agent

This is the actual binary that maintains the system resources based on policy files.  

cf-serverd

Only the server can share files with other agent hosts and also receive requests to execute policies. It is not possible to send (push) new information to CFEngine from outside.

cf-execd

This is the scheduling daemon (which can either supplement or replace cron). It works as a wrapper, executing and collecting the output of cf-agent and E-mailing it to system account. 

cf-runagent

This is a helper program that can talk to cf-serverd and request that it execute cf-agent with its existing policy. It can thus be used to simulate a push of changes to CFEngine hosts, if their policy includes that they check for updates. 

cf-report

Generates summary and other reports in a variety of formats for export or integration with other systems. 

cf-know

This agent can generate an ISO standard Topic Map from a number of promises about system knowledge. It is used for rendering documentation as a `semantic web'.

Error code 19 in Windows Device Manager for DVD

Remove/Delete the LowFilter and/or UpperFilter (you might have only one od these) entries from this registry area:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4D36E965-E325-11CE-BFC1-08002BE10318}

Note: You will have several identical keys like this {4D36E965-E325-11CE-BFC1-08002BE10318}. The one you are looking for will have DVD/CD Rom Drives as the top entry. Reboot after deleting the LowFilter and/or UpperFilter entries.

Accessing MS SQL DATABASE FROM COMMAND PROMPT

To access MS SQL database from command prompt follow below steps in line:-

1.) Locate sqlcmd on your machine and trace down to its path from command prompt.

2.) To access named instance, use below command refer help for switches explanation.

SQLCMD -S SERVERNAME\INSTANCENAME –E

3.) Once connection is established it will prompt screen as


4.) Use below command to check out DB instances.

sp_databases
Go




on command prompt to execute any query we need to set go and then press enter.

Building and Installing Cfengine from Source

Pre-requisites: 
1. OpenSSL - Encryption System. http://www.openssl.org 
2. Tokyo Cabinet - Flat File Database. http://fallabs.com/tokyocabinet/ 
3. PCRE - Perl Compatible Regular Expression Library. http://www.pcre.org/ 

Sample commands to Install the pre-requisites under ubuntu: 

1. sudo apt-get update
2. sudo apt-get install libtokyocabinet-dev
3.  sudo apt-get install libssl-dev
4. sudo apt-get install libpcre3-dev


Downloading and Installing Cfengine
If you have not purchased ready to use cfengine binaries, you need you build them from source code. This is pretty easy once you have the required development environment setup ie., installed looks like gcc, flex, bison. 

Here are the steps you need to follow for building cfengine from source. 

1.  Download Latest Cfengine source to a directory. http://cfengine.com/source-code
2.  tar zxf cfengine-x.x.x.tar.gz
3.  ./configure
4.  make
5.  make install

The last step actually installs the built binaries onto your system. Be default all gets installed in /var/cfengine/

That's all you need to do for installing cfengine. 

Easy.. Isn't It? 

Cfengine Features - A Quick Look.

Cfengine is the only configuration tool that is based on actual Research.

Why Cfengine against other configuration tool?

• Longest Track Record. 
• Based on actual research. 
• Works equally well on 1 as well as 1000's of servers. 
• A very helpful and large community. 
• Supports largest number of platforms. 
• Idempotent ie., Multiple applications of the operation does not change result. 
• Capable of making each and every host autonomous. 
• Non-reliant on a working network to function correctly. 
• Lightweight and generic.
• Open source software (GPL or COSL)

Latest version of Cfengine as of writing this article is 3.3.5


The primary principle of Cfengine is automatic convergence from Clean or Unknown states back to Configured. It is Resilient meaning, it pulls config from a Policy server. If server is unavailable or there are network issues it reads the cached copy from /var/cfengine/inputs

It does not have to worry about how many hosts... Each host is responsible for updating itself. 

It is hardly 2 MB in size and consumes very little network bandwidth. Have been used on embedded systems too. 

How to download file using cURL?

Simple.. use the -O option which tells that the downloaded file should be named as that of the original file that are we are trying to download. 

For e.g., 

curl -O http://mydomain.com/example.zip

This will download the file example.zip into the same directory from where you are running this command and will name the downloaded file as example.zip. 

There are a lot of options which can be used with curl. For a list of them see curl --help. 

An excellent link which I found that details various curl day to day examples is http://www.thegeekstuff.com/2012/04/curl-examples/. 

How to Unzip/extract an .tar.gz file?

Use command: 

tar -xvzf community_images.tar.gz

x : Extracts the files. 
v : Prints verbose output.
z : Tells tar to decompress using gzip.
f : Tells the name of the file to be Unzip and extracted.

pure-ftpd: relocation error: pure-ftpd: symbol my_make_scrambled_password, version libmysqlclient_15 not defined in file libmysqlclient.so.15 with link time reference

There are several solutions to this.. for me following worked: 

yum install mysql-devel

Running an Infinite loop in Bash

#!/bin/bash
for (( ; ; ))
do
   echo "infinite loops [ hit CTRL+C to stop]"
done

Alert: the_dns_template_doesnt_exist

Have not been able to find any solution for this... 

How to check your CentOS version

Easiest way is to use command line.. 

cat /etc/centos-release
Or
cat /etc/redhat-release

CentOS version history follows that of Red Hat
If you want to know more details on your kernel etc.. use the below command: 
uname -a
Sample Output
Linux sample.com 2.6.18-194.26.1.el5xen #1 SMP Tue Nov 9 13:35:30 EST 2010 x86_64 x86_64 x86_64 GNU/Linux

There you could be newer kernels installed on your CentOS server but not actually running. You can list the installed kernels using yum or rpm. You can then see which kernel is set to start on boot by checking your /etc/grub.conf.

Another great way to get all this information would be to use lsb-release..: 


lsb_release -a
LSB Version: :core-3.1-amd64:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-noarch
Distributor ID: CentOS
Description: CentOS release 5.5 (Final)
Release: 5.5
Codename: Final

How to view Events of a MySql Schema

To see various events that are there in MySql Schema use the below command:

show events; 

If you want to see events on a particular schema include the From clause..

show events from MySchema; 

How to Create New User Account in Ubuntu Linux from command line?

Use useradd command create a new user or update default new user information from command line.

useradd testuser
passwd testuser

The above command will create a user named testuser and set its password to testuser. 

Some important files that define various properties etc for this user: 

• /etc/passwd - User account information.
• /etc/shadow - Secure user account information such as password.
• /etc/group - Group account information.
• /etc/default/useradd - Default values for account creation.
• /etc/skel/ - Directory containing default files.
• /etc/login.defs - Shadow password suite configuration.

Above command useradd will not create a home directory for this user. To create those try the below command that will create the user as well as home directory for it. 

sudo useradd -m -k /etc/skel -d /home/testuser testuser

This will create a user named test user with a home directory /home/testuser and will have the same files and settings as in /etc/skel.

To delete the user follow command: 

sudo userdel testuser

"Too many connections" - Error when trying to connect to Mysql

If you get "Too many connections" error when trying to connect to Mysql Server that means all available connections have been allocated and used by other clients.




Actually the number of connections is limited by max_connections variable. To whatever value this variable is set, that many connections can be opened by clients to mysql. In actual it is set to max_connections + 1. The +1 is actually reserved for the Super User so that they can login to mysql and troubleshoot problems even if the usable connections have exhausted. 




The maximum number of connections MySQL can support depends on the quality of the thread library on a given platform, the amount of RAM available, how much RAM is used for each connection, the workload from each connection, and the desired response time.