Saturday 3 May 2014

Hadoop Installation Using Cloudera Package - Pseudo Distributed Mode (Single Node)

[Previous Post]

Hadoop can be installed using cloudera also with less steps in an easy way .The difference is Cloudera packed Apache Hadoop and some ecosystem projects into one package.And they have set all the configuration to localhost and we need not want to set the configuration files.

Installation using Cloudera Package.

Prerequistie

1.Java


Installation Steps

Step 1: Set Java home in /etc/profile

unmesha@unmesha-hadoop-virtual-machine:~$ java -version
java version "1.7.0_55"
Java(TM) SE Runtime Environment (build 1.7.0_55-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.55-b03, mixed mode)

Check your current location of java 

unmesha@unmesha-hadoop-virtual-machine:~$ sudo update-alternatives --config java
[sudo] password for unmesha: 
There is only one alternative in link group java: /usr/lib/jvm/java-7-oracle/jre/bin/java
Nothing to configure.

Set JAVA_HOME

export JAVA_HOME=/usr/lib/jvm/java-7-oracle
unmesha@unmesha-hadoop-virtual-machine:~$ source ~/.bashrc 

Step 2: Download the package for your system under "On Ubuntu and other Debian systems, do the following:" heading from here.

Step 3: Extract the package

unmesha@unmesha-hadoop-virtual-machine:~$sudo dpkg -i cdh4-repository_1.0_all.deb


Step 4: Install Hadoop

unmesha@unmesha-hadoop-virtual-machine:~$sudo apt-get update 
unmesha@unmesha-hadoop-virtual-machine:~$sudo apt-get install hadoop-0.20-conf-pseudo


Step 5: Format Namenode

unmesha@unmesha-hadoop-virtual-machine:~$sudo -u hdfs hdfs namenode -format


Step 6: Start HDFS

unmesha@unmesha-hadoop-virtual-machine:~$for x in `cd /etc/init.d ; ls hadoop-hdfs-*` ; do sudo service $x start ; done


Step 7: Create the /tmp Directory

unmesha@unmesha-hadoop-virtual-machine:~$sudo -u hdfs hadoop fs -mkdir /tmp 
unmesha@unmesha-hadoop-virtual-machine:~$sudo -u hdfs hadoop fs -chmod -R 1777 /tmp


Step 8: Create the MapReduce system directories

unmesha@unmesha-hadoop-virtual-machine:~$sudo -u hdfs hadoop fs -mkdir -p /var/lib/hadoop-hdfs/cache/mapred/mapred/staging

unmesha@unmesha-hadoop-virtual-machine:~$sudo -u hdfs hadoop fs -chmod 1777 /var/lib/hadoop-hdfs/cache/mapred/mapred/staging

unmesha@unmesha-hadoop-virtual-machine:~$sudo -u hdfs hadoop fs -chown -R mapred /var/lib/hadoop-hdfs/cache/mapred


Step 9: Verify the HDFS File Structure

unmesha@unmesha-hadoop-virtual-machine:~$sudo -u hdfs hadoop fs -ls -R /


Step 10: Start MapReduce

unmesha@unmesha-hadoop-virtual-machine:~$for x in `cd /etc/init.d ; ls hadoop-0.20-mapreduce-*` ; do sudo service $x start ; done


Step 11: Set up user directory

unmesha@unmesha-hadoop-virtual-machine:~$sudo -u hdfs hadoop fs -mkdir /user/<your username>unmesha@unmesha-hadoop-virtual-machine:~$sudo -u hdfs hadoop fs -chown <user> /user/<your username> 
unmesha@unmesha-hadoop-virtual-machine:~$sudo -u hdfs hadoop fs -mkdir /user/unmesha/new


Step 12: Run grep example, you can also try out wordcount example

unmesha@unmesha-hadoop-virtual-machine:~$/usr/bin/hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-examples.jar grep input output 'dfs[a-z.]+'

Step 13: You can also stop the services

unmesha@unmesha-hadoop-virtual-machine:~$for x in `cd /etc/init.d ; ls hadoop-hdfs-*` ; do sudo service $x stop ; done

unmesha@unmesha-hadoop-virtual-machine:~$for x in `cd /etc/init.d ; ls hadoop-0.20-mapreduce-*` ; do sudo service $x stop ; done


Happy Hadooping ...


14 comments:

  1. Fortunately, Apache Hadoop is a tailor-made solution that delivers on both counts, by turning big data insights into actionable business enhancements for long-term success. To know more, visit Hadoop Training Bangalore

    ReplyDelete
  2. So is there any true magical matka of numbers that can help you to turn your fortunes around? Are there some unique combinations of numbers that you can learn? Are there some numerology rules that you should know about that can help you win big each time with guaranteed success?

    It may sound a bit odd to you but there is no such magical Satta matka of numbers or specific numbers that you can bet on for guaranteed success. Even those websites that promise you to make the winner in each round of the Indian matka are completely vague and absurd, to say the least.

    Instead, here we will try to elaborate on some more logical tips that can help you win the matka. Some of these tips may sound very simple and absurd as you know them. But the ground rules in the matka of Indian matka are the same and using some basic principles you can have more chances of success than failure.

    ReplyDelete
  3. Forex Broker Canada Best Forex Brokers For Your Forex Trading And Fx. The Profanity Review And Forex Today Reviews The Forex Brokers And Provides The Best Fx Ratings And Reviews.

    ReplyDelete
  4. Trading Directory is an experienced and reliable Forex Trading company serves as an informational guide for anyone who wants to find the right online broker. On Trading Directory you can compare the best online brokers by sorting the search results by price, regulation or any other selected criteria. The search results showcase detailed broker profiles that include information such as fees and minimum deposit requirements.

    ReplyDelete
  5. Market Live Rates Is A One-stop Shop For Traders On The Lookout For The Best Brokers And Strategies For Online Trading. We Also Offer Content Marketing Services, That Includes Guest Post And Blog, Link Building Services And Content Writing Services. Our Services Are Extremely Cost Effective And We Provide You With The Best Of Results To Help Make Your Business Flourish.

    ReplyDelete
  6. Track your stocks in all major markets instantly with our unique live stock overview. See Bfarf stock live prices changes as they occur and view change details, including volume and share changes.

    ReplyDelete
  7. Mmatf Stock Real-Time Overview Of A Stock, Including Recent And Historical Price Charts, News, Events, Analyst Rating Changes And Other Key Stock Information.

    ReplyDelete
  8. At Ati Testing Login , you will find everything that you need related to your account and your choices as as a customer. Complete information about, News & Events, Credit Cards, Bill Pay, Rewards, Customer Service and much more.

    ReplyDelete
  9. Morning after breakfast, drive to Badrinath. Check in to the hotel arrival in Badrinath. Pilgrims after having a bath in the Taptkund have the Darshan of Badrivishal & Aarti in evening. Brahamakapal is significant for Pinddan Shraddh of ancestors (Pitrus). There are other interesting sightseeing spot like Mana, Vyas Gufa, Maatamoorti, Charanpaduka, Bhimkund and the "Mukh" of the Saraswati River. Just within the three kms of Badrinathjee. Badrinath tour package by helicopter.

    ReplyDelete
  10. Morning after breakfast, drive to Badrinath. Check in to the hotel arrival in Badrinath. Pilgrims after having a bath in the Taptkund have the Darshan of Badrivishal & Aarti in evening. Brahamakapal is significant for Pinddan Shraddh of ancestors (Pitrus). There are other interesting sightseeing spot like Mana, Vyas Gufa, Maatamoorti, Charanpaduka, Bhimkund and the "Mukh" of the Saraswati River. Just within the three kms of Badrinathjee. Badrinath tour package by helicopter.

    ReplyDelete
  11. The Cheapest White Label Forex Broker is a sum of money which forex companies offer their partners for the use of their brand to sell the company's instruments on the partner's platform.

    ReplyDelete
  12. This is very good content you share on this blog. it's very informative and provide me future related information. thnaks For Sharing

    ReplyDelete
  13. Considering the Chardham Yatra by Helicopter from Hyderabad and the potential time-saving benefits, understanding the pricing is on my priority list. I believe that a well-structured pricing model will enhance the overall appeal of this unique pilgrimage experience.
    chardham yatra by helicopter

    ReplyDelete