{"id":1292,"date":"2013-09-03T09:10:52","date_gmt":"2013-09-03T16:10:52","guid":{"rendered":"http:\/\/blog.light42.com\/wordpress\/?p=1292"},"modified":"2013-09-25T07:44:31","modified_gmt":"2013-09-25T14:44:31","slug":"ampcamp-3-hdfs","status":"publish","type":"post","link":"http:\/\/blog.light42.com\/wordpress\/?p=1292","title":{"rendered":"AmpCamp 3 – HDFS"},"content":{"rendered":"

\"Hadoop-blue-200x200\"<\/a><\/p>\n

As described in a previous post<\/a>, I used Cloudera<\/strong> .debs to install Hadoop\/HDFS on an Ubuntu 12.04 ‘Precise’ single node. Now, to put some data into the HDFS system and use it. (note- this did not work in a 32bit VM<\/em>)<\/p>\n

The Hadoop\/HDFS install consisted of two steps: obtain and install .deb cdh4-repository<\/code>, which enables a suite of other packages (perhaps auto-magically updated); then use those packages to install the features you want. <\/p>\n

apt-get update<\/code> shows the Cloudera repository in the list.
\n
\nHit http:\/\/archive.cloudera.com precise-cdh4 Release.gpg
\nHit http:\/\/archive.cloudera.com precise-cdh4 Release
\nHit http:\/\/archive.cloudera.com precise-cdh4\/contrib Sources
\nHit http:\/\/archive.cloudera.com precise-cdh4\/contrib amd64 Packages
\nIgn http:\/\/archive.cloudera.com precise-cdh4\/contrib TranslationIndex
\nIgn http:\/\/archive.cloudera.com precise-cdh4\/contrib Translation-en_US
\nIgn http:\/\/archive.cloudera.com precise-cdh4\/contrib Translation-en
\n<\/code><\/p>\n

Packages look like this:
\n\"CDH4-all-pkgs\"<\/a><\/p>\n

Some quick reading shows I have at least two choices to easily interface to HDFS. One is the hue suite<\/a>, and another is an HDFS Fuse<\/a> interface. More options are listed on this MountableHDFS page<\/a>.<\/p>\n

Cloudera supplies an HDFS Fuse mount<\/a> with their system. Instructions on how to use the FUSE extension are here<\/a>.<\/p>\n

When HDFS is running with this system (with or without FUSE) you can view a web interface at port 50070<\/code>. A port list here<\/a>.<\/p>\n

after mucking with the environment a bit
\n
\nException in thread \"main\" java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS):
\n<\/code>
\nI was getting an error when trying to start the secondarynamenode, that the namenode had an invalid address. so, investigating:
\n
\nupdate-alternatives --get-selections | grep hadoop
\nhadoop-conf auto \/etc\/hadoop\/conf.empty<\/p>\n

less \/etc\/hadoop\/conf.empty\/core-site.xml
\n<\/code><\/p>\n

core-site.xml turned out to be missing the actual namenode name,
\nwhich was not apparent in the doc I was reading.. So, edit
\n\/etc\/hadoop\/conf.empty\/core-site.xml<\/code> and add a name and value property xml tags. The system starts up after this change.<\/p>\n

\r\n<configuration>\r\n <property>\r\n  <name>fs.default.name<\/name>\r\n  <value>hdfs:\/\/127.0.1.1:50070<\/value>\r\n <\/property>\r\n<\/configuration>\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"
As described in a previous post, I used Cloudera .debs to install Hadoop\/HDFS on an Ubuntu 12.04 ‘Precise’ single node. Now, to put some data into the HDFS system and use it. (note- this did not work in a 32bit VM) The Hadoop\/HDFS install consisted of two steps: obtain and install .deb cdh4-repository, which enables […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11],"tags":[],"_links":{"self":[{"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/1292"}],"collection":[{"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1292"}],"version-history":[{"count":16,"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/1292\/revisions"}],"predecessor-version":[{"id":1373,"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/1292\/revisions\/1373"}],"wp:attachment":[{"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1292"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1292"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1292"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}