tag:blogger.com,1999:blog-2893711224871666839.post4758632319861979787..comments2024-03-27T22:31:33.557-07:00Comments on Here is Something !: How To Set Counters In Hadoop MapReduceUnmesha Sreevenihttp://www.blogger.com/profile/08020142046300098477noreply@blogger.comBlogger6125tag:blogger.com,1999:blog-2893711224871666839.post-6841358872310376802022-01-17T22:46:44.780-08:002022-01-17T22:46:44.780-08:00The Mmatf Stock Overview offers real time stock ...The <a href="https://servlogin.com/tag/mmatf-stock/" rel="nofollow"> Mmatf Stock </a> Overview offers real time stock price updates. All you need to do is open the application and you can view Streaming stock prices of your favorite stocks.<br />Peter Schiffhttps://www.blogger.com/profile/07836606862131770975noreply@blogger.comtag:blogger.com,1999:blog-2893711224871666839.post-27005590805060451392016-01-30T20:43:43.783-08:002016-01-30T20:43:43.783-08:00For latest and updated Cloudera certification dump...For latest and updated Cloudera certification dumps in PDF format contact us at completeexamcollection@gmail.com. <br />Refer our blog for more details http://completeexamcollection.blogspot.in/2015/04/cloudera-hadoop-certification-dumps.htmlCompleteExamCollectionhttps://www.blogger.com/profile/03710906599208048343noreply@blogger.comtag:blogger.com,1999:blog-2893711224871666839.post-30952968474452712112015-12-30T08:08:34.020-08:002015-12-30T08:08:34.020-08:00But the prblm is everytime the whole individual xm...But the prblm is everytime the whole individual xml file is getting appended. Instead I need only the nodes to be appended<br /><br />Problematic output:<br /><?xml version="1.0" encoding="UTF-8" standalone="no"?><br /><TransactionLog><br /> <Attributes><br /> <TS>AccumulationRule</TS><br /> <TSD>113|3600024151|3|30|Watermelon|200|20151112|113|3600024151|23|100|Jujubi|201|20151113|113|3600024152|2|40|Blackberry|202|20151114|</TSD><br /> </Attributes><br /></TransactionLog><br /><br /><?xml version="1.0" encoding="UTF-8" standalone="no"?><br /><TransactionLog><br /> <Attributes><br /> <TS>ApplyMathRule</TS><br /> <TSD>113|3600024151|3|30|Watermelon|200|20151112|113|3600024151|23|100|Jujubi|201|20151113|113|3600024152|2|40|Blackberry|202|20151114|</TSD><br /> </Attributes><br /></TransactionLog><br /><br /><br /><br />Desired output is :<br /><?xml version="1.0" encoding="UTF-8" standalone="no"?><br /><TransactionLog><br /> <Attributes><br /> <TS>AccumulationRule</TS><br /> <TSD>113|3600024151|3|30|Watermelon|200|20151112|113|3600024151|23|100|Jujubi|201|20151113|113|3600024152|2|40|Blackberry|202|20151114|</TSD><br /> </Attributes><br /> Attributes><br /> <TS>ApplyMathRule</TS><br /> <TSD>113|3600024151|3|30|Watermelon|200|20151112|113|3600024151|23|100|Jujubi|201|20151113|113|3600024152|2|40|Blackberry|202|20151114|</TSD><br /> </Attributes><br /></TransactionLog><br />Abhinayhttps://www.blogger.com/profile/05536702145348764344noreply@blogger.comtag:blogger.com,1999:blog-2893711224871666839.post-88097407232269093332015-12-30T08:00:13.898-08:002015-12-30T08:00:13.898-08:00This comment has been removed by the author.Abhinayhttps://www.blogger.com/profile/05536702145348764344noreply@blogger.comtag:blogger.com,1999:blog-2893711224871666839.post-27441733812295173752015-12-30T07:58:53.236-08:002015-12-30T07:58:53.236-08:00This comment has been removed by the author.Abhinayhttps://www.blogger.com/profile/05536702145348764344noreply@blogger.comtag:blogger.com,1999:blog-2893711224871666839.post-41640261363221376062015-12-30T07:58:18.630-08:002015-12-30T07:58:18.630-08:00Hi Unmesha
Abhinay again here.
we count the no.o...Hi Unmesha<br />Abhinay again here. <br /><br />we count the no.of bad records received by our map() and write it into file.<br /><br />We do this in 10 mr jobs.<br /><br />so every jobs writes one bad records file into hdfs and also every job has it own counter.<br /><br />finally all these counters and bad records of 10 jobs has to be written into a XML file.<br /><br />For that I have created a static final class and call it in every MR job (total 10 jobs) like <br />LoggingCounter.LogMessage(arg1,arg2)<br /><br />At the end of each job i call the function in that class, so this function is called 10 times<br /><br />Below is the code:<br /><br />public final class LoggingCounter {<br /><br /> <br /> public static int LogMessage (String Record, String Component ) throws IOException, ParserConfigurationException, TransformerException<br /> { <br /> Configuration conf = new Configuration();<br /> FileSystem fs = FileSystem.get(conf);<br /> <br /> Path inPath = new Path("hdfs://nameservice1/user/abhime01/haadoop/Rules/AccumulationRule/op/BadMapper-m-00000");<br /> Path outPath = new Path("hdfs://nameservice1/user/abhime01/logging.xml");<br /> <br /> if (!fs.exists(inPath))<br /> {<br /> System.err.println("Input Path " + inPath.toString() + " does not exist.");<br /> return 1;<br /> }<br /> <br /> DocumentBuilderFactory documentBuilderFactory =DocumentBuilderFactory.newInstance();<br /> DocumentBuilder documentBuilder =documentBuilderFactory.newDocumentBuilder();<br /> Document document; document = documentBuilder.newDocument();<br /> <br /> <br /> <br /> FSDataOutputStream fos;<br /> if (!fs.exists(outPath))<br /> { <br /> fos = fs.create(outPath);<br /> <br /> }<br /> else<br /> fos= fs.append(outPath);<br /> <br /> final String root = "TransactionLog";<br /> final String attribute = "Attributes";<br /> final String elementTS ="TS"; <br /> final String elementTSD ="TSD";<br /> <br /> <br /> <br /> Element rootElement = document.createElement(root); // <br /> document.appendChild(rootElement);<br /> <br /> Element subrootElement = document.createElement(attribute); // <br /> rootElement.appendChild(subrootElement);<br /> <br /> Element ts = document.createElement(elementTS); // <br /> ts.appendChild(document.createTextNode(Component));<br /> subrootElement.appendChild(ts);<br /> <br /> Element tsd = document.createElement(elementTSD); // <br /> BufferedReader br=new BufferedReader(new InputStreamReader(fs.open(inPath)));<br /> try<br /> { <br /> String writeBadRcrd=null;<br /> String badRcrdInputline = br.readLine();<br /> while (badRcrdInputline != null)<br /> {<br /> writeBadRcrd = badRcrdInputline.replaceAll(";","|");<br /> tsd.appendChild(document.createTextNode(writeBadRcrd));<br /> badRcrdInputline = br.readLine(); //Read the next line to avoid infinite loop<br /> <br /> }<br /> <br /> <br /> }<br /> catch(Exception e)<br /> {<br /> }<br /> finally<br /> {<br /> br.close();<br /> }<br /> subrootElement.appendChild(tsd);<br /> <br /> TransformerFactory transformerFactory = TransformerFactory.newInstance();<br /> Transformer transformer = transformerFactory.newTransformer();<br /> DOMSource source = new DOMSource(document);<br /> StreamResult result = new StreamResult(new StringWriter()); //Read the generated XML and write into HDFS<br /> transformer.setOutputProperty(OutputKeys.INDENT, "yes");<br /> transformer.setOutputProperty("{http://xml.apache.org/xslt}indent-amount", "5");<br /> transformer.transform(source, result);<br /> <br /> try<br /> { <br /> String xmlString = result.getWriter().toString();<br /> fos.writeBytes(xmlString+"\n");<br /> }<br /> catch(Exception e)<br /> {<br /> }<br /> finally<br /> {<br /> fos.close();<br /> }<br /> return 0;<br /> }<br />}<br />Abhinayhttps://www.blogger.com/profile/05536702145348764344noreply@blogger.com