You are here

namenode ?

Dear rekans idBigData ...

Apakah spesifikasi hardware untuk namenode harus tinggi ? termasuk menggunakan memory RAM yang besar, pertanyaan ini berhubungan dengan gagalnya proses mapreduce yang coba saya jalankan menggunakan baik PIG / HIVE dengan total data sekitar 11G file txt. Namun jika saya coba menggunakan filter ( LIMIT ... ) maka proses berjalan seperti semestinya tanpa error.

Hal lain yang juga selalu gagal saya jalankan yaitu apabila menggunakan HIVE untuk filter query menggunakan opsi WHERE, apakah masih ada hubungannya juga dengan pertanyaan di atas.

Mohon pencerahannya ..., atas jawabannya saya ucapkan banyak terima kasih ...

bolon9

selamat mencoba, di tunggu kabar baiknya senang bisa membantu mas bro

baik .... matur nuwun mas Zastrow, saya coba config ulang juga hadoopnya ... mudah mudahan ada pencerahan. kita lanjut diskusinya kalau saya sudah dapet kondisi lain. matur nuwun ...

sepertinya resourcenya habis mas, jadi kalau di filter datanya yang diproses lebih kecil sehingga sukses. kalau hardware memorynya cuma 2 GB minim sekali. sisakan buat OS dan daemon hadoop yaitu 1 GB utk daemon dan OS 1 GB utk yarn meskipun minim. semoga

Sebelumnya saya ucapkan terima kasih atas responnya sdr. Zastrow. mungkin perlu di ketahui saja sebagai informasi, jadi dalam uji coba ini saya menggunakan komputer desktop branded HP dan DELL masing masing menggunakan memori RAM 2G, dan proc. core to duo, hardisk 2T, dengan konfigurasi menggunakan hadoop 2.7.2, 1 namenode dan 15 datanode, berikut dibawah config mapred-site.xml dan log error saat query :

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx2048m</value>
</property>
</configuration>

hduser@mksdellnn01:~$ hive
Logging initialized using configuration in jar:file:/opt/hive/lib/hive-common-2.1.0.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. tez, spark) or using Hive 1.X releases.
hive> SELECT * FROM datadsp.dspcollect_20161009 WHERE regional='KALIMANTAN';
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. tez, spark) or using Hive 1.X releases.
Query ID = hduser_20161011154559_2424e077-7d58-4631-91aa-5519012f1bd3
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1476169222465_0002, Tracking URL = http://mksdellnn01:8088/proxy/application_1476169222465_0002/
Kill Command = /opt/hadoop/bin/hadoop job -kill job_1476169222465_0002
Hadoop job information for Stage-1: number of mappers: 82; number of reducers: 0
2016-10-11 15:46:14,685 Stage-1 map = 0%, reduce = 0%
2016-10-11 15:46:23,142 Stage-1 map = 7%, reduce = 0%
2016-10-11 15:46:24,231 Stage-1 map = 0%, reduce = 0%
2016-10-11 15:46:29,607 Stage-1 map = 100%, reduce = 0%
Ended Job = job_1476169222465_0002 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1476169222465_0002_m_000001 (and more) from job job_1476169222465_0002
Examining task ID: task_1476169222465_0002_m_000068 (and more) from job job_1476169222465_0002
Examining task ID: task_1476169222465_0002_m_000051 (and more) from job job_1476169222465_0002
Examining task ID: task_1476169222465_0002_m_000012 (and more) from job job_1476169222465_0002
Examining task ID: task_1476169222465_0002_m_000014 (and more) from job job_1476169222465_0002
Examining task ID: task_1476169222465_0002_m_000026 (and more) from job job_1476169222465_0002
Examining task ID: task_1476169222465_0002_m_000022 (and more) from job job_1476169222465_0002
Examining task ID: task_1476169222465_0002_m_000015 (and more) from job job_1476169222465_0002
Examining task ID: task_1476169222465_0002_m_000009 (and more) from job job_1476169222465_0002
Examining task ID: task_1476169222465_0002_m_000047 (and more) from job job_1476169222465_0002
Examining task ID: task_1476169222465_0002_m_000067 (and more) from job job_1476169222465_0002
Examining task ID: task_1476169222465_0002_m_000058 (and more) from job job_1476169222465_0002
Examining task ID: task_1476169222465_0002_m_000028 (and more) from job job_1476169222465_0002
Examining task ID: task_1476169222465_0002_m_000039 (and more) from job job_1476169222465_0002
Examining task ID: task_1476169222465_0002_m_000066 (and more) from job job_1476169222465_0002
Examining task ID: task_1476169222465_0002_m_000042 (and more) from job job_1476169222465_0002
Examining task ID: task_1476169222465_0002_m_000068 (and more) from job job_1476169222465_0002
Examining task ID: task_1476169222465_0002_m_000004 (and more) from job job_1476169222465_0002
Examining task ID: task_1476169222465_0002_m_000049 (and more) from job job_1476169222465_0002
Examining task ID: task_1476169222465_0002_m_000075 (and more) from job job_1476169222465_0002
Examining task ID: task_1476169222465_0002_m_000059 (and more) from job job_1476169222465_0002
Examining task ID: task_1476169222465_0002_m_000032 (and more) from job job_1476169222465_0002
Examining task ID: task_1476169222465_0002_m_000038 (and more) from job job_1476169222465_0002

Task with the most failures(4):
-----
Task ID:
task_1476169222465_0002_m_000016

URL:
http://mksdellnn01:8088/taskdetails.jsp?jobid=job_1476169222465_0002&tipid=task_1476169222465_0002_m_000016
-----
Diagnostic Messages for this Task:
Container launch failed for container_1476169222465_0002_01_000235 : java.lang.IllegalArgumentException: java.net.UnknownHostException: mkshpdn04.telkomsel.co.id
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:377)
at org.apache.hadoop.security.SecurityUtil.setTokenService(SecurityUtil.java:356)
at org.apache.hadoop.yarn.util.ConverterUtils.convertFromYarn(ConverterUtils.java:238)
at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProtocolProxy.java:266)
at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.<init>(ContainerManagementProtocolProxy.java:244)
at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:129)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:409)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:375)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: mkshpdn04.ngoprek.co.id
... 12 more

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1: Map: 82 HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
hive>

Sebelumnya saya ucapkan terima kasih atas responnya sdr. Zastrow. mungkin perlu di ketahui saja sebagai informasi, jadi dalam uji coba ini saya menggunakan komputer desktop branded HP dan DELL masing masing menggunakan memori RAM 2G, dan proc. core to duo, hardisk 2T,

seharusnya mapreduce adalah yg paling hemat memory karena hanya dengan setting minimal ram 512 MB jobs biasanya jalan normal. bisa minta tolong di upload log job yarn nya utk melihat lebih detail beserta setting file mapred-site.xml filenya. Error jobnya seperti apa.