localhost:9870 or 50070 is not running, however, localhost:8088 is working fine after installing Hadoop on windows - hadoop3

I am a beginner, and I have installed Hadoop on Windows 10 with help of various blogs and site, which are given below:
https://github.com/MuhammadBilalYar/Hadoop-On-Window/wiki/Step-by-step-Hadoop-2.8.0-installation-on-Window-10
https://www.joe0.com/2017/02/02/how-to-install-a-hadoop-single-node-cluster-on-windows-10/
localhost:8088 is working fine, however, 50070 or 9870 is not working at all.
Moreover, I have also created a new rule for 50070 and 9870, under the firewall protocol.
-> netsh advfirewall firewall add rule name = "Hadoop" dir=in action=allow prsomeoneCP localport=50070.
I have tried this for 6th time the whole process to make it work. Can some one please let me know what am I doing wrong.

Related

Using port number in Windows host file

After installing TeamViewer, I have changed the wampserver port to 8080, so the address is http://localhost:8080.
For the host file located at C:\WINDOWS\system32\drivers\etc\, I have also made the change as below
BEFORE
127.0.0.1 www.example.com
AFTER
127.0.0.1:8080 www.example.com
When I access www.example.com, it haven't direct to my wampserver, how can I fix it?
The hosts file is for host name resolution only (on Windows as well as on Unix-like systems). You cannot put port numbers in there, and there is no way to do what you want with generic OS-level configuration - the browser is what selects the port to choose.
So use bookmarks or something like that.
(Some firewall/routing software might allow outbound port redirection, but that doesn't really sound like an appealing option for this.)
What you want can be achieved by modifying the hosts file through Fiddler 2 application.
Follow these steps:
Install Fiddler2
Navigate to Fiddler2 menu:- Tools > HOSTS.. (Click to select)
Add a line like this:-
localhost:8080 www.mydomainname.com
Save the file & then checkout www.mydomainname.com in browser.
I managed to achieve this by using Windows included Networking tool netsh.
As Mat points out : The hosts file is for host name resolution only, so a combination of the two did the trick for me.
Example
Overview
example.app:80
| <--Link by Hosts File
+--> 127.65.43.21:80
| <--Link by netsh Utility
+--> localhost:8081
Actions
Started my server on localhost:8081
Added my "local DNS" in the hosts file as a new line
127.65.43.21 example.app
Any free address in the network 127.0.0.0/8 (127.x.x.x) can be used.
Note: I am assuming 127.65.43.21:80 is not occupied by another service.
You can check with netstat -a -n -p TCP | grep "LISTENING"
added the following network configuration with netsh command utility
netsh interface portproxy add v4tov4 listenport=80 listenaddress=127.65.43.21 connectport=8081 connectaddress=127.0.0.1
I can now access the server at http://example.app
Notes:
- These commands/file modifications need to be executed with Admin rights
- netsh portproxy needs ipv6 libraries even only to use v4tov4, typically they will also be included by default, otherwise install them using the following command: netsh interface ipv6 install
You can see the entry you have added with the command:
netsh interface portproxy show v4tov4
You can remove the entry with the following command:
netsh interface portproxy delete v4tov4 listenport=80 listenaddress=127.65.43.21
Links to Resources:
Using Netsh
Netsh commands for Interface IP
Netsh commands for Interface Portproxy
Windows Port Forwarding Example
Fiddler2 -> Rules -> Custom Rules
then find function OnBeforeRequest on put in the next script at the end:
if (oSession.HostnameIs("mysite.com")){
oSession.host="localhost:39901";
}
The simplest way is using Ergo as your reverse proxy:
https://github.com/cristianoliveira/ergo
You set your services and its IP:PORT and ergo routes it for you :).
You can achieve the same using nginx or apache but you will need to configure them.
-You can use any free address in the network 127.0.0.0/8 , in my case needed this for python flask and this is what I have done :
add this line in the hosts file (you can find it is windows under : C:\Windows\System32\drivers\etc ) :
127.0.0.5 flask.dev
Make sure the port is the default port "80" in my case this is what in the python flask: app.run("127.0.0.5","80")
now run your code and browse flask.dev
If what is happening is that you have another server running on localhost and you want to give this new server a different local hostname like
http://teamviewer/
I think that what you are actually looking for is Virtual Hosts functionality. I use Apache so I do not know how other web daemons support this. Maybe it is called Alias. Here is the Apache documentation:
Apache Virtual Hosts examples
Using netsh with connectaddress=127.0.0.1 did not work for me.
Despite looking everywhere on the internet I could not find the solution which solved this for me, which was to use connectaddress=127.x.x.x (i.e. any 127. ipv4 address, just not 127.0.0.1) as this appears to link back to localhost just the same but without the restriction, so that the loopback works in netsh.
You need NGNIX or Apache HTTP server as a proxy server for forwarding http requests to appropriate application -> which listens particular port (or do it with CNAME which provides Hosting company). It is most powerful solution and this is just a really easy way to keep adding new subdomains, or to add new domains automatically when DNS records are pointed at the server.
Apache era call it Virtual host ->
httpd.apache.org/docs/trunk/vhosts/examples.html
NGINX -> Server Block
https://www.nginx.com/resources/wiki/start/topics/examples/server_blocks/

How to see what is reserving ephemeral port ranges on Windows?

I have a Windows application that needs to use ports 50005 and 50006 but it is being blocked.
I see the following when I run netsh int ip show excludedportrange protocol=tcp:
Protocol tcp Port Exclusion Ranges
Start Port End Port
---------- --------
5357 5357
49709 49808
49809 49908
49909 50008
50009 50108
50109 50208
50280 50379
* - Administered port exclusions.
So something on my machine is reserving ports 49909 to 50008, which is presumably what is causing my application to fail. I've tried deleting this excludedportrange with the following command:
netsh int ip delete excludedportrange protocol=tcp numberofports=100 startport=49909
But I see an error Access is denied., which makes me think that whatever is reserving this ports is actively running, but I have no idea what that could be.
What's also weird is that after running that command, even though I saw an error, if I reboot the excludedportrange will be different.
As a sanity check I've also run resmon.exe and confirmed that there is nothing running on ports 50005 and 50006.
How can I tell what is adding the excludedportrange?
EDIT: I've narrowed this down to Hyper-V. If I disable Hyper-V then those ports are not excluded.

Hadoop/Yarn port configuration with firewall enabled

I have a small Hadoop/Yarn cluster that is running on a system with a firewall that must be enabled. We are trying to submit Spark jobs that fail because of the port allocations.
I've configured the firewall for all the standard Hadoop/Yarn/Spark ports that need to be opened, as well as set what I thought was all the configurations to restrict port ranges. But the application manager still creates containers on random ports that get blocked.
The one setting I thought would do the trick was yarn.app.mapreduce.am.job.client.port-range set in mapred-site.xml, but is doesn't seem to be respected or making difference.
Any thoughts/help would be greatly appreciated. Banged my head on the wall way too long on this one.
Edit Forgot versions - Hadoop/Yarn 2.8.0, Spark 2.1.0, CentOS7

elasticsearch on Ec2 cannot hit public IP(timeout)

I have elasticsearch running on EC2,
I can hit form local IP address(ex. curl -XGET localhost:9200)
I cannot hit from public IP address, whether on the same machine, or from our network, it always times out,
IPtables are allowing
port is open(to itself as well as private network)
Elasticsearch http.cors is enabled and allows "*"
aside from Iptables, amazon security config, elasticsearch config could there be anything I am overlooking? (we can access 443 and get kibana up, it just times out on the elasticsearch ajax call or if I try to access 9200 directly)
been working on this for over a day so I humbly come to you all!
thank you
I had exactly the same issue.
I managed to solve it as follows:
Do what TJ said in his comment, + restart the instance. I wasn't sure if this was/is necessary, but I did it for good measure.
I made sure that the following is set in the elasticsearch.yml file:
a. http.enabled: true
b. http.cors.enabled: true
c. http.cors.allow-origin: "*"
Restarted elasticsearch (service elasticsearch restart)
Then when I tried to access elasticsearch from the public IP it worked - http://[PUBLIC IP OF INSTANCE]:9200
Hope this helps.
I just spent lots of time trying to get this working and just succeeded.
Setup: Elasticsearch 6.2.4, running on a Windows Server 2012, EC2 instance.
I also installed the discovery-ec2 plugin, not sure now if it is required, my assumption is, yes it is required although some of the settings it allows were not necessary to get it working.
Config (.yml). I tried tons of different .yml config settings which in the end did not help, in the end I think the main setting is:
network.host: 0.0.0.0
I tried setting the network.host to ec2:privateIpv4 and ec2:publicIpv4 (plugin settings) but they didn't help.
I had added the required Custom TCP Rules (allowing 9200 and 9300...not sure if 9300 is needed).
Either it failed to start (usually with a binding to 9300 error) or started but was not publicly accessible.
The Fix. What got it working in the end is you must also open the port in windows firewall. As soon as I added the inbound rule, boom it connected :)
I then stripped out all the extra configs I had been trying, restarted Elasticsearch... and it still worked!

Apache Cassandra: Unable to gossip with any seeds

I have built Cassandra server 2.0.3, then run it. It is starting and then stopped with messages:
X:\MyProjects\cassandra\apache-cassandra-2.0.3-src\bin>cassandra.bat >log.txt
java.lang.RuntimeException: Unable to gossip with any seeds
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1160)
at org.apache.cassandra.service.StorageService.checkForEndpointCollision
(StorageService.java:416)
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageServ
ice.java:608)
at org.apache.cassandra.service.StorageService.initServer(StorageService
.java:576)
at org.apache.cassandra.service.StorageService.initServer(StorageService
.java:475)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.ja
va:346)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon
.java:461)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.jav
a:504)
What I can change to run it?
I had a similar problem with my cassandra v2.0.4 cluster running a single node.
Check your cassandra.yaml and make sure that your "listen_address" and "seeds" values match, with the exception that the seeds value requires quotes around it.
You might get this problem if your private IP address is different than the public one (like on AWS). For example, the host thinks it's "172.31.0.2" when it's visible as "55.70.33.10".
The solution to this problem is:
listen_address: 172.31.0.2
broadcast_address: 55.70.33.10
in cassandra.yaml
Make sure your cluster_name entry match on all the nodes in the cluster
(you may need to delete your storage if you changed the cluster name)
Verify that all nodes can ping to each other
broadcast_rpc_address and listen_address should be set to local IP
(not localhost or 127.0.0.1)
seeds should point to the IP address of the seed(s)
If you are on AWS and use the Ec2MultiRegionSnitch you will need to set the seeds to the public IP addresses rather than the private IPs.
For a quick single node setup on RHEL, I did the following:
Get info about your network interface setup:
# /sbin/ifconfig -a
It will list the interfaces and the ip addresses they are attached to.
Usually it will show an "Ethernet" interface and a "Local Loopback".
Get the associated ip addresses.
Then edit conf/cassandra.yaml:
rpc_address: [Local Loopback address]
broadcast_rpc_address: [Ethernet address]
listen_address: [Local Loopback address]
broadcast_address: [Ethernet address]
listen_on_broadcast_address: true
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "[Ethernet address]"
Then also, open the correct ports on Linux firewall, being 9042, 7000 and 7001. More info about opening ports on Linux here:
http://ask.xmodulo.com/open-port-firewall-centos-rhel.html
I had the same problem on Ubuntu 16.04. I'm not sure which of these changes made it work, where XXX.XXX.XXX.XXX is your public facing IP address, below are selections from cassandra.yaml
seed_provider:
# Addresses of hosts that are deemed contact points.
# Cassandra nodes use this list of hosts to find each other and learn
# the topology of the ring. You must change this if you are running
# multiple nodes!
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
# seeds is actually a comma-delimited list of addresses.
# Ex: "<ip1>,<ip2>,<ip3>"
- seeds: "XXX.XXX.XXX.XXX"
listen_address: XXX.XXX.XXX.XXX
broadcast_address: XXX.XXX.XXX.XXX
broadcast_rpc_address: XXX.XXX.XXX.XXX
listen_on_broadcast_address: true
start_rpc: true
rpc_address: XXX.XXX.XXX.XXX
I also needed to restart my Virtual Machine for some reason. ¯_(ツ)_/¯
in cassandra.yaml, I update the seed from domain name to IP address. and it works.
Happened to me because in my configuration the "intial_token" settings was specified (I think because I just copied to configuration file over from another cluster member). After clearing the data directory, commenting out the setting and restarting the node, it worked fine for me.
I experienced this error today...
I could not find any reason for the error other than timing issues.
I restarted many times and after a while it stick. It looks like they expect a bi-directional communication on the gossip channel and if it does not happen quickly enough (which looks like a very small amount of time to me) then they drop the line and generate that error.
In my case I just upgraded my software and restarted the computer. So it was clearly not a connection issue between the computers (I have firewalls and SSL, to complicate matters) and the node was connected before... So the one entry I found in that regard from datastax did not apply...
https://support.datastax.com/hc/en-us/articles/209691483-Bootstap-fails-with-Unable-to-gossip-with-any-seeds-yet-new-node-can-connect-to-seed-nodes
I got the same error. There can be more than one solution. Hope my mistake is what you have done.
I had my localhost IP pointing to some domain name (and I did that in order that my Spring boot application's server context is some domain name like www.example.com:8080 instead of localhost:8080, and I had the following entry in my hosts file on Windows system).
127.0.0.1 www.example.com
While my cassandra batch file was looking for localhost which it didn't find. So, I made another entry for localhost too in my hosts file as:
127.0.0.1 localhost
127.0.0.1 www.example.com
After adding it, I opened new command prompt, ran cassandra batch from the cassandra bin directory and it then worked.
Disable the firewall and SELINUX and try again
In our case ssl was enabled, and cassandra.yaml configuration looks fine as per above comments. Then we enabled ssl debugging by by adding below jvm paramter in cassandra-env.sh -Djavax.net.debug=ssl:handshake
After starting the node again we noticed below in cassandra log file
MessagingService-Outgoing-geo2_host/xx.xx.xx.xx, Exception while
waiting for close javax.net.ssl.SSLHandshakeException: Received fatal
alert: certificate_unknown
After further investigating the ssl debug logs we got to know that the certificate was not valid. After fixing this ssl issue node was able to join the cluster.
Thanks to elvingt
His answer just remind me , I need to verify that all node needs to be able to talk to each other.
https://support.datastax.com/hc/en-us/articles/209691483-Bootstap-fails-with-Unable-to-gossip-with-any-seeds-yet-new-node-can-connect-to-seed-nodes
Gossip communications must be bi-directional.
To verify use this commnd, and you need test from BOTH SIDE
nc -vz {your_node_ip} 7000
Then I recollect that I turned on my ubuntu firewall last night. I open it by
sudo ufw allow 7000/tcp
And it is working now
I had same issue, I checked port, used tcpdump, netcat to test connections and finally it comes to expired SSL certificates on internode_encryption. I modified internode_encryption to make it 'none', restarted all nodes and it worked.
Before all neighbor nodes were down. And node repair command was failing with:
"Did not get positive replies from all endpoints"
P.S Dont leave internode_encryption as none for a long time, just regenerate certs and enable it back.

Resources