amazon ec2 - Which core-site.xml do I add my AWS access keys to? -

-

I want to run spark code on EC2 against the data stored in my S3 bucket. And according to both, I need to add my AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to the core-site.xml file. However, when I open my master EC2 node, I have several core-sites. XML files appear. 

  $ find -name core-site.xml ./mapreduce/conf/core-site.xml./persistent-hdfs/share/hadoop/templates/conf/core-site.xml / Persistent-hdfs / src / packages / templates / conf / core-site. Xml ./persistent-hdfs/src/contrib/test/core-site.xml./persistent-hdfs/src/test/core-site.xml./persistent-hdfs/src/c++/libhdfs/ test/conf / core -site.xml ./persistent-hdfs/conf/core-site.xml./ephemeral-hdfs/share/hadoop/templates/conf/core-site.xml./hehemeral-hdfs/src/packages/templates/config/ Core-site.xml ./ephemeral-hdfs/src/contrib/test/core-site.xml./ephemeral-hdfs/src/test/core-site.xml./ephemeral-hdfs/src/ C ++ / libhdfs / Tests / conf / core-site.xml ./ephemeral-hdfs/conf/core-site.xml./spark-ec2/templates/root/mapreduce/conf/core-site.xml./spark-ec2/templates/ Root / persistent-hdfs / conf / core-site.xml ./spark-ec2/templates/root/ephemeral-hdfs/conf/core-site.xml./spark-ec2/templates/root/spark/conf / core- Site.xml ./spark/conf/core-site.xml   
 After some experiment, I decided that I could use a s3n ur from Spark Evl  s3n: like //mcneill-scratch/GR.txt  if I Matret / Conf / core-site.xml and spark / conf / core-site.xml I introduce your credentials in both.  
 It does not seem to be wrong for me, and I can not find anything in the documentation which says that you have to add your credentials to several files.  
 Multiple sites have to modify the correct way to set S3 credentials through core-site. ? Is there such a document anywhere that it tells?   
 
 
  ./spark / conf / core-site Xml  should be the correct location

Comments

Post a Comment

Popular posts from this blog

java - ImportError: No module named py4j.java_gateway -

-
I'm trying to call the Python using the Java program py4j . I've been installing the plug-in Eclipse and test name Piidvi project. I'm trying to execute the following part of the code found on py4j webpage: Import from py4j.java_gateway to JavaGateway, java_import gateway = JavaGateway () jvm = gateway.jvm java_import (jvm, '' Org.eclipse Kkorkrisorsej. * ") Vrkspes_rut = Jvankresourkesplginkgetvrkspas (). GetRoot (Gateway .help (workspace_root, '* Projects *') project_names = [project.getName () (for projects workpace_root.getProjects))] print (Projekt_nam) But I There is an error in import. I have checked that the P4JJ is present in the Jar Eclipse plugin directory. Can anyone help please? I had to install the py4j application
Read more

python - Receiving "KeyError" after decoding json result from url -

-
I am new to Python I am trying to parse JSON result from a URL. Basically, I was using the following: response = urllib.request.urlopen (url) json_obj = json.load (response) It should be a stroke "str 'not' bytes' in the lines of a given" JSON object, so after searching on the StackoverView Flo, I decode the response in this way: F = urllib.request.urlopen (Url) charset = f.info (). Get_param ('charset', 'utf8') data = f.read () decoded = json.loads (data.decode (charset)) If I print "decode" I is as follows: { 'link': { 'summary data': 'https: // localhost / piwebapi / streams / p0_7qHaW4UHU-RlCaz8tpasAAQAAAAU0hJTExNQU42NDIwXFNJTlVTT0lE / summary' 'value': 'https: // localhost / Piwebapi / streams / P0_7qHaW4UHU-RlCaz8tpasAAQAAAAU0hJTExNQU42NDIwXFNJTlVTT0lE / price ',' InterpolatedData ':' https: // localhost / Piwebapi / streams / P0_7qHaW4UHU-RlCaz8tpasAAQAAAAU0hJTE...
Read more

.net - Creating a new Queue Manager and Queue in Websphere MQ (using C#) -

-
"itemprop =" text "> I am writing applications that use WebSphere MQ to send messages. My unittests (for flow), I want to verify that I have put the correct message on the response queue. I am trying to figure out how to do this. My main obstacle is that I think it might be scary to clear the queue before running in my queue because the same queue can be used by other applications, I thought a decent solution would be a new line The manager will remove the queue for my solidarity and after using it. So my question is: Is it possible to use C # by using queue manager and queue? After the For future reference and future people, who want to make queues. I thought how to make and remove IBM MQ QE (not quenhers) with PCM messaging. It is not very simple, but it can be done. We have implemented it in a library and it is being used to create and delete commands before and after integration tests. The most important part of the code in this library is shown in the...
Read more