java - How do I set an environment variable in a YARN Spark job? -

-

Using an AccumuloInputFormat , trying to reach a job (written in Java) I am newAPIHadoopRDD . To do this, you have to ask me to locate the ZooKeeper by calling AccumuloInputFormat by calling the setZooKeeperInstance method. This method takes a client configuration object which specifies the various relevant properties.

I considered the ClientConfiguration object static loadDefault method to look for a client.conf file in different locations on this method , Which is loading its default. This is such a place which is going to see $ ACCUMULO_CONF_DIR / client.conf .

Therefore, I am trying to set the ACCUMULO_CONF_DIR environment variable in this way that there is a way that when the spark works, it will appear (for reference, i yarn -cluster trying to run in deployment mode) I have not found any way to do this yet.

So far I have tried:

  • Calling setExecutorEnv ("ACCUMULO_CONF_DIR", "/ etc
  • export spark-env.sh in ACCUMULO_CONF_DIR
  • setting Spark.executorEnv.ACCUMULO_CONF_DIR in spark-defaults.conf

    None of them have worked. When I set setZooKeeperInstance < If I print the environment before calling on / code>, ACCUMULO_CONF_DIR is not visible.

    If this is relevant, then I Here's an example of what I'm trying to do (import and exception handling left for concise and concise behavior): 

     < Code> Public Square MySpacz {Public Static Zero Main (String [] args) {SparkConf Spark Confe = New Spark Confe; sparkConf.setAppName ("MySparkJob"); SparkConf.setExecutorEnv ("ACcUMULO_CONF_DIR", "/ etc / accumulo / Conf "); JavaSparkContext sc = new JavaSparkContext (sparkConf); Job accumuloJob = Job.getInstance (sc.hadoop configuration ()); // Foreach loop to print the environment, no ACCUMULO_CONF_DIR client does not show configuration cache Configuration = Client Configuration. Load default (); AccumuloInputFormat.setZooKeeperInstance (accumuloJob, Deposit Configuration); // AccumuloInputFormat Other calls for static functions To configure it correctly, JavaPairRDD & lt; Key, value & gt; AccumuloRDD = sc.newAPIHadoopRDD (accumuloJob.getConfiguration (), AccumuloInputFormat.class, Key.class, Value.class); }}    
     
     
     So I answered the answer to the question (sorry) , Reputation seekers). The problem is that the CDH5 uses Spark 1.0.0, and that I was running a job through the yarn. Clearly, the Yarn mode ignores the executable environment and instead uses the environmental variable  SPARK_YARN_USER_ENV  to control its environment, then visible in the environment at the point of view in the source of the question  SPARK_YARN_USER_ENV  in  ACCUMULO_CONF_DIR = / etc / accumulo / conf  works, and  ACCUMULO_CONF_DIR  ensures example.  
     How the standalone mode and yarn mode work is different, which is fixed in SPARC 1.1.0.

Comments

Post a Comment

Popular posts from this blog

java - ImportError: No module named py4j.java_gateway -

-
I'm trying to call the Python using the Java program py4j . I've been installing the plug-in Eclipse and test name Piidvi project. I'm trying to execute the following part of the code found on py4j webpage: Import from py4j.java_gateway to JavaGateway, java_import gateway = JavaGateway () jvm = gateway.jvm java_import (jvm, '' Org.eclipse Kkorkrisorsej. * ") Vrkspes_rut = Jvankresourkesplginkgetvrkspas (). GetRoot (Gateway .help (workspace_root, '* Projects *') project_names = [project.getName () (for projects workpace_root.getProjects))] print (Projekt_nam) But I There is an error in import. I have checked that the P4JJ is present in the Jar Eclipse plugin directory. Can anyone help please? I had to install the py4j application
Read more

python - Receiving "KeyError" after decoding json result from url -

-
I am new to Python I am trying to parse JSON result from a URL. Basically, I was using the following: response = urllib.request.urlopen (url) json_obj = json.load (response) It should be a stroke "str 'not' bytes' in the lines of a given" JSON object, so after searching on the StackoverView Flo, I decode the response in this way: F = urllib.request.urlopen (Url) charset = f.info (). Get_param ('charset', 'utf8') data = f.read () decoded = json.loads (data.decode (charset)) If I print "decode" I is as follows: { 'link': { 'summary data': 'https: // localhost / piwebapi / streams / p0_7qHaW4UHU-RlCaz8tpasAAQAAAAU0hJTExNQU42NDIwXFNJTlVTT0lE / summary' 'value': 'https: // localhost / Piwebapi / streams / P0_7qHaW4UHU-RlCaz8tpasAAQAAAAU0hJTExNQU42NDIwXFNJTlVTT0lE / price ',' InterpolatedData ':' https: // localhost / Piwebapi / streams / P0_7qHaW4UHU-RlCaz8tpasAAQAAAAU0hJTE...
Read more

java - Session timeout does't work vaadin -

-
I am developing a vaadin web application and I have added the following snippet of code in my web.xml. & lt; Session-Config & gt; & Lt; Session timeout & gt; 30 & lt; / Session timeout & gt; & Lt; / Session-config & gt; Now I have noticed after 30 minutes that my users are capable of using appllication and I do not want this. I read something about this problem on the book Wadin but I do not believe in anything. From the Wadin book: The session expires after user deactivation In the normal servlet operation, the session time defines the allowed time of inactivity After which the server should clear the session. Inactivity is measured by the last server request. Different servicelists use different default for container timeouts, such as 30 minutes for Apache Tomcat. You can set the timeout with: In a web.xml: & lt; Session-config & gt; & Lt; Session timeout & gt; 30 & lt; / Session timeout & gt; & Lt; / S...
Read more