java - How do I set an environment variable in a YARN Spark job? -

-

Using an AccumuloInputFormat , trying to reach a job (written in Java) I am newAPIHadoopRDD . To do this, you have to ask me to locate the ZooKeeper by calling AccumuloInputFormat by calling the setZooKeeperInstance method. This method takes a client configuration object which specifies the various relevant properties.

I considered the ClientConfiguration object static loadDefault method to look for a client.conf file in different locations on this method , Which is loading its default. This is such a place which is going to see $ ACCUMULO_CONF_DIR / client.conf .

Therefore, I am trying to set the ACCUMULO_CONF_DIR environment variable in this way that there is a way that when the spark works, it will appear (for reference, i yarn -cluster trying to run in deployment mode) I have not found any way to do this yet.

So far I have tried:

  • Calling setExecutorEnv ("ACCUMULO_CONF_DIR", "/ etc
  • export spark-env.sh in ACCUMULO_CONF_DIR
  • setting Spark.executorEnv.ACCUMULO_CONF_DIR in spark-defaults.conf

    None of them have worked. When I set setZooKeeperInstance < If I print the environment before calling on / code>, ACCUMULO_CONF_DIR is not visible.

    If this is relevant, then I Here's an example of what I'm trying to do (import and exception handling left for concise and concise behavior): 

     < Code> Public Square MySpacz {Public Static Zero Main (String [] args) {SparkConf Spark Confe = New Spark Confe; sparkConf.setAppName ("MySparkJob"); SparkConf.setExecutorEnv ("ACcUMULO_CONF_DIR", "/ etc / accumulo / Conf "); JavaSparkContext sc = new JavaSparkContext (sparkConf); Job accumuloJob = Job.getInstance (sc.hadoop configuration ()); // Foreach loop to print the environment, no ACCUMULO_CONF_DIR client does not show configuration cache Configuration = Client Configuration. Load default (); AccumuloInputFormat.setZooKeeperInstance (accumuloJob, Deposit Configuration); // AccumuloInputFormat Other calls for static functions To configure it correctly, JavaPairRDD & lt; Key, value & gt; AccumuloRDD = sc.newAPIHadoopRDD (accumuloJob.getConfiguration (), AccumuloInputFormat.class, Key.class, Value.class); }}    
     
     
     So I answered the answer to the question (sorry) , Reputation seekers). The problem is that the CDH5 uses Spark 1.0.0, and that I was running a job through the yarn. Clearly, the Yarn mode ignores the executable environment and instead uses the environmental variable  SPARK_YARN_USER_ENV  to control its environment, then visible in the environment at the point of view in the source of the question  SPARK_YARN_USER_ENV  in  ACCUMULO_CONF_DIR = / etc / accumulo / conf  works, and  ACCUMULO_CONF_DIR  ensures example.  
     How the standalone mode and yarn mode work is different, which is fixed in SPARC 1.1.0.

Comments

Post a Comment

Popular posts from this blog

java - ImportError: No module named py4j.java_gateway -

-
I'm trying to call the Python using the Java program py4j . I've been installing the plug-in Eclipse and test name Piidvi project. I'm trying to execute the following part of the code found on py4j webpage: Import from py4j.java_gateway to JavaGateway, java_import gateway = JavaGateway () jvm = gateway.jvm java_import (jvm, '' Org.eclipse Kkorkrisorsej. * ") Vrkspes_rut = Jvankresourkesplginkgetvrkspas (). GetRoot (Gateway .help (workspace_root, '* Projects *') project_names = [project.getName () (for projects workpace_root.getProjects))] print (Projekt_nam) But I There is an error in import. I have checked that the P4JJ is present in the Jar Eclipse plugin directory. Can anyone help please? I had to install the py4j application
Read more

python - Receiving "KeyError" after decoding json result from url -

-
I am new to Python I am trying to parse JSON result from a URL. Basically, I was using the following: response = urllib.request.urlopen (url) json_obj = json.load (response) It should be a stroke "str 'not' bytes' in the lines of a given" JSON object, so after searching on the StackoverView Flo, I decode the response in this way: F = urllib.request.urlopen (Url) charset = f.info (). Get_param ('charset', 'utf8') data = f.read () decoded = json.loads (data.decode (charset)) If I print "decode" I is as follows: { 'link': { 'summary data': 'https: // localhost / piwebapi / streams / p0_7qHaW4UHU-RlCaz8tpasAAQAAAAU0hJTExNQU42NDIwXFNJTlVTT0lE / summary' 'value': 'https: // localhost / Piwebapi / streams / P0_7qHaW4UHU-RlCaz8tpasAAQAAAAU0hJTExNQU42NDIwXFNJTlVTT0lE / price ',' InterpolatedData ':' https: // localhost / Piwebapi / streams / P0_7qHaW4UHU-RlCaz8tpasAAQAAAAU0hJTE...
Read more

c++ - Qt::make_shared for creating QSharedPtr as std::make_shared for creating std::shared_ptr -

-
As stated in the Barjini Stroustrup's "A Tour of C ++", and a known C ++ 14 practice As such, someone should avoid the naked code in the new and delete . Provides std :: make_shared and std :: make_unique to create smart pointers to store allocated items in the standard library. However, it is not possible to use these routines for non-standard smart pointers, such as QT's own memory management model (with parent), but also smart pointer class like Provides QSharedPointer and QPointer (although later it is not actually a brochure indicator). My question is: Is not it convenient to create Qt analog of std :: make_shared ? To like it, QSharedPtr : Namespace Qt {template & lt; Class T, squares ... Args & gt; QSharedPointer & LT; T & gt; Make_shared (erg & amp; ... args) {return QSharedPointer & lt; T & gt; (New t (std :: forward & lt; ergos & gt; (args) ...)); }} or to create this, QPointer : named place ...
Read more