Can't read data correctly through beeline when data is save by HiveContext

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Can't read data correctly through beeline when data is save by HiveContext

licl
This post has NOT been accepted by the mailing list yet.
This post was updated on .
Hi,

        Here is my javacode;

      SparkConf sparkConf = Constance.getSparkConf();
        JavaSparkContext sc = new JavaSparkContext(sparkConf);
        SQLContext sql = new SQLContext(sc);
        HiveContext sqlContext = new HiveContext(sc.sc());
        List<StructField> fields = new ArrayList<StructField>();
        fields.add(DataTypes.createStructField("id", DataTypes.IntegerType,
                                true));
        List<Integer> data = Arrays.asList(1, 2, 3, 4, 5);
        JavaRDD<Integer> distData = sc.parallelize(data);
        StructType schema = DataTypes.createStructType(fields);
        JavaRDD<Row> rdd = distData.map(new Function<Integer, Row>() {
                        @Override
                        public Row call(Integer v1) throws Exception {
                                return RowFactory.create(v1);
                        }
                });
        DataFrame peopleDataFrame = sqlContext.createDataFrame(rdd, schema);
        peopleDataFrame.write().mode(SaveMode.Append).saveAsTable("shop.id");
        System.out.println("**************"+sqlContext.sql("select * from shop.id").toJavaRDD()
        .collect());


         bin/spark-submit --class com.bjdv.spark.demo.Copy_2_of_Fenlei  ~/sparkjob.jar

        I run this shell on my server to create a table named "shop.id" in Hive,the table have 5 records;
 
        And then I run "select * from shop.id" through beeline;

        It works well;
   
        But when I run the shell second or above;

        The javacode could read the data well,bug beeline still show only 5 records that insert by the first time;

        Why???

        I'm sure that beeline and shell read the same hive-site.xml ,the metastore is saved with mysql;