1. 24 Jun, 2020 1 commit
    • Jeff Zhang's avatar
      [ZEPPELIN-4893]. Upgrade to spark 3.0.0 · 82e4057d
      Jeff Zhang authored
      ### What is this PR for?
      
      Simple PR to upgrade to official released spark 3.0.0.
      
      ### What type of PR is it?
      [ Improvement ]
      
      ### Todos
      * [ ] - Task
      
      ### What is the Jira issue?
      * https://issues.apache.org/jira/browse/ZEPPELIN-4893
      
      ### How should this be tested?
      * CI pass
      
      ### Screenshots (if appropriate)
      
      ### Questions:
      * Does the licenses files need update? No
      * Is there breaking changes for older versions? No
      * Does this needs documentation? No
      
      Author: Jeff Zhang <zjffdu@apache.org>
      
      Closes #3809 from zjffdu/ZEPPELIN-4893 and squashes the following commits:
      
      e34cd7984 [Jeff Zhang] save
      0918aebfb [Jeff Zhang] [ZEPPELIN-4893]. Upgrade to spark 3.0.0
      82e4057d
  2. 23 Jun, 2020 4 commits
    • Philipp Dallig's avatar
      [ZEPPELIN-4902] Pom cleanup · cf527200
      Philipp Dallig authored
      ### What is this PR for?
      This PR fixes the problem, that hadoop libraries are not in every submodule.
      
      ### What type of PR is it?
       - Improvement
      
      ### What is the Jira issue?
      * https://issues.apache.org/jira/browse/ZEPPELIN-4902
      
      ### How should this be tested?
      * **Travis-CI**: https://travis-ci.org/github/Reamer/zeppelin/builds/700094748
      
      ### Questions:
      * Does the licenses files need update? No
      * Is there breaking changes for older versions? No
      * Does this needs documentation? No
      
      Author: Philipp Dallig <philipp.dallig@gmail.com>
      
      Closes #3817 from Reamer/pom_cleanup and squashes the following commits:
      
      fa221660a [Philipp Dallig] We need kerberos-libraries with hadoop3
      54db1215d [Philipp Dallig] Plugins should not need extra hadoop jars
      933069ca3 [Philipp Dallig] Remove global dependency for all submodules, if you run with hadoop3 profile
      d2e4569cc [Philipp Dallig] test dependencies should be in test scope
      cf527200
    • Alex Ott's avatar
      [ZEPPELIN-4818] Additional refactoring of Cassandra interpreter for new driver versions · 3a31d6a9
      Alex Ott authored
      ### What is this PR for?
      
      New DataStax Java Driver allows to override configuration options by different ways - via config file, system properties, etc.
      
      This PR implements following:
      
      * Allow to override any option of the Java driver by putting additional options into interpreter's configuration.  For example, this allows to use DataStax Astra (Cassandra as a Service) without need to make any changes in the code of the interpreter.  It also allows to configure any additional parameter not exposed directly by interpreter
      * Upgrades driver to latest released version
      * Configures embedded Cassandra for faster startup when executing unit tests
      
      ### What type of PR is it?
      
      Improvement
      
      ### What is the Jira issue?
      
      * https://issues.apache.org/jira/browse/ZEPPELIN-4818
      
      ### How should this be tested?
      
      * https://travis-ci.org/github/alexott/zeppelin/builds/700969578
      * tested manually
      
      Author: Alex Ott <alexott@gmail.com>
      
      Closes #3815 from alexott/ZEPPELIN-4818 and squashes the following commits:
      
      f34738f75 [Alex Ott] [ZEPPELIN-4818] Upgrade to driver 4.7.2
      5ef5c8823 [Alex Ott] [ZEPPELIN-4818] Allow to provide any Java driver setting
      a458df7ba [Alex Ott] [ZEPPELIN-4818] Explicitly set timeout for DDL statements
      42dc4e65c [Alex Ott] [ZEPPELIN-4818] Upgrade driver to 4.7.1
      db3950f1b [Alex Ott] [ZEPPELIN-4818] Speedup unit test
      3a31d6a9
    • Nathaniel Troutman's avatar
      [ZEPPELIN-4896] Add S3 Canned ACL Option for S3NotebookRepo · 5b3b819e
      Nathaniel Troutman authored
      ### What is this PR for?
      When S3NotebookRepo saves a notebook into S3 it does so without granting the bucket owner full control of the object, this means that owning AWS account cannot read the notebook, only the creating AWS account. This causes issues when a notebook bucket is shared across AWS accounts.
      
      This PR introduces the configuration value "zeppelin.notebook.s3.cannedAcl" which
      will override the default object permissions when saving a notebook to S3. This allows
      granting the bucket owner full control when the writer and the bucket owner are
      different AWS accounts.
      
      ### What type of PR is it?
      Improvement/Feature
      
      ### Todos
      * [ ] - N/A
      
      ### What is the Jira issue?
      https://issues.apache.org/jira/browse/ZEPPELIN-4896
      
      ### How should this be tested?
      #### Unit-Tests
      Unfortunately Unit-Tests are not possible as gaul.S3Proxy does not support Object ACL throwing a NotImplemented exception if you attempt to use PutRequest.withCannedAcl for any value other than public or private (see https://github.com/gaul/s3proxy#limitations).
      
      #### Manual Testing
      1. Have two AWS accounts, "account-a" and "account-b"
      2. Create a bucket "notebook-bucket" in "account-a" and grant "account-b" permissions to write to it
      3. Setup zeppelin-site.xml for S3NotebookRepor WITHOUT Canned ACL Feature and AWS Credentials for "account-b"
      4. Launch zeppelin, create a new note titled 'WithoutACL', and shutdown zeppelin
      5. Verify that the permissions on the S3 Object do NOT grant the bucket owner "account-a" permissions
      ```
      aws s3api get-object-acl --bucket notebook-bucket --key test-user/notebook/WithoutACL_2FD4NFYTU.zpln
      {
          "Owner": {
              "DisplayName": "account-b",
              "ID": "1e9...e4"
          },
          "Grants": [
              {
                  "Grantee": {
                      "DisplayName": "account-b",
                      "ID": "1e9...e4",
                      "Type": "CanonicalUser"
                  },
                  "Permission": "FULL_CONTROL"
              }
          ]
      }
      ```
      6. Edit zeppelin-site.xml enabling "zeppelin.notebook.s3.cannedAcl" as "BucketOwnerFullControl"
      7. Launch zeppelin, create a new note titled "WithACL", and shutdown zeppelin
      8. Verify that the permissions on the S3 object DO GRANT the bucket owner, "account-a", full control
      ```
      aws s3api get-object-acl --bucket notebook-bucket --key test-user/notebook/WithACL_2FCXTUS3M.zpln
      {
          "Owner": {
              "DisplayName": "account-b",
              "ID": "1e9...e4"
          },
          "Grants": [
              {
                  "Grantee": {
                      "DisplayName": "account-b",
                      "ID": "1e9...e4",
                      "Type": "CanonicalUser"
                  },
                  "Permission": "FULL_CONTROL"
              },
              {
                  "Grantee": {
                      "DisplayName": "account-a",
                      "ID": "f60...ee",
                      "Type": "CanonicalUser"
                  },
                  "Permission": "FULL_CONTROL"
              }
          ]
      }
      ```
      
      ### Screenshots (if appropriate)
      
      ### Questions:
      * Does the licenses files need update? No
      * Is there breaking changes for older versions? No
      * Does this need documentation? Yes, added relevant documentation along side existing S3 Notebook Repo documentation.
      
      Author: Nathaniel Troutman <troutma@amazon.com>
      
      Closes #3811 from ntroutman/canned-acl and squashes the following commits:
      
      6ebe608a1 [Nathaniel Troutman] [ZEPPELIN-4896] Add S3 Canned ACL Option for S3NotebookRepo
      5b3b819e
    • Jeff Zhang's avatar
      [ZEPPELIN-4899]. Injected variables in ZeppelinServer are lazy loaded · 22ff37b6
      Jeff Zhang authored
      ### What is this PR for?
      
      Trivial PR to create these injected variables when starting zeppelin server. Now these variables are
      created when user open zeppelin in browser if we don't get it explicitly here.
      Lazy loading will cause paragraph recovery and cron job initialization is delayed.
      
      ### What type of PR is it?
      [Improvement ]
      
      ### Todos
      * [ ] - Task
      
      ### What is the Jira issue?
      * https://issues.apache.org/jira/browse/ZEPPELIN-4899
      
      ### How should this be tested?
      * CI pass
      
      ### Screenshots (if appropriate)
      
      ### Questions:
      * Does the licenses files need update? No
      * Is there breaking changes for older versions? No
      * Does this needs documentation? No
      
      Author: Jeff Zhang <zjffdu@apache.org>
      
      Closes #3816 from zjffdu/ZEPPELIN-4899 and squashes the following commits:
      
      32e47278e [Jeff Zhang] [ZEPPELIN-4899]. Injected variables in ZeppelinServer are lazy loaded
      22ff37b6
  3. 22 Jun, 2020 2 commits
    • Jeff Zhang's avatar
      [ZEPPELIN-4839]. Update flink interpreter doc · b09d57e9
      Jeff Zhang authored
      ### What is this PR for?
      
      This PR update the flink interpreter doc for the recently improvement in flink interpreter and add some screenshot.
      
      ### What type of PR is it?
      [Documentation]
      
      ### Todos
      * [ ] - Task
      
      ### What is the Jira issue?
      * https://issues.apache.org/jira/browse/ZEPPELIN-4839
      
      ### How should this be tested?
      * CI pass
      
      ### Screenshots (if appropriate)
      
      ### Questions:
      * Does the licenses files need update? No
      * Is there breaking changes for older versions? No
      * Does this needs documentation? No
      
      Author: Jeff Zhang <zjffdu@apache.org>
      
      Closes #3779 from zjffdu/ZEPPELIN-4839 and squashes the following commits:
      
      ddc774078 [Jeff Zhang] [ZEPPELIN-4839]. Update flink interpreter doc
      b09d57e9
    • Jeff Zhang's avatar
      [ZEPPELIN-4875]. Add document for interpreter recovery · 7bc7c63d
      Jeff Zhang authored
      ### What is this PR for?
      
      Add document explaining what is interpreter recovery and how to use it.
      
      ### What type of PR is it?
      [ Documentation ]
      
      ### Todos
      * [ ] - Task
      
      ### What is the Jira issue?
      * https://issues.apache.org/jira/browse/ZEPPELIN-4875
      
      ### How should this be tested?
      * CI pass
      
      ### Screenshots (if appropriate)
      
      ### Questions:
      * Does the licenses files need update? No
      * Is there breaking changes for older versions? No
      * Does this needs documentation? No
      
      Author: Jeff Zhang <zjffdu@apache.org>
      
      Closes #3803 from zjffdu/ZEPPELIN-4875 and squashes the following commits:
      
      9c2e71ef0 [Jeff Zhang] [ZEPPELIN-4875]. Add document for interpreter recovery
      7bc7c63d
  4. 20 Jun, 2020 1 commit
  5. 19 Jun, 2020 2 commits
  6. 18 Jun, 2020 6 commits
    • Philipp Dallig's avatar
      [ZEPPELIN-4892] Delete plugin dir in clean phase · 2e51c397
      Philipp Dallig authored
      ### What is this PR for?
      This PR will delete the content of the plugin folders. Unfortunately the main folder is still present after a `mvn clean`
      
      ### What type of PR is it?
      - Bug Fix
      
      ### What is the Jira issue?
      * https://issues.apache.org/jira/browse/ZEPPELIN-4892
      
      ### How should this be tested?
      * **Travis-CI:** https://travis-ci.org/github/Reamer/zeppelin/builds/699667326
      
      ### Questions:
      * Does the licenses files need update? No
      * Is there breaking changes for older versions? No
      * Does this needs documentation? No
      
      Author: Philipp Dallig <philipp.dallig@gmail.com>
      
      Closes #3808 from Reamer/clean_plugin_dir and squashes the following commits:
      
      a0dab671c [Philipp Dallig] Delete plugin dir in clean phase
      2e51c397
    • Philipp Dallig's avatar
      [ZEPPELIN-4891] Remove duplicate declaration · 54c8ce89
      Philipp Dallig authored
      ### What is this PR for?
       Fixes a Maven warning because maven-resources-plugin and net.jodah:concurrentunit is defined twice in flink/interpreter/pom.xml
      
      ### What type of PR is it?
      Bug Fix
      
      ### What is the Jira issue?
      * https://issues.apache.org/jira/browse/ZEPPELIN-4891
      
      ### How should this be tested?
      * **Travis-CI**: https://travis-ci.org/github/Reamer/zeppelin/builds/699634998
      
      ### Questions:
      * Does the licenses files need update? No
      * Is there breaking changes for older versions? No
      * Does this needs documentation? No
      
      Author: Philipp Dallig <philipp.dallig@gmail.com>
      
      Closes #3807 from Reamer/mvn_warning_flink_interpreter and squashes the following commits:
      
      f5e945060 [Philipp Dallig] Fixes a Maven warning because maven-resources-plugin and net.jodah:concurrentunit is defined twice in this pom.xml
      54c8ce89
    • Jeff Zhang's avatar
      [ZEPPELIN-4850]. Include charting option for single value · a58902c4
      Jeff Zhang authored
      ### What is this PR for?
      
      This PR to support single row result displaying in frontend. For now we always use table format for sql output, but sometimes user want to just display a single value or single row in frontend (e.g. displaying KPI value). This PR use `SingleRowInterpreterResult` to represent such sql output, and refactor existing code to leverage this class to display single row in frontend. Besides that this PR also support the single row result in spark sql.
      
      ### What type of PR is it?
      [ Improvement | Feature | Documentation ]
      
      ### Todos
      * [ ] - Task
      
      ### What is the Jira issue?
      * https://github.com/zjffdu/zeppelin/compare/ZEPPELIN-4850
      
      ### How should this be tested?
      * CI pass
      
      ### Screenshots (if appropriate)
      
      ### Questions:
      * Does the licenses files need update? No
      * Is there breaking changes for older versions? No
      * Does this needs documentation? No
      
      Author: Jeff Zhang <zjffdu@apache.org>
      
      Closes #3791 from zjffdu/ZEPPELIN-4850 and squashes the following commits:
      
      10b992e70 [Jeff Zhang] [ZEPPELIN-4850]. Include charting option for single value
      a58902c4
    • Alex Ott's avatar
      [ZEPPELIN-4821] Spark interpreter uses incorrect property name · a282e786
      Alex Ott authored
      ### What is this PR for?
      
      Spark interpreter did use incorrect property name for Spark Master - `master`, although a lot of code was dependent on `spark.master` - it's better to use only one name everywhere
      
      ### What type of PR is it?
      Bug Fix
      
      ### What is the Jira issue?
      * ZEPPELIN-4821
      
      ### How should this be tested?
      * https://travis-ci.org/github/alexott/zeppelin/builds/699582429
      
      Author: Alex Ott <alexott@apache.org>
      
      Closes #3773 from alexott/ZEPPELIN-4821 and squashes the following commits:
      
      82b8321f1 [Alex Ott] [ZEPPELIN-4821] Spark interpreter uses incorrect property name
      a282e786
    • dijie's avatar
      [ZEPPELIN-4889] [Flink] Error logs in catch exception · 9294062f
      dijie authored
      ### What is this PR for?
      Fix error logs in  catch exception
      
      ### What type of PR is it?
      [Bug Fix ]
      
      ### Todos
      * [ ] - Task
      
      ### What is the Jira issue?
      https://issues.apache.org/jira/browse/ZEPPELIN-4889
      
      ### How should this be tested?
      
      ### Screenshots (if appropriate)
      
      ### Questions:
      No questions.
      
      Author: dijie <dijie@mogu.com>
      
      Closes #3804 from lonelyGhostisdog/ZEPPELIN-4889 and squashes the following commits:
      
      636b69809 [dijie] [ZEPPELIN-4889] [Flink] Error logs in catch exception
      9294062f
    • Jeff Zhang's avatar
      [hotfix] use proper row equals for flink 1.11 · cfb64a7c
      Jeff Zhang authored
      cfb64a7c
  7. 17 Jun, 2020 5 commits
  8. 16 Jun, 2020 4 commits
    • Jeff Zhang's avatar
      [ZEPPELIN-4878]. Unable to run flink 1.10.1 in yarn mode due to FLINK-17788 · 951da429
      Jeff Zhang authored
      ### What is this PR for?
      
      Flink interpreter doesn't work with flink 1.10.1 in yarn mode due to FLINK-17788, this PR fix it in zeppelin side. Because in scala shell, flink only support yarn session mode, so when it is yarn mode, we can set deployment target to be yarn-session directly.
      
      ### What type of PR is it?
      [Bug Fix ]
      
      ### Todos
      * [ ] - Task
      
      ### What is the Jira issue?
      * https://issues.apache.org/jira/browse/ZEPPELIN-4878
      
      ### How should this be tested?
      * CI pass and manually tested
      
      ### Screenshots (if appropriate)
      
      ### Questions:
      * Does the licenses files need update? No
      * Is there breaking changes for older versions? No
      * Does this needs documentation? No
      
      Author: Jeff Zhang <zjffdu@apache.org>
      
      Closes #3795 from zjffdu/ZEPPELIN-4878 and squashes the following commits:
      
      498325cc8 [Jeff Zhang] use flink1.10.version & flink1.11.version
      fb1a761b5 [Jeff Zhang] [ZEPPELIN-4878]. Unable to run flink 1.10.1 in yarn mode due to FLINK-17788
      951da429
    • Jeff Zhang's avatar
      [ZEPPELIN-4837]. Add property to only check some packages for udf finding · 37953f4f
      Jeff Zhang authored
      ### What is this PR for?
      
      Add property `flink.udf.jars.packages` to specify the packages that would be searched for udf, otherwise all the classes will be check, which might be very time consuming especially when the udf jars are large (if your udf depends on other third party libraries)
      
      ### What type of PR is it?
      [Feature ]
      
      ### Todos
      * [ ] - Task
      
      ### What is the Jira issue?
      * https://issues.apache.org/jira/browse/ZEPPELIN-4837
      
      ### How should this be tested?
      * CI pass and manually tested
      
      ### Screenshots (if appropriate)
      ![image](https://user-images.githubusercontent.com/164491/84347276-b3767700-abe4-11ea-9688-f4334dc4d395.png)
      
      ### Questions:
      * Does the licenses files need update? No
      * Is there breaking changes for older versions? No
      * Does this needs documentation? No
      
      Author: Jeff Zhang <zjffdu@apache.org>
      
      Closes #3798 from zjffdu/ZEPPELIN-4837 and squashes the following commits:
      
      78816e548 [Jeff Zhang] [ZEPPELIN-4837]. Add property to only check some packages for udf finding
      37953f4f
    • Jeff Zhang's avatar
      [ZEPPELIN-4873]. Display rich duration info for insert into flink job · 3139ed6a
      Jeff Zhang authored
      ### What is this PR for?
      
      Trivial PR which display rich duration info instead of just x seconds. See screenshot below.
      
      ### What type of PR is it?
      [ Improvement ]
      
      ### Todos
      * [ ] - Task
      
      ### What is the Jira issue?
      * https://issues.apache.org/jira/browse/ZEPPELIN-4873
      
      ### How should this be tested?
      * CI pass
      
      ### Screenshots (if appropriate)
      ![image](https://user-images.githubusercontent.com/164491/84286308-19291b80-ab71-11ea-96ef-b237d2463b8c.png)
      
      ### Questions:
      * Does the licenses files need update? No
      * Is there breaking changes for older versions? No
      * Does this needs documentation? No
      
      Author: Jeff Zhang <zjffdu@apache.org>
      
      Closes #3794 from zjffdu/ZEPPELIN-4873 and squashes the following commits:
      
      459f181c5 [Jeff Zhang] add java doc
      419818e4a [Jeff Zhang] address comment
      9691eae82 [Jeff Zhang] [ZEPPELIN-4873]. Display rich duration info for insert into flink job
      3139ed6a
    • Jeff Zhang's avatar
      [ZEPPELIN-4844]. Support yarn interpreter launch mode · df54df9e
      Jeff Zhang authored
      ### What is this PR for?
      
      This PR is to support yarn interpreter launch mode, that means to launch the interpreter process in yarn container. So that we can mitigate the memory pressure of zeppelin server machine.
      This PR add new launcher module `yarn` for that. I manually tested shell, python, jdbc and flink interpreted.
      
      ### What type of PR is it?
      [Feature]
      
      ### Todos
      * [ ] - Task
      
      ### What is the Jira issue?
      * https://issues.apache.org/jira/browse/ZEPPELIN-4844
      
      ### How should this be tested?
      * CI pass and manually tested
      
      ### Screenshots (if appropriate)
      
      ### Questions:
      * Does the licenses files need update? No
      * Is there breaking changes for older versions? No
      * Does this needs documentation? No
      
      Author: Jeff Zhang <zjffdu@apache.org>
      
      Closes #3786 from zjffdu/ZEPPELIN-4844 and squashes the following commits:
      
      899c4e1c0 [Jeff Zhang] [ZEPPELIN-4844]. Support yarn interpreter launch mode
      df54df9e
  9. 14 Jun, 2020 1 commit
    • Alex Ott's avatar
      [ZEPPELIN-4870][hotfix] fix edge case, and remove duplicate parser · d679bb1b
      Alex Ott authored
      ### What is this PR for?
      
      Tests didn't cover all of the edge cases, that lead to error when parsing specific message.
      
      Also, was found that parsing is also duplicated in the `Paragraph` class - it was replaced with call to `ParagraphTextParser`.
      
      The most visible change is that `%interpreter(some text` will know fail as there is no matching `)` character. But it's works the same if we change it to `%interpreter (some text`...
      
      ### What type of PR is it?
      Hot Fix
      
      ### What is the Jira issue?
      * ZEPPELIN-4870
      
      ### How should this be tested?
      * Added one more test
      * https://travis-ci.org/github/alexott/zeppelin/builds/697944179
      
      Author: Alex Ott <alexott@gmail.com>
      
      Closes #3800 from alexott/ZEPPELIN-4870-hotfix and squashes the following commits:
      
      5942c5d35 [Alex Ott] [ZEPPELIN-4870][hotfix] fix edge case, and remove duplicate parser
      d679bb1b
  10. 12 Jun, 2020 1 commit
    • Alex Ott's avatar
      [ZEPPELIN-4870] Improve parsing of the paragraph properties · 1d1b0583
      Alex Ott authored
      ### What is this PR for?
      
      We can provide properties that are local to the paragraph, that could be used to pass an additional information for interpreter that could affect its behavior.  Unfortunately existing parsing functionality relies on the fact that key/value pairs need to be separated by `,` character, and doesn't handle values with special characters (`,`, `=`, ...) inside, like this:
      
      ```
      %cassandra(locale=ruRU, timeFormat="E, d MMM yy", floatPrecision = 5, outputFormat=cql)
      ```
      
      This PR changes the parsing logic to perform character-by-character parsing, and handling of the quoted keys & values, escaping of the special characters, etc.
      
      ### What type of PR is it?
      
      Improvement
      
      ### What is the Jira issue?
      * https://issues.apache.org/jira/browse/ZEPPELIN-4870
      
      ### How should this be tested?
      * https://travis-ci.org/github/alexott/zeppelin/builds/697522260
      * additional unit tests were added
      
      Author: Alex Ott <alexott@gmail.com>
      
      Closes #3799 from alexott/ZEPPELIN-4870 and squashes the following commits:
      
      5fb6ee84e [Alex Ott] [ZEPPELIN-4870] Improve parsing of the paragraph properties
      1d1b0583
  11. 10 Jun, 2020 4 commits
    • Jeff Zhang's avatar
      [ZEPPELIN-4865]. Allow specify jobName as paragraph local properties · 6bea350b
      Jeff Zhang authored
      ### What is this PR for?
      
      Minor PR which allow user to specify job name as paragraph local properties, by default it is the sql statement if user don't specify it.
      
      ### What type of PR is it?
      [Improvement]
      
      ### Todos
      * [ ] - Task
      
      ### What is the Jira issue?
      
      https://issues.apache.org/jira/browse/ZEPPELIN-4865
      
      ### How should this be tested?
      * CI pass and manually tested.
      
      ### Screenshots (if appropriate)
      ![image](https://user-images.githubusercontent.com/164491/84017337-15e73180-a9b1-11ea-94de-baf605d6a422.png)
      
      ### Questions:
      * Does the licenses files need update? No
      * Is there breaking changes for older versions? No
      * Does this needs documentation? No
      
      Author: Jeff Zhang <zjffdu@apache.org>
      
      Closes #3792 from zjffdu/ZEPPELIN-4865 and squashes the following commits:
      
      7c6e2f5c9 [Jeff Zhang] [ZEPPELIN-4865]. Allow specify jobName as paragraph local properties
      6bea350b
    • Alex Ott's avatar
      [ZEPPELIN-4864] Cleanup Travis-CI config · 5cdb51bf
      Alex Ott authored
      ### What is this PR for?
      
      Existing Travis-CI config contained references to previous distributions, etc. This PR fixes this.
      It also fixes a problem with Livy interpreter, where tests were failing because R wasn't enabled for them
      
      ### What type of PR is it?
      Bug Fix / Improvement
      
      ### What is the Jira issue?
      * https://issues.apache.org/jira/browse/ZEPPELIN-4864
      
      ### How should this be tested?
      * https://travis-ci.org/github/alexott/zeppelin/builds/696385230
      
      Author: Alex Ott <alexott@apache.org>
      
      Closes #3793 from alexott/ZEPPELIN-4864 and squashes the following commits:
      
      3fccb1465 [Alex Ott] [ZEPPELIN-4864] enable R for Livy tests
      91a30a303 [Alex Ott] [ZEPPELIN-4864] improve debugability of Livy integration test
      82233446e [Alex Ott] [ZEPPELIN-4864] Cleanup Travis-CI config
      5cdb51bf
    • Jeff Zhang's avatar
      [ZEPPELIN-4816]. Support Flink 1.11 · c1d6297b
      Jeff Zhang authored
      ### What is this PR for?
      Although flink 1.11 is not released yet,  it is already feature cut, so its api is stable now. This PR is to make flink interpreter of zeppelin to support flink 1.11. Due to api change of flink 1.11, this PR introduce flink-shim which will use different api for different version of flink.
      
      ### What type of PR is it?
      [Feature ]
      
      ### Todos
      * [ ] - Task
      
      ### What is the Jira issue?
      * https://jira.apache.org/jira/browse/ZEPPELIN-4816
      
      ### How should this be tested?
      * CI pass and tested on flink 1.11 snapshot
      
      ### Screenshots (if appropriate)
      
      ### Questions:
      * Does the licenses files need update? No
      * Is there breaking changes for older versions? no
      * Does this needs documentation? No
      
      Author: Jeff Zhang <zjffdu@apache.org>
      
      Closes #3782 from zjffdu/ZEPPELIN-4816 and squashes the following commits:
      
      6ce065221 [Jeff Zhang] [ZEPPELIN-4816]. Support Flink 1.11
      c1d6297b
    • Jeff Zhang's avatar
      [ZEPPELIN-4845]. Recover running paragraph when recovery is enabled · 446608f1
      Jeff Zhang authored
      ### What is this PR for?
      
      This PR is to recover the running paragraph when recovery is enabled.
      1. Get all the running Interpreter processes from RecoveryStorage.
      2. Reconnect to the running interpreter process.
      3. Resubmit the paragraph to interpreter process.
      
      ### What type of PR is it?
      [ Feature]
      
      ### Todos
      * [ ] - Task
      
      ### What is the Jira issue?
      * https://issues.apache.org/jira/browse/ZEPPELIN-4845
      
      ### How should this be tested?
      * CI pass and manully tested the recovery scenairo.
      
      ### Screenshots (if appropriate)
      
      ![ezgif com-video-to-gif (1)](https://user-images.githubusercontent.com/164491/83387264-013df300-a41f-11ea-9170-091af76037e0.gif)
      
      ### Questions:
      * Does the licenses files need update? No
      * Is there breaking changes for older versions? No
      * Does this needs documentation? No
      
      Author: Jeff Zhang <zjffdu@apache.org>
      
      Closes #3781 from zjffdu/ZEPPELIN-4845 and squashes the following commits:
      
      2302471de [Jeff Zhang] [ZEPPELIN-4845]. Recover running paragraph when recovery is enabled
      446608f1
  12. 09 Jun, 2020 1 commit
    • Jeff Zhang's avatar
      [ZEPPELIN-4859]. Use PooledObject for the connection from interpreter process to zeppelin server · 7eb125d5
      Jeff Zhang authored
      ### What is this PR for?
      
      This is a improvement and refactoring PR which just use PooledObject for the connection from interpreter process to zeppelin server. Otherwise, once the connection is lost, interpreter process can never connect to zeppelin sever again. In this PR, I create class `PooledRemoteClient` which represent the pooled connection objects and this class will manage the life cycle of the connection between thrift client to thrift server.
      
      ### What type of PR is it?
      [ Improvement |  Refactoring]
      
      ### Todos
      * [ ] - Task
      
      ### What is the Jira issue?
      * https://issues.apache.org/jira/browse/ZEPPELIN-4859
      
      ### How should this be tested?
      * CI pass
      
      ### Screenshots (if appropriate)
      
      ### Questions:
      * Does the licenses files need update? No
      * Is there breaking changes for older versions? No
      * Does this needs documentation? No
      
      Author: Jeff Zhang <zjffdu@apache.org>
      
      Closes #3788 from zjffdu/ZEPPELIN-4859 and squashes the following commits:
      
      1c965ce7f [Jeff Zhang] [ZEPPELIN-4859]. Use PooledObject for the connection from interpreter process to zeppelin server
      7eb125d5
  13. 08 Jun, 2020 3 commits
    • Jeff Zhang's avatar
      [ZEPPELIN-4833] misleading logging when fail to load plugin from classpath directly · 11780d38
      Jeff Zhang authored
      ### What is this PR for?
      
      Previous plugin loading strategy is to load them from system classloader first, if fails, then fallback to plugin folder classloader. This would produce misleading logging which make user think that the plugin is failed to load. This PR would load plugin from system classloader only when they are builtin plugins.
      
      ### What type of PR is it?
      [Improvement]
      
      ### Todos
      * [ ] - Task
      
      ### What is the Jira issue?
      * https://issues.apache.org/jira/browse/ZEPPELIN-4833
      
      ### How should this be tested?
      * CI pass
      
      ### Screenshots (if appropriate)
      
      ### Questions:
      * Does the licenses files need update? No
      * Is there breaking changes for older versions? No
      * Does this needs documentation? No
      
      Author: Jeff Zhang <zjffdu@apache.org>
      
      Closes #3784 from zjffdu/ZEPPELIN-4833 and squashes the following commits:
      
      62cd4db59 [Jeff Zhang] [ZEPPELIN-4833] misleading logging when fail to load plugin from classpath directly
      11780d38
    • Jeff Zhang's avatar
      [ZEPPELIN-4848]. Switch for enabling Spark REPL output for each paragraph · 4d4d532d
      Jeff Zhang authored
      ### What is this PR for?
      
      This PR add paragraph local property `printREPLOutput` for spark interpreter, so that user can switch for enabling Spark REPL output for each paragraph, e.g.
      
      ```
      %spark(printREPLOutput=false)
      
      System.out.println("hello ")
      println("hello world")
      ```
      
      ### What type of PR is it?
      [ Feature ]
      
      ### Todos
      * [ ] - Task
      
      ### What is the Jira issue?
      * https://issues.apache.org/jira/browse/ZEPPELIN-4848
      
      ### How should this be tested?
      * CI pass
      
      ### Screenshots (if appropriate)
      
      ### Questions:
      * Does the licenses files need update? No
      * Is there breaking changes for older versions? No
      * Does this needs documentation? No
      
      Author: Jeff Zhang <zjffdu@apache.org>
      
      Closes #3787 from zjffdu/ZEPPELIN-4848 and squashes the following commits:
      
      b6997d99b [Jeff Zhang] [ZEPPELIN-4848]. Switch for enabling Spark REPL output for each paragraph
      4d4d532d
    • Jeff Zhang's avatar
      [ZEPPELIN-4852]. Add name to RemoteInterpreterProcess · 2ab8f9ec
      Jeff Zhang authored
      ### What is this PR for?
      
      This is a trivial PR which add `getInterpreterGroupId` to `InterpreterClient`, and use `interpreterGroupId` as the identifier of RemoteInterpreterProcess because `interpreterGroupId` is also unique identifier of InterpreterGroup
      
      ### What type of PR is it?
      [Improvement]
      
      ### Todos
      * [ ] - Task
      
      ### What is the Jira issue?
      * https://issues.apache.org/jira/browse/ZEPPELIN-4852
      
      ### How should this be tested?
      * CI pass
      
      ### Screenshots (if appropriate)
      
      ### Questions:
      * Does the licenses files need update? No
      * Is there breaking changes for older versions? No
      * Does this needs documentation? No
      
      Author: Jeff Zhang <zjffdu@apache.org>
      
      Closes #3785 from zjffdu/ZEPPELIN-4852 and squashes the following commits:
      
      01ec89373 [Jeff Zhang] address comment
      4fc92cc60 [Jeff Zhang] address comment
      a482faa39 [Jeff Zhang] [ZEPPELIN-4852]. Add name to RemoteInterpreterProcess
      2ab8f9ec
  14. 07 Jun, 2020 1 commit
  15. 05 Jun, 2020 1 commit
    • Jeff Zhang's avatar
      [ZEPPELIN-4840]. Set zeppelin.spark.concurrentSQL to be true by default · a50df08a
      Jeff Zhang authored
      ### What is this PR for?
      
      Trivial PR which just make `zeppelin.spark.concurrentSQL` to be `true` by default, this is what user expect usually.
      
      ### What type of PR is it?
      [Improvement]
      
      ### Todos
      * [ ] - Task
      
      ### What is the Jira issue?
      * https://issues.apache.org/jira/browse/ZEPPELIN-4840
      
      ### How should this be tested?
      CI pass
      
      ### Screenshots (if appropriate)
      
      ### Questions:
      * Does the licenses files need update? no
      * Is there breaking changes for older versions? no
      * Does this needs documentation? no
      
      Author: Jeff Zhang <zjffdu@apache.org>
      
      Closes #3783 from zjffdu/ZEPPELIN-4840 and squashes the following commits:
      
      7ece99b83 [Jeff Zhang] [ZEPPELIN-4840]. Set zeppelin.spark.concurrentSQL to be true by default
      a50df08a
  16. 01 Jun, 2020 3 commits
    • Alex Ott's avatar
      [ZEPPELIN-4792] Help needs to be updated in Cassandra interpreter · def30ee4
      Alex Ott authored
      ### What is this PR for?
      
      When updating  for the new DataStax Java driver, I forgot to update template for a `HELP` command. This PR fixes this problem, and simplifies maintenance by reading driver version from the resource file generated during compilation, so HELP command will provide information based on the actual driver version used for building interpreter.
      
      ### What type of PR is it?
      Bug Fix
      
      ### What is the Jira issue?
      
      https://issues.apache.org/jira/browse/ZEPPELIN-4792
      
      ### How should this be tested?
      
      * https://travis-ci.org/github/alexott/zeppelin/builds/693123862 ()
      * Tested manually
      
      Author: Alex Ott <alexott@gmail.com>
      
      Closes #3780 from alexott/ZEPPELIN-4792 and squashes the following commits:
      
      f62bc5c51 [Alex Ott] [ZEPPELIN-4792] Help needs to be updated in Cassandra interpreter
      def30ee4
    • Jeff Zhang's avatar
      [ZEPPELIN-4802]. pyspark warnings with spark-3.0.0-preview - Sync of properties between JVM and PVM · df86e502
      Jeff Zhang authored
      ### What is this PR for?
      
      This PR will first set env `PYSPARK_PIN_THREAD` to be `true` and then suppress the warning message in `pyspark.util`.
      
      ### What type of PR is it?
      [Improvement ]
      
      ### Todos
      * [ ] - Task
      
      ### What is the Jira issue?
      * https://issues.apache.org/jira/browse/ZEPPELIN-4802
      
      ### How should this be tested?
      * CI pass
      * Manually tested
      
      ### Screenshots (if appropriate)
      
      ### Questions:
      * Does the licenses files need update? no
      * Is there breaking changes for older versions? No
      * Does this needs documentation?
      
      Author: Jeff Zhang <zjffdu@apache.org>
      
      Closes #3778 from zjffdu/ZEPPELIN-4802 and squashes the following commits:
      
      d6a65cbde [Jeff Zhang] [ZEPPELIN-4802]. pyspark warnings with spark-3.0.0-preview - Sync of properties between JVM and PVM
      df86e502
    • Jeff Zhang's avatar
      [ZEPPELIN-4836]. set zeppelin.spark.sql.stacktrace to true as default value · 3034a44d
      Jeff Zhang authored
      ### What is this PR for?
      
      Trivial PR to set `zeppelin.spark.sql.stacktrace` to `true` as default value
      
      ### What type of PR is it?
      [Improvement]
      
      ### Todos
      * [ ] - Task
      
      ### What is the Jira issue?
      * https://issues.apache.org/jira/browse/ZEPPELIN-4836
      
      ### How should this be tested?
      * CI pass
      
      ### Screenshots (if appropriate)
      
      ### Questions:
      * Does the licenses files need update? No
      * Is there breaking changes for older versions? No
      * Does this needs documentation? No
      
      Author: Jeff Zhang <zjffdu@apache.org>
      
      Closes #3777 from zjffdu/ZEPPELIN-4836 and squashes the following commits:
      
      547f1381f [Jeff Zhang] [ZEPPELIN-4836]. set zeppelin.spark.sql.stacktrace to true as default value
      3034a44d