livy.session

class livy.session.LivySession(url, auth=None, verify=True, kind=<SessionKind.PYSPARK: 'pyspark'>, proxy_user=None, jars=None, py_files=None, files=None, driver_memory=None, driver_cores=None, executor_memory=None, executor_cores=None, num_executors=None, archives=None, queue=None, name=None, spark_conf=None, echo=True, check=True)[source]

Manages a remote Livy session and high-level interactions with it.

The py_files, files, jars and archives arguments are lists of URLs, e.g. [“s3://bucket/object”, “hdfs://path/to/file”, …] and must be reachable by the Spark driver process. If the provided URL has no scheme, it’s considered to be relative to the default file system configured in the Livy server.

URLs in the py_files argument are copied to a temporary staging area and inserted into Python’s sys.path ahead of the standard library paths. This allows you to import .py, .zip and .egg files in Python.

URLs for jars, py_files, files and archives arguments are all copied to the same working directory on the Spark cluster.

The driver_memory and executor_memory arguments have the same format as JVM memory strings with a size unit suffix (“k”, “m”, “g” or “t”) (e.g. 512m, 2g).

See https://spark.apache.org/docs/latest/configuration.html for more information on Spark configuration properties.

Parameters

  • url (str) – The URL of the Livy server.

  • auth (Union[AuthBase, Tuple[str, str], None]) – A requests-compatible auth object to use when making requests.

  • verify (Union[bool, str]) – Either a boolean, in which case it controls whether we verify the server’s TLS certificate, or a string, in which case it must be a path to a CA bundle to use. Defaults to True.

  • kind (SessionKind) – The kind of session to create.

  • proxy_user (Optional[str]) – User to impersonate when starting the session.

  • jars (Optional[List[str]]) – URLs of jars to be used in this session.

  • py_files (Optional[List[str]]) – URLs of Python files to be used in this session.

  • files (Optional[List[str]]) – URLs of files to be used in this session.

  • driver_memory (Optional[str]) – Amount of memory to use for the driver process (e.g. ‘512m’).

  • driver_cores (Optional[int]) – Number of cores to use for the driver process.

  • executor_memory (Optional[str]) – Amount of memory to use per executor process (e.g. ‘512m’).

  • executor_cores (Optional[int]) – Number of cores to use for each executor.

  • num_executors (Optional[int]) – Number of executors to launch for this session.

  • archives (Optional[List[str]]) – URLs of archives to be used in this session.

  • queue (Optional[str]) – The name of the YARN queue to which submitted.

  • name (Optional[str]) – The name of this session.

  • spark_conf (Optional[Dict[str, Any]]) – Spark configuration properties.

  • echo (bool) – Whether to echo output printed in the remote session. Defaults to True.

  • check (bool) – Whether to raise an exception when a statement in the remote session fails. Defaults to True.

start()[source]

Create the remote Spark session and wait for it to be ready.

Return type

None

property state

The state of the managed Spark session.

Return type

SessionState

close()[source]

Kill the managed Spark session.

Return type

None

run(code)[source]

Run some code in the managed Spark session.

Parameters

code (str) – The code to run.

Return type

Output

read(dataframe_name)[source]

Evaluate and retrieve a Spark dataframe in the managed session.

Parameters

dataframe_name (str) – The name of the Spark dataframe to read.

Return type

DataFrame

read_sql(code)[source]

Evaluate a Spark SQL satatement and retrieve the result.

Parameters

code (str) – The Spark SQL statement to evaluate.

Return type

DataFrame