livy.session

class livy.session.LivySession(url, auth=None, kind=<SessionKind.PYSPARK: 'pyspark'>, proxy_user=None, spark_conf=None, echo=True, check=True)

Manages a remote Livy session and high-level interactions with it.

Parameters

• url (str) – The URL of the Livy server.

• kind (SessionKind) – The kind of session to create.

• proxy_user (Optional[str]) – User to impersonate when starting the session.

• spark_conf (Optional[Dict[str, Any]]) – Spark configuration properties.

• echo (bool) – Whether to echo output printed in the remote session. Defaults to True.

• check (bool) – Whether to raise an exception when a statement in the remote session fails. Defaults to True.

start()

Create the remote Spark session and wait for it to be ready.

Return type

None

property state

The state of the managed Spark session.

Return type

SessionState

close()

Kill the managed Spark session.

Return type

None

run(code)

Run some code in the managed Spark session.

Parameters

code (str) – The code to run.

Return type

Output

read(dataframe_name)

Evaluate and retrieve a Spark dataframe in the managed session.

Parameters

dataframe_name (str) – The name of the Spark dataframe to read.

Return type

DataFrame

read_sql(code)

Evaluate a Spark SQL satatement and retrieve the result.

Parameters

code (str) – The Spark SQL statement to evaluate.

Return type

DataFrame