V3IO Frames ("Frames") is a Golang based remote data frames access (over gRPC or HTTP stream)

Related tags

Network frames
Overview

V3IO Frames

GoDoc License

V3IO Frames ("Frames") is a multi-model open-source data-access library that provides a unified high-performance DataFrame API for working with different types of data sources (backends). The library was developed by Iguazio to simplify working with data in the Iguazio Data Science Platform ("the platform"), but it can be extended to support additional backend types.

Note: For a full API reference of the Frames platform backends, including detailed examples, see the Frames API reference in the platform documentation.

In This Document

Client Python API Reference

Overview

Python Version

The current version of Frames supports Python 3.6 and 3.7.

Initialization

To use Frames, you first need to import the v3io_frames Python library. For example:

import v3io_frames as v3f

Then, you need to create and initialize an instance of the Client class; see Client Constructor. You can then use the client methods to perform different data operations on the supported backend types.

Backend Types

All Frames client methods receive a backend parameter for setting the Frames backend type. Frames currently supports the following backend types:

  • nosql | kv — a platform NoSQL (key/value) table. See the platform NoSQL backend API reference.

    Note: The documentation uses the "nosql" alias to the "kv" type, which was added in Frames v0.6.10-v0.9.13; "kv" is still supported for backwards compatibility with earlier releases.

  • stream — a platform data stream [Tech Preview].
  • tsdb — a time-series database (TSDB). See the platform TSDB backend API reference.
  • csv — a comma-separated-value (CSV) file. This backend type is used only for testing purposes.

Client Methods

The Client class features the following methods for supporting operations on a data collection, such as a NoSQL or TSDB table or a data stream:

  • create — creates a new collection.
  • delete — deletes a collection or specific items of the collection.
  • read — reads data from a collection into pandas DataFrames.
  • write — writes data from pandas DataFrames to a collection.
  • execute — executes a backend-specific command on a collection. Each backend may support multiple commands.
  • history — returns information about requests made to the service.

Note: Some methods or method parameters are backend-specific, as detailed in this reference.

User Authentication

When creating a Frames client, you must provide valid credentials for accessing the backend data, which Frames will use to identify the identity of the user. This can be done by using any of the following alternative methods (documented in order of precedence). For more information about the user authentication for the platform backends, see the platform documentation:

  • Provide the authentication credentials in the call to the Client constructor — either by setting the token parameter to a valid authentication token (access key) or by setting the user and password parameters to a username and password. Note that you cannot set the token parameter concurrently with the username and password parameters.

  • Provide the authentication credentials in environment variables — either by setting the V3IO_ACCESS_KEY variable to an authentication token or by setting the V3IO_USERNAME and V3IO_PASSWORD variables to a username and password.

    Note:

    • When V3IO_ACCESS_KEY is defined, V3IO_USERNAME and V3IO_PASSWORD are ignored.
    • When the client constructor is called with authentication parameters (option #1), the authentication-credentials environment variables (if defined) are ignored.

Client Constructor

All Frames operations are executed via an object of the Client class.

Syntax

Client(address=""[, data_url=""], container=""[, user="", password="", token=""])

Parameters and Data Members

  • address — The address of the Frames service (framesd). Use the grpc:// prefix for gRPC (default; recommended) or the http:// prefix for HTTP. When running locally on the platform, set this parameter to framesd:8081 to use the gRPC (recommended) or to framesd:8080 to use HTTP; for more information, see the platform documentation.

    • Type: str
    • Requirement: Required
  • data_url — A web-API base URL for accessing the backend data. By default, the client uses the data URL that's configured for the Frames service; for the platform backends, this is typically the HTTPS URL of the web-APIs service of the parent tenant.

    • Type: str
    • Requirement: Optional
  • container — The name of the data container that contains the backend data. For example, "bigdata" or "users".

    • Type: str
    • Requirement: Required
  • user — The username of a user with permissions to access the backend data. See User Authentication.

    • Type: str
    • Requirement: Required when neither the token parameter or the authentication environment variables are set.
      When the user parameter is set, the password parameter must also be set to a matching user password.
  • password — A valid password for the user configured in the user parameter. See User Authentication.

    • Type: str
    • Requirement: Required when the user parameter is set.
  • token — A valid token that allows access to the backend data, such as a platform access key for the platform backends. See User Authentication.

    • Type: str
    • Requirement: Required when neither the user or password parameters or the authentication environment variables are set.

Return Value

Returns a new Frames Client data object.

Examples

The following examples, for local platform execution, both create a Frames client for accessing data in the "users" container by using the authentication credentials of user "iguazio"; the first example uses token (access-key) authentication while the second example uses username and password authentication (see User Authentication):

import v3io_frames as v3f
client = v3f.Client("framesd:8081", token="e8bd4ca2-537b-4175-bf01-8c74963e90bf", container="users")
import v3io_frames as v3f
client = v3f.Client("framesd:8081", user="iguazio", password="mypass", container="users")

Common Client Method Parameters

All client methods receive the following common parameters; additional, method-specific parameters are described for each method.

  • backend — The backend data type for the operation. See Backend Types.

    • Type: str
    • Requirement: Required
    • Valid Values: "nosql" | "stream" | "tsdb" | "csv" (for testing)
  • table — The relative path to a data collection of the specified backend type in the target data container (as configured for the client object). For example, "mytable" or "/examples/tsdb/my_metrics".

    • Type: str
    • Requirement: Required unless otherwise specified in the method-specific documentation

create Method

Creates a new data collection in the configured client data container, according to the specified backend type.

Note: The create method isn't applicable to the nosql backend, because NoSQL tables in the platform don't need to be created prior to ingestion; when ingesting data into a table that doesn't exist, the table is automatically created.

Syntax

create(backend, table, schema=None, if_exists=FAIL, **kw)

Common create Parameters

All Frames backends that support the create method support the following common parameters:

  • if_exists — Determines whether to raise an error when the specified collection (table) already exists.

    • Type: pb.ErrorOptions enumeration. To use the enumeration, import the frames_pb2 module; for example:

      from v3io_frames import frames_pb2 as fpb
    • Requirement: Optional
    • Valid Values: FAIL to raise an error when the specified collection already exist; IGNORE to ignore this
    • Default Value: FAIL
  • schema — a schema for describing unstructured collection data. This parameter is intended to be used only for testing purposes with the csv backend.

    • Type: Backend-specific or None
    • Requirement: Optional
    • Default Value: None
  • kw — This parameter is used for passing a variable-length list of additional keyword (named) arguments. For more information, see the backend-specific method parameters.

    • Type: ** — variable-length keyword arguments list
    • Requirement: Optional

tsdb Backend create Parameters

The following create parameters are specific to the tsdb backend and are passed as keyword arguments via the kw parameter; for more information and examples, see the platform's Frames TSDB-backend reference:

  • rate — metric-samples ingestion rate.

    • Type: str
    • Requirement: Required
    • Valid Values: A string of the format "[0-9]+/[smh]" — where 's' = seconds, 'm' = minutes, and 'h' = hours. For example, "1/s" (one sample per minute), "20/m" (20 samples per minute), or "50/h" (50 samples per hour).
  • aggregates — A list of aggregation functions for real-time aggregation during the samples ingestion ("pre-aggregation").

    • Type: str
    • Requirement: Optional
    • Valid Values: A string containing a comma-separated list of supported aggregation functions — avg| count| last| max| min| rate| stddev| stdvar| sum. For example, "count,avg,min,max".
  • aggregation_granularity — Aggregation granularity; applicable when the aggregates parameter is set.

    • Type: str
    • Requirement: Optional
    • Valid Values: A string of the format "[0-9]+[mhd]" — where 'm' = minutes, 'h' = hours, and 'd' = days. For example, "30m" (30 minutes), "2h" (2 hours), or "1d" (1 day).
    • Default Value: "1h" (1 hour)

stream Backend create Parameters

The following create parameters are specific to the stream backend and are passed as keyword arguments via the kw parameter:

  • shards — The number of stream shards to create.

    • Type: int
    • Requirement: Optional
    • Default Value: 1
    • Valid Values: A positive integer (>= 1). For example, 100.
  • retention_hours — The stream's retention period, in hours.

    • Type: int
    • Requirement: Optional
    • Default Value: 24
    • Valid Values: A positive integer (>= 1). For example, 2 (2 hours).

create Examples

tsdb Backend
client.create("tsdb", table="mytsdb", rate="10/m")
client.create("tsdb", table="/tsdb/my_metrics", rate="1/s", aggregates="count,avg,min,max", aggregation_granularity="1h")

stream Backend
client.create("stream", table="/mystream", shards=3)
client.create("stream", table="/my_streams/stream1", retention_hours=2)

write Method

Writes data from a DataFrame to a data collection, according to the specified backend type.

Syntax

write(backend, table, dfs, expression='', condition='', labels=None,
    max_rows_in_msg=0, index_cols=None, save_mode='createNewItemsOnly',
    partition_keys=None):

Note: The expression parameter isn't supported in the current release.

Common write Parameters

All Frames backends that support the write method support the following common parameters:

  • dfs — One or more DataFrames containing the data to write.

    • Type: A single DataFrame, a list of DataFrames, or a DataFrames iterator
    • Requirement: Required
  • index_cols — A list of column (attribute) names to be used as index columns for the write operation, regardless of any index-column definitions in the DataFrame. By default, the DataFrame's index columns are used.

    Note: The significance and supported number of index columns is backend specific. For example, the nosql backend supports only a single index column for the primary-key item attribute, while the tsdb backend supports additional index columns for metric labels.

    • Type: []str
    • Requirement: Optional
    • Default Value: None
  • labels — This parameter is currently applicable only to the tsdb backend (although it's available for all backends) and is therefore documented as part of the write method's tsdb backend parameters.

    • Type: dict
    • Requirement: Optional
  • save_mode — This parameter is currently applicable only to the nosql backend, and is therefore documented as part of the write method's nosql backend parameters.

    • Type: str
    • Requirement: Optional
  • max_rows_in_msg — Maximum number of rows to write in each message (write chunk size).

    • Type: int
    • Requirement: Optional
    • Default Value: 0

nosql Backend write Parameters

The following write parameters are specific to the nosql backend; for more information and examples, see the platform's Frames NoSQL-backend reference:

  • condition — A platform condition expression that defines conditions for performing the write operation.

    • Type: str
    • Requirement: Optional
  • save_mode — Save mode, which determines in which circumstances to write new item to the table.

    • Type: str
    • Requirement: Optional
    • Valid Values:
      • createNewItemsOnly — write only new items; don't replace or update any existing table item with the same name (primary-key attribute value) as a written item.
      • "updateItem" — update items; add new items and update the attributes of existing table items.
      • "overwriteItem" — overwrite items; add new items and replace any existing table item with the same name as a written item.
      • "errorIfTableExists" — create a new table only; only write items if the target table doesn't already exist.
      • "overwriteTable" — overwrite the table; replace all existing table items (if any) with the written items.
    • Default Value: createNewItemsOnly

tsdb Backend write Parameters

The following write parameter descriptions are specific to the tsdb backend; for more information and examples, see the platform's Frames TSDB-backend reference:

  • labels — A dictionary of metric labels of the format { to apply to all the DataFrame rows. For example, {"os": "linux", "arch": "x86"}.

    • Type: dict
    • Requirement: Optional
    • Default Value: None

write Examples

nosql Backend
data = [["tom", 10, "TLV"], ["nick", 15, "Berlin"], ["juli", 14, "NY"]]
df = pd.DataFrame(data, columns = ["name", "age", "city"])
df.set_index("name", inplace=True)
client.write(backend="nosql", table="mytable", dfs=df, condition="age>14")

tsdb Backend
from datetime import datetime
df = pd.DataFrame(data=[[30.1, 12.7]], index=[[datetime.now()], ["1"]],
                  columns=["cpu", "disk"])
df.index.names = ["time", "node"]
client.write(backend="tsdb", table="mytsdb", dfs=df)

stream Backend
import numpy as np
df = pd.DataFrame(np.random.rand(9, 3) * 100,
                  columns=["cpu", "mem", "disk"])
client.write("stream", table="mystream", dfs=df)

read Method

Reads data from a data collection to a DataFrame, according to the specified backend type.

Syntax

read(backend, table='', query='', columns=None, filter='', group_by='',
    limit=0, data_format='', row_layout=False, max_rows_in_msg=0, marker='',
    iterator=False, get_raw=False, **kw)

Note: The limit, data_format, row_layout, and marker parameters aren't supported in the current release, and get_raw is for internal use only.

Common read Parameters

All Frames backends that support the read method support the following common parameters:

  • iterator — set to True to to return a pandas DataFrames iterator; False (default) returns a single DataFrame.

    • Type: bool
    • Requirement: Optional
    • Default Value: False
  • filter — A query filter. For example, filter="col1=='my_value'".
    This parameter is currently applicable only to the nosql and tsdb backends, and cannot be used concurrently with the query parameter of the tsdb backend.

    • Type: str
    • Requirement: Optional
  • columns — A list of attributes (columns) to return.
    This parameter is currently applicable only to the nosql and tsdb backends, and cannot be used concurrently with the query parameter of the tsdb backend.

    • Type: []str
    • Requirement: Optional
  • kw — This parameter is used for passing a variable-length list of additional keyword (named) arguments. For more information, see the backend-specific method parameters.

    • Type: ** — variable-length keyword arguments list
    • Requirement: Optional

nosql Backend read Parameters

The following read parameters are specific to the nosql backend; for more information and examples, see the platform's Frames NoSQL-backend reference:

  • max_rows_in_msg — The maximum number of rows per message.

    • Type: int
    • Requirement: Optional

The following parameters are passed as keyword arguments via the kw parameter:

  • reset_index — Set to True to reset the index column of the returned DataFrame and use the auto-generated pandas range-index column; False (default) sets the index column to the table's primary-key attribute.

    • Type: bool
    • Requirement: Optional
    • Default Value: False
  • sharding_keys [Tech Preview] — A list of specific sharding keys to query, for range-scan formatted tables only.

    • Type: []str
    • Requirement: Optional

tsdb Backend read Parameters

The following read parameters are specific to the tsdb backend; for more information and examples, see the platform's Frames TSDB-backend reference:

  • group_by [Tech Preview] — A group-by query string.
    This parameter cannot be used concurrently with the query parameter.

    • Type: str
    • Requirement: Optional
  • query [Tech Preview] — A query string in SQL format.

    Note:

    • When setting the query parameter, you must provide the path to the TSDB table as part of the FROM clause in the query string and not in the read method's table parameter.
    • This parameter cannot be set concurrently with the following parameters: aggregators, columns, filter, or group_by parameters.
    • Type: str
    • Requirement: Optional

The following parameters are passed as keyword arguments via the kw parameter:

  • start — Start (minimum) time for the read operation.

    • Type: str
    • Requirement: Optional
    • Valid Values: A string containing an RFC 3339 time, a Unix timestamp in milliseconds, a relative time of the format "now" or "now-[0-9]+[mhd]" (where m = minutes, h = hours, and 'd' = days), or 0 for the earliest time. For example: "2016-01-02T15:34:26Z"; "1451748866"; "now-90m"; "0".
    • Default Value: - 1h
  • end — End (maximum) time for the read operation.

    • Type: str
    • Requirement: Optional
    • Valid Values: A string containing an RFC 3339 time, a Unix timestamp in milliseconds, a relative time of the format "now" or "now-[0-9]+[mhd]" (where m = minutes, h = hours, and 'd' = days), or 0 for the earliest time. For example: "2018-09-26T14:10:20Z"; "1537971006000"; "now-3h"; "now-7d".
    • Default Value: now
  • step — The query aggregation or downsampling step. The default step is the query's time range, which can be configured via the start and end parameters.

    • Type: str
    • Requirement: Optional
  • aggregators — Aggregation information to return, as a comma-separated list of supported aggregation functions ("aggregators").
    This parameter cannot be used concurrently with the query parameter.

    • Type: str
    • Requirement: Optional
    • Valid Value: The following aggregation functions are supported for over-time aggregation (across each unique label set); for cross-series aggregation (across all metric labels), add "_all" to the end of the function name:
      avg | count | last | max | min | rate | stddev | stdvar | sum
  • aggregation_window [Tech Preview] — Aggregation interval for applying over-time aggregation functions, if set in the aggregators or query parameters.

    • Type: str
    • Requirement: Optional
    • Valid Values: A string of the format "[0-9]+[mhd]" where 'm' = minutes, 'h' = hours, and 'd' = days. For example, "30m" (30 minutes), "2h" (2 hours), or "1d" (1 day).
    • Default Value: The query's aggregation step
  • multi_index — set to True to display labels as index columns in the read results; False (default) displays only the metric's sample time as an index column.

    • Type: bool
    • Requirement: Optional
    • Default Value: False

stream Backend read Parameters

The following read parameters are specific to the stream backend and are passed as keyword arguments via the kw parameter:

  • seek — Seek type.
    When the "seq" or "sequence" seek type is set, you must set the sequence parameter to the desired record sequence number.
    When the time seek type is set, you must set the start parameter to the desired seek start time.

    • Type: str
    • Requirement: Required
    • Valid Values: "time" | "seq" | "sequence" | "latest" | "earliest"
  • shard_id — The ID of the stream shard from which to read.

    • Type: str
    • Requirement: Required
    • Valid values: "0" ... " - 1"
  • sequence — The sequence number of the record from which to start reading.

    • Type: int64
    • Requirement: Required
  • start — The earliest record ingestion time from which to start reading.

    • Type: str
    • Requirement: Required when seek = "time"
    • Valid Values: A string containing an RFC 3339 time, a Unix timestamp in milliseconds, a relative time of the format "now" or "now-[0-9]+[mhd]" (where m = minutes, h = hours, and 'd' = days), or 0 for the earliest time. For example: "2016-01-02T15:34:26Z"; "1451748866"; "now-90m"; "0".

Return Value

  • When the value of the iterator parameter is False (default) — returns a single DataFrame.
  • When the value of the iterator parameter is True — returns a DataFrames iterator.

read Examples

nosql Backend
df = client.read(backend="nosql", table="mytable", filter="col1>666")

tsdb Backend
df = client.read("tsdb", table="mytsdb" start="0", multi_index=True)
df = client.read(backend="tsdb", query="select avg(cpu) as cpu, avg(disk) from 'mytsdb' where node='1'", start="now-1d", end="now", step="2h")

stream Backend
df = client.read(backend="stream", table="mystream", seek="latest", shard_id="5")

delete Method

Deletes a data collection or specific collection items, according to the specified backend type.

Syntax

delete(backend, table, filter='', start='', end='', if_missing=FAIL

Common delete Parameters

  • if_missing — Determines whether to raise an error when the specified collection (table) doesn't exist.

    • Type: pb.ErrorOptions enumeration. To use the enumeration, import the frames_pb2 module; for example:

      from v3io_frames import frames_pb2 as fpb
    • Requirement: Optional
    • Valid Values: FAIL to raise an error when the specified collection doesn't exist; IGNORE to ignore this
    • Default Value: FAIL

nosql Backend delete Parameters

The following delete parameters are specific to the nosql backend; for more information and examples, see the platform's Frames NoSQL-backend reference:

  • filter — A filter expression that identifies specific items to delete.

    • Type: str
    • Requirement: Optional
    • Default Value: "" — delete the entire table and its schema file

tsdb Backend delete Parameters

The following delete parameters are specific to the tsdb backend; for more information and examples, see the platform's Frames TSDB-backend reference:

  • start — Start (minimum) time for the delete operation — i.e., delete only items whose data sample time is at or after (>=) the specified start time.

    • Type: str
    • Requirement: Optional
    • Valid Values: A string containing an RFC 3339 time, a Unix timestamp in milliseconds, a relative time of the format "now" or "now-[0-9]+[mhd]" (where m = minutes, h = hours, and 'd' = days), or 0 for the earliest time. For example: "2016-01-02T15:34:26Z"; "1451748866"; "now-90m"; "0".
    • Default Value: "" when neither start nor end are set — delete the entire table and its schema file (.schema); 0 when end is set
  • endstr — End (maximum) time for the delete operation — i.e., delete only items whose data sample time is before or at (<=) the specified end time.

    • Type: str
    • Requirement: Optional
    • Valid Values: A string containing an RFC 3339 time, a Unix timestamp in milliseconds, a relative time of the format "now" or "now-[0-9]+[mhd]" (where m = minutes, h = hours, and 'd' = days), or 0 for the earliest time. For example: "2018-09-26T14:10:20Z"; "1537971006000"; "now-3h"; "now-7d".
    • Default Value: "" when neither start nor end are set — delete the entire table and its schema file (.schema); 0 when start is set

Note:

  • When neither the start nor end parameters are set, the entire TSDB table and its schema file are deleted.
  • Only full table partitions within the specified time frame (as determined by the start and end parameters) are deleted. Items within the specified time frames that reside within partitions that begin before the delete start time or end after the delete end time aren't deleted. The partition interval is calculated automatically based on the table's ingestion rate and is stored in the TSDB's partitionerInterval schema field (see the .schema file).

delete Examples

nosql Backend
client.delete(backend="nosql", table="mytable", filter="age > 40")

tsdb Backend
client.delete(backend="tsdb", table="mytsdb", start="now-1d", end="now-5h")

stream Backend
from v3io_frames import frames_pb2 as fpb
client.delete(backend="stream", table="mystream", if_missing=fpb.IGNORE)

execute Method

Extends the basic CRUD functionality of the other client methods via backend-specific commands for performing operations on a data collection.

Note: Currently, no execute commands are available for the tsdb backend.

Syntax

execute(backend, table, command="", args=None)

Common execute Parameters

All Frames backends that support the execute method support the following common parameters:

  • command — The command to execute.

    • Type: str
    • Requirement: Required
    • Valid Values: Backend-specific
  • args — A dictionary of : pairs for passing command-specific parameters (arguments).

    • Type: dict
    • Requirement and Valid Values: Backend-specific
    • Default Value: None

nosql Backend execute Commands

The following execute commands are specific to the nosql backend; for more information and examples, see the platform's Frames NoSQL-backend reference:

  • infer | infer_schema — Infers the data schema of a given NoSQL table and creates a schema file for the table.

    Example:

    client.execute(backend="nosql", table="mytable", command="infer")

stream Backend execute Commands

The following execute commands are specific to the stream backend:

  • put — Adds records to a stream shard.

    Example:

    client.execute('stream', table="mystream", command='put',
                   args={'data': '{"cpu": 12.4, "mem": 31.1, "disk": 12.7}',
                         "client_info": "my custom info", "partition_key": "PK1"})

history Method

By default every command ran with frames is logged.
History returns information about requests made to the service as a pandas DataFrame.

Syntax

history(backend='', container='', table='', user='', action='', min_start_time='', max_start_time='',
min_duration=0, max_duration=0):

Common history Parameters

  • backend — filter logs by backend.

    • Type: str
    • Requirement: Optional
  • container — filter logs by container.

    • Type: str
    • Requirement: Optional
  • table — filter logs by table.

    • Type: str
    • Requirement: Optional
  • user — filter logs by the user that executed the command.

    • Type: str
    • Requirement: Optional
  • action — filter logs by frames action.

    • Type: str
    • Requirement: Optional
    • Valid Values: "create" | "delete" | "execute" | "read" | "write"
  • min_start_time — specify start time of the desired logs.

    • Type: str
    • Requirement: Optional
    • Valid Values: A string containing an RFC 3339 time, a Unix timestamp in milliseconds, a relative time of the format "now" or "now-[0-9]+[mhd]" (where m = minutes, h = hours, and 'd' = days), or 0 for the earliest time. For example: "2016-01-02T15:34:26Z"; "1451748866"; "now-90m"; "0".
  • max_start_time — specify end time of the desired logs.

    • Type: str
    • Requirement: Optional
    • Valid Values: A string containing an RFC 3339 time, a Unix timestamp in milliseconds, a relative time of the format "now" or "now-[0-9]+[mhd]" (where m = minutes, h = hours, and 'd' = days), or 0 for the earliest time. For example: "2016-01-02T15:34:26Z"; "1451748866"; "now-90m"; "0".
  • min_duration — specify minimum duration in milliseconds for the desired logs.

    • Type: int
    • Requirement: Optional
  • max_duration — specify maximum duration in milliseconds for the desired logs.

    • Type: int
    • Requirement: Optional

Return Value

Returns a single DataFrame.

history Examples

df = client.history()
df = client.history(backend='tsdb', action='read')
df = client.history(container='test-0', min_start_time='now-1d', min_duration='50')

Contributing

To contribute to V3IO Frames, you need to be aware of the following:

Components

The following components are required for building Frames code:

  • Go server with support for both the gRPC and HTTP protocols
  • Go client
  • Python client

Development

The core is written in Go. The development is done on the development branch and then released to the master branch.

Before submitting changes, test the code:

  • To execute the Go tests, run make test.
  • To execute the Python tests, run make test-python.

Adding and Changing Dependencies

  • If you add Go dependencies, run make update-go-deps.
  • If you add Python dependencies, update clients/py/Pipfile and run make update-py-deps.

Travis CI

Integration tests are run on Travis CI. See .travis.yml for details.

The following environment variables are defined in the Travis settings:

  • Docker Container Registry (Quay.io)
    • DOCKER_PASSWORD — a password for pushing images to Quay.io.
    • DOCKER_USERNAME — a username for pushing images to Quay.io.
  • Python Package Index (PyPI)
    • V3IO_PYPI_PASSWORD — a password for pushing a new release to PyPi.
    • V3IO_PYPI_USER — a username for pushing a new release to PyPi.
  • Iguazio Data Science Platform
    • V3IO_SESSION — a JSON encoded map with session information for running tests. For example:

      '{"url":"45.39.128.5:8081","container":"mitzi","user":"daffy","password":"rabbit season"}'
      

      Note: Make sure to embed the JSON object within single quotes ('{...}').

Docker Image

Building the Image

Use the following command to build the Docker image:

make build-docker

Running the Image

Use the following command to run the Docker image:

docker run \
	-v /path/to/config.yaml:/etc/framesd.yaml \
	quay.io/v3io/frames:unstable

LICENSE

Apache 2

Comments
  • Support Rapids cudf

    Support Rapids cudf

    We'd like to support using Rapids cudf in frames. Both on read and write.

    Support

    • [x] Categoracial data
    • [ ] strings (see here)
    • [x] Read/write
    • Run on Iguazio platform
    python cudf 
    opened by tebeka 12
  • "pip install v3io_frames" fails

    @yaronha reported:

    Collecting v3io_frames
      Downloading https://files.pythonhosted.org/packages/3c/fb/cbba02edf998747d7065a8a0121c26a2ac727cf7d48cdefefdf8d8fa0af0/v3io_frames-0.3.3-py3-none-any.whl
    Requirement already satisfied: backcall==0.1.0 in /conda/lib/python3.6/site-packages (from v3io_frames) (0.1.0)
    Requirement already satisfied: webencodings==0.5.1 in /conda/lib/python3.6/site-packages (from v3io_frames) (0.5.1)
    Requirement already satisfied: nbconvert==5.4.0 in /conda/lib/python3.6/site-packages (from v3io_frames) (5.4.0)
    Collecting python-dateutil==2.7.5 (from v3io_frames)
      Downloading https://files.pythonhosted.org/packages/74/68/d87d9b36af36f44254a8d512cbfc48369103a3b9e474be9bdfe536abfc45/python_dateutil-2.7.5-py2.py3-none-any.whl (225kB)
        100% |################################| 235kB 37.5MB/s 
    Requirement already satisfied: pexpect==4.6.0 in /conda/lib/python3.6/site-packages (from v3io_frames) (4.6.0)
    Requirement already satisfied: wcwidth==0.1.7 in /conda/lib/python3.6/site-packages (from v3io_frames) (0.1.7)
    Requirement already satisfied: parso==0.3.1 in /conda/lib/python3.6/site-packages (from v3io_frames) (0.3.1)
    Requirement already satisfied: pickleshare==0.7.5 in /conda/lib/python3.6/site-packages (from v3io_frames) (0.7.5)
    Requirement already satisfied: entrypoints==0.2.3 in /conda/lib/python3.6/site-packages (from v3io_frames) (0.2.3)
    Requirement already satisfied: jedi==0.13.1 in /conda/lib/python3.6/site-packages (from v3io_frames) (0.13.1)
    Requirement already satisfied: tornado==5.1.1 in /conda/lib/python3.6/site-packages (from v3io_frames) (5.1.1)
    Collecting grpcio==1.16.1 (from v3io_frames)
      Downloading https://files.pythonhosted.org/packages/0e/4f/e9e84e4600c43cae7ce58489c6e73ff4c864557bc4d4d0f0029c79e07f31/grpcio-1.16.1-cp36-cp36m-manylinux1_x86_64.whl (9.7MB)
        100% |################################| 9.7MB 7.0MB/s 
    Collecting googleapis-common-protos==1.5.5 (from v3io_frames)
      Downloading https://files.pythonhosted.org/packages/ae/94/a256572abf5d10347301c638d5df552fab1515497270726e6b56698d2e99/googleapis-common-protos-1.5.5.tar.gz
    Collecting grpcio-tools==1.16.1 (from v3io_frames)
      Downloading https://files.pythonhosted.org/packages/9a/30/b7e6dd587c68199835cf863663882dd2dccb0c0ce17c170123bc047ba7d8/grpcio_tools-1.16.1-cp36-cp36m-manylinux1_x86_64.whl (22.8MB)
        100% |################################| 22.8MB 3.4MB/s 
    Collecting requests==2.20.1 (from v3io_frames)
      Downloading https://files.pythonhosted.org/packages/ff/17/5cbb026005115301a8fb2f9b0e3e8d32313142fe8b617070e7baad20554f/requests-2.20.1-py2.py3-none-any.whl (57kB)
        100% |################################| 61kB 33.2MB/s 
    Requirement already satisfied: ipython-genutils==0.2.0 in /conda/lib/python3.6/site-packages (from v3io_frames) (0.2.0)
    Collecting certifi==2018.11.29 (from v3io_frames)
      Downloading https://files.pythonhosted.org/packages/9f/e0/accfc1b56b57e9750eba272e24c4dddeac86852c2bebd1236674d7887e8a/certifi-2018.11.29-py2.py3-none-any.whl (154kB)
        100% |################################| 163kB 36.5MB/s 
    Collecting prompt-toolkit==2.0.7 (from v3io_frames)
      Downloading https://files.pythonhosted.org/packages/d1/e6/adb3be5576f5d27c6faa33f1e9fea8fe5dbd9351db12148de948507e352c/prompt_toolkit-2.0.7-py3-none-any.whl (338kB)
        100% |################################| 348kB 46.5MB/s 
    Requirement already satisfied: pandas==0.23.4 in /conda/lib/python3.6/site-packages (from v3io_frames) (0.23.4)
    Requirement already satisfied: protobuf==3.6.1 in /conda/lib/python3.6/site-packages (from v3io_frames) (3.6.1)
    Requirement already satisfied: mistune==0.8.4 in /conda/lib/python3.6/site-packages (from v3io_frames) (0.8.4)
    Collecting ipython==7.2.0 (from v3io_frames)
      Downloading https://files.pythonhosted.org/packages/f0/b4/a9ea018c73a84ee6280b2e94a1a6af8d63e45903eac2da0640fa63bca4db/ipython-7.2.0-py3-none-any.whl (765kB)
        100% |################################| 768kB 31.9MB/s 
    Requirement already satisfied: terminado==0.8.1 in /conda/lib/python3.6/site-packages (from v3io_frames) (0.8.1)
    Collecting notebook==5.7.2 (from v3io_frames)
      Downloading https://files.pythonhosted.org/packages/a2/5d/d1907cd32ac00b5ead56f6e61d9794fa60ef105a22ac5da6e7556011580f/notebook-5.7.2-py2.py3-none-any.whl (9.0MB)
        100% |################################| 9.0MB 11.5MB/s 
    Collecting numpy==1.15.4 (from v3io_frames)
      Downloading https://files.pythonhosted.org/packages/ff/7f/9d804d2348471c67a7d8b5f84f9bc59fd1cefa148986f2b74552f8573555/numpy-1.15.4-cp36-cp36m-manylinux1_x86_64.whl (13.9MB)
        100% |################################| 13.9MB 7.6MB/s 
    Collecting pygments==2.3.0 (from v3io_frames)
      Downloading https://files.pythonhosted.org/packages/fc/41/4f900a7852e25bb9350b4e3ee8c4aba0ee32abefd401456962b25f954823/Pygments-2.3.0-py2.py3-none-any.whl (845kB)
        100% |################################| 849kB 35.5MB/s 
    Requirement already satisfied: idna==2.7 in /conda/lib/python3.6/site-packages (from v3io_frames) (2.7)
    Requirement already satisfied: ipykernel==5.1.0 in /conda/lib/python3.6/site-packages (from v3io_frames) (5.1.0)
    Requirement already satisfied: jsonschema==2.6.0 in /conda/lib/python3.6/site-packages (from v3io_frames) (2.6.0)
    Requirement already satisfied: nbformat==4.4.0 in /conda/lib/python3.6/site-packages (from v3io_frames) (4.4.0)
    Requirement already satisfied: defusedxml==0.5.0 in /conda/lib/python3.6/site-packages (from v3io_frames) (0.5.0)
    Collecting markupsafe==1.1.0 (from v3io_frames)
      Downloading https://files.pythonhosted.org/packages/08/04/f2191b50fb7f0712f03f064b71d8b4605190f2178ba02e975a87f7b89a0d/MarkupSafe-1.1.0-cp36-cp36m-manylinux1_x86_64.whl
    Requirement already satisfied: send2trash==1.5.0 in /conda/lib/python3.6/site-packages (from v3io_frames) (1.5.0)
    Requirement already satisfied: ptyprocess==0.6.0 in /conda/lib/python3.6/site-packages (from v3io_frames) (0.6.0)
    Requirement already satisfied: six==1.11.0 in /conda/lib/python3.6/site-packages (from v3io_frames) (1.11.0)
    Requirement already satisfied: chardet==3.0.4 in /conda/lib/python3.6/site-packages (from v3io_frames) (3.0.4)
    Requirement already satisfied: pyzmq==17.1.2 in /conda/lib/python3.6/site-packages (from v3io_frames) (17.1.2)
    Requirement already satisfied: jupyter-core==4.4.0 in /conda/lib/python3.6/site-packages (from v3io_frames) (4.4.0)
    Collecting urllib3==1.24.1 (from v3io_frames)
      Downloading https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl (118kB)
        100% |################################| 122kB 47.2MB/s 
    Requirement already satisfied: testpath==0.4.2 in /conda/lib/python3.6/site-packages (from v3io_frames) (0.4.2)
    Collecting pytz==2018.7 (from v3io_frames)
      Downloading https://files.pythonhosted.org/packages/f8/0e/2365ddc010afb3d79147f1dd544e5ee24bf4ece58ab99b16fbb465ce6dc0/pytz-2018.7-py2.py3-none-any.whl (506kB)
        100% |################################| 512kB 39.9MB/s 
    Requirement already satisfied: traitlets==4.3.2 in /conda/lib/python3.6/site-packages (from v3io_frames) (4.3.2)
    Requirement already satisfied: jupyter-client==5.2.3 in /conda/lib/python3.6/site-packages (from v3io_frames) (5.2.3)
    Requirement already satisfied: prometheus-client==0.4.2 in /conda/lib/python3.6/site-packages (from v3io_frames) (0.4.2)
    Requirement already satisfied: decorator==4.3.0 in /conda/lib/python3.6/site-packages (from v3io_frames) (4.3.0)
    Collecting bleach==3.0.2 (from v3io_frames)
      Downloading https://files.pythonhosted.org/packages/d4/0d/4696373c3b714f6022d668fbab619690a42050dbeacede6d10ed97fbd3e2/bleach-3.0.2-py2.py3-none-any.whl (148kB)
        100% |################################| 153kB 49.3MB/s 
    Requirement already satisfied: pandocfilters==1.4.2 in /conda/lib/python3.6/site-packages (from v3io_frames) (1.4.2)
    Requirement already satisfied: jinja2==2.10 in /conda/lib/python3.6/site-packages (from v3io_frames) (2.10)
    Requirement already satisfied: setuptools in /conda/lib/python3.6/site-packages (from protobuf==3.6.1->v3io_frames) (40.4.3)
    Building wheels for collected packages: googleapis-common-protos
      Running setup.py bdist_wheel for googleapis-common-protos ... done
      Stored in directory: /igz/.cache/pip/wheels/22/57/2c/fdf1a5d9f14360bd1123c6dbac26eb784eba25223800239a69
    Successfully built googleapis-common-protos
    tensorflow-tensorboard 1.5.1 has requirement bleach==1.5.0, but you'll have bleach 3.0.2 which is incompatible.
    Installing collected packages: python-dateutil, grpcio, googleapis-common-protos, grpcio-tools, urllib3, certifi, requests, prompt-toolkit, pygments, ipython, notebook, numpy, markupsafe, pytz, bleach, v3io-frames
      Found existing installation: python-dateutil 2.7.3
        Uninstalling python-dateutil-2.7.3:
          Successfully uninstalled python-dateutil-2.7.3
      Found existing installation: urllib3 1.23
        Uninstalling urllib3-1.23:
          Successfully uninstalled urllib3-1.23
      Found existing installation: certifi 2018.10.15
    Cannot uninstall 'certifi'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
    
    python 
    opened by tebeka 4
  • [DOC-REVIEW] Frames README & help text (#3): Overview, user-auth, Client constructor, Create, and Write params doc imprv; misc cosmetics

    [DOC-REVIEW] Frames README & help text (#3): Overview, user-auth, Client constructor, Create, and Write params doc imprv; misc cosmetics

    The current plan (approved by Adi) is to integrate the Frames doc into the Iguazio Data Science Platform doc, finish the doc-review edits as part of the platform doc, and replace the doc in the Frames GitHub repo with a reference to the platform doc. But in the meantime, it would be good to merge the doc-review updates already prepared for the Frames repo in this PR.

    opened by Sharon-iguazio 2
  • Client should hold session information

    Client should hold session information

    Currently, the server holds the connection information per backend. We'd like to change that and have the client pass connection information with every request (very much like regular database driver).

    opened by tebeka 2
  • Bump py from 1.8.2 to 1.10.0 in /clients/py

    Bump py from 1.8.2 to 1.10.0 in /clients/py

    Bumps py from 1.8.2 to 1.10.0.

    Changelog

    Sourced from py's changelog.

    1.10.0 (2020-12-12)

    • Fix a regular expression DoS vulnerability in the py.path.svnwc SVN blame functionality (CVE-2020-29651)
    • Update vendored apipkg: 1.4 => 1.5
    • Update vendored iniconfig: 1.0.0 => 1.1.1

    1.9.0 (2020-06-24)

    • Add type annotation stubs for the following modules:

      • py.error
      • py.iniconfig
      • py.path (not including SVN paths)
      • py.io
      • py.xml

      There are no plans to type other modules at this time.

      The type annotations are provided in external .pyi files, not inline in the code, and may therefore contain small errors or omissions. If you use py in conjunction with a type checker, and encounter any type errors you believe should be accepted, please report it in an issue.

    Commits
    • e5ff378 Update CHANGELOG for 1.10.0
    • 94cf44f Update vendored libs
    • 5e8ded5 testing: comment out an assert which fails on Python 3.9 for now
    • afdffcc Rename HOWTORELEASE.rst to RELEASING.rst
    • 2de53a6 Merge pull request #266 from nicoddemus/gh-actions
    • fa1b32e Merge pull request #264 from hugovk/patch-2
    • 887d6b8 Skip test_samefile_symlink on pypy3 on Windows
    • e94e670 Fix test_comments() in test_source
    • fef9a32 Adapt test
    • 4a694b0 Add GitHub Actions badge to README
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 1
  • Bump pygments from 2.6.1 to 2.7.4 in /clients/py

    Bump pygments from 2.6.1 to 2.7.4 in /clients/py

    Bumps pygments from 2.6.1 to 2.7.4.

    Release notes

    Sourced from pygments's releases.

    2.7.4

    • Updated lexers:

      • Apache configurations: Improve handling of malformed tags (#1656)

      • CSS: Add support for variables (#1633, #1666)

      • Crystal (#1650, #1670)

      • Coq (#1648)

      • Fortran: Add missing keywords (#1635, #1665)

      • Ini (#1624)

      • JavaScript and variants (#1647 -- missing regex flags, #1651)

      • Markdown (#1623, #1617)

      • Shell

        • Lex trailing whitespace as part of the prompt (#1645)
        • Add missing in keyword (#1652)
      • SQL - Fix keywords (#1668)

      • Typescript: Fix incorrect punctuation handling (#1510, #1511)

    • Fix infinite loop in SML lexer (#1625)

    • Fix backtracking string regexes in JavaScript/TypeScript, Modula2 and many other lexers (#1637)

    • Limit recursion with nesting Ruby heredocs (#1638)

    • Fix a few inefficient regexes for guessing lexers

    • Fix the raw token lexer handling of Unicode (#1616)

    • Revert a private API change in the HTML formatter (#1655) -- please note that private APIs remain subject to change!

    • Fix several exponential/cubic-complexity regexes found by Ben Caller/Doyensec (#1675)

    • Fix incorrect MATLAB example (#1582)

    Thanks to Google's OSS-Fuzz project for finding many of these bugs.

    2.7.3

    ... (truncated)

    Changelog

    Sourced from pygments's changelog.

    Version 2.7.4

    (released January 12, 2021)

    • Updated lexers:

      • Apache configurations: Improve handling of malformed tags (#1656)

      • CSS: Add support for variables (#1633, #1666)

      • Crystal (#1650, #1670)

      • Coq (#1648)

      • Fortran: Add missing keywords (#1635, #1665)

      • Ini (#1624)

      • JavaScript and variants (#1647 -- missing regex flags, #1651)

      • Markdown (#1623, #1617)

      • Shell

        • Lex trailing whitespace as part of the prompt (#1645)
        • Add missing in keyword (#1652)
      • SQL - Fix keywords (#1668)

      • Typescript: Fix incorrect punctuation handling (#1510, #1511)

    • Fix infinite loop in SML lexer (#1625)

    • Fix backtracking string regexes in JavaScript/TypeScript, Modula2 and many other lexers (#1637)

    • Limit recursion with nesting Ruby heredocs (#1638)

    • Fix a few inefficient regexes for guessing lexers

    • Fix the raw token lexer handling of Unicode (#1616)

    • Revert a private API change in the HTML formatter (#1655) -- please note that private APIs remain subject to change!

    • Fix several exponential/cubic-complexity regexes found by Ben Caller/Doyensec (#1675)

    • Fix incorrect MATLAB example (#1582)

    Thanks to Google's OSS-Fuzz project for finding many of these bugs.

    Version 2.7.3

    (released December 6, 2020)

    ... (truncated)

    Commits
    • 4d555d0 Bump version to 2.7.4.
    • fc3b05d Update CHANGES.
    • ad21935 Revert "Added dracula theme style (#1636)"
    • e411506 Prepare for 2.7.4 release.
    • 275e34d doc: remove Perl 6 ref
    • 2e7e8c4 Fix several exponential/cubic complexity regexes found by Ben Caller/Doyensec
    • eb39c43 xquery: fix pop from empty stack
    • 2738778 fix coding style in test_analyzer_lexer
    • 02e0f09 Added 'ERROR STOP' to fortran.py keywords. (#1665)
    • c83fe48 support added for css variables (#1633)
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 1
  • Bump pyyaml from 5.3.1 to 5.4 in /clients/py

    Bump pyyaml from 5.3.1 to 5.4 in /clients/py

    Bumps pyyaml from 5.3.1 to 5.4.

    Changelog

    Sourced from pyyaml's changelog.

    5.4 (2021-01-19)

    Commits
    • 58d0cb7 5.4 release
    • a60f7a1 Fix compatibility with Jython
    • ee98abd Run CI on PR base branch changes
    • ddf2033 constructor.timezone: _copy & deepcopy
    • fc914d5 Avoid repeatedly appending to yaml_implicit_resolvers
    • a001f27 Fix for CVE-2020-14343
    • fe15062 Add 3.9 to appveyor file for completeness sake
    • 1e1c7fb Add a newline character to end of pyproject.toml
    • 0b6b7d6 Start sentences and phrases for capital letters
    • c976915 Shell code improvements
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 1
  • Bump jinja2 from 2.11.2 to 2.11.3 in /clients/py

    Bump jinja2 from 2.11.2 to 2.11.3 in /clients/py

    Bumps jinja2 from 2.11.2 to 2.11.3.

    Release notes

    Sourced from jinja2's releases.

    2.11.3

    This contains a fix for a speed issue with the urlize filter. urlize is likely to be called on untrusted user input. For certain inputs some of the regular expressions used to parse the text could take a very long time due to backtracking. As part of the fix, the email matching became slightly stricter. The various speedups apply to urlize in general, not just the specific input cases.

    Changelog

    Sourced from jinja2's changelog.

    Version 2.11.3

    Released 2021-01-31

    • Improve the speed of the urlize filter by reducing regex backtracking. Email matching requires a word character at the start of the domain part, and only word characters in the TLD. :pr:1343
    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 1
  • Bump bleach from 3.1.5 to 3.3.0 in /clients/py

    Bump bleach from 3.1.5 to 3.3.0 in /clients/py

    Bumps bleach from 3.1.5 to 3.3.0.

    Changelog

    Sourced from bleach's changelog.

    Version 3.3.0 (February 1st, 2021)

    Backwards incompatible changes

    • clean escapes HTML comments even when strip_comments=False

    Security fixes

    • Fix bug 1621692 / GHSA-m6xf-fq7q-8743. See the advisory for details.

    Features

    None

    Bug fixes

    None

    Version 3.2.3 (January 26th, 2021)

    Security fixes

    None

    Features

    None

    Bug fixes

    • fix clean and linkify raising ValueErrors for certain inputs. Thank you @Google-Autofuzz.

    Version 3.2.2 (January 20th, 2021)

    Security fixes

    None

    Features

    • Migrate CI to Github Actions. Thank you @hugovk.

    Bug fixes

    • fix linkify raising an IndexError on certain inputs. Thank you @Google-Autofuzz.

    Version 3.2.1 (September 18th, 2020)

    ... (truncated)

    Commits
    • 79b7a3c Merge pull request from GHSA-vv2x-vrpj-qqpq
    • 842fcb4 Update for v3.3.0 release
    • 1334134 sanitizer: escape HTML comments
    • c045a8b Merge pull request #581 from mozilla/nit-fixes
    • 491abb0 fix typo s/vnedoring/vendoring/
    • 10b1c5d vendor: add html5lib-1.1.dist-info/REQUESTED
    • cd838c3 Merge pull request #579 from mozilla/validate-convert-entity-code-points
    • 612b808 Update for v3.2.3 release
    • 6879f6a html5lib_shim: validate unicode points for convert_entity
    • 90cb80b Update for v3.2.2 release
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 1
  • [DOC] README: Update links to the Iguazio doc site in light of the move to separate versioned doc site

    [DOC] README: Update links to the Iguazio doc site in light of the move to separate versioned doc site

    <base URL>/<section>/<version>/ > <base URL>/<version>/<section>/ (The old URLs still work because we added redirect rules, but it's better to link to the new URLs.)

    opened by Sharon-iguazio 1
  • Bump github.com/valyala/fasthttp from 1.2.0 to 1.34.0

    Bump github.com/valyala/fasthttp from 1.2.0 to 1.34.0

    Bumps github.com/valyala/fasthttp from 1.2.0 to 1.34.0.

    Release notes

    Sourced from github.com/valyala/fasthttp's releases.

    v1.34.0

    • 59f94a3 Update github.com/klauspost/compress (#1237) (Mikhail Faraponov)
    • 62c15a5 Don't reset RequestCtx.s (#1234) (Erik Dubbelboer)
    • 7670c6e Fix windows tests (#1235) (Erik Dubbelboer)
    • f54ffa1 feature: Keep the memory usage of the service at a stable level (#1216) (Rennbon)
    • 15262ec Warn about unsafe ServeFile usage (#1228) (Erik Dubbelboer)
    • 1116d03 Fix panic while reading invalid trailers (Erik Dubbelboer)
    • 856ca8e Update dependencies (#1230) (Mikhail Faraponov)
    • 6b5bc7b Add windows support to normalizePath (Erik Dubbelboer)
    • f0b0cfe Don't log ErrBadTrailer by default (Erik Dubbelboer)
    • 6937fee fix: (useless check), skip Response body if http method HEAD (#1224) (Pavel Burak)
    • b85d2a2 Fix http proxy behavior (#1221) (Aoang)
    • ad8a07a RequestHeader support set no default ContentType (#1218) (Jack.Ju)
    • c94581c support configure HostClient (#1214) (lin longhjui)
    • 632e222 Client examples (#1208) (Sergey Ponomarev)
    • 6a3cc23 uri_test.go use example.com for clearness (#1212) (Sergey Ponomarev)
    • 9d665e0 Update dependencies (#1204) (Mikhail Faraponov)
    • 8d7953e Fix scheme check for not yet parsed requests (#1203) (ArminBTVS)

    v1.33.0

    • 61aa8b1 remove redundant code (#1202) (tyltr)
    • 4369776 fix(hijack): reuse RequestCtx (#1201) (Sergio VS)
    • 2aca3e8 fix(hijack): reset userValues after hijack handler execution (#1199) (Sergio VS)
    • 9123060 Updated dependencies (#1194) (Mikhail Faraponov)

    v1.32.0

    • 7eeb00e Make tests less flaky (#1189) (Erik Dubbelboer)
    • d19b872 Update tcpdialer.go (#1188) (Mikhail Faraponov)
    • c727b99 Release UseHostHeader in ReleaseRequest() (#1185) (Tolyar)
    • 6c0518b Fix UseHostHeader for DoTimeout + tests (#1184) (Tolyar)
    • 6b55811 Add MaxIdleWorkerDuration to Server. (#1183) (Kilos Liu)
    • 4517204 Allow to set Host header for Client (#1169) (Tolyar)
    • 258a4c1 fix: reset response after reset user values on keep-alive connections (#1176) (Sergio VS)
    • e9db537 Use %w to wrap errors (#1175) (Erik Dubbelboer)
    • 7db0597 Fix bad request trailer panic (Erik Dubbelboer)
    • 4aadf9a Fix parseTrailer panic (Erik Dubbelboer)
    • da7ff7a Add trailer support (#1165) (ichx)
    • 017f0aa fix: reset request after reset user values on keep-alive connections (#1162) (Sergio VS)
    • 3b117f8 feat: close idle connections when server shutdown (#1155) (ichx)
    • a94a2c3 Remove redundant code (#1154) (ichx)
    • f7c354c Fix race condition in Client.mCleaner (Erik Dubbelboer)
    • c078a9d Add string and bytes buffer convert trick in README (#1151) (ichx)
    • 3ff6aaa uri: isHttps() and isHttp() (#1150) (Sergey Ponomarev)
    • 8febad0 http.go: Request.SetURI() (Fix #1141) (#1148) (Sergey Ponomarev)
    • 2ca01c7 fix: Status Line parsing and writing (#1135) (Shivansh Vij)
    • 931d0a4 Fix lint (Erik Dubbelboer)
    • d613502 use sync.map is better (#1145) (halst)
    • c15e642 Don't run all race tests on windows (#1143) (Erik Dubbelboer)
    • 6006c87 chore (#1137) (tyltr)
    • 6d4db9b Fix race condition in getTCPAddrs (Erik Dubbelboer)

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    go dependencies 
    opened by dependabot[bot] 0
  • Feature: support Dask Dataframes for larger than memory returns

    Feature: support Dask Dataframes for larger than memory returns

    Would it be possible to not only support a pandas dataframe serialization/deserialization for the return, but also a dask dataframe for when the v3io return is much larger than memory?

    I was thinking that perhaps if we are reading chunks of any of these data sources into in memory arrow, then could serialize to parquet (I would prefer pure arrow files, but dask doesn't support that right now), and finally read those into a dask dataframe with read_parquet. Would love more thoughts on this, so we could support larger than memory dataframe operations.

    opened by ihs-nick 0
  • travis upload to github fails on wrong credentials

    travis upload to github fails on wrong credentials

    See this job for example:

    /home/travis/.rvm/gems/ruby-2.4.1/gems/octokit-4.6.2/lib/octokit/response/raise_error.rb:16:in `on_complete': GET https://api.github.com/user: 401 - Bad credentials // See: https://developer.github.com/v3 (Octokit::Unauthorized)
    
    opened by tebeka 0
Releases(v0.10.2)
Owner
null
protoc-gen-grpc-gateway-ts is a Typescript client generator for the grpc-gateway project. It generates idiomatic Typescript clients that connect the web frontend and golang backend fronted by grpc-gateway.

protoc-gen-grpc-gateway-ts protoc-gen-grpc-gateway-ts is a Typescript client generator for the grpc-gateway project. It generates idiomatic Typescript

gRPC Ecosystem 94 Dec 19, 2022
A major platform Remote Access Terminal Tool based by Blockchain/P2P.

NGLite A major platform Remote Access Terminal Tool based by Blockchain/P2P. No public IP address required.More anonymity Example Detection Warning!!!

null 323 Jan 2, 2023
Access more HTTP ports over CDN with this application.

More-Ports More Ports is a proxy service to establish all web-based applications on different ports on the server-side over a well known TCP port. It

Ahmet ÖZER 4 May 8, 2022
Fast time-series data storage server accessible over gRPC

tstorage-server Persistent fast time-series data storage server accessible over gRPC. tstorage-server is lightweight local on-disk storage engine serv

Bartlomiej Mika 6 Sep 14, 2022
Serve traffic (HTTP/gRPC) over SSH using Domain Sockets

Serve On SSH Introduction There is often a need to offer services for administrative purposes on servers or even for microservices that are running on

John Doak 8 Nov 10, 2022
Compiler as a Service is a compiler that is available over http/https and gRPC

BlakBoks(CaaS) Elasticsearch but for compiling untrusted code Compiler as a Service is a compiler that is available over http/2 and gRPC. Setup First

Nathan Kimutai 0 Nov 24, 2021
grpc-http1: A gRPC via HTTP/1 Enabling Library for Go

grpc-http1: A gRPC via HTTP/1 Enabling Library for Go This library enables using all the functionality of a gRPC server even if it is exposed behind a

StackRox 89 Dec 17, 2022
A remote access tool & CNC

⚠️ ⚠️ Disclaimer just use this with good intentions ⚠️ ⚠️ An useless rat (remote acces tool in develop) web client you want to use it? download pairat

pai 16 Dec 14, 2022
A remote access tool & CNC

⚠️ ⚠️ Disclaimer just use this with good intentions ⚠️ ⚠️ An useless rat (remote acces tool in develop) web client you want to use it? download pairat

pai 16 Dec 14, 2022
An event driven remote access trojan for experimental purposes.

erat An event driven remote access trojan for experimental purposes. This example is very simple and leverages ssh failed login events to trigger erat

siovador 0 Jan 16, 2022
A Go package for sending and receiving ethernet frames. Currently supporting Linux, Freebsd, and OS X.

ether ether is a go package for sending and receiving ethernet frames. Currently supported platform: BPF based OS X FreeBSD AF_PACKET based Linux Docu

Song Gao 78 Sep 27, 2022
Package ethernet implements marshaling and unmarshaling of IEEE 802.3 Ethernet II frames and IEEE 802.1Q VLAN tags. MIT Licensed.

ethernet Package ethernet implements marshaling and unmarshaling of IEEE 802.3 Ethernet II frames and IEEE 802.1Q VLAN tags. MIT Licensed. For more in

Matt Layher 249 Dec 29, 2022
Go based grpc - grpc gateway micro service example

go-grpc-gateway-server This repository provides an example for go based microservice. Go micro services developed based on gRPC protobuf's and also us

Suresh Yekasiri 0 Dec 8, 2021
The devs are over here at devzat, chat over SSH!

Devzat Where are the devs at? Devzat! Devzat is chat over SSH Try it out: ssh sshchat.hackclub.com Add this to ~/.ssh/config: Host chat HostName s

Ishan Goel 1.4k Jan 7, 2023
Access Google Ads API via GRPC

google-ads-pb You can use the golang library to interact with the Google Ads API across grpc. This library is not the official Google Ads API library.

null 13 Dec 9, 2022
Go-grpc - This is grpc server for golang.

go-grpc This is grpc server for golang. protocのインストール brew install protoc Golang用のプラグインのインストール go install google.golang.org/protobuf/cmd/protoc-gen-go

jotaro yuza 1 Jan 2, 2022
Go-grpc-template - A small template for quickly bootstrapping a, developer platform independent gRPC golang application

go-grpc-template A small template for quickly bootstrapping a developer platform

null 1 Jan 20, 2022
Toy gRPC Tunnel over CloudFlare (Proof of Concept)

gun You know what it means. Guide Server Go to your domain in CloudFlare. In "Network" tab, turn on gRPC.

Qv2ray Workgroup 146 Jan 6, 2023
gRPC over WebRTC

gRPC over WebRTC Just a proof of concept, please be kind. How to Start all the things Client, create-react-app + grpc-web signaling + webrtc extension

Jean-Sebastien Mouret 261 Dec 16, 2022