fedn.network.storage.s3 package

Module handling storage of objects in S3-compatible object storage. This functionality is used by the controller to store global models in the model trail in persistent storage. Currently implemented for MinIO, but a ‘ developer can extend the framwork by implemeting the interface in base.py.

Submodules

fedn.network.storage.s3.base module

Base class for artifacts repository implementations.

class fedn.network.storage.s3.base.RepositoryBase

Bases: ABC

Base class for artifacts repository implementations.

abstract get_artifact(instance_name: str, bucket: str) → IO

Retrieve object with name instance_name.

Parameters:

• instance_name (str) – The name of the object to retrieve

• bucket (str) – The bucket name

Returns:

The retrieved object

Return type:

Any

abstract get_artifact_stream(instance_name: str, bucket: str) → IO

Return a stream handler for object with name instance_name.

Parameters:

• instance_name (str) – The name of the object

• bucket (str) – The bucket name

Returns:

Stream handler for object instance_name

Return type:

IO

abstract set_artifact(instance_name: str, instance: IO, bucket: str) → None

Set object with name instance_name.

Parameters:

• instance_name (str) – The name of the object

• instance (Any) – The object

• bucket (str) – The bucket name

fedn.network.storage.s3.boto3repository module

Module implementing Repository for Amazon S3 using boto3.

class fedn.network.storage.s3.boto3repository.Boto3Repository(config: dict)

Bases: RepositoryBase

Class implementing Repository for Amazon S3 using boto3.

__init__(config: dict) → None

Initialize object.

Parameters:

config (dict) – Dictionary containing configuration for credentials and bucket names.

create_bucket(bucket_name: str) → None

Create a new bucket. If bucket exists, do nothing.

Parameters:

bucket_name (str) – The name of the bucket

delete_artifact(instance_name: str, bucket: str) → None

Delete object with name instance_name from bucket.

Parameters:

• instance_name (str) – The object name

• bucket (str) – Bucket to delete from

get_artifact(instance_name: str, bucket: str) → bytes

Retrieve object with name instance_name.

Parameters:

• instance_name (str) – The name of the object to retrieve

• bucket (str) – The bucket name

Returns:

The retrieved object

Return type:

bytes

get_artifact_stream(instance_name: str, bucket: str) → BytesIO

Return a stream handler for object with name instance_name.

Parameters:

• instance_name (str) – The name of the object

• bucket (str) – The bucket name

Returns:

Stream handler for object instance_name

Return type:

io.BytesIO

list_artifacts(bucket: str) → List[str]

List all objects in bucket.

Parameters:

bucket (str) – Name of the bucket

Returns:

A list of object names

Return type:

List[str]

set_artifact(instance_name: str, instance: IO, bucket: str, is_file: bool = False) → bool

Set object with name instance_name.

Parameters:

• instance_name (str) – The name of the object

• instance (Any) – The object

• bucket (str) – The bucket name

• is_file (bool, optional) – Whether the instance is a file, defaults to False

Returns:

True if the artifact was set successfully

Return type:

bool

fedn.network.storage.s3.miniorepository module

fedn.network.storage.s3.repository module

Interface for storing model objects and compute packages in S3 compatible storage.

class fedn.network.storage.s3.repository.Repository(config: dict, init_buckets: bool = True, storage_type: str | None = None)

Bases: object

Interface for storing model objects and compute packages in S3 compatible storage.

__init__(config: dict, init_buckets: bool = True, storage_type: str | None = None) → None

Initialize the repository.

Parameters:

• config (dict) – Configuration dictionary for credentials and bucket names.

• init_buckets (bool, optional) – Whether to initialize buckets, defaults to True

• storage_type – Type of storage to use, defaults to an empty string which falls back to FEDN_OBJECT_STORAGE_TYPE

delete_compute_package(compute_package: str) → None

Delete a compute package from storage.

Parameters:

compute_package (str) – The name of the compute package

delete_model(model_id: str) → None

Delete model.

Parameters:

model_id (str) – The id of the model to delete

get_compute_package(compute_package: str) → bytes

Retrieve compute package from object store.

Parameters:

compute_package (str) – The name of the compute package.

Returns:

Compute package.

Return type:

bytes

get_model(model_id: str) → bytes

Retrieve a model with id model_id.

Parameters:

model_id (str) – Unique identifier for model to retrieve.

Returns:

The model object

Return type:

bytes

get_model_stream(model_id: str) → bytes

Retrieve a stream handle to model with id model_id.

Parameters:

model_id (str) – Unique identifier for model to retrieve.

Returns:

Handle to model object

Return type:

bytes

presigned_get_url(bucket: str, object_name: str, expires: timedelta = datetime.timedelta(seconds=3600)) → str

Generate a presigned URL for a download object request.

Parameters:

• bucket (str) – The bucket name

• object_name (str) – The object name

• expires (datetime.timedelta) – The time the URL is valid

Returns:

The URL

Return type:

str

presigned_put_url(bucket: str, object_name: str, expires: timedelta = datetime.timedelta(seconds=3600)) → str

Generate a presigned URL for an upload object request.

Parameters:

• bucket (str) – The bucket name

• object_name (str) – The object name

• expires (datetime.timedelta) – The time the URL is valid

Returns:

The URL

Return type:

str

set_compute_package(name: str, compute_package: bytes | str, is_file: bool = True) → None

Upload compute package.

Parameters:

• name (str) – The name of the compute package.

• compute_package (Union[bytes, str]) – The compute package

• is_file (bool, optional) – True if model is a file name, else False

set_model(model: bytes | str, is_file: bool = True) → str

Upload model object.

Parameters:

• model (Union[bytes, str]) – The model object

• is_file (bool, optional) – True if model is a file name, else False

Returns:

id for the uploaded object

Return type:

str

fedn.network.storage.s3.saasrepository module

Implementation of the Repository interface for SaaS deployment using boto3.

class fedn.network.storage.s3.saasrepository.SAASRepository(config: dict)

Bases: RepositoryBase

Class implementing Repository for SaaS deployment using boto3.

__init__(config: dict) → None

Initialize object.

Parameters:

config (dict) – Dictionary containing configuration for credentials and bucket names.

create_bucket(bucket_name: str) → None

Create a new bucket. If bucket exists, do nothing.

Parameters:

bucket_name (str) – The name of the bucket

delete_artifact(instance_name: str, bucket: str) → None

Delete object with name instance_name from bucket.

Parameters:

• instance_name (str) – The object name

• bucket (str) – Bucket to delete from

get_artifact(instance_name: str, bucket: str) → bytes

Retrieve object with name instance_name.

Parameters:

• instance_name (str) – The name of the object to retrieve

• bucket (str) – The bucket name

Returns:

The retrieved object

Return type:

bytes

get_artifact_stream(instance_name: str, bucket: str) → BytesIO

Return a stream handler for object with name instance_name.

Parameters:

• instance_name (str) – The name of the object

• bucket (str) – The bucket name

Returns:

Stream handler for object instance_name

Return type:

io.BytesIO

list_artifacts(bucket: str) → List[str]

List all objects in bucket.

Parameters:

bucket (str) – Name of the bucket

Returns:

A list of object names

Return type:

List[str]

set_artifact(instance_name: str, instance: IO, bucket: str, is_file: bool = False) → bool

Set object with name instance_name.

Parameters:

• instance_name (str) – The name of the object

• instance (Any) – The object

• bucket (str) – The bucket name

• is_file (bool, optional) – Whether the instance is a file, defaults to False

Returns:

True if the artifact was set successfully

Return type:

bool