fedn.network.storage.s3 package

Module handling storage of objects in S3-compatible object storage. This functionality is used by the controller to store global models in the model trail in persistent storage. Currently implemented for MinIO, but a ‘ developer can extend the framwork by implemeting the interface in base.py.

Submodules

fedn.network.storage.s3.base module

class fedn.network.storage.s3.base.RepositoryBase

Bases: object

abstract get_artifact(instance_name, bucket)

Retrive object with name instance_name.

Parameters:

• instance_name (str) – The name of the object to retrieve

• bucket (str) – The bucket name

abstract get_artifact_stream(instance_name, bucket)

Return a stream handler for object with name instance_name.

Parameters:

• instance_name (str) – The name if the object

• bucket (str) – The bucket name

Returns:

stream handler for object instance name

abstract set_artifact(instance_name, instance, bucket)

Set object with name object_name

Parameters:

• instance_name – The name of the object

• instance – the object

• bucket (str) – The bucket name

Tyep insance_name:

str

fedn.network.storage.s3.miniorepository module

class fedn.network.storage.s3.miniorepository.MINIORepository(config)

Bases: RepositoryBase

Class implementing Repository for MinIO.

client = None

create_bucket(bucket_name)

Create a new bucket. If bucket exists, do nothing.

Parameters:

bucket_name (str) – The name of the bucket

delete_artifact(instance_name, bucket)

Delete object with name instance_name from buckets.

Parameters:

• instance_name – The object name

• bucket (str) – Buckets to delete from

get_artifact(instance_name, bucket)

Retrive object with name instance_name.

Parameters:

• instance_name (str) – The name of the object to retrieve

• bucket (str) – The bucket name

get_artifact_stream(instance_name, bucket)

Return a stream handler for object with name instance_name.

Parameters:

• instance_name (str) – The name if the object

• bucket (str) – The bucket name

Returns:

stream handler for object instance name

list_artifacts(bucket)

List all objects in bucket.

Parameters:

bucket (str) – Name of the bucket

Returns:

A list of object names

set_artifact(instance_name, instance, bucket, is_file=False)

Set object with name object_name

Parameters:

• instance_name – The name of the object

• instance – the object

• bucket (str) – The bucket name

Tyep insance_name:

str

fedn.network.storage.s3.repository module

class fedn.network.storage.s3.repository.Repository(config, init_buckets=True)

Bases: object

Interface for storing model objects and compute packages in S3 compatible storage.

delete_compute_package(compute_package)

Delete a compute package from storage.

Parameters:

compute_package (str) – The name of the compute_package

delete_model(model_id)

Delete model.

Parameters:

model_id (str) – The id of the model to delete

get_compute_package(compute_package)

Retrieve compute package from object store.

Parameters:

compute_package – The name of the compute package.

Returns:

Compute package.

get_model(model_id)

Retrieve a model with id model_id.

Parameters:

model_id – Unique identifier for model to retrive.

Returns:

The model object

get_model_stream(model_id)

Retrieve a stream handle to model with id model_id.

Parameters:

model_id

Returns:

Handle to model object

presigned_get_url(bucket: str, object_name: str, expires: timedelta = datetime.timedelta(seconds=3600)) → str

Generate a presigned URL for a download object request.

Parameters:

• bucket (str) – The bucket name

• object_name (str) – The object name

• expires (datetime.timedelta) – The time the URL is valid

Returns:

The URL

Return type:

str

presigned_put_url(bucket: str, object_name: str, expires: timedelta = datetime.timedelta(seconds=3600))

Generate a presigned URL for an upload object request.

Parameters:

• bucket (str) – The bucket name

• object_name (str) – The object name

• expires (datetime.timedelta) – The time the URL is valid

Returns:

The URL

Return type:

str

set_compute_package(name, compute_package, is_file=True)

Upload compute package.

Parameters:

• name (str) – The name of the compute package.

• compute_package – The compute package

• is_file – True if model is a file name, else False

set_model(model, is_file=True)

Upload model object.

Parameters:

• model (BytesIO or str file name.) – The model object

• is_file – True if model is a file name, else False

Returns:

id for the uploaded object (str)