(S3) Storage Lifecycle #

A storage lifecycle is used to automatically perform actions on the objects in your buckets. One example, is that of automatically expiring objects after a certain time.

In general, a new lifecycle will run almost immediately when applied and then continue to be re-evaluated every hour. Depending on how large your bucket is and the number of objects contained, the lifecycle may run for longer than that and generally need a significant amount of time to complete.

There are no guarantees provided how fast it runs and when it runs. If you need to delete objects right away, please use another way. Also, please note that debugging capabilities are very small, however, if you use an SDK to create the lifecycle, it should be accepted by our object storage.

Files deleted in the object storage cannot be recovered unless bucket versioning is setup.

Use case: Expiring objects #

In order to expire objects, the lifecycle supports a minimum interval of 1 day.

The following is an example using AWS’ SDK for Go, to create this policy and expire all objects in it, that are older than 14 days.

The lifecycle itself is an XML structure, but with the power of one of the SDKs this is of lesser importance to us and well hidden. Examples for other configurations and programming languages are available in the official AWS docs.

Code #

package main

import (
	"fmt"
	"log"
	"os"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/credentials"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3"
)

var (
	// bucket, prefix for the rule and expiration
	bucket     string = "bucket-name.example.org"
	prefix     string = ""
	expireDays int64  = 14
)

func main() {
	// initialize the credential session
	credentialSession := session.Must(session.NewSession(&aws.Config{
		Region:                    aws.String("unused"),
		Credentials:               credentials.NewStaticCredentials("your-key-id", "your-secret", ""),
		S3ForcePathStyle:          aws.Bool(true),
		DisableEndpointHostPrefix: aws.Bool(true),
	}))

	// create service client
	svc := s3.New(credentialSession, aws.NewConfig().WithEndpoint("https://s3.storage.planetary-networks.de"))

	enabled := aws.String("Enabled")

	_, err := svc.PutBucketLifecycleConfiguration(&s3.PutBucketLifecycleConfigurationInput{
		Bucket: &bucket,
		LifecycleConfiguration: &s3.BucketLifecycleConfiguration{
			Rules: []*s3.LifecycleRule{
				{
					// expire objects in "prefix"
					ID:     aws.String(fmt.Sprintf("expire_after_%d_days", expireDays)),
					Status: enabled,
					Filter: &s3.LifecycleRuleFilter{
						Prefix: aws.String(prefix),
					},
					Expiration: &s3.LifecycleExpiration{
						Days: aws.Int64(expireDays),
					},
					NoncurrentVersionExpiration: &s3.NoncurrentVersionExpiration{
						NoncurrentDays: aws.Int64(expireDays),
					},
					AbortIncompleteMultipartUpload: &s3.AbortIncompleteMultipartUpload{
						DaysAfterInitiation: aws.Int64(20),
					},
				},
				{
					ID:     aws.String("delete_markers"),
					Status: enabled,
					Filter: &s3.LifecycleRuleFilter{
						Prefix: aws.String(prefix),
					},
					Expiration: &s3.LifecycleExpiration{
						ExpiredObjectDeleteMarker: aws.Bool(true),
					},
				},
			},
		},
	})
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println("saved/updated policy: "+bucket+"/"+prefix)

	output, err := svc.GetBucketLifecycleConfiguration(&s3.GetBucketLifecycleConfigurationInput{
		Bucket: &bucket,
	})
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(output.String())
}

To use this example:

init a go project: go init my-lifecycle
paste the code into a main.go file and adjust the bucket and credentials
run go mod tidy to download dependencies
finally, run with go run main.go.

Output will look similar to this:

$ go run main.go
saved/updated policy: bucket-name.example.org/
{
  Rules: [{
      AbortIncompleteMultipartUpload: {
        DaysAfterInitiation: 20
      },
      Expiration: {
        Days: 14
      },
      Filter: {
        Prefix: ""
      },
      ID: "expire_after_14_days",
      NoncurrentVersionExpiration: {
        NoncurrentDays: 14
      },
      Status: "Enabled"
    },{
      Expiration: {
        ExpiredObjectDeleteMarker: true
      },
      Filter: {
        Prefix: ""
      },
      ID: "delete_markers",
      Status: "Enabled"
    }]
}

Next steps #

In addition to apply the lifecycle to everything, you may want to filter which objects the lifecycle is applied to. To address this use case, our example code contains a prefix variable which allows you to define a files this lifecycle will apply to. The prefix variable will be used to setup the filter when set.

Without the prefix our code example applies to everything in the bucket.