Listing contents of a bucket with boto3

We Are Going To Discuss About Listing contents of a bucket with boto3. So lets Start this Python Article.

Listing contents of a bucket with boto3

How to solve Listing contents of a bucket with boto3

One way to see the contents would be:
for my_bucket_object in my_bucket.objects.all(): print(my_bucket_object)

Listing contents of a bucket with boto3

One way to see the contents would be:
for my_bucket_object in my_bucket.objects.all(): print(my_bucket_object)

Solution 1

One way to see the contents would be:

for my_bucket_object in my_bucket.objects.all():
    print(my_bucket_object)

Original Author garnaat Of This Content

Solution 2

This is similar to an ‘ls’ but it does not take into account the prefix folder convention and will list the objects in the bucket. It’s left up to the reader to filter out prefixes which are part of the Key name.

In Python 2:

from boto.s3.connection import S3Connection

conn = S3Connection() # assumes boto.cfg setup
bucket = conn.get_bucket('bucket_name')
for obj in bucket.get_all_keys():
    print(obj.key)

In Python 3:

from boto3 import client

conn = client('s3')  # again assumes boto.cfg setup, assume AWS S3
for key in conn.list_objects(Bucket='bucket_name')['Contents']:
    print(key['Key'])

Original Author cgseller Of This Content

Solution 3

I’m assuming you have configured authentication separately.

import boto3
s3 = boto3.resource('s3')

my_bucket = s3.Bucket('bucket_name')

for file in my_bucket.objects.all():
    print(file.key)

Original Author Tushar Niras Of This Content

Solution 4

My s3 keys utility function is essentially an optimized version of @Hephaestus’s answer:

import boto3


s3_paginator = boto3.client('s3').get_paginator('list_objects_v2')


def keys(bucket_name, prefix='/', delimiter='/', start_after=''):
    prefix = prefix[1:] if prefix.startswith(delimiter) else prefix
    start_after = (start_after or prefix) if prefix.endswith(delimiter) else start_after
    for page in s3_paginator.paginate(Bucket=bucket_name, Prefix=prefix, StartAfter=start_after):
        for content in page.get('Contents', ()):
            yield content['Key']

In my tests (boto3 1.9.84), it’s significantly faster than the equivalent (but simpler) code:

import boto3


def keys(bucket_name, prefix='/', delimiter='/'):
    prefix = prefix[1:] if prefix.startswith(delimiter) else prefix
    bucket = boto3.resource('s3').Bucket(bucket_name)
    return (_.key for _ in bucket.objects.filter(Prefix=prefix))

As S3 guarantees UTF-8 binary sorted results, a start_after optimization has been added to the first function.

Original Author Sean Summers Of This Content

Conclusion

So This is all About This Tutorial. Hope This Tutorial Helped You. Thank You.

Also Read,

ittutorial team

I am an Information Technology Engineer. I have Completed my MCA And I have 4 Year Plus Experience, I am a web developer with knowledge of multiple back-end platforms Like PHP, Node.js, Python and frontend JavaScript frameworks Like Angular, React, and Vue.

Leave a Comment