Can't get dynamodb scan to work with boto

Question

I use boto to access the dynamodb table. Everything went well until I tried to perform a scan operation.

I tried a couple of syntaxes that I found after doing repeated searches on the internet but with no luck:

def scanAssets(self, asset): results = self.table.scan({('asset', 'EQ', asset)}) -or- results = self.table.scan(scan_filter={'asset':boto.dynamodb.condition.EQ(asset)}) 

The attribute I'm viewing is called an asset, and the asset is a string.

Odd is the call to table.scan, which always ends through this function:

 def dynamize_scan_filter(self, scan_filter): """ Convert a layer2 scan_filter parameter into the structure required by Layer1. """ d = None if scan_filter: d = {} for attr_name in scan_filter: condition = scan_filter[attr_name] d[attr_name] = condition.to_dict() return d 

I am not a python expert, but I do not understand how this will work. That is, what structure will scan_filter have to go through this code?

Again, maybe I'll just call it wrong. Any suggestions?

Answer 1

OK, it looks like I had a problem with import. Just using:

 import boto 

and specifying boto.dynamodb.condition does not cut it. I had to add:

 import dynamodb.condition 

to get the type of condition to receive. My working code is:

 results = self.table.scan(scan_filter={'asset': dynamodb.condition.EQ(asset)}) 

Not that I fully understood why, but now it works for me. :-)

Answer 2

Or you can do it

 exclusive_start_key = None while True: result_set = self.table.scan( asset__eq=asset, # The scan filter is explicitly given here max_page_size=100, # Number of entries per page limit=100, # You can divide the table by n segments so that processing can be done parallelly and quickly. total_segments=number_of_segments, segment=segment, # Specify which segment you want to process exclusive_start_key=exclusive_start_key # To start for last key seen ) dynamodb_items = map(lambda item: item, result_set) # Do something with your item, add it to a list for later processing when you come out of the while loop exclusive_start_key = result_set._last_key_seen if not exclusive_start_key: break 

This is applicable for any field.

segmentation: suppose you have a higher script in test.py

you can run in parallel, for example

 python test.py --segment=0 --total_segments=4 python test.py --segment=1 --total_segments=4 python test.py --segment=2 --total_segments=4 python test.py --segment=3 --total_segments=4 

on different screens