Python has tons of cool idioms and features that are often overlooked or underutilized. List comprehensions to cut back on the use of unnecessary loops, decorators to wrap functions with annotations, and generator functions are just some that can be applied to working with the DFP API.
In this post, we'll tackle one of our most asked questions using decorators : "Why am I running into CONCURRENT_MODIFICATION or QUOTA_EXCEEDED errors, and how do I avoid this?".
Addressing these errors using decorators
CONCURRENT_MODIFICATION and QUOTA_EXCEEDED errors are similar in nature - the requests you’re making are failing, but not necessarily because the data you’re sending over is bad. In the first case, one of the objects you’re trying to modify is being updated elsewhere, but you likely want to try again after pulling down the same set of objects. You could certainly write code that retries your operations in all of your services for each object, but it may get a bit hard to maintain (especially with duplicated code). A much cleaner implementation would be to use a decorator!
The Python wiki has an entry under the decorators section that shows how a generic decorator might work for retrying a call. With a few modifications, we can tailor this to capture the two types of errors that might arise:
import time
from functools import wraps
RESPONSES_TO_RETRY = ['CONCURRENT_MODIFICATION', 'QUOTA_EXCEEDED']
def retry(tries=4, delay=3, backoff=2):
''' Decorator that implements an exponential backoff for retrying on errors.
Args:
tries: int number of times to execute the wrapped function before failing
delay: int time to delay in seconds before the FIRST retry
backoff: int multiplier to extend the initial delay by for each retry
'''
def decorated_function_with_retry(func):
@wraps(func)
def function_to_retry(*args, **kwargs):
local_tries, local_delay = tries, delay
while local_tries > 1:
try:
return func(*args, **kwargs)
except Exception, e:
if [response for response in RESPONSES_TO_RETRY
if response in e.fault['faultstring']]:
print '%s, Retrying in %d seconds...' % (str(e),
local_delay)
time.sleep(local_delay)
local_tries -= 1
local_delay *= backoff
return func(*args, **kwargs)
return function_to_retry
return decorated_function_with_retrySay you were making a call to update line items - with large networks, it’s not unlikely that someone might be editing the line item at the same time. Since you’d want to pull down the most recent copy of the line item any time the update fails, you would want to abstract out the update method to include the getLineItemsByStatement call, e.g.,
@retry()
def fetch_and_update_line_item(statement):
# call to get the line item in question
response = line_item_service.getLineItemsByStatement(
statement.ToStatement())
updated_line_items = []
if 'results' in response:
for line_item in response['results']:
# Do something with your line items here and add them to
# updated_line_items.
line_item_service.updateLineItems(updated_line_items)This would effectively allow you to, in the event of the update failing due to concurrent modification, pull down and update a new copy of the line item. Using the default constructor will retry 4 times with 3, 6, and 12 second delays in between.
To wrap things up, decorators are incredibly useful constructs in Python and are useful for the DFP API for several reasons:
Your application will be less flaky and less affected by intermittent application issues.
You’re less likely to run into quota errors.
This would prevent overwriting other changes (in the case of retrying failed calls on concurrent modification errors).
You could also use something like this to log errors on your end, which could help reveal poor code health or inefficient processes.
Make use of decorators in your code, and you'll soon be sitting pretty.
Nicholas Chen , DFP API Team
