Unicodeencodeerror For Google App Engine Datastore To Bigquery Process

I'm trying to follow along with this Codelab that shows you how to take data from your Google App Engine Datastore and move it through Google Cloud Storage and on to BigQuery by se

Solution 1:

You are converting Unicode data to a bytestring:

ctx.get_pool("file_pool").append(self._filename, str(data))

When you do that without specifying an encoding, Python falls back to the default, which is ASCII. You'll need to settle on a different encoding instead, one that can handle all Unicode codepoints your data contains.

For most text, UTF-8 is a good choice for that; if you have a lot of non-western text (Arabic, Asian, etc.) then UTF-16 might be more efficient. In either case, you'll have to explicitly encode:

ctx.get_pool("file_pool").append(self._filename, data.encode('utf8'))

When reading back the data from that file, use filedata.decode('utf8') to decode back to Unicode.

See the Python Unicode HOWTO for more information on Python and Unicode.

Solution 2:

ctx.get_pool("file_pool").append(self._filename, str(data))

if data contains unicode characters, this will fail. Try

ctx.get_pool("file_pool").append(self._filename, unicode(data))