Beautifulsoup Returning [] When I Run It

I am using Beautiful soup with python to retrieve weather data from a website. Here's how the website looks like:

Solution 1:

The main API page on the Singapore NEA site shows clearly that the response you get is an XML document:

2-hour Nowcast
Data Description: Weather forecast for next 2 hours
Last API Update: 1-Mar-2016
Frequency Hourly
File Type: XML

You are looking at a HTML representation of the data in Chrome; Chrome transformed the XML to make it presentable in some way, but your Python code is still accessing the XML directly. The PDF documentation and your own question show the actual XML contents, parse those.

If you want to use BeautifulSoup with XML, make sure you have the lxml project installed and use the 'xml' parser type. Then simply access the text content of the validTime element:

soup = BeautifulSoup(r.content, "xml")
valid_time = soup.find('validTime').string

Demo:

>>> import requests
>>> from bs4 import BeautifulSoup
>>> r = requests.get('http://www.nea.gov.sg/api/WebAPI/?dataset=2hr_nowcast&keyref=<private_api_key>')
>>> soup = BeautifulSoup(r.content, "xml")
>>> soup.find('validTime').string
u'4.00 pm to 6.00 pm'

If you are trying to write to an XML file, you'd have to make sure it is writing valid XML however; this is outside the scope of BeautifulSoup.

Alternatively, use the ElementTree API that comes with Python by default; it can both parse the XML and produce new XML.

Share