Introduction¶

The Japan Meteorological Agency (JMA) monitors and records rainfall, wind speed and direction, temperature and relative humidity from over 840 sites across Japan through their Automated Meteorological Data Acquisition System (AMEDAS) [1]. Much of this data are publicly available online, and some datasets can be downloaded directly as CSV files. However, learning to scrape weather data directly from their website is useful because:

• Not all JMA datasets are available as CSV downloads
• Downloading hundreds of days of data from multiple weather stations manually can be painful

Data scraping allows you to retrieve any weather data available on the JMA website, automatically. This means that once the code is set up, you can collect weather data in a consistent and reproducible manner. You can easily modify the code to scrape data from different dates or different weather stations as well.

In this tutorial, I’ll walk through a method to scrape and parse 10-minute interval weather data of the Tokyo JMA weather station using Python libraries requests and Beautiful Soup.

What You’ll Learn in This Tutorial¶

By the end of this tutorial, you’ll learn how to:

• Scrape tabular weather data from JMA’s website using requests library
• Parse and give structure to the raw data with Beautiful Soup library
• Clean and save the data as a CSV file

If you prefer to skip the explanations and jump straight to the implementation, you can download this blog post as a Jupyter notebook. You can also download the extended code which is already set up to:

• Scrape 10-minute interval weather data from JMA
• Retrieve data from two JMA observation sites (Tokyo and Edogawa Base)
• Pull weather records for several dates in March 2024
• Translate wind direction from Japanese to English
• Save the cleaned results to Google Drive as CSV files

Just make sure to adjust the variables to your needs and that the appropriate Google Drive folders exist.

Alright — let’s get started.

Prerequisites¶

• A copy of this blog as a Jupyter notebook

Jargon¶

Data scraping: Using code to extract information from a web page or other source that’s primarily designed for human viewing. In this blog, we’ll be pulling data from an HTML table on the JMA site.

Data parsing: Converting raw or unstructured text into something structured and easy to work with — for example, turning HTML table rows into a tidy pandas DataFrame.

Scrape, Parse and Save Japan Weather Data¶

The general outline for collecting weather data from the JMA website is as follows:

1. Find the URL you want to retrieve data from
2. Use requests library to scrape data
3. Use an HTML parser and Beautiful Soup library to give structure to the scraped data
4. Navigate the HTML to extract the desired data
5. Convert the desired data into a pandas Dataframe
6. Save the DataFrame as a CSV file

Let’s start with step 1:

Step 1: Designate the URL Containing the Weather Data¶

First, we’ll need the URL of the JMA webpage that contains the data we want to scrape. You can navigate the records of past weather data from here. Below, I’ve provided a screenshot to help you:

1. Narrow down on the JMA weather site by designating the prefecture (red box)
2. Pinpoint the observation site within the prefecture (orange box, activated once you designate the prefecture)
3. Choose the year, month and day for the desired data (green box)
4. Select the type of weather data to access (blue box)

You may need a combination of Google maps, a translation app as well as trial and error to navigate to your desired URL. You’ll also notice that the column headers in the tables differ depending on the type of data you’ve selected (blue box).

For the remainder of this blog post, we’ll work with the 10-minute interval weather records. You can select the “１０分ごとの値” option located on the left column within the blue box, second from the bottom for a JMA observation site and date of your preference. Alternatively, I’ve provided a default test URL, which accesses 10-minute interval weather records from the Tokyo site on March 9, 2024.

Accessing this URL, we find the tabular weather data which we will be scraping:

Step 2: Data Scraping with the requests library.¶

Import the requests library.

The requests.get() function inside this library sends an HTTP GET request to the designated URL:

This returns a Response object, a class defined by the requests library that stores the server’s reply when accessing the designated URL.

The Response object comes with several useful attributes, which are handy for basic error handling. Refer to my short blog post for details.

For now, we’ll parse the HTML content using Beautiful Soup.

Step 3: Initial HTML Parsing with Beautiful Soup¶

With the response in hand, we can now parse the raw HTML. Import the BeautifulSoup() function from bs4:

Parse the contents of the Response object from step 2 with Python’s built-in html parser, html.parser. This will return a BeautifulSoup object.

By parsing HTML with Beautiful Soup, we can navigate the contents of the HTML. Namely, HTML consists of tags that define specific sections within the content. Most elements follow the pattern:

<tag_name> ...content... </tag_name>

These tags are nested in a form of hierarchy. For example, every webpage can be broadly divided into two major sections:

• <head> – metadata, scripts, page title
• <body> – the visible content, including the tabular data

Let us print just the <head> section to see a concise example of structured HTML:

We can see the nested hierarchy structure in lines 3 to 5 above with:

<title>
気象庁｜過去の気象データ検索
</title>

In a similar manner, the tabular data to extract is contained within <body> ...tabular data... </body>.

Step 4: Navigate the HTML and Extract the Desired Table¶

Unfortunately, the HTML body contains much more than the tabular data. If you are curious as to just how much content is inside the HTML body, try running the following code on your own:

print(step3_soup.body)

You can scroll through the whole HTML and manually search out the tags containing the tabular data. However, I will show you a slightly more sophisticated search method, which takes into account the following facts:

• Target data is likely within <table> ... </table> tags
• Target data contains '00:10' as a data element (as seen from the screenshot of the table)

By selectively searching for the '00:10' string nested inside <table> tags, we have successfully isolated the meteorological data buried inside the long HTML <body>. The contents within the first few table rows (as specificed by <tr> tags) confirms that the desired data is indeed contained inside this table.

By printing out the attributes of the table, the relevant table has:

• ID of tablefix1
• class of data2_s

We can now select just the one desired table by specifying either #id or .class in the soup.select() method:

To process the table row-by-row, re-write the soup.select() function to find the <tr> tag elements within <table class=data2_s>:

Using len() on the returned list shows that the table contains 146 rows. Inspecting the first few confirms we’ve successfully isolated the data we need.

Note that the first two rows in the table are column headers. The actual data starts on row 3. Additionally, the wind direction data is recorded in Kanji. We’d like to convert these values into a pandas DataFrame in the next step.

Step 5: Create a DataFrame with the Fully Parsed Data Using for Loops¶

From the isolated table rows, we extract the weather data and place them into a structured pandas DataFrame using a for loop to work through each row.

First, import the pandas library.

Then, create an empty DataFrame. I’ve provided the English translation of the column headers below:

Referring back to the sample table rows from the previous step, a few patterns stand out:

1. The first two table rows contain no data, only headers.
2. Every data row begins with a <td> or table data tag containing the time in HH:MM format.
3. All remaining meteorological values appear inside <td class="data_0_0"> elements.

We can use these patterns to guide our parsing when working row-by-row in the for loop.

Step 5-1: Skip Rows Corresponding to Column Headers¶

First, we’ll filter out the rows containing just the column headers. These are characterized by rows that do not contain any data_0_0 class data. As such, the method call of soup.select('td.data_0_0') will return an empty list, resulting in a safe method to skip the column headers table rows:

Step 5-2: Extract the Timestamps¶

With a method to safely skip the column headers set in the for loop, we’d like to extract the timestamps at which the weather data were collected next. Since the first <td> tag in each valid row always contains the timestamp, we can

• Use soup.select_one('td') to pull out just the first element
• Clean the timestamp with .text.strip()
• Assign it to the first column of the DataFrame using df.loc[] attribute

Let’s add this code to the for loop:

You can verify that all timestamps from 00:10 to 24:00 have been assigned to the DataFrame by inspecting the first and last few time entries:

Step 5-3: Retrieve the Meterological Data¶

The loop can skip the column headrs and collect the time information, so now we’d like to extract the remaining meteorological data from the data_0_0 class elements. To do so, we’ll loop through each <td class="data_0_0">, clean the text, and insert it into the DataFrame.

However, the keen-eyed reader would have noticed that the wind direction data is recorded as Japanese Kanji. We will translate these into English compass notation (NESW notation) within the for loop.

I’ve provided you with the ConvertKanji2NESW() function that translates the Japanese kanji into English compass notation:

A quick check of the first few lines in the DataFrame confirms that all data values of time, atmospheric pressure, temperature, humidity, wind speed and others, have been successfully extracted, and wind direction has been converted from kanji to English.

Step 6: Save DataFrame as a CSV File¶

In the final step, we save the fully populated and cleaned DataFrame as a CSV file. We can use pandas’ built-in pandas.to_csv() method to write the data out. Just specify the desired file path, disable the index column, and choose an appropriate text encoding (UTF-8 is a safe default):

If you are working in Google Colaboratory, and would like to save your files directly to Google Drive, you will need to mount to your drive with the following code snippet:

This will allow you to save directly to your Google Drive. Just make sure that the directory which you are trying to save to, exists:

Summary¶

Congratulations! You now have a full toolkit for scraping and parsing structured data from the JMA website using a combination of the requests, Beautiful Soup, and pandas libraries. A quick summary of the steps we took:

1. Designated the URL to access data from
2. Accessed the URL using requests.get() and obtained the raw HTML
3. Parsed the raw HTML using BeautifulSoup()
4. Navigated the complex HTML structure and isolated the target table using soup.select()
5. Manually parsed each table row whilst

• Skipping rows with no data
• Extracting and cleaning the timestamps
• Pulling out meteorological measurements and assigned them to the correct DataFrame columns
  • Translating the wind direction into English

6. Exported the fully parsed DataFrame to a CSV file using df.to_csv().

Next Steps¶

• Handling multiple dates

• Handling multiple JMA sites

• Adding a progress bar

• Save and load tables with datetime objects

  • Timestamps as dateime objects (before saving)
  • Attributing time zones to datetime objects (loading csv)

Further Reading¶

• Pause between requests
• Exception handling

References¶

[1] Japan Meteorological Agency, “地域気象観測システム（アメダス）” [Japanese]. Available: https://www.jma.go.jp/jma/kishou/know/amedas/kaisetsu.html. [Accessed March 30, 2026]