This page looks best with JavaScript enabled

IMDLIB

 ·  ☕ 5 min read  ·  🤖 Saswata Nandi

Description

This is a python package to download and handle binary grided data from Indian Meterological department (IMD).

Installation

IMDLIB can be installed via pip, conda, or using the source file from Github. It is tested for both Windows and Linux platforms with 64 bit architecture only.

Installation using pip.

1
pip install imdlib

Installation using conda.

1
conda install -c iamsaswata imdlib

Installation from source file.

1
pip install git+https://github.com/iamsaswata/imdlib.git

IMDLIB is currently built with the following plugins. You need to have a python version >= 3.5 and install the below dependencies before installing IMDLIB from source.

PluginVersion No
certifi[2019.11.28]
matplotlib[3.1.3]
numpy[1.18.1]
pandas[0.25.3]
python-dateutil[ 2.8.1 ]
pytest[5.3.4]
pytz[ 2019.3]
requests[2.23.0]
scipy[1.4.1]
six[1.14.0]
xarray[0.14.1]
rioxarray (optional)[0.1.1]

Usage

Downloading data from IMD

IMDLIB is capable of downloading gridded rainfall and temperature data (min and max) from IMD Pune.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import imdlib as imd

# Downloading 8 years of rainfall data for India
start_yr = 2010
end_yr = 2018
variable = 'rain' # other options are ('tmin'/ 'tmax')
file_dir = 'dataFolder'
"""
fn_format   : str or None
        fn_format represent filename format. 
        Default vales is None.  Which means filesnames are accoding to the IMD naming convention
        If we specify fn_format = 'yearwise', filenames are renamed like <year.grd>

file_dir   : str or None
        Directory for downliading the files.
        If None, the currently working directory is used.

sub_dir : bool
		True : if you need subdirectory for each variable type;
        False: Files will be saved directly under main directory
proxies : dict
        Give details in curly bracket as shown in the example below
        e.g. proxies = { 'http' : 'http://uname:password@ip:port'}
"""
data = imd.get_data(variable, start_yr, end_yr, fn_format='yearwise', file_dir=file_dir)

Reading downloaded binary dataset

One major purposes of IMDLIB is to process gridded IMD meterological dataset. The original data is available in .grd file format. IMDLIB can read .grd file in xarray style and will create a IMD class objetct. For the already downloaded data, one should use the imd.open_data() functionality for further analysis and processing.

1
2
3
4
5
6
7
8
9
import imdlib as imd

start_yr =2018
end_yr = 2018
variable = 'rain' # other options are ('tmin'/ 'tmax')
file_format = 'yearwise' # other option (None), which will assume deafult imd naming convention
file_dir = '/home/data/imd' # other option keep it blank
data = imd.open_data(variable, start_yr, end_yr,'yearwise', file_dir)
data
<imdlib.core.IMD at 0x7f39b0960750>

file_dir should refer to top-lev directory. It should contain 3 sub-directories. rain, tmin, and tmax.

if file_dir exist, but no subdirectory, it will try to find the files in file_dir. But be careful if you are using file_format = 'yearwise', as it will not differentiate between 2018.grd for rainfall and 2018.grd for tmin.

if file_dir is not given, it will look for the associate subdirectories and files from the current directory.

Get shape of IMD objecct

1
data.shape
(365, 135 129)

Get numpy. ndarray

1
np_array = data.data

Get xarray object

1
2
ds = data.get_xarray()
type(ds)
xarray.core.dataarray.DataArray

Display mean rainfall in Map

1
2
ds = ds.where(ds['rain'] != -999.) #Remove NaN values
ds['rain'].mean('time').plot()

png

Processing and Saving

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
# Get data for a given coordinare and convert to csv file
lat = 20.03
lon = 77.23
out_dir='/home/downloads/data'

data.to_csv(file_name, lat, lon, out_dir)

# Convert to netcdf file
data.to_netcdf(file_name, out_dir)

# Convert to GeoTIFF format
data.to_geotiff(file_name, out_dir)

For converting to GeoTIFF, we are using [`rioxarray`](https://corteva.github.io/rioxarray/stable/) package, but it has not been considered as a dependency for IMDLIB. So, if a call is made to `to_geotiff()` functionaality, IMDLIB will check if *rioxarray* module is available to the system python path and if *rioxarray* is found, the corresponding GeoTIFF file will be generated or else a error will be thrown saying *rioxarray* is not installed.

NetCDF File Convention

The IMDLIB focuses on producing netCDF (network Common Data Form) based final output as netCDF is the format most commonly used for climate model generated data. The netCDF Climate and Forecast (CF) Metadata Conventions, Version 1.7, has been adopted by IMDLIB for its efficient and consistent use with other standard netCDF based tool/applications. The epsg:4326 coordinate reference systems (CRS) is considered in CF naming convention and is vital for the function to_geotiff to work correctly. For more information on the CF convention, users are requested to visit CF Conventions Home Page and XArray & CF integration resources.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
test@test:~/data$ ncdump -h test.nc 
netcdf test {
dimensions:
	time = 365 ;
	lat = 31 ;
	lon = 31 ;
variables:
	double tmax(time, lat, lon) ;
		tmax:_FillValue = NaN ;
		tmax:units = "C" ;
		tmax:long_name = "Maximum Temperature" ;
	double lat(lat) ;
		lat:_FillValue = NaN ;
		lat:axis = "Y" ;
		lat:standard_name = "latitude" ;
		lat:long_name = "latitude" ;
		lat:units = "degrees_north" ;
	double lon(lon) ;
		lon:_FillValue = NaN ;
		lon:axis = "X" ;
		lon:long_name = "longitude" ;
		lon:units = "degrees_east" ;
	int64 time(time) ;
		time:standard_name = "time" ;
		time:long_name = "time" ;
		time:units = "days since 2010-01-01" ;
		time:calendar = "proleptic_gregorian" ;

// global attributes:
		:Conventions = "CF-1.7" ;
		:title = "IMD gridded data" ;
		:source = "https://imdpune.gov.in/" ;
		:history = "2020-12-29 22:16:07.359709 Python" ;
		:references = "" ;
		:comment = "" ;
		:crs = "epsg:4326" ;
}

License

IMDLIB is available under MIT the open source license.

Share on

Saswata Nandi
WRITTEN BY
Saswata Nandi
PhD Student in Hydrology