PPS Frequently Asked Questions (FAQ)

The Frequently Asked Questions (FAQ) help page has been organized into the following topics questions (Click on the topic of interest to see the answer):

Contents:

User Account and General Information
STORM- Data Search, Custom Subsets and Subscriptions- PPS Data Ordering Interface
THOR- Orbit Viewer Information
Retrieving and Reading PPS Data Products

 

PPS User Accounts and General Information

 

STORM- Data Search, Custom Subsets and Subscriptions- PPS Data Ordering Interface

 

THOR- Orbit Viewer Information

 

Retrieving and Reading PPS Data Products

 

PPS User Accounts and General Information

 

  1. How do I register to obtain TRMM, GPM and partner data from PPS?

    2. PPS no longer requires users to have a special science user account which provided a dedicated personal ftp area and STORM account. Instead, GPM, TRMM and Partner data users can acquire data and use STORM after completing a brief and simple registration process. Go to:

    https://registration.pps.eosdis.nasa.gov/registration/

    All access to the online PPS data archive for GPM, TRMM and partner data currently at:
    https://arthurhouhttps.pps.eosdis.nasa.gov

    FTP was replaced at the start of 2021 with FTPS/ HTTPS access as noted below.

    Access will require a registered email address to be entered. Please note that BOTH your 'User name' and 'Password' are your registered email address.
    Attention: ftp data transfer access will end at the beginning of 2021 due to NASA security mandates. You must access GPM data through FTPS or HTTPS (direct FTPS/ HTTPS and STORM access) . Users must access the GPM production archives using:

    arthurhouftps.pps.eosdis.nasa.gov
    or
    https://arthurhouhttps.pps.eosdis.nasa.gov/
    for browser access.

    Data can also be ordered using our PPS Data Ordering Interface (STORM).
    https://storm.pps.eosdis.nasa.gov/storm/ after registering

    The following documents describe how to access arthurhou and jsimpson (NRT) after the switch to FTPS/ HTTPS:

    https://gpm.nasa.gov/tutorial-how-access-new-pps-ftps-servers

    https://pps.gsfc.nasa.gov/Documents/FTP-FTPS-Transition.pdf
    https://pps.gsfc.nasa.gov/Documents/jsimpsonhttps_retrieval.pdf
    https://spdf.gsfc.nasa.gov/ftps_readme.html
    https://pps.gsfc.nasa.gov/Documents/arthurhouhttps_retrieval.pdf

    You can register now at:

    https://registration.pps.eosdis.nasa.gov/

    When you register and submit your information, you will get a confirmation e-mail with a 'verification' or 'update' link. Please use this to complete the PPS registration process.

    If you don't receive this e-mail in one hour, please check in your spam folder, and then contact helpdesk@mail.pps.eosdis.nasa.gov to resolve the issue.

     

  • What are TRMM, GPM, PPS/TSDIS, and the DAAC / DISC

    • The Tropical Rainfall Measuring Mission (TRMM) is a satellite mission which ended in 2015 and provided the first detailed and comprehensive data set on the four dimensional distribution of rainfall and latent heating over vastly undersampled oceanic and tropical continental regimes. To get additional information, see the TRMM homepage:

    TRMM has its own unique Precipitation Processing System (PPS), formerly known as the TRMM Science Data and Information System (TSDIS) at GSFC to process the information from three satellite instruments and four ground radars. PPS analyzes the rainfall data and also provides validation from nearly a dozen ground radar sites. The PPS homepage is located at:

    GPM (Global Precipitation Measurement) is an international satellite mission launched by NASA and JAXA on Feb. 27, 2014, that will set new standards for precipitation measurements worldwide using a network of satellites united by the GPM Core Observatory. See the GPM homepage located at:

    The operational archiving and distribution of TRMM science data products is provided by the Goddard Distributed Active Archive Center (DAAC), now known as DISC (Data and Information Services Center). In addition to archiving and distributing the TRMM science data, the DISC also provides necessary information and support for manipulating these data files, which are provided in NCSA's Hierarchical Data Format (HDF). The Goddard DISC homepage is located at:

     

  • How do I obtain TRMM, GPM and partner data?

    All GPM data users must complete a simple registration process as detailed above in "How do I register to obtain TRMM, GPM and partner data from PPS?"
    Once registered, users can obtain data through our arthurhou server:'arthurhouftps.pps.eosdis.nasa.gov' by using their registered email address as BOTH 'Username' and 'Password'.
    Due to a NASA/ GSFC mandate, PPS will transition from 'FTP' data access to 'FTPS' and 'HTTPS' transfer at the end of Dec 2020 or early 2021. GPM/ Partner data transfer will be via FTPS or HTTPS as an alternative method (See "How do I register to obtain TRMM, GPM and partner data from PPS" in this webpage -above- for all details).

     

    Registered users can also order data through PPS's STORM data ordering interface: https://storm.pps.eosdis.nasa.gov/storm/ . Using STORM, you can search for data, create custom subsets and establish data subscriptions.

 

 

  • How do I obtain GPM near realtime (NRT) data?

    Please register or re-register at: https://registration.pps.eosdis.nasa.gov/registration/

    • Once you access the registration portal, please see:" FTPS Access in addition to Standard Research Products". and CHECK "Near-Realtime Products".

    Both your Username and Password are your registered email address (all lowercase)

    Please also see the following webpage "GPM NRT Documentation" for the most current information on how to obtain/ download the GPM NRT data products from PPS's jsimpson:

    ‘jsimpsonftps.pps.eosdis.nasa.gov.'

    'https://jsimpsonhttps.pps.eosdis.nasa.gov/'

 

 

STORM- Data Search, Custom Subsets and Subscriptions- PPS Data Ordering Interface

 

  • Where can I find more information on how to use PPS's STORM?

    • Please go directly to the STORM web portal at: https://storm.pps.eosdis.nasa.gov/storm/   For complete instructions, user should dowload the Adobe PDF format document "PPS STORM User Guide"

     

  • Can the PPS Order Notifications be sent to more than one person and/or place?

    • The PPS ordering system is designed to send notices to a single user, presumably the registered user as established through PPS's simple registration process.. That single "user" could also be a mailing list alias defined on your computer. Using an alias is the simplest method, although a bit cumbersome since a system administrator is needed to make changes to the mailing list. You may also want to consider setting up a majordomo list or using mail filtering.   To set up the local mailing list alias using Linux, complete the following:

      -Set up a local mailing list alias in /etc/aliases:

      -Example: pps-users: user1, user2@their.computer.edu, user3

      -Run "newaliases" so your send mail will be aware of the changes.

      -Verify the alias is working.

     

  • I tried to order a large quantity of data or broad range of dates but receive a warning message that there are too many granules that meet the search criteria. What is the best way get my desired data?

    • In STORM, the size of the search results is limited by 2,000 granules.  Please use a narrower date range, select fewer product types, or select a smaller geographic area. For example, a user could split a large order with an expansive date range or an order that contains many data products into smaller multiple chronological orders for a shorter period of time or place multiple orders with fewer data types. 

     

  • Can I preview data images from the STORM web site before ordering?

    • Once you search for data products and obtain search results, certain GPM, TRMM and partner data products can be previewed directly from the "Search Results" area under the column "Download/View".  Click on the camera icon to see a preview low resolution browse image of the selected file. A variety of data products can also be analyzed and viewed with PPS's THORonline which is a web based version of the "Orbit Viewer-THOR" .  To see the selected file in THOR, please click on the "THOR" icon also in the Download/View" column.

     

  • What is the difference between a Standing Order and a Special Request?

    A Special Request is made when ordering a product that has already been produced. Special Requests are filled soon after placing your order and you will receive an E-mail data notification to your registered email address.

    A Standing Order is submitted for a product that has not yet been produced and is being requested for an extended period of time. Standing Orders are processed daily and are filled as the files are produced. Data products will be delivered via attached scripts and these will also be placed in a dedicated "pgs/so" directory. A data notice will be emailed to you at the provided email address.

    If you are requesting both past and future products, or only future products, use the following method:

    1. Enter Product Type, Start Date/Time, and Stop Date/Time from the Order TRMM Products Page
    2. Select the desired products by checking the boxes to the left or using the "Select All Granules" button. At any time during this step you can start anew by clicking on the "Clear All Selections"
    3. It is very IMPORTANT to note, that if you are only interested in establishing a Standing Order for future products, you will still need to select a past date/time first and the desired product type. For each data type you must at least select one granule to establish a Standing Order for TRMM future products.
    4. Once you are satisfied with your selections select "Continue Order" at the bottom of the page.
    5. You are now presented with two ordering options
    6. Enter Yes for "Submit standing order for future products with these criteria" if you want the selected products produced in the past plus all future products of the type/types indicated.
    7. You also have the additional options of selecting the only the "product" or "browse" files or both at the "Check the product types you would like included with this order" option.
    8. Select "Submit Request" to place your order

    You will be presented with the "Request Status" confirmation page which will indicate if your request was successful along with your order number "Request ID" number. At this stage you can "logout" , go back and order additional TRMM data products, or explore additional features by selecting the "Services" button.

    Additionally, from the "STORM Services" page you can select "Track Order Status" which will provide a detailed list and history of your order/orders.

THOR- Orbit Viewer Information

 

  • Does PPS provide any software to read and visualize the dta that I have downloaded?

    • Yes, PPS develops and provides  the free "Orbit Viewer THOR" software written in the IDL language which is a tool for displaying the standard data products of the Global Precipitation Measurement (GPM) mission, partner satellite data and for the Tropical Rainfall Measuring Mission (TRMM). THOR stands for the Tool for High-resolution Observation Review. THOR displays GPM and TRMM products on a map of the Earth when possible, and otherwise, as 2D plots or text output. Since 1997, the Precipitation Processing System (PPS) at NASA Goddard has developed and distributed many versions of Orbit Viewer THOR.    

 

Retrieving and Reading PPS Data Products

 

  • I've placed an order through STORM and received a data notice, but have questions on how to download the data?

    • After you place your order, you will receive a "PPS Order Notification" Email from the Helpdesk containing details on where the files can be found and downloaded from our "arthurhou" server via FTPS, etc. Depending on what you chose to recieve through STORM, there will also be several text based attachments included; one file (ftps_url) contains the ftps_url of all the files in that order; while the others (ftps_get) are scripts which can be used to automate the download/s. The other attached file (RequestParameters) documents the specific parameters for your requested files.

    You will receive further instructions on how to use the attached scripts, etc. with your "PPS Order Notification" Email after your data order/s is processed.

    The following are examples of how to use the scripts, etc. attached to your "PPS Order Notification

    STORM Orders

    STORM provides options to receive scripts to assist in retrieving ordered data via FTPS. Three options are available, the
    first of which is recommended for reasons described below:

    • Python script
    • FTPS script
    • FTPS URL

    The Python script is run with the Python 3 interpreter:
    python script.py

    The Python script hides the details of the file transfer from the user. It attempts to connect to the PPS FTPS server but
    will fall back to the HTTPS server if needed. It will also perform ‘sha’ hash tests against downloaded data files to confirm
    their contents. The script will skip over files that have already been downloaded, which allows the scripts to be stopped
    and started again later. Due to these benefits, the Python script is the recommended method of obtaining data from
    PPS.

    The FTPS script is meant to use used with lftp on Linux / mac
    lftp script

    The FTPS url file can be directly used with wget
    wget -i urlfile

    Unfortunately, curl cannot use FTPS URLs. To convert the PPS FTPS URL file to FTP URLs, run (on Linux/mac)
    sed -i 's/ftps:/ftp:/g' urlfile

    The URL file can then be used with curl like:
    xargs -n 1 curl --ftp-ssl-control -O < urlfile

    Python

    It is strongly recommended that Python 3 rather than Python 2.7 be used when accessing the PPS FTPS server. However,
    PPS has been successful using Python 2.7. The following initialization should be done before use of the Python 2.7
    retrieval.
    from ftplib import FTP_TLS
    import ssl

    ftp_site = "arthurhouftps.pps.eosdis.nasa.gov"
    FTP_TLS.ssl_version = ssl.PROTOCOL_TLSv1_2
    ftps = FTP_TLS()
    ftps.debugging = 2
    ftps.connect(ftp_site,21)
    ftps.login(’uname’,’password’)
    ftps.prot_p()

    The Python script provided by the STORM ordering system is intended to be run under Python 3 and has not been tested
    using Python 2.7.

    -PPS data users can also use a dedicated FTP/ FTPS client to access the PPS archive files.

    -For complete details on obtaining data from our 'arthurhou' production or 'jsimpson' NRT data archives, please see: "Downloading PPS Production and Realtime Data" (4th header) on the page:
    https://arthurhou.pps.eosdis.nasa.gov/GPMprelimdocs.html


    If you require any further assistance, please send Email to: helpdesk@mail.pps.eosdis.nasa.gov

     

  • How do I get retreive my daily Standing Order files using wget?

    • To get all generated files, the following command may be run repeatedly (daily).  It will retrieve new files (only) as they are added to the ftps area. 

    wget --mirror --no-parent --no-host-directories --cut-dirs=3 --user=<email_addr> --password=<email_addr> ftps://arthurhouftps.pps.eosdis.nasa.gov/gpmuser/<email_addr>/pgs/

    For security reasons, PPS recommends users create a wget config file named something like 'wget.pps.config' with the following contents:

    user=<email_addr>
    password=<email_addr>
    mirror=on
    no_parent=on
    add_hostdir=off
    cut_dirs=3

    and run with:

    wget --config=wget.pps.config ftps://arthurhouftps.pps.eosdis.nasa.gov/gpmuser/<email_addr>/pgs/

    Some people might find this easier than using a gui app like 'FileZilla' to sync a user's ftps area to a local directory tree.

     

  • I am using the Safari web browser on a macOS computer and cannot login to PPS's ftp data archives such as: https://gpmweb2https.pps.eosdis.nasa.gov/ using the browser interface. Is this a known problem and is there a solution?

    • There was a known compatibility issue with the macOS Safari web browser and how it could not resolve the ftp login challange on our arthurhou and other ftp servers such as PPS's GPM NRT server 'jsimpson'. Please note: PPS was now discontinued FTP access.simpson.  PPS web development and system programming staff currently recommend that macOS users use Mozilla's Firefox web browser or the Google Chrome browser for full compatibility when visiting these ftp areas via a web browser on a macOS system. PPS will continue to investigate this issue.

     

  • I use a special script or other automated method to download GPM data from PPS's FTP site, but am encountering access problems.

    • Although PPS does not block our ftps/ https users from running their own scripts, etc.for data retrieval, etc., PPS does limit the number of concurrent ftp sessions per user (registered Email address) to three. We recommend either modifying your code or make sure that you try to minimize the number of active (concurrent) sessions.  For instance; if you have your web browser open when visiting the "arthurhou" ftps/ https site and then open another session to run your code, you may be exceeding this connection limit.  We limit concurrent ftps sessions to 3 connections per registered user.   This measure is neccessary to ensure that our network maintains an acceptible throughput and avoids prolonged periods of slowness due to an excessive number of open sessions.

    It is also important to note that users who employ scripts or other methods to obtain PPS data enable passive mode, “PASV” as the correct option.

     

  • How long will my ordered data be available for retrieval?

    • Once you have received your PPS Order Notification message/s, the files will be available for a period of approximately 2 weeks from the time of the Email/s. It is best to retreive your data as soon as possible once the PPS Oder Notifications are delivered to you. If you are away, or have not yet obtained the data after the preset time of the data purge, then you will have to place a new data order through STORM for the missed products.

     

  • How can I read the HDF files?

    • Orbital TRMM and GPM data products are in HDF format, which may be accessed in several ways:

    1. Algorithm developers use the PPS TKIO Science Algorithm toolkits to write the TRMM and GPM files, so the toolkit is ideal for reading the files into your own programs. The PPS Toolkit (TKIO) for GPM is available from our FTP location:

      https://gpmweb2https.pps.eosdis.nasa.gov/pub/PPStoolkit/GPM/

    2. The PPS TRMM toolkit is callable from both C and FORTRAN. You can download this from:
    3. The PPS Orbit Viewer-THOR is a tool written in IDL to display all GPM,TRMM and some partner satellite HDF files. This is a viewing tool only, and does not allow any data manipulation. Please see the "tutorial.pdf" document for background information, instructions on how to install the software and how to use THOR. The Orbit Viewer can be downloaded from the PPS THOR FTP area. Downloading the most current version of THOR for your respective operating system is recommended:
    4. One can also use Python to read GPM HDF 5 files.Please see "Using Python to Read HDF5 Files from the Global Precipitation Measurement (GPM) Mission".
    1. NASA's Atmospheric Science Data Center has a page devoted to HDF format:

      2. The "Public Domain and Commercial HDF Tools" section also has more information on viewing and manipilating HDF data.

       

  • What Type of File Transfer Protocals are supported by PPS?

    • Currently, PPS supports the following file transfer protocols: FTPS. HTTPS. PPS has phased out FTP access at the start of 2021. Many FTP software clients already provide for secure FTPS transfers (sftp, etc.) such as "Filezilla" or "WS_FTP Professional" , to name a few.

     

  • I would like to obtain GPM data from the PPS "arthurhou" FTPS server directly, but am unfamiliar with the various directories.

    • Within the GPM Production Archive Directory Structure located at:
    https://arthurhouhttps.pps.eosdis.nasa.gov/

    Provide your registered Email address (lower case) for both Username and Password

    Please see the tutorial on how to access the arthurhou FTPS server using an FTP client.:

    https://gpm.nasa.gov/tutorial-how-access-new-pps-ftps-servers

    arthurhouftps.pps.eosdis.nasa.gov

    To better understand the PPS GPM '/gpmdata' data archive directory in regard to structure, product types, etc.. please see these documents; https://arthurhouhttps.pps.eosdis.nasa.gov/gpmdata/documents/

    The '/trmmdata' directory is the archive that the TRMM and GPM data ordering system- PPS's "STORM" uses to obtain TRMM data.  The most user-friendly way to obtain TRMM data files may be to use STORM by going to: https://storm.pps.eosdis.nasa.gov/storm/.  

    If you know which TRMM HDF4 data product you are interested in, you can obtain the granules of that data product in the ByDate subdirectory of '/trmmdata.  Other specialized directories that may be of interest are the 3G subdirectory, which contains the 3G family of ASCII gridded products, and the GIS subdirectory, which contains the TMPA 3-hour gridded surface precipitation product converted into a GIS data-format (TIFF+WorldFile).

    An alphabetical list of the subdirectories of the top-level /trmmdata directory is listed below.  Not all of the directories are of interest to first time TRMM researchers.

    • 3G: Contains the "3G" family of data products which contain ASCII text gridded TRMM data. Each file contains either 24 one-hour grids to summarize one day or contains one grid that summarizes an entire month.  The file format described in the following documents available on ftp://trmmopen.pps.eosdis.nasa.gov/pub/ : README.3G68, README.3G01, README.3G68.QuarterDegree, and README.3G68Land .
    • Auxiliary: Contains orbit-definition files not intended for use by typical TRMM researchers.
    • Browse: Contains one PNG browse image for each HDF4 granule of many TRMM standard data products.  Browse images are rarely downloaded directly from the FTP site and are instead viewed from a link in the Search Result table of the STORM data ordering system: https://storm-pps.gsfc.nasa.gov/storm/ .
    • ByDate: Perhaps the most popular FTP directory for obtaining TRMM standard products.  Organized by reprocessing data version, year, month, and day.  The file format of these files are documented in the V7 file specification: http://pps.gsfc.nasa.gov/Documents/filespec.TRMM.V7.pdf .
    • ByInstrument: An alternative FTP directory for obtaining TRMM standard products.  Organized the same way as the ByDate subdirectory, except that an instrument-specific subdirectory occurs in between data version and year. 
    • CSI: This directory is no longer active.  Prior to the launch of GPM, it contained subsets of TRMM standard products over pre-defined geographic regions.
    • Climatology: Contains files in the TRMM Composite Climatology developed by Wang, Adler, Hoffman, and Bolvin.  A journal article on this topic is available here: http://journals.ametsoc.org/doi/abs/10.1175/JCLI-D-13-00331.1 .
    • GIS: Contains a GIS-friendly reformatted version of the TMPA data product (aka TRMM 3B42).  TMPA is a 3-hourly average global grid of surface rainfall estimates from a collection of microwave radiometers.  The GIS-reformatted version is a 2-byte TIFF image plus an ESRI WorldFile text metadata file.  This format is documented here: ftp://trmmopen.gsfc.nasa.gov/pub/trmmdata/GIS/README.GIS.pdf
    • L0: This subdirectory contains low-level packet-data downloaded from the TRMM satellite.  Typical TRMM researchers are unlikely to be interested in this data. Furthermore, this data is not written in HDF4, so it cannot be read by PPS tools such as the THOR data viewer.
    • PF: The files of the TRMM Precipitation Feature database.  These files are a useful way to summarize rain events observed by TRMM, although they are not designated as a TRMM standard product.  Documentation of the TRMM Precipitation Feature database is available here: http://trmm.chpc.utah.edu/ .
    • SpecialLH: Contains latent heating products of interest only to latent-heating researchers.
    • shortTermAuxiliary, shortTermGV, and shortTermMOC: Does not contain files of interest to beginning TRMM researchers.

      •  

  • Is there a GPM data product equivalent to the TRMM TMPA (3B4x) Products?

    • Yes, The Integrated Multi satellitE Retrievals for GPM (IMERG) Final Run, which is the post real-time research product in the IMERG suite of products , is now available at PPS at https://storm.pps.eosdis.nasa.gov and 'arthurhouftps.pps.eosdis.nasa.gov'.

    IMERG is presently available for the period mid-March 2014 to the present (delayed by about 3 months), which is the most current data given the latency of the ECMWF input data required for the production runs. The half-hourly products have the prefix “3B-HHR”and the monthly products have the prefix “3B-MO” . The complete file naming convention can be found at:  https://pps.gsfc.nasa.gov/Documents/FileNamingConventionForPrecipitationProductsForGPMMission.pdf

    Example:

    half-hourly 10.5067/GPM/IMERG/HH/3B
    monthly 10.5067/GPM/IMERG/MONTH/3B

    Note that you will need to be a registered user to access the data. Register online at https://registration.pps.eosdis.nasa.gov/registration/

     

  • I have a general question regarding TRMM or GPM File specifications and range values as stated in the documentation?

    • For example, if the filespec says the range is 0-3000 but the data is always within 0-50, then the range is still correct.
    If a datum was 4000, then either the data is in error or the range should be changed.

    Only if the data is outside the stated range could the range be questioned.

     

  • I am using a script or other automated method to download large volumes of GPM and/or partner data and am encountering problems with the way my username/ password address is resolved?

    • there is no ability to register an account without using an email address as the login username and password. However, certain script and automation problems can occur because of the way the "@" symbols in the Email addresses are resolved. One solution is to modify the the username/password (address) in the URL. The first two "@" symbols have to be replaced with their Unicode representation "%40". For example; if you are trying to obtain files from our near realtime "jsimpson" server in an automated fashion, you would use something similar to the example given below:

    ftp://scienceuser%40gmail.com:scienceuser%40gmail.com@jsimpson.pps.eosdis.nasa.gov/NRTPUB/imerg/gis/2015/12/3B-HHR-L.MS.MRG.3IMERG.20151203-S000000-E002959.0000.V03E.7day.tif.gz