Page tree
Skip to end of metadata
Go to start of metadata

This page contains information to help you understand how to use the SOAR, where all the Solar Orbiter data are stored and distributed.


Contents:



Basic Search and Results pages

This video shows the basic search functionality of the SOAR and how the results are shown to the user.

SOAR - Basic search and results - 1280X720.mp4



Sending files to visualisation tools

This video shows how to send files found in the archive to external applications using the SAMP protocol (https://www.ivoa.net/documents/SAMP/). Some of the SAMP-enabled applicatins that can be used to visualise Solar Orbiter data are:

SOAR - SAMP Usage - 1280X720.mp4

SOAR - SAMP Usage - Remote Sensing Data - 1280X720.mp4


Time series visualisation for Low Latency data

This video shows how to previsualise data from the In-Situ instruments in the form of time series (1D, 2D, Spectrograms). 

SOAR - Time-series display - HR.mp4



Machine Interface

This section informs you on how to use in a simple way the quite powerful SOAR API which fully implements the IVOA TAP, but which details you do not  have to understand in order to access the SOAR data bypassing the User Interface. At the archive pages, there is a help section with documentation that describes in more details how to fully exploit all the TAP functionalities implemented in the SOAR (http://soar.esac.esa.int/soar/#aio). 

Main public access table

The main public access table of the SOAR is called "v_public_files". In a later section, it will be shown how to use an external tool (like Topcat) to access more detailed SOAR database information using the TAP interface.

This table provides access to information on which files are stored in the archive. For programmatic access to get files, you should perform two steps within your code:


  1. Get the names of the available files that satisfy your criteria (for instance: "all the MAG files archived later than 1 April 2021")
  2. For each of the files in the response (which will appear in the "file_name_ variable, see below), get the file.


The contents of the v_public_files table are the following:


v_public_files

archived_on

char

Date when file was archived

(e.g.: 2021-04-24 19:15:44.679)

begin_time

char

Begin time of observation

(e.g.: 2021-04-24 00:01:02.0)

data_type

char

LL (Low Latency), KERNEL (Spice Kernels), SCI (Science)

file_name

char

Name of file (e.g.: solo_LL02_epd-het-asun-rates_20210424T000102-20210425T000102_V01I.cdf)

file_size

long

Size of file in bytes

instrument

char

EPD, EUI, MAG, RPW, SWA

item_id

char

Relevant internally to SOAR, e.g.: solo_LL02_epd-ept-asun-rates_20210424T000102-20210425T000102

item_version

char

Version of the product, e.g.: V01, V02, etc.

processing_level

char

LL01, LL02, LL03, L1, L2, L3, etc.


Queries to this table can be done via an HTTP request in the following way. In the example, a query to obtain all available files for MAG archived later than 1 April 2021 is done:

http://soar.esac.esa.int/soar-sl-tap/tap/sync?REQUEST=doQuery&LANG=ADQL&FORMAT=json&QUERY=SELECT * FROM v_public_files WHERE instrument='MAG' AND archived_on >'2021-04-01'

(please note that when inserting this type of query manually in your browser, the URL (the "http sentence") will be added some "escape" codes from the browser, like "pkus" signs, or "%22" signs, etc; this is normal, and is the way that the Browser translates the plain queries to understand them internally; the above query might look like: 

http://soar.esac.esa.int/soar-sl-tap/tap/sync REQUEST=doQuery&LANG=ADQL&FORMAT=json&QUERY=%22SELECT%20*%20%20FROM%20v_public_files%20WHERE%20instrument=%27MAG%27%20AND%20archived_on%20%3E%272021-04-01%27%22

)


The above query can be decomposed in the following three main parts:

Compulsory bit (server definition)FORMATQUERY
http://soar.esac.esa.int/soar-sl-tap/tap/sync?REQUEST=doQuery&LANG=ADQL&FORMAT=json&QUERY=SELECT * FROM v_public_files WHERE instrument='MAG' AND archived_on >'2021-04-01'
This part shall not be modified unless the user is an expert on TAP access. Please consult the TAP API pages if that is the case

This is the desired format for the results. Currently, the following formats are available:

votable_plain

votable (binary format)

csv

json

This is the query part. The SOAR API implements the IVOA ADQL language, which is basically the same as SQL92 with some things added (and some others removed). Basic standard database queries are accepted. If you know nothing about SQL you can consult for instance https://en.wikipedia.org/wiki/Select_(SQL). For more specific details on the ADQL lenguage implemented by the SOAR service, you can access https://www.ivoa.net/documents/latest/ADQL.html. For more information on the TAP interface implemented by the SOAR API you can access https://www.ivoa.net/documents/TAP/.

Running the example above, we currently get a list of files: 1619452697345OPE-result.json in the json format. Now, we can get the files one by one. In the example, the HTTP request is given to access the first file:


http://soar.esac.esa.int/soar-sl-tap/data?retrieval_type=PRODUCT&data_item_id=solo_LL02_mag_20210331T000059-20210401T000058&product_type=LOW_LATENCY

This request will return all the versions corresponding to the identifier solo_LL02_mag_20210331T000059-20210401T000058 in a tar file. If you are only interested in receiving the LATEST VERSION you only have to substitute "retrieval_type=PRODUCT" for "retrieval_type=LAST_PRODUCT":

http://soar.esac.esa.int/soar-sl-tap/data?retrieval_type=LAST_PRODUCT&data_item_id=solo_LL02_mag_20210331T000059-20210401T000058&product_type=LOW_LATENCY

 you will see how in this case, only the latest version of the product is downloaded.



Python Access

There are two libraries which implement access to Solar Orbiter data in Python: Heliopy (https://docs.heliopy.org/en/stable/) and Sunpy (https://sunpy.org), by David Stansby. Both can request data from Solar Orbiter via the aforementioned "machine interface" whose details are hidden from the User, so h/she does not need to delve into them.

An example of how to plot a time series from the MAG instrument can be seen in the Heliopy pages:

https://docs.heliopy.org/en/stable/auto_examples/plot_solo_mag.html#sphx-glr-auto-examples-plot-solo-mag-py


Python code to get files

In what follows, a simple snippet of code from David Stansby is inlined so you can just copy and paste into your Python interpreter. You can find details here:

https://github.com/dstansby/sunpy-soar

The following are the steps to run the code:

  • Get the sunpy-soar extension to sunpy. This is done via:
pip install sunpy-soar
  • Put the following code in your Python interpreter:

===========================================

import sunpy_soar
from sunpy.net import Fido

from sunpy.net.attrs import Instrument, Level, Time
from sunpy_soar.attrs import Identifier

# Create search attributes
instrument = Instrument('MAG')
time = Time('2020-12-01', '2020-12-05')
level = Level(2)
identifier = Identifier('MAG-RTN-NORMAL-1-MINUTE')

# Do search
result = Fido.search(instrument, time, level, identifier)
print(result)

# Download files
files = Fido.fetch(result)
print(files)

===========================================

In the example, MAG instrument  1 minute resolution RTN coordinates Magnetic field vector files available from 1 December 2020 to 5 Dec 2020 are downloaded to your file system. Then you can work with them at your convenience. The "identifier"s that are available can be seen in the SOAR itself by querying on the different instruments and extracting the relevant information from the "Descriptor" column:

The "Level" identifier is 



The following example is identical to the previous but downloading EUI images:

===========================================

import sunpy_soar
from sunpy.net import Fido

from sunpy.net.attrs import Instrument, Level, Time
from sunpy_soar.attrs import Identifier

# Create search attributes
instrument = Instrument('EUI')
time = Time('2021-02-01', '2021-02-02')
level = Level(1)
identifier = Identifier('EUI-FSI174-IMAGE')

# Do search
result = Fido.search(instrument, time, level, identifier)
print(result)

# Download files
files = Fido.fetch(result)
print(files)

===========================================




Access to more information on SOAR data

The SOAR implements the TAP protocol of the International Virtual Observatory Alliance (https://www.ivoa.net). This is a powerful protocol yet safe way to access more details on the data contents and structure of the metadata from Solar Orbiter. It might be useful for users that are more technically knowledgeable and want to make more complicated programmatic queries to the contents of the archive. There is ample information on the TAP access of SOAR in the SOAR "Programmatic access" pages (http://soar.esac.esa.int/soar/#aio). Here, we describe how to use an external client (TOPCAT) to get more details about the contents of the SOAR.

A video on how to access the TOPCAT client to see all SOAR DB information can be seen here (in low resolution to save bandwidth; probably good enough as it has audio information, otherwise, get the same video in higher resolution below):

SOAR - TAP Usage from TOCAT Client - II - Lower Res.mp4


(A higher quality video can be downloaded from here (243 MB):

http://vospace.esac.esa.int/vospace/sh/a5d2e439ebd7e763b8c0c47328bb7e4ad7766f7?dl=1)




Private access data

The SOAR hosts products which are used by the PIs to interchange data among them for calibration purposes or even for internal matters within one PI and the corresponding collaborators. Currently, CDAG (Calibration Data Access Group) and "MAG Private" groups are defined.

To access those data, you only have to make sure to login into the SOAR application at the top rightmost part of the screen "SIGN IN":

Once logged in, if you are member of a group that can access protectede data, you will be able to select it from the search menu, as in the image where in the "Include also" section, there are now two new checkboxes for "CDAG" and "MAG Private" data:

Access to this data are granted by the PIs. If you think you should have the right to access those, either contact the relevant PI or send an email to the SOAR Archive Scientist (currently yours truly, Pedro.Osuna at esa.int) who will contact the relevant people to check and take eventual actions.









  • No labels