This page contains information to help you understand how to use the SOAR, where all the Solar Orbiter data are stored and distributed.

Contents:

• Basic Search and Result 
• Pre-visualising data
• Time series Low Latency visualisation
• Programmatic access ("machine interface")
• Python snippet to access data
• Access to more details on the Solar Orbiter data contents via TAP and external client application
• Access to non-public data (e.g., CDAG and "formagonly") 

Basic Search and Results pages

This video shows the basic search functionality of the SOAR and how the results are shown to the user.

SOAR - Basic search and results - 1280X720.mp4

Sending files to visualisation tools

These videos show how to send files found in the archive to external applications using the SAMP protocol (https://www.ivoa.net/documents/SAMP/). Some of the SAMP-enabled applications that can be used to visualise Solar Orbiter data are:

• JHelioviewer (https://www.jhelioviewer.org)
• Autoplot (http://autoplot.org)
• Aladin (https://aladin.u-strasbg.fr/)
• Topcat (http://www.star.bris.ac.uk/~mbt/topcat/)

SOAR - SAMP Usage - 1280X720.mp4

SOAR - SAMP Usage - Remote Sensing Data - 1280X720.mp4

Note that currently, you may experience difficulties using this facility using Chrome. Try using a different browser or follow these workaround instructions:

1. Go to chrome://flags/ in the browser
2. Set "Block insecure private network requests" to Disabled
3. Relaunch Chrome

Time series visualisation for Low Latency data

This video shows how to pre-visualise data from the In-Situ instruments in the form of time series (1D, 2D, Spectrograms). 

SOAR - Time-series display - HR.mp4

Machine Interface

This section informs you on how to use in a simple way the quite powerful SOAR API which fully implements the IVOA TAP, but which details you do not  have to understand in order to access the SOAR data bypassing the User Interface. At the archive pages, there is a help section with documentation that describes in more details how to fully exploit all the TAP functionalities implemented in the SOAR (http://soar.esac.esa.int/soar/#aio). 

Main public access table

The main public access table of the SOAR is called "v_public_files". In a later section, it will be shown how to use an external tool (like Topcat) to access more detailed SOAR database information using the TAP interface.

This table provides access to information on which files are stored in the archive. For programmatic access to get files, you should perform two steps within your code:

1. Get the names of the available files that satisfy your criteria (for instance: "all the MAG files archived later than 1 April 2021")
2. For each of the files in the response (which will appear in the "file_name_ variable, see below), get the file.

The contents of the v_public_files table are the following:

Queries to this table can be done via an HTTP request in the following way. The example below gives the ADQL query to obtain all available files for MAG archived later than 1 April 2021:

SELECT * FROM v_public_files WHERE instrument='MAG' AND archived_on >'2021-04-01'

I.e., in plain language: SELECT all columns FROM the v_public_files table WHERE instrument is MAG AND it was archived after 2021-04-01

In order to use this in a browser, we need to say where we want to get this information:

http://soar.esac.esa.int/soar-sl-tap/tap/sync?REQUEST=doQuery&LANG=ADQL

in what format (e.g., CSV is another option):

&FORMAT=json

and then add the ADQL query in a way that the browser can understand it - the spaces and some other characters need to be replaced:

&QUERY=SELECT+*+FROM+v_public_files+WHERE+instrument='MAG'+AND+archived_on>'2021-04-01'

This can be copied and pasted into a browser address bar to get the results:

http://soar.esac.esa.int/soar-sl-tap/tap/sync?REQUEST=doQuery&LANG=ADQL&FORMAT=csv&QUERY=SELECT+*+FROM+v_public_files+WHERE+instrument='MAG'+AND+archived_on>'2021-04-01'

SOME BUGS FOUND IN THESE INSTRUCTIONS - THESE WILL BE FIXED ASAP, HOWEVER, THE TEXT ABOVE WILL WORK WHEN COPIED AND PASTED INTO A BROWSER

(please note that when inserting this type of query manually in your browser, the URL (the "http sentence") will be added some "escape" codes from the browser, like "plus" signs, or "%22" signs, etc; this is normal, and is the way that the browser translates the plain queries to understand them internally; the above query might look like: 

http://soar.esac.esa.int/soar-sl-tap/tap/sync?REQUEST=doQuery&LANG=ADQL&FORMAT=json&QUERY=SELECT%20*%20FROM%20v_public_files%20WHERE%20instrument=%27MAG%27%20AND%20archived_on%3E%272021-04-01%27

http://soar.esac.esa.int/soar-sl-tap/tap/sync? REQUEST=doQuery&LANG=ADQL&FORMAT=json&QUERY=%22SELECT%20*%20%20FROM%20v_public_files%20WHERE%20instrument=%27MAG%27%20AND%20archived_on%20%3E%272021-04-01%27%22

)

The above query can be decomposed in the following three main parts:

Running the example above, we currently get a list of files: 1619452697345OPE-result.json in the json format. Now, we can get the files one by one. In the example, the HTTP request is given to access the first file:

http://soar.esac.esa.int/soar-sl-tap/data?retrieval_type=PRODUCT&data_item_id=solo_LL02_mag_20210331T000059-20210401T000058&product_type=LOW_LATENCY

This request will return all the versions corresponding to the identifier solo_LL02_mag_20210331T000059-20210401T000058 in a tar file. If you are only interested in receiving the LATEST VERSION you only have to substitute "retrieval_type=PRODUCT" for "retrieval_type=LAST_PRODUCT":

http://soar.esac.esa.int/soar-sl-tap/data?retrieval_type=LAST_PRODUCT&data_item_id=solo_LL02_mag_20210331T000059-20210401T000058&product_type=LOW_LATENCY

 you will see how in this case, only the latest version of the product is downloaded.

Python Access

There are two libraries which implement access to Solar Orbiter data in Python: HelioPy (https://docs.heliopy.org/en/stable/) and SunPy (https://sunpy.org), by David Stansby. Both can request data from Solar Orbiter via the aforementioned "machine interface" whose details are hidden from the User, so h/she does not need to delve into them.

An example of how to plot a time series from the MAG instrument can be seen in the HelioPy pages:

https://docs.heliopy.org/en/stable/auto_examples/plot_solo_mag.html#sphx-glr-auto-examples-plot-solo-mag-py

Python code to get files

In what follows, a simple snippet of code from David Stansby is inlined so you can just copy and paste into your Python interpreter. You can find details here:

https://github.com/dstansby/sunpy-soar

The following are the steps to run the code:

• Get the sunpy-soar extension to sunpy. This is done via:

pip install sunpy-soar

• Put the following code in your Python interpreter:

===========================================

import sunpy_soar
from sunpy.net import Fido

from sunpy.net.attrs import Instrument, Level, Time
from sunpy_soar.attrs import Identifier

# Create search attributes
instrument = Instrument('MAG')
time = Time('2020-12-01', '2020-12-05')
level = Level(2)
identifier = Identifier('MAG-RTN-NORMAL-1-MINUTE')

# Do search
result = Fido.search(instrument, time, level, identifier)
print(result)

# Download files
files = Fido.fetch(result)
print(files)

===========================================

In the example, MAG instrument  1 minute resolution RTN coordinates Magnetic field vector files available from 1 December 2020 to 5 Dec 2020 are downloaded to your file system. Then you can work with them at your convenience. The "identifier"s that are available can be seen in the SOAR itself by querying on the different instruments and extracting the relevant information from the "Descriptor" column:

The "Level" identifier is 

The following example is identical to the previous but downloading EUI images:

===========================================

import sunpy_soar
from sunpy.net import Fido

from sunpy.net.attrs import Instrument, Level, Time
from sunpy_soar.attrs import Identifier

# Create search attributes
instrument = Instrument('EUI')
time = Time('2021-02-01', '2021-02-02')
level = Level(1)
identifier = Identifier('EUI-FSI174-IMAGE')

# Do search
result = Fido.search(instrument, time, level, identifier)
print(result)

# Download files
files = Fido.fetch(result)
print(files)

===========================================

Access to more information on SOAR data

The SOAR implements the TAP protocol of the International Virtual Observatory Alliance (https://www.ivoa.net). This is a powerful protocol yet safe way to access more details on the data contents and structure of the metadata from Solar Orbiter. It might be useful for users that are more technically knowledgeable and want to make more complicated programmatic queries to the contents of the archive. There is ample information on the TAP access of SOAR in the SOAR "Programmatic access" pages (http://soar.esac.esa.int/soar/#aio). Here, we describe how to use an external client (TOPCAT) to get more details about the contents of the SOAR.

A video on how to access the TOPCAT client to see all SOAR DB information can be seen here (in low resolution to save bandwidth; probably good enough as it has audio information, otherwise, get the same video in higher resolution below):

SOAR - TAP Usage from TOPCAT Client - II - Lower Res.mp4

(A higher quality video can be downloaded from here (243 MB):

http://vospace.esac.esa.int/vospace/sh/a5d2e439ebd7e763b8c0c47328bb7e4ad7766f7?dl=1)

Private access data

The SOAR hosts products which are used by the PIs to interchange data among themselves for calibration purposes or even for internal matters within one team. Currently, CDAG (Calibration Data Access Group) and "MAG Private" groups are defined.

Access to these data are granted by the PIs. To apply for access rights, either contact the relevant PI or the SOAR Archive Scientist (currently Helen.Middleton at esa.int).

Once authorised, to access those data, log in into the SOAR application at the top rightmost part of the screen "SIGN IN":

Once logged in, the additional options (e.g., Operational data search, CDAG, MAG Private) will appear: