The SOCCI Confluence installation will be unavailable due to scheduled maintenance on Wednesday, December 11, from 10:00 to 12:00.
Page tree
Skip to end of metadata
Go to start of metadata
Meeting Date20190115 - 20190116RefHELIO-MOM-0002
Meeting PlaceTeleconChairmanHarra, L., MSSL, UK 
Minute's Date20190123Participants

Chen, C., QMW college, UK
De Marchi, ESTEC/ESA
Dressing, N., FMI, Finland
Fleck, B., GSFC/ESA
Génot, V., IRAP/CNRS, France
Martinez, B., ESAC/ESA
Masson, A., ESAC/ESA (secretary)
Osuna, P., ESAC/ESA
Roth, M., Freiburg University, Germany 
Taylor, M., ESTEC/ESA


SubjectMinutes of the 3rd Heliophysics Archives USer group meetingCopy

Scope of this meeting

  This was the third meeting but the second meeting on the completeness and possible curation of some of the datasets contained in the Ulysses and SOHO archives (see detailed list of experiments reviewed in the more details section below, together with guidelines). Additionally, three more EU funded FP7/H2020 projects have been reviewed to answer the following  question: when based on ESA heliophysics archives, would the HAUS group recommend to store any of the value-added datasets together with their related archives as done for Cluster?  Each HAUS group member was invited to review at least one experiment from either Ulysses or SOHO, FP7/H2020 EU funded projects. Additionally, the scientific interest of a possible link between the ESA helio archives and solar ground based measurements was investigated. By the end of this meeting, all experiments on SOHO and Ulysses have been reviewed except one experiment (SOHO/SWAN), to be reviewed at the next meeting.

 More details (inc. review guidelines)

SOHO
Experiments reiewed during this 3rd HAUS group meeting: CELIAS, UVCS  
Experiments reiewed during the 2nd HAUS group meeting: CDS, COSTEP, EIT, ERNE, GOLF, LASCO, MDI, SUMER, VIRGO
Remaining experiment to review: SWAN

ULYSSES (complete)
Experiments reiewed during this 3rd HAUS group meeting: EPAC, URAP    
Experiments reiewed during this 2nd HAUS group meeting: COSPIN, DUST, FGM/VHM, GAS, GRB, SWICS, SWOOPS 

Curation/Completeness review guidelines

For each experiment 

List of data products available on the archive per instrument
Which datasets are available?
What is not calibrated? 
Can it be calibrated? 
How?  through community software package for instance?
If not, who/which group could do it? with eventually an estimate of the manpower needed when applicable.
What is missing? Please compare to what is available on PI or PI team member website(s)?
What would you recommend to archive in the long term?
Any ground-based data essential to support data analysis that shall be stored as well?
Are datasets stored in format compatible with community (visualization/data analysis) tools?

Be aware, that for instance on SOHO, there are various (PI) websites containing up-to-date derived products that are not stored in the archives yet. Which one of those, if any, shall be stored in ESA archives?

  

FP7/H2020 EU funded projects

EU funded projects related to Heliophysics 
-----------------------------------------------------
H2020    HESPERIA   
H2020    Flarecast
FP7       SEPServer       

Review guidelines: shall we archive anything of the value-added datasets produced by the various Heliophysics archives related FP7/H2020 projects listed below? If yes, which one(s)? 

General HAUS recommendation 

For long term preservation, the HAUS group recommends that the best practice is to store a calibrated version of each dataset, together with raw data. Clear guidelines on how these calibrated datasets have been generated shall of course be made available. 

B. Fleck (SOHO/IRIS/HINODE MM and PS) participated to this meeting, commenting on all SOHO related action items (AI) raised at the last meeting. He also presenting the status of the SOHO archive at GSFC and the long term archiving of HINODE and IRIS data at ESA (see detailed minutes and presentations).
G. DeMarchi presented the approved long term strategy of the ESDC (see detailed minutes and presentation).
P. Osuna (SOLO archive scientist) presented a first outlook of the Solar Orbiter archive under development (see presentation).
A. Masson presented the recent setup of the International Heliophysics Data Environment Alliance(IHDEA) and its relevance for the ESA heliophysics archives (see presentation).  


 Agenda

Agenda

Agenda ItemNotes
 Status of action items related to Ulysses A. Masson, 20 mn
 Review of Ulysses URAP experiment  C. Chen, 30 mn
 Review of Ulysses EPAC experiment N. Dresing, 30 mn
The International Heliophysics Data Environment Alliance (IHDEA)A. Masson, 30 mn
Status of SOHO archive at GSFC and long term HINODE and IRIS archivesB. Fleck, 30 mn
Status of action items related to SOHOB. Fleck, 30 mn
Science Archives Long-term StrategyG. De Marchi, 45 mn
Review of FP7/H2020 heliophysics projectsV. Génot, 30 mn
Review of SOHO CELIAS instrumentN. Dresing, 30 mn
Solar Orbiter archive: development statusP. Osuna, 30 mn
Interest of interconnecting ESA heliophysics archives with DKIST and other GB solar observatoriesM .Roth, 30 mn
 HAUS group Recommendations

HAUS group Recommendations

Major
HAUS 03, MAR-1 about UFA support: The HAUS group acknowledged the unique scientific value of the data available through the ESAC Science Data Center Ulysses Final Archive (UFA). It recommends to provide ASAP the ESDC heliophysics archives’ group enough manpower not only to maintain UFA online, but also ingest new (best calibrated) data and develop up-to-date command line tools/APIs to improve UFA's interoperability.  These tools are key to support the scientific community not only to access the Ulysses data but also compare them with data collected by the Parker Solar Probe and upcoming Solar Orbiter missions. Moreover, these data could be used by Solar Orbiter PI teams in their science activity plan, in particular for out-of-the ecliptic science plan. 

HAUS 03, MAR-2 about Calibrated EIT files, EIT PI team to deliver ASAP to SOHO archives at ESAC and Goddard EIT level 2 calibrated files to avoid Solarsoft dependence for data calibration, wider data exploitation and long term preservation..

Minor
HAUS 03, MIR-1 on UFA interoperability: SAMP is implemented on the Ulysses server but a contribution by the ESDC for the most popular Heliophysics Python 
libraries to be able to easily download Ulysses data from these libraries 
would be an asset. A TAP server may also be considered.

 Actions

Actions

IDDescriptionAssigneeDue DateStatus
 HAUS 02, AI-01

To a review the content of VIRGO datasets, software and value-added products at PMOD/WRC and the scientific interest of their possible inclusion in the SSA at ESDC. 
SOHO PS will work with the VIRGO team to provide calibrated data for all three sensors (TSI, SPM, and LOI). Value added PMOD products have been produced in the course of the SOLID FP7 project, which ran from 2012 until 2015 (https://projects.pmodwrc.ch/solid/). The related datasets are considered to be beyond the scope of the ESAC heliophysics archive. Beside, these datasets are already outdated (or will be so soon). Action closed. See related action item HAUS 03-AI-08 for a follow-up action on VIRGO mission long files.

M. Roth/B. Fleck/ESDCHAUS group meeting #4Closed
 HAUS 02, AI-02To check with the SOHO PS if any of the dozens MDI datasets available at Stanford, http://soi.stanford.edu but not at ESAC SSA shall be included in the SSA for long term preservation. A detailed list of datasets is available in Annex B (minutes of 2nd HAUS group meeting).
Complete MDI datasets will be part of the SOHO archive at GSFC once copy of SDO archive to SDAC is completed. SOHO PS will discuss with M. Roth which of the many (and large!) MDI datasets should be included in ESAC archive.
B. Fleck/M. RothHAUS group meeting #4Open
 HAUS 02, AI-03To check with the SOHO PS if the calibrated version of GOLF data from the PI data at CEA Saclay and Nice observatory shall be eventually ingested in the ESAC SSA for long term preservation.
The action has been completed. See HAUS 03-AI-09 for a follow-up action on the ingestion and availibility of a new GOLF mission-long file.
A. Masson/B. FleckHAUS group meeting #3Closed
 HAUS 02, AI-04To check with the SOHO PS if the following value added datasets suggested by M. Roth, highly complementary to SOHO data, shall be archived within the ESDC SSA
  • ingestion of HMI datasets during the overlapping period when both instruments were operating,
  • "Mark-I" solar spectrophotometer database (IAC, Spain)

SOHO PS does not see this as a high priority. The HMI data in particular is a huge data set, several orders of magnitude bigger than the complete SOHO archive, and the Mark-I instrument is only one of the global BISON network. ESDC SOHO archive shall not replicate services offered by the Virtual Solar Observatory (VSO), which provides access to data from all these instruments. Action closed, See HAUS 03-AI-10 for a follow-up action

A. Masson/B. FleckHAUS group meeting #3Closed
 HAUS 02, AI-05To check if a calibrated version of all SUMER data exists at Max Planck MPS and shall be ingested within the ESDC SSA, together with the raw data presently available. Thumbnails of the SUMER data only available at MPS may also be considered to be ingested in order to browse these data.
A complete new set of SUMER level-0 and level-1 data is expected very soon but not yet available. It will be organized by “studies” and provided as tar balls of individual FITS files for each exposure, and a “summary” FITS header describing the tar ball. Ingestion of the thumbnails of spectra shall be considered by ESDC even though not considered as a high priority by the SOHO PS. Acton closed.
A.Masson/B. FleckHAUS group meeting #3Closed
 HAUS 02, AI-06

To discuss with SOHO PS on possible ingestion of LASCO level 1 data, a calibrated version of all LASCO data and CME catalogs (e.g. CACTUS, ARTEMUS, CORIMP, CDAW, NRL LASCO CME list incomplete, COR1 CME catalog, HI1, catalogs genertaed by the HELCATS project).

Level 1 data does not seem to be a high priority for the team, as almost all of LASCO science can be and is done with level 0.5 data. Hence no final calibrated Level 1 data are available yet. The various CME catalogs are still “living” objects, and it may be premature and the community might prefer the original source. These catalogs will be ingested in the SSA once the SOHO mission is in post-ops.

A. Masson/B.FleckHAUS group meeting #3Closed
 HAUS 02, AI-07To ingest new level 3 COSTEP/EPHIN data set under development at Kiel university (protons, helium) and documentation when received, expected end of August 2018. Possibility of visualizing the data shall be considered.
Data downloaded at ESDC but not yet made available yet in the SOHO archive at ESDC. Data visualization to be estimated in terms of manpower.
ESDCHAUS group meeting #4Open
 HAUS 02, AI-08

To ingest new ERNE .SL2 data set with H and He intensities in preparation by Turku group and documentation when received, expected end of August 2018.
The delivery of the same data with a finer energy resolution is also envisaged by this group. Possibility of visualizing the data shall be considered.
Data ingested and available at ESDC. Data visualization to be estimated in terms of manpower. 

ESDCSeptember 2018Closed
 HAUS 02, AI-09

Basic visualization is very useful for spectroscopic data. An 'atlas' is included at the RAL CDS website (http://solar.bnsc.rl.ac.uk/atlas/atlas.shtml). To be investigated.
Yet to be investigated in term of ESDC manpower, not a priority. 

ESDCHAUS group meeting #4Open
 HAUS 02, AI-10

Access to calibrated CDS data is a priority. ESDC SSA shall store calibrated data. At bare minimum clear information about the calibration procedure with solarsoft shall be made available. A simple calibration version of the data assuming single component shift for each spectral line enables to produce a calibrated version of  velocity and intensity maps
Reprocessed calibrated CDS/NIS level 1 expected soon. HIgh priority.

ESDCHAUS group meeting #4Open
 HAUS 02, AI-11EIT Calibration info on https://umbra.nascom.nasa.gov/eit/eit_guide/gui de.html shall at least linked, best transferred.
Calibrated (level 2) EIT data highest priority as regards EIT but PI team has not yet finalized its calibration (see recommendations)
ESDCHAUS group meeting #4Open
 HAUS 02, AI-12EIT movies from CME catalogues shall be included, this is a useful resource.
AI rejected as this can be done through for instance JHelioviewer, time animator at ESDC SSA or SOHO movie theater. Action closed 
ESDCHAUS group meeting #3Closed
 HAUS 02, AI-13

For imaging data, the EIT daily movies are critical.  They can be used by users from any community easily. And can give basic info
(e.g. is there a coronal hole, is there a flare, are there active regions?). Movies available at IAS shall be ingested in the ESDC SOHO
science archive. 
https://www.ias.u-psud.fr/eit/movies/

AI rejected as this can be done through for instance JHelioviewer, time animator at ESDC SSA or SOHO movie theater. Action closed 

ESDCHAUS group meeting #3Closed
 HAUS 02, AI-14Update all hyperlinks to instrument team pages of the Ulysses COSPIN experiment on the Ulysses Final Archive. All hyperlinks updated. Action closedESDCHAUS group meeting #3Closed
 HAUS 02, AI-15

To follow up on the delivery of Ulysses KET and HET intensity files to ESDC 
B. Heber, PI on the KET instrument will eventually deliver a version of the KET files in intensities instead of rates. May take until the launch of Solar Orbiter. For HET, there is no response from the PI despite a contact by email in June 2018.

 N. Dresing/A. Masson N/ASuspended
 HAUS 02, AI-16

 To check any data is missing from Ulysses HFT (until 2008 244) and AT (until end 2008) for prime resolution and HET for high resolution data (until 2004 173)
There is no response from the PI despite a contact by email in June 2018.

 N. Dresing/A. Masson N/ASuspended
HAUS 02, AI-17

For Ulysses DUST, Ingest V3.1 PDS3 DUST dataset available at the NASA PDS small bodies sub node.
Ingestion of new data can only be done after the upgrade and migration of the Ulysses Final Archive hardware, DB, java version, OS (planned around Jan. 2019). New data (e.g. DUST) will be ingested afterwards on best effort basis.

ESDCIngestion of new data after UFA upgrade, best effort basisOpen
HAUS 02, AI-18

To interact with the Ulysses FGM/VHM PI team for a possible delivery of a CDF ISTP compliant version of the files, together with a more logical dataset tree
(all data from one instrument in one place) with unzipped files  labelled in clear consistent way, use 4-digit years in filenames, use of consistent time stamps in files.
A version in CDF is available at NASA/GSFC/SPDF. To be further investigated.

ESDCIngestion of new data after UFA upgrade, best effort basisOpen
HAUS 02, AI-19For the Ulysses GAS experiment, fix the broken link to the PI team at Max Planck MPS; ingest newest calibrated version dataset, recalibrated in 2013.
Ingestion of new data can only be done after the upgrade and migration of the Ulysses Final Archive hardware, DB, java version, OS (planned around Jan. 2019). New data (e.g. DUST) will be ingested afterwards on best effort basis. 
ESDCIngestion of new data after UFA upgrade, best effort basisOpen
HAUS 02, AI-20

For the Ulysses GRB experiment, to correct the data description (wrong labelling of the schematic view of the detectors); compile C and Fortran programs to generate the solar flares (prime resolution) and GRB datasets (high time resolution) in a readable format; ingest them in the UFA; Data visualization with meaningful time resolution. 
Data description has been corrected with proper labelling of the schematic view of the detectors. The rest of the actions are still pending, on best effort basis.

ESDCIngestion of new data after UFA upgrade, best effort basisSome progress, action open
HAUS 02, AI-21For the Ulysses GWE experiment, to investigate why no data is stored in the archive.
Not done. To be investigated by A. Masson with support from C. Tranquille on best effort basis. 
A. MassonHAUS group meeting #4Open
HAUS 02, AI-22For the Ulysses SWICS experiment, to make available the latest version of the latest calibration software (version 2.3), available on the Ulysses cosmos page.
Detailed test of this software shall be performed. 
Not done. To be first tested by A. Masson with C. Tranquille (on best effort basis), before a possible ingestion into UFA.
ESDCBest effort basis after UFA upgradeOpen
HAUS 02, AI-23

For the Ulysses SWOOPS experiment, to interact with Bruce Goldstein and Anna Tenerani to ingest the following missing datasets in the UFA 

  • missing ion distributions: important to include (Bruce Goldstein)
  • updated ion moments: moments with anisotropy/drift at NASA, should be included 
  • electron distributions  (Bruce Goldstein)
  • full electron moments, including heat flux? (Goldstein and Anna Tenerani)
  • If possible: DSN +operational reports
  • spacecraft pointing for 3D distributions

Discuss with PI and NASA their conversion to CDF

A. Tenerani has been contacted and delivered new data collected, including three types of data per species (protons and alpha particles):
1) ancillary data and moments from the distribution function of each species
2) ancillary data and the distribution function (2-dimensional matrix)
3) ancillary data and the parameters of bi-maxwellian fits to distribution function

To be ingested on UFA.

ESDC/C. ChenNew data has been requested and collected. To be ingested after UFA upgrade on best effort basis.Some progress, action open
HAUS 03, AI-01Implement clear on-line documentation on how to use the command line on the Ulysses archiveESDCBest effort basisOpen
HAUS 03, AI-02URAP user notes say sounder data is for calibration and will be published in the literature FES data will be archived at a later date.
Hence, ESDC to check if URAP Sounder data and/or Fast Envelope Sampler data exist and could eventually be stored in the ESA Archive
ESDCBest effort basisOpen
HAUS 03, AI-03To check with the PI team if there is any URAP higher resolution data that is not on the archive, or better calibrated. Contact URAP US website to be linked to UFA (https://urap.gsfc.nasa.gov/links.html)ESDCBest effort basisOpen
HAUS 03, AI-04To estimate the manpower necessary to implement the following recommendation of the HAUS group reviewer on the Ulysses URAP experiment
CDF files (ISTP compliant meta-data) preferred, + ascii files are helpful too
• Directory structure could be made clearer
–no need to split into prime resolution and additional data
–logical file structure, labelled in consistent way
–documentation files in one place
• Make available by machine interface, e.g. ftp
• Prefer unzipped files 
ESDCBest effort basisOpen
HAUS 03, AI-05To fix typo and broken links on the EPAC instrument description on the Ulysses Final Archive (Title to be corrected to Energetic PArticle Composition instead of The Ulysses Particle Investigation) and in the Data available section: Full resolution of Electron and heavy ions fluxes instead of Full resolution Electron and Proton intensities) + fix broken link. Moreover, in the documentation (UDSepacdata_EPAC_Doc.pdf, update the link http://www.mps.mpg.de/en/projekte/ulysses to https://www2.mps.mpg.de/de/projekte/ulysses/epac/ ESDCBest effort basisOpen
HAUS 03, AI-06To double check with PI and/or Ulysses expert if the data content of the EPAC PHA product in the UFA is complete wrt to mission operations. At the moment, it only covers from 1990 DOY 292 to 1998 DOY 291.ESDCBest effort basisOpen
HAUS 03, AI-07

To recover a number of "Related Documents mentionned in the Ulysses EPAC documentation (UDSepacdata_EPAC_Doc.pdf)

ESDCBest effort basisOpen
HAUS03-AI-08VIRGO mission long calibrated datasets of Total Solar Irradiance (TSI) are available as single ASCII files in the SOHO Goddard archive as well as mission-long photospheric activity index S_Ph from CEA Saclay. At ESDC, only daily files are available, those mission long files need to be ingested and made clearly available in the Proc Level submenu of the search page as Mission long. Moreover, there are also two processing levels versions for VIRGO files: 1.0 and 1.02 while 1.02 is only applicable to SPM. Hence, 1.02 shall be changed to 1.02 (SPM only) in the Search page Instrument panel Proc Level submenu. To be clarified with the VIRGO team Finally, a detector named N/A shall be changed to N/A (housekeeping).ESDC/B. FleckHAUS group meeting #4Open
HAUS 03, AI-09

Three new GOLF mission-long calibrated line-of-sight Doppler velocities, described in Appourchaux et al. 2018A&A...617A.108Aare part of the SOHO archives (both at Goddard and ESAC). At the SOHO Goddard archive, old (deprecated) data have been moved to a private area on the server and will be available upon request (but not searchable through the GUI). This work needs to be done at SOHO ESDC archive, including

  • Deprecated mission-long calibrated files, of processing levels 2A, 2B, 2C, shall not appear as a result of a search of calibrated files for GOLF. The processing levels 2A, 2B and 2C shall be hidden in the Proc Level combo box. Instead, Calibrated mission-long series (level 2) shall be visible in this combo box.
  • Raw data (level 1) shall replace 1 in the Proc level combo box.
ESDCHAUS group meeting #4Open
HAUS 03, AI-10To ingest of a new version of the SUMER data composed of level-0 and level-1 data expected to be delivered very soon, but not yet available. It will be organized by “studies” and provided as tar balls of individual FITS files for each exposure, and a “summary” FITS header describing the tar ball. Ingestion of the thumbnails of SUMER spectra available as Max Plack MPS shall be estimated in terms of ESDC manpower needed. ESDCHAUS group meeting #4Open
HAUS 03, AI-11

SOHO PS to contact the UVCS PI team to clarify which of the files delivered to SDAC are calibrated and transfer those files to ESDC for their ingestion and availability.

B. Fleck/UVCS PI team/ESDCHAUS group meeting #4Open
HAUS 03, AI-12


Most urgent action related to CELIAS data


ingest on the SSA mission long file bundles available for (STOF) Solar EUV Monitor and (MTOF) Proton Montor


Proton Monitor data
Two mission long tar files are available on the Virtual Solar Observatory for the calibrated CELIAS Proton Monitor data https://vso.nascom.nasa.gov/new/web_patch/SOHO_ML.html


    • CELIAS_Proton_Monitor_5min.tar.gz: bundle of year long files containing measurement at 5 minutes time resolution, from 1996 to 2016, please note that data for the year 2016 only, data are available up to 28 Sep. 
    • CELIAS_Proton_Monitor_30s.tar.gz: bundle of year long files containing measurement at 30s minutes time resolution, from 1996 to 2015.
    • These two files shall be ingested on the SSA with 
      • a processing level set as "Calibrated (L1)" (internal SSA metadata)
      • a calibration status set as true (internal SSA metadata), essential for command line access and GUI usability
      • an observation type set to "Solar wind param. 30s" or "Solar wind param. 5mn" instead of Proton
      • a Temp Res internal metadata parameter set at either 30s or 5mn


 Solar EUV Monitor


Two mission long tar files are available on the Virtual Solar Observatory for the calibrated CELIAS Proton Monitor data https://vso.nascom.nasa.gov/new/web_patch/SOHO_ML.html


CELIAS_SEM_15sec_avg.tar.gz, data from 1996 to 2017-05-11 stored in either monthly or daily ascii files per year, 15s resolution
CELIAS_SEM_daily_avg.tar.gz, data  from 1996 to 2017-05-11 stored in yearly files


 For these two files shall be ingested on the SSA with 


  • a processing level set as "Calibrated (L1)" (internal SSA metadata)
  • a calibration status set as true (internal SSA metadata), essential for command line access and GUI usability
  • a detector set to Solar EUV Monitor and not STOF
  • an observation type set to "EUV Irradiance 15s avg." or "EUV Irradiance 1 day avg." instead of Solar EUV Monitor
  • a Temp Res internal metadata parameter set at either 15s or daily


Point of discussions with the SOHO PS


    • The two PM tar files superseed all the historical 1304 daily CDF files from 21/01/1996 to 24/02/2000, 5 min time resolution, available now at the SSA. These CDF files shall not be downloadable anymore. To be double checked with SOHO PS.
    • The two SEM files superseed the 49 monthly SEM CDF files of type SO_CEL_STOF_SEM_L1_V01-YYYYMMDDHHmm.cdf of 5 mn time resolution, covering the period from 1996 to 1999. These files shall not be available anymore for download.
    • Is there any way to get more up-to-date data bundles?
    • SOHO PS to confirm if SEM files can be associated with the SEM and not STOF as detector, even if SEM is a subunit physically attached on top of the STOF sensor; at the moment all the SEM files available are associated with STOF hence no data are available if SEM is selected; which is confusing. If STOF is the sensor decided, all these files shall be re-ingested accordingly.
    • Possible ingestion of all (up-to-date) PM calibrated files per Carrington Rotation at 1h time resolution available at the University of Maryland: http://umtof.umd.edu/pm/crn/; please note that these files do not content any (predicted not measured) Helium speed/velocity.
    • Up-to-date SEM files 15s and daily averages are available here: https://dornsifecms.usc.edu/space-sciences-center/download-sem-data/ ; shall SSA ingest them on a daily basis?


For CELIAS/CTOF

(i) CTOF 221 CDF « Level 1 Science Data » calibrated files at ESDC/SSA need to be reiingested as follows

  • Processing level renamed as « Calibrated (L1) » instead of « Level 1 Science Data » 
  • Calibrated metadata set to true, essential to improve usability and accuracy of the GUI and the SAIO, also called API or command line. 
  • Temp Res keyword set as 5 minutes
  • Modify Observation type metadata as follows
Observation type (actual)Updated observation typeCTOF Calibrated files affected
O densitiesO6+, O7+, O densities138 files of type SO_CEL_CTOF_ODENSITIES_L1_V01.cdf
Fe DensitiesFe, S, Si densities80 files of type SO_CEL_CTOF_FEDENSITIES_L1_V01.cdf
Fe and Proton DensityFe, p densities, Vsw, V_kinSO_CEL_CTOF_FEPDENSITY_L1_V01-199604212355.cdf
Fe freeze-in temperature

Fe freeze-in temperature

SO_CEL_CTOF_FEFTEMP_L1_V01-199605282359.cdf 
Heavy ion Freeze-in Temp…O freeze-in temperature, n_O, v_O, v_Si, v_FeSO_CEL_CTOF_HIONFTEMP_L1_V01199605290000.cdf



(ii) Re-ingest all 5,616 raw data files with a Processing level set to « Raw data » instead of « Level ASCII » which is meaningless.


(iii) The availibility of raw data files up to 7 Sep. 2018 is confusing while only 4 months of science data available (April 1 to August 16 1996); documentation would help. PI team may provide it, to be discussed between SOHO PS and PI team.


(iv) New high level data (velocity spectra of heavy ions+n,T) could be provided by the PI group (at some point), as mentioned by the reviewer. Hence, a discussion between the SOHO PS and CELIAS PI team of data delivery timeline shall be held.


For CELIAS/MTOF

Status of raw data content: up to 7 September 2018 (25,146 files on February 20, 2019), to be re-ingested with Processing level set to « Raw data »
Status of science data content: 1 file, SO_CEL_MTOF_CME_L1_V01-199701082357.cdf, to be re-ingested with

    • Processing level renamed as « Calibrated (L1) » instead of « Level 1 Science Data » 
    • Calibrated metadata set to true, essential to improve usability and accuracy of the GUI and the SAIO, also called API or command line. 
    • Temp Res keyword set as 5 minutes
    • Modify Observation type set to Heavy ion densties and SW param. instead of "coronal mass ejection"

To be further discussed with SOHO PS and PI team if any other calibrated file of this type is available.


STOF

  • Status of raw data content: up to 7 September 2018 (29,767 files on February 26, 2019), to be re-ingested with Processing level set to « Raw data »
  • Status of science data content: from 1996 to 1999, 49 monthly SEM files, 5mn average; most likely superseeded by the SDAC mission long data bundles and up-to-date data available at https://dornsifecms.usc.edu/space-sciences-center/download-sem-data/ 

Measures ions in the range from 20 keV to 3 MeV per charge through two separate sections of the electrostatic deflection system, one with curved (STOF) and one with flat deflection plates at higher energies (HSTOF). Energy measurement is done via a solid state detector. The Solar EUV Monitor (SEM) is physically mounted on top of the STOF detector. it measures the full solar disk absolute photon flux for the He II 30.4 nm line, as well as the absolute integral flux between 17 and 70 nm. The detector associated with the SEM data is STOF, hence no calibrated SEM data appear with SEM is selected as detector


HSTOF

  • Status of raw data content: up to 7 September 2018 (9,341 files on February 26, 2019), to be re-ingested with Processing level set to « Raw data »
  • Status of science data content: 1 file, SO_CEL_HSTOF_EHA_L1_V01-199602290000.cdf, to be re-ingested with
      • Processing level renamed as « Calibrated (L1) » instead of « Level 1 Science Data » 
      • Calibrated metadata set to true, essential to improve usability and accuracy of the GUI and the SAIO, also called API or command line. 
      • Modify Observation type set to Hydrogen ENA flux instead of "Energetic Neutral Hydrogen Atom"


Content of HSTOF (high energy part of STOF) science data for the whole mission is 1 CDF file, covering period from 29/02/1996 to 23/12/1997. HSTOF has been working for years and is presently the only ENA instrument with energy range overlapping with Voyager Low Energy Charged Particle measurements of the energetic ion fluxes in the heliosheath, according to this article https://www.aanda.org/articles/aa/pdf/2018/10/aa32432-17.pdf. Until 2003, the whole range of the ecliptic longitude was scanned yearly. After 2005, only the flank sectors are accessible to observations. Hence, it is not understandable that only 2 years of calibrated data only are available on the SSA. To be further investigated.


Data Completeness wrt to GSFC


Here is the status on 26 February 2019


- CTOF: Goddard 20360 files (max (imported_date) 2019-01-11 07:58:54) /  SSA 15837 files
- MTOF: Goddard 33823 files (max (imported_data) 2019-01-11 07:59:13  /  SSA 21147 files
- STOF: Goddard 20463 files (max (imported_data) 2019-01-11 07:59:31)  /  SSA 29816 files
- HSTOF: Goddard 6768 files (max (imported_data) 2019-01-11 07:59:00)  /  SSA 9342 files


Discrepencies shall be investigated


Detectors Ordering on the SSA GUI

The ordering of the CELIAS detectors on the SSA is in alphabetical order and could be improved by listing them by sensors as suggested by the reviewer from high to low energy detectors, to be discussed with the SOHO PS. At the moment there are listed alphabetically.

Alternatively it could be listed as follows

Proton Monitor
Solar EUV Monitor
CTOF
MTOF
STOF
HSTOF
DPU 

To be discussed.


B. Fleck/UVCS PI team/ESDCHAUS group meeting #4Open


Detailed minutes

 1. Ulysses archive at ESDC: experiment reviews and recommendations

1. Ulysses archive at ESDC: experiment reviews and recommendations

See Annexes 1, 2 and 3

1.1    URAP review

See Annex 1 

Short experiment description
The Unified Radio And Plasma wave experiment (URAP) measures the electric field from DC to 1 MHz and the magnetic field from 0.22 to 450 Hz. It is composed of a 72.5 dipoloe antenna in the spin plane, a 7.5 m monopole along the spin axis and 2 orthogonal search coil magnetometers. The PI is Bob MacDowall (GSFC) and data contact point is Roger Hess (GSFC). For more details on instrument charcteristics and main scientific objectives and first results, please see Annex 2.

PI website
https://urap.gsfc.nasa.gov

Data content
URAP at the Ulysses Final Archive at ESDC contains science data (in physical units)and quicklook plots for the whole mission. Science data are composed of the following folders

df: direction finding results, monthly .zip files, within each is daily ascii .txt file, 10 min resolution, there is good documentation file for this product
pfra/pfrp/rara/rarp/wfba/wfbp/wfea/wfep: frequency spectra in different ranges, yearly .zip files, within each is daily ascii .ulyfile, 10 min resolution, good documentation
rara144: same as raraexcept 144 second resolution, and uses .ravfile extension

Some additional data sets and, documentation and software.

HAUS 03, AI-2 on ESDC, URAP user notes say sounder data is for calibration and will be published in the literature FES data will be archived at a later date. Hence, ESDC to check if URAP Sounder data and/or Fast Envelope Sampler data exist and could eventually be stored in the ESA Archive
HAUS 03, AI-3 on ESDC, to check with the PI team if there is any URAP higher resolution data that is not on the archive, or better calibrated. Contact URAP US website to be linked to UFA (https://urap.gsfc.nasa.gov/links.html)
HAUS 03, AI-4 on ESDC, to estimate the manpower necessary to implement the following recommendation of the HAUS group reviewer on the Ulysses URAP experiment

CDF files (ISTP compliant meta-data) preferred, + ascii files are helpful too
Directory structure could be made clearer
no need to split into prime resolution and additional data
logical file structure, labelled in consistent way
documentation files in one place
Make available by machine interface, e.g. ftp
Prefer unzipped files

1.2    EPAC review

See Annex 2

Short experiment description
The Energetic PArticle Composition (EPAC) experiment measures the energetic particle fluxes (electrons, protons, heavuy ions), PI: N. Krupp (MPS, Göttingen, Germany). In more details, EPAC measures the flux anisotropy and chemical compostion so particles with energy between 300 keV and 25 MeV per nucleus. It is composed of four identical telescopes with different inclination angles with respect to the spin axis of Ulysses.

PI website
https://www2.mps.mpg.de/de/projekte/ulysses/epac/ ; no data provided at the PI website.

HAUS 03, AI-5 on ESDC, To fix typo and broken links on the EPAC instrument description on the Ulysses Final Archive (Title to be corrected to Energetic PArticle Composition instead of The Ulysses Particle Investigation) and in the Data available section: Full resolution of Electron and heavy ions fluxes instead of Full resolution Electron and Proton intensities) + fix broken link. Moreover, in the documentation (UDSepacdata_EPAC_Doc.pdf, update the link http://www.mps.mpg.de/en/projekte/ulysses to https://www2.mps.mpg.de/de/projekte/ulysses/epac/

Data content
Prime resolution PHA: 1990 292 - 1998, 291 (binary files), calibrated, documented

Prime resolution ulykep-xx: whole mission (1990-2007), calibrated, documented
No high resolution data

HAUS 03, AI-6 on ESDC, To double check with PI and/or Ulysses expert if the data content of the EPAC PHA product in the UFA is complete wrt to mission operations. At the moment, it only covers from 1990 DOY 292 to 1998 DOY 291.

HAUS 03, AI-7 on ESDC, To recover a number of "Related Documents mentionned in the Ulysses EPAC documentation (UDSepacdata_EPAC_Doc.pdf)

1.3   New AIs and recommendations

Ulysses Final Archive (UFA) is a legacy archive hence any work is done on best effort basis. The most urgent action at the moment is to upgrade of hardware, DB, java version, OS (planned around Jan. 2019). New data (e.g. DUST) will be ingested afterwards (2019). We are also happy to report that Cecil Tranquille, Ulysses expert, recently joined the Cluster SOC team at ESAC and is willing to support the HAUS group on a best effort basis. All AIs related to Ulysses are reviewed in Annex 3.

New Action Items (AIs)
HAUS 03, AI-1 on ESDC, implement clear on-line documentation on how to use the command line on the Ulysses archive

HAUS 03, AI-2 on ESDC, URAP user notes say sounder data is for calibration and will be published in the literature FES data will be archived at a later date. Hence, ESDC to check if URAP Sounder data and/or Fast Envelope Sampler data exist and could eventually be stored in the ESA Archive

HAUS 03, AI-3 on ESDC, to check with the PI team if there is any URAP higher resolution data that is not on the archive, or better calibrated. Contact URAP US website to be linked to UFA (https://urap.gsfc.nasa.gov/links.html)

HAUS 03, AI-4 on ESDC, to estimate the manpower necessary to implement the following recommendation of the HAUS group reviewer on the Ulysses URAP experiment

• CDF files (ISTP compliant meta-data) preferred, + ascii files are helpful too
• Directory structure could be made clearer
–no need to split into prime resolution and additional data
–logical file structure, labelled in consistent way
–documentation files in one place
• Make available by machine interface, e.g. ftp
• Prefer unzipped files

HAUS 03, AI-5 on ESDC, To fix typo and broken links on the EPAC instrument description on the Ulysses Final Archive (Title to be corrected to Energetic PArticle Composition instead of The Ulysses Particle Investigation) and in the Data available section: Full resolution of Electron and heavy ions fluxes instead of Full resolution Electron and Proton intensities) + fix broken link. Moreover, in the documentation (UDSepacdata_EPAC_Doc.pdf, update the link http://www.mps.mpg.de/en/projekte/ulysses to https://www2.mps.mpg.de/de/projekte/ulysses/epac/

HAUS 03, AI-6 on ESDC, To double check with PI and/or Ulysses expert if the data content of the EPAC PHA product in the UFA is complete wrt to mission operations. At the moment, it only covers from 1990 DOY 292 to 1998 DOY 291.

HAUS 03, AI-7 on ESDC, To recover a number of "Related Documents mentionned in the Ulysses EPAC documentation (UDSepacdata_EPAC_Doc.pdf)

Recommendations
HAUS 03, MIR-1 on UFA interoperability: SAMP is implemented on the Ulysses server but a contribution by the ESDC for the most popular Heliophysics Python libraries to be able to easily download Ulysses data from these libraries would be an asset. A TAP server may also be considered.

HAUS 03, MAR-1 on UFA support: The HAUS group acknowledged the unique scientific value of the data available through the ESDC Ulysses Final Archive. It recommends to provide ASAP the ESDC heliophysicsarchives’ group enough manpower to maintain, ingest new (best calibrated) data and develop up-to-date command line tools/APIs.  These tools are key to support the scientific community not only to access the Ulysses data but also compare them with data collected by the PSP and upcoming Solar Orbiter missions.

Please note that all action items related to Ulysses at the last meeting have been either closed or commented in the Actions section.

Ulysses

experiment

Ingestion status

Ok = latest version at ESAC 

Additional datasets / software

Complementary datasets

Thumbnails/QL plots

Visualization needed?

Visualization available?

COSPIN

Double check completeness of HFT, AT and HET datasets

First calibrated data from HET and KET are expected
KET intensities instead of event rates data are expected from the PI team by 2020. No feedback from HET PI

None

Yes

Yes

Moments only

DUST

Newer data at NASA PDS small bodies subnode (V3.1 PDS3)

None

None

None

No

No

EPACOk, double check completeness of PHANoneNoneYesYesYes

FGM/VHM

Ok

CDF ISTP version would help
CDF files found, to be ingested 

None

Yes

Yes

Better resolution neeeded

GAS

Recalibrated data at MPS shall be ingested 

Other extra datasets available at MPS

None

Yes

Yes

QL plot viewer

GRB

Ok

Compile C and Fortran 77 routines to provide calibrated data

Solar X-flares catalogue over the Ulysses mission

Yes but time resolution meaningless for Solar flares and GRB

Yes

Yes but time resolution => meaningless

GWE

No data

None

None

No

No

No

SWICS

Ok

Update SWICS software (v.2.3)


QL plots available

Yes

Moments only

SWOOPS

Missing files available at NASA

Interact with Bruce Goldstein and Anna Tenerani for double checking data completeness
 

None

QL plots available

Yes

Moments only

URAPOkCheck availibility of Sounder and Fast Evnvelope Sampler dataNoneYesYesYes
 2. SOHO archive at ESDC: review and recommendations

2. SOHO archive at ESDC: experiment reviews and recommendations

Reviewers
Louis Harra (UVCS), see annex 4
Nina Dresing (CELIAS), see annex 5

2.1   UVCS

See Annex 4

Short experiment description
UVCS (UltraViolet Coronograph Spectrometer) made measurements of the solar corona between 1.3 and 12 solar radii from sun center with high spectral and spatial resolution. The instrument creates an artificial solar eclipse in ultraviolet light, and allowed observation of the less intense light from the extended corona.The corona emits most of its ultraviolet intensity in spectral emission lines, containing information of the highly-ionized coronal plasma. At UV wavelengths:

• proton velocity distribution
• proton outflow velocity
• electron temperature
• O VI, Si XII, and Mg X outflow velocities and densities
• O VI, Si XII, and Mg X velocity distributions
• electron density and minor ion densities

UVCS has observed about 37 spectral lines in the extended solar corona for H I and ions of C, N, O, Mg, Al, Si, S, Fe, and Ni.

The purpose of the UVCS is to provide a detailed description of the extended solar corona that can be used to address a broad range of scientific questions regarding the nature of the solar corona and the generation of the solar wind.

PI team information
John L. Kohl: Principal Investigator (Harvard, USA)
Giancarlo Noci: Co-Principal Investigator (Universita di Firenze, Italy)
Website: https://www.cfa.harvard.edu/uvcs/

The UVCS PI team has defined the levels of its processed data as follows. As described on the PI website, the output from the UVCS instrument and delivered to the UVCS project by NASA, is called level 0 data. Level 0 data is not useful for any purpose except processing into level-1 data. 

Level-1 data is processed raw data which has been decompressed and repacked into data files in a logical form representative of the data products as delivered by the UVCS instrument. Level-1 data is stored in a standard external storage protocol (FITS files). Types of Level-1 data are the following: 

  1. Spectral Data Files: These are FITS files containing uncalibrated detector count data (counts per pixel), as well as instrument configuration and exposure timing data. Each Exposure Sequence produces one Spectral Data File per detector. Each file includes a header that contains a brief, standard description for use by investigators.  

  2. Visible Light Data Files: These are FITS files containing the time series of count rates from the White Light Channel, along with pointing information and information on the polarizer and neutral density filter. Each visible light data (VLD) file contains the data corresponding to one exposure sequence. Each file includes a header that contains a brief, standard description for use by investigators. 

Level-2 data are calibrated level-1 data which has been decompressed and are stored in a standard external storage protocol (FITS files). Types of Level-2 data are the following: 

  1. Calibrated Spectral Data Files: These are FITS files containing calibrated detector intensities (photons cm -2 s-1 sr -1), as well as instrument configuration and exposure timing data. All sequential exposures in a given file for a given height, grating position and common instrument configuration are added together except in the case of Star and CME observations. Each Exposure Sequence produces one Spectral Data File per detector. Each file includes a header that contains a brief, standard description for use by investigators.  

  2. Calibrated Visible Light Data Files: These are FITS files containing polarized brightness (pB) values from the White Light Channel, along with pointing information and information on the polarizer and neutral density filter. Each VL Data File contains the data corresponding to one exposure sequence. Each file includes a header that contains a brief, standard description for use by investigators. 

Calibration Data Files contain the following: a) Parameter files (ascii) for pointing, spatial, wavelength and radiometric calibrations, and b) Flat field and dark count data (FITS). 

The UVCS Data Analysis Software (DAS version 5.1) allows the user to perform calibrations and obtain spectra that are wavelength calibrated and converted to photons cm -2 s-1 sr -1 nm-1. Calibrated pointing information can also be derived using this software. The software allows the user to write out processed spectral data files and processed visible light data files. Temporary processed spectral and visible light data files are produced at the EOF to be used as Quick Look data for observation planning. Processed data files are examples of level-2 data.

In addition, the UVCS program produces level-3 data which has undergone a higher level of processing. For example, image data files, including synoptic maps, are FITS files such as HI Ly-alpha maps of the corona that are provided as part of the Summary Data. There are also Data Catalogs to permit users to select observations based on pointing, observation date, type of feature observed, or other criteria.  They also include references to Mission Log Files, which record the observing sequences and other commands sent to the instrument. Most level-3 data will be produced by scientists in the course of analyzing the data using software and data that is publicly available. 

The UVCS/SOHO Data Analysis Software (DAS) and the calibration and help files are included in the SolarSoft package. 

Data content at ESDC
Only Level 1 data are available at ESDC SOHO archive. SOHO data stored at Goddard, in the Solar Data Analysis Center (SDAC) contain the level 2 files (final calibration).
However, level 2 or calibration status never appear in the result page when requesting UVCS data from SDAC, through the Virtual Solar Observatory. Moreover, the file naming convention of these FITS files are not standard.

HAUS 03, AI-11 on B. Fleck/UVCS PI team/ESDC, SOHO PS to contact the UVCS PI team to clarify which of the files delivered to SDAC are calibrated and transfer those files to ESDC for their ingestion and availability.

2.2   CELIAS

See Annex 5

Actions related to CELIAS (see HAUS 03, AI-12 on ESDC/B. Fleck)

Short experiment description
CELIAS (Charge, ELement, and Isotope Analysis System) measures the mass, ionic charge and energy of the solar wind, supra-thermal ions, and low energy energetic particles. The CELIAS instrument consists of three different sensor units (CTOFMTOF, and STOF) coupled to a Digital Processing Unit (DPU).

Most urgent action related to CELIAS data

ingest on the SSA mission long file bundles available for (STOF) Solar EUV Monitor and (MTOF) Proton Montor

Proton Monitor data
Two mission long tar files are available on the Virtual Solar Observatory for the calibrated CELIAS Proton Monitor data https://vso.nascom.nasa.gov/new/web_patch/SOHO_ML.html

    • CELIAS_Proton_Monitor_5min.tar.gz: bundle of year long files containing measurement at 5 minutes time resolution, from 1996 to 2016, please note that data for the year 2016 only, data are available up to 28 Sep. 
    • CELIAS_Proton_Monitor_30s.tar.gz: bundle of year long files containing measurement at 30s minutes time resolution, from 1996 to 2015.
    • These two files shall be ingested on the SSA with 
      • a processing level set as "Calibrated (L1)" (internal SSA metadata)
      • a calibration status set as true (internal SSA metadata), essential for command line access and GUI usability
      • an observation type set to "Solar wind param. 30s" or "Solar wind param. 5mn" instead of Proton
      • a Temp Res internal metadata parameter set to either 30s or 5mn

 Solar EUV Monitor

Two mission long tar files are available on the Virtual Solar Observatory for the calibrated CELIAS Proton Monitor data https://vso.nascom.nasa.gov/new/web_patch/SOHO_ML.html

CELIAS_SEM_15sec_avg.tar.gz, data from 1996 to 2017-05-11 stored in either monthly or daily ascii files per year, 15s resolution
CELIAS_SEM_daily_avg.tar.gz, data  from 1996 to 2017-05-11 stored in yearly files

 For these two files shall be ingested on the SSA with 

  • a processing level set as "Calibrated (L1)" (internal SSA metadata)
  • a calibration status set as true (internal SSA metadata), essential for command line access and GUI usability
  • a detector set to Solar EUV Monitor and not STOF
  • an observation type set to "EUV Irradiance 15s avg." or "EUV Irradiance 1 day avg." instead of Solar EUV Monitor
  • a Temp Res internal metadata parameter set to either 15s or daily

Point of discussions with the SOHO PS

    • The two PM tar files superseed all the historical 1304 daily CDF files from 21/01/1996 to 24/02/2000, 5 min time resolution, available now at the SSA. These CDF files shall not be downloadable anymore. To be double checked with SOHO PS.
    • The two SEM files superseed the 49 monthly SEM CDF files of type SO_CEL_STOF_SEM_L1_V01-YYYYMMDDHHmm.cdf of 5 mn time resolution, covering the period from 1996 to 1999. These files shall not be available anymore for download.
    • Is there any way to get more up-to-date data bundles?
    • SOHO PS to confirm if SEM files can be associated with the SEM and not STOF as detector, even if SEM is a subunit physically attached on top of the STOF sensor; at the moment all the SEM files available are associated with STOF hence no data are available if SEM is selected; which is confusing. If STOF is the sensor decided, all these files shall be re-ingested accordingly.
    • Possible ingestion of all (up-to-date) PM calibrated files per Carrington Rotation at 1h time resolution available at the University of Maryland: http://umtof.umd.edu/pm/crn/; please note that these files do not content any (predicted not measured) Helium speed/velocity.
    • Up-to-date SEM files 15s and daily averages are available here: https://dornsifecms.usc.edu/space-sciences-center/download-sem-data/ ; shall SSA ingest them on a daily basis?


Processing level (general remark)

There are two processing levels for all these detectors: Level 1 Science data and Level ASCII, both are unclear and shall be renamed, for instance by

Calibrated (L1) instead of « Level 1 Science Data »
Raw data instead of « Level ASCII »

Calibrated files shall be the ones selected by default when searching any data from the CELIAS sensors and their subunits.
To be discussed confirmed with the SOHO PS.
 


CTOF 

Status of CTOF data content at ESDC
CTOF sensor has produced 221 calibrated data during around 4 months (April 1 to August 16 1996) including the physical parameters
O densities
Fe densities
Heavy ion
freeze-in temperatures 

Their processing level is Level 1 Science data. A mention of the time resolution (5 minutes) would help (see HAUS 03, AI12 (i))

The other processing level is called Level ASCII (unclear name, see HAUS3-AI12(ii)). It is related the rest of the (raw) data products: Pulse Height Data called Pulse Height Analyzer, Spectral Matrix Rates called Rates and Matrix Elements called ME. Additionally, Housekeeping data files are available.


For CELIAS/CTOF

(i) CTOF 221 CDF « Level 1 Science Data » calibrated files at ESDC/SSA need to be reingested as follows

  • Processing level renamed as « Calibrated (L1) » instead of « Level 1 Science Data » 
  • Calibrated metadata set to true, essential to improve usability and accuracy of the GUI and the SAIO, also called API or command line. 
  • Temp Res keyword set to 5 mn
  • Modify Observation type metadata as follows
Observation type (actual)Updated observation typeCTOF Calibrated files affected
O densitiesO6+, O7+, O densities138 files of type SO_CEL_CTOF_ODENSITIES_L1_V01.cdf
Fe DensitiesFe, S, Si densities80 files of type SO_CEL_CTOF_FEDENSITIES_L1_V01.cdf
Fe and Proton DensityFe, p densities, Vsw, V_kinSO_CEL_CTOF_FEPDENSITY_L1_V01-199604212355.cdf
Fe freeze-in temperature

Fe freeze-in temperature

SO_CEL_CTOF_FEFTEMP_L1_V01-199605282359.cdf 
Heavy ion Freeze-in Temp…O freeze-in temperature, n_O, v_O, v_Si, v_FeSO_CEL_CTOF_HIONFTEMP_L1_V01199605290000.cdf

(ii) Re-ingest all 15,616 raw data files with a Processing level set to « Raw data » instead of « Level ASCII » which is meaningless.

(iii) The availibility of raw data files up to 7 Sep. 2018 is confusing while only 4 months of science data available (April 1 to August 16 1996); documentation would help. PI team may provide it, to be discussed between SOHO PS and PI team.

(iv) New high level data (velocity spectra of heavy ions+n,T) could be provided by the PI group (at some point), as mentioned by the reviewer. Hence, a discussion between the SOHO PS and CELIAS PI team of data delivery timeline shall be held.

MTOF 

Status of raw data content: up to 7 September 2018 (25,146 files on February 20, 2019), to be re-ingested with Processing level set to « Raw data »
Status of science data content: 1 file, SO_CEL_MTOF_CME_L1_V01-199701082357.cdf, to be re-ingested with

    • Processing level renamed as « Calibrated (L1) » instead of « Level 1 Science Data » 
    • Calibrated metadata set to true, essential to improve usability and accuracy of the GUI and the SAIO, also called API or command line. 
    • Temp Res keyword set as 5 minutes
    • Modify Observation type set to Heavy ion densties and SW param. instead of "coronal mass ejection"

Content of MTOF science data for the whole mission is 1 file (except Proton Monitor data), covering a week in 1997 (08th to 16th January), a CME event subject of several scientific publication! This file contains calibrated measurements of densities (Ne, Np, O. Mg, Si, S, Ca, Fe, Na, Al, Ar, He, C, N) and velocities (Vsw, Vkinetic). The observation type is "coronal mass ejection", which has no obvious scientific meaning in this context. According to the PI team, there does not exist any level 1 data due to instrumental problems (but counts yes with many papers on the discovery of rare elements and isotopes in the solar wind). To be further discussed with SOHO PS and PI team if any other calibrated file of this type is available.


STOF

  • Status of raw data content: up to 7 September 2018 (29,767 files on February 26, 2019), to be re-ingested with Processing level set to « Raw data »
  • Status of science data content: from 1996 to 1999, 49 monthly SEM files, 5mn average; most likely superseeded by the SDAC mission long data bundles and up-to-date data available at https://dornsifecms.usc.edu/space-sciences-center/download-sem-data/ 

Measures ions in the range from 20 keV to 3 MeV per charge through two separate sections of the electrostatic deflection system, one with curved (STOF) and one with flat deflection plates at higher energies (HSTOF). Energy measurement is done via a solid state detector. The Solar EUV Monitor (SEM) is physically mounted on top of the STOF detector. it measures the full solar disk absolute photon flux for the He II 30.4 nm line, as well as the absolute integral flux between 17 and 70 nm. The detector associated with the SEM data is STOF, hence no calibrated SEM data appear with SEM is selected as detector


HSTOF

  • Status of raw data content: up to 7 September 2018 (9,341 files on February 26, 2019), to be re-ingested with Processing level set to « Raw data »
  • Status of science data content: 1 file, SO_CEL_HSTOF_EHA_L1_V01-199602290000.cdf, to be re-ingested with
      • Processing level renamed as « Calibrated (L1) » instead of « Level 1 Science Data » 
      • Calibrated metadata set to true, essential to improve usability and accuracy of the GUI and the SAIO, also called API or command line. 
      • Modify Observation type set to Hydrogen ENA flux instead of "Energetic Neutral Hydrogen Atom"


Content of HSTOF (high energy part of STOF) science data for the whole mission is 1 CDF file, covering period from 29/02/1996 to 23/12/1997. HSTOF has been working for years and is presently the only ENA instrument with energy range overlapping with Voyager Low Energy Charged Particle measurements of the energetic ion fluxes in the heliosheath, according to this article https://www.aanda.org/articles/aa/pdf/2018/10/aa32432-17.pdf. Until 2003, the whole range of the ecliptic longitude was scanned yearly. After 2005, only the flank sectors are accessible to observations. Hence, it is not understandable that only 2 years of calibrated data only are available on the SSA. To be further investigated.


Data Completeness wrt to GSFC


Here is the status on 26 February 2019


- CTOF: Goddard 20360 files (max (imported_date) 2019-01-11 07:58:54) /  SSA 15837 files
- MTOF: Goddard 33823 files (max (imported_data) 2019-01-11 07:59:13  /  SSA 21147 files
- STOF: Goddard 20463 files (max (imported_data) 2019-01-11 07:59:31)  /  SSA 29816 files
- HSTOF: Goddard 6768 files (max (imported_data) 2019-01-11 07:59:00)  /  SSA 9342 files


Discrepencies shall be investigated


Detectors Ordering on the SSA GUI

The ordering of the CELIAS detectors on the SSA is in alphabetical order and could be improved by listing them by sensors as suggested by the reviewer from high to low energy detectors, to be discussed with the SOHO PS. At the moment there are listed alphabetically.

Alternatively it could be listed as follows

Proton Monitor
Solar EUV Monitor
CTOF
MTOF
STOF
HSTOF
DPU 

To be discussed.

 

2.3   New AIs and recommendations

Here is the list of actions related to the review of the ESDC SOHO archive related to the following SOHO instruments: CDS, COSTEP, EIT, ERNE, GOLF, LASCO, MDI, SUMER, VIRGO

HAUS 03, AI-11 on B. Fleck/UVCS PI team/ESDC, SOHO PS to contact the UVCS PI team to clarify which of the files delivered to SDAC are calibrated and transfer those files to ESDC for their ingestion and availability.

HAUS 03, AI-12 on B. Fleck/UVCS PI team/ESDC, for CELIAS/CTOF,

(i) All CTOF files of processing level "Level 1 Science data" are calibrated. Hence, these files shall be re-ingested to eventually be selectable as calibrated products. These files shall be the ones selected by default.
(ii) Observation types of the calibrated CTOF files shall better reflect their file content, including their time resolution
(iii) The processing level entitled Level ASCII is unclear and must to be corrected, even though extracted automatically from the metadata of these (raw data) files.
(iv) The availibility of raw data files after August 15, 1996 is confusing and not in line with the 4 months of science data available (April 1 to August 15 1996) .
(v) New high level data (velocity spectra of heavy ions+n,T) could be provided by the PI group (at some point). Hence a discussion of data delivery timeline shall be considered.


 

SOHO

experiment

Ingestion status

Ok means

same data content at ESAC and GSFC

Additional datasets 

Complementary datasets

Thumbnails/QL

Visualization needed?

Visualization available?

CDS

Latest data at 

SSA/ESAC: 2013-03-18

GSFC: 2013-05-02

Generate and ingest calibrated version (Level 2)[1]

None

None, possible ingestion of the "Atlas" catalogue (CDS website at RAL)

Yes

No

CELIASRe-ingestion of all GSFC data neeeded at ESACUp-to-date mission long tar files at 15s/daily time resolution (SEM) and 30s/5mn time resolution (PM) shall be made available at GSFC and then transferred to ESACUpdated CTOF calibrated dataNoneNoNo

COSTEP

Ok

Ingest new level 3 COSTEP/EPHIN (protons,helium) and documentation when available (08/2018) 

None

None

Yes

No

EIT

Ok

Calibrated version of the images shall be produced and ingested

Calibration information with Solarsoft

Thumbnails available through time animator; possible ingestion of daily EIT movies and from CME catalogues 

Yes

Yes, time animator

ERNE

Ok

Ingest new ERNE .SL2 data set with H and He intensities once available

None

None

Yes

No

GOLF

Ok

Possible ingestion of calibrated version from CEA Saclay or Nice observatory

None

None

Yes

No

LASCO

Ok

Level 1 data and Calibrated version of the images to produce & ingest

Calibration info with Solarsoft 

CME catalogues

Thumbnails available through time animator

Yes

Yes, time animator

MDI

Ok

Possible ingestion of MDI datasets available at Stanford university

HMI data when both instruments were operated

Thumbnails available through time animator

Yes

Yes, time animator

SUMER

SSA/ESAC: 2009-11-10

GSFC: 2014-10-24

Ingest calibrated version of the data available at MPS

None

Ingest thumbnails available at MPS

Yes

No

UVCSRaw data (level 1) only at ESDCCalibrated files to be ingested at ESDCNoneNoneNoNo

VIRGO

Ok

Complete

PMOD/WRC datasets to investigate

N/A

Yes

No

[1]CDS calibration: assuming single component shift for each spectral line enables to produce a calibrated version of  velocity and intensity maps

 

 3. SOHO archive at GSFC and long term archives of HINODE and IRIS

3. SOHO archive at GSFC and long term archives of HINODE and IRIS

See Annex 6

GOLF

  • Most relevant scientific products at GSFC: Level 2, 3 mission-long files (11 April 1996 – 10 April 2018), newly calibrated line-of-sight Doppler velocities files, hence 3 files; GOLF_22y_PM1 (from photmultiplier 1), GOLF_22y_PM2, GOLF_22y_MEAN with timing as good as ±5s, as sdescribed in Appourchaux et al. 2018A&A...617A.108A. Old (deprecated) data have been moved to a private area on the GSFC server and will be available upon request (but not searchable through the GUI/VSO).
  • Other products: daily photomultiplier data (not really useful for community, since raw counts)
  • Coming up: New series (with improved FITS header info) expected later this year (2019)

VIRGO

  • Most important scientific products at GSFC: Level 2, mission-long, calibrated Total Solar Irradiance (TSI) time series (daily + hourly averages)
  • Other products: daily raw data (named level 0) for TSI (PMO6 and DIARAD), SPM, and LOI
  • Coming up:Mission long, detrended SPM files (blue, red and green channel) expected as soon as FITS headers are fixed, VIRGO team also working on calibration of LOI files

MDI

  • Most important scientific products at GSFC: all MDI magnetograms, in particular the 96-min full disk synoptic magnetograms. However, no Dopplergram series are available, because these files were transferred directly to Stanford and the size of the MDI helioseismology data was considered too big at that time (MDI high-rate telemetry 160 kb/s vs 40 kb/s for rest of SOHO). MDI helioseismology series currently served by JSOC at Stanford Univ., as part of the SDO archive 
  • Coming up: to comply with long-term archiving NASA requirements, the complete SDO archive is being copied to the SDAC at GSFC, i.e. in the future the SOHO archive at GSFC will include all MDI data.

SUMER

  • Coming up: full reprocessing of SUMER data, both level-0 and level-1 (calibrated) completed by the PI team. New organization of SUMER data sets, now bundled by “studies” and provided as tar balls of individual FITS files for each exposure, and a FITS header describing the tar ball. SUMER team is currently working on some minor consistency issues and a description of the new data sets. Ingestion of new data expected very soon. Old (deprecated) data will be moved to private area on server and will be available upon request (but not searchable through GUI). SUMER off since 4 April 2017.

CDS

  • Coming up: Complete reprocessing of CDS/NIS data, both level-0 and level-1 (calibrated) completed by the PI team. However, inconsistencies detected in pointing info in headers during final quality check. Last telecon with CDS team to resolve these issues on 20 Dec 2018. Hopefully, these problems can be fixed quickly and we have the final, complete and consistent level-0 and level-1 data in the archive before summer 2019. Old (deprecated) data will be moved to private area on server and will be available upon request (but not searchable through GUI). CDS hibernated since 5 September 2014

EIT

  • Most important scientific products at GSFC: all raw data that can be calibrated with SolarSoft.
  • PI still working on “final” calibration of level-2 files, see recommendation HAUS 03, MAR-2

UVCS


  • Most important scientific products at GSFC: final calibrated data
  • No changes to UVCS data expected
  • 23 Jan 2013 end of UVCS operations

LASCO

  • No new data product


SWAN

  • No new data product


CELIAS

  • Most important scientific products at GSFC: calibrated mission-long 
    • solar wind Proton Monitor (PM) data (proton speed, density, thermal speed, direction), 5-min and 30s averages
    • SEM EUV flux (daily and 15 s averages)
  • No new data expected

COSTEP

  • Most important scientific products at GSFC: since early October 2018, new EPHIN Level-3 data product available including

- Mission-long, GEANT4 calibrated proton and helium fluxes for all four energy channels
- Daily, hourly, 30 min, 10 min, 5 min, and 1 min averages
- Served as mission-long tar balls

  • Coming up: level-3 electron data – much more difficult to calibrate, but COSTEP/EPHIN team at Kiel working on that

ERNE

  • Most important scientific products at GSFC: On 30 August 2018, the entire ERNE data set has been replaced with new, corrected data.ERNE team had noticed some errors in part of the ERNE helium data collected before 18 April 2000. For completeness and consistency, ERNE team re-processed entire ERNE data set.To avoid confusion and duplication, all old ERNE records were deleted (moved to private area of server) and the complete, new ERNE data ingested (both at GSFC and ESAC).


Long-term plans for Hinode and IRIS archive

Hinode and IRIS data currently served from Hinode SDC Europe at Univ. of Oslo
Hinode and IRIS data also served from several other places, e.g. LMSAL
Archive structure rather simple, just a directory tree with FITS files
After operational phase of Hinode, complete set of FITS files will be transferred to ESAC and ingested into heliophysics archive for long-term preservation


Summary

SOHO archive at GSFC in good shape overall

GOLF, LASCO, and SWAN teams a bit late in delivering latest data
LASCO quicklook data served to community by FTP
UVCS finalized
SUMER and CDS being finalized
Complete MDI data sets being copied to SDAC
Access to SOHO archive at GSFC through VSO and IDL/Python APIs
GUI to GSFC archive had to be taken down because of security issues in the 20+ year old Perl code we were using
Same Perl scripts also used for real-time images, movie theater, bibliographies
Some services not up yet due to limited manpower to rewrite software


 4. FP7 projects related to ESA heliophysics missions

4. FP7 projects related to ESA heliophysics missions

See Annex 7

Reviewer: V. Génot

FP7 projects reviewed at this meeting: HESPERIA, FLARECAST, SEPSERVER
FP7 projects previously: IMPEX, HELIO, HELCATS, CASSIS, Spaceinn

4.1 HESPERIA

High Energy Solar Particle Events foRecastIng and Analysis (HESPERIA), https://www.hesperia.astro.noa.gr/; H2020 project from May 2015 to April 2017. HESPERIA exploits available large datasets stored into databases including the Neutron Monitor Database (NMDB) and SEPServer that have been developed under FP7 projects from 2008 to 2013

Model 1: Release 

  • Uses the fact that electrons arrive 30 to 90 minutes faster at L1 than the protons to predict proton flux by utilizing the actual electron flux and the increase of the electron flux in the last 60 minutes
  • Based on Posner, A., Up to 1-hour forecasting of radiation hazards from solar energetic ion events with relativistic electrons, Space Weather, Vol. 5, S05001, doi:10.1029/2006SW000268, 2007
  • The model uses SOHO/EPHIN (0.25 to 1.0 MeV ) and ACE/EPAM (0.175 to 0.315 MeV) electron fluxes to forecast proton fluxes in 30, 60 and 90 minutes
  • The same model runs at NASA/GSFC with SOHO/EPHIN only (https://costep2.nascom.nasa.gov/). An alerts system is in place; registration by email

Model 2: UMASEP

  • predictor of >500 MeV SEP proton events at geostationary orbit
  • >500 MeV solar protons are so energetic that they usually have effects on the ground, producing what is called a Ground Level Event (GLE).
  • Based on [Núñez, 2011, 2015]
  • Did not work at the time of the review

Catalogue

  • Catalogue of events to investigate SEP spectrum prediction using microwave observation.
  • Investigates if the hardness or softness of the proton spectrum in interplanetary space can be predicted from the shape of the microwave spectrum (see report)
  • Catalogue with 64 events between Nov 1997 and Jan 2016
  • GLE inversion software
  • Relativistic (>500 MeV) Solar Energetic Particles (SEPs) enter the Earth’s atmosphere sporadically leading to an increase of the intensities recorded by the worldwide network of neutron monitors (NMs) on the ground. These events are known as ground level events or ground level enhancements (GLEs).
  • HESPERIA developed the first inversion software for NM observations that infers the release timescales of relativistic SEPs at or near the Sun and the characteristics of their transport in the interplanetary space. 
  • The Python code can be used to estimate the particle fluxes at 1 AU to assess e.g. radiation dose rates at flight altitudes

Potential value added product based on SOHO data

The result of REleASE code for the (30-90 min) forecasts based on electron measurements by SOHO/EPHIN. However, no obvious scripting way to access the results/plots


4.2 FLARECAST

Flare Likelihood And Region Eruption foreCASTing (FLARECAST) EU programme ran from January2015 to December2017; http://flarecast.eu/
Flare prediction API works for 2013-2017, computes predictions of flares of different GOES class (X, M, C) in different forecast windows with several algorithm used
Results are fully integrated in theAPI: https://api.flarecast.eu/
A REST API is available and can be included in (Python, IDL, JAVA) codes to embed the prediction algorithm
Can certainly be used as a prototype for future automatic detection of flares but not directly for SOHO and Ulysses post-event analysis

4.3 SEPSERVER

FP7; finished since a few years, some links already do not work
SEPServer is a tool developed for the analysis of solar energetic particle (SEP) events.
http://server.sepserver.eu/
SEPServer provides access to SEP datasets, related electromagnetic observations and browsing, plotting and analysis tools facilitating the use of these data.
Access to an “inversion program” (linux on IDL, last version from 2013) which enables to infer SEP origins by inverting processes due to detection and interplanetary transport

SEP inversion: IDL Software for Analyzing Solar Energetic Particle Events

http://server.sepserver.eu/inc/SEPinversion/index.htm
Running the program for one event produces 4 PDF (~100p) of plots
Here: 24/9/2001 for Ulysses

Certainly usefull but difficult to use without reading all the documentation and references given on the server
Edit after N. Dresing’s intervention: it would be beneficial to, at least, store the code (+Green’s functions and other useful data) as the code is unique and well documented

SEPServer value added products of interest

SOHO and ULYSSES proton event lists (2)
Associated plots (rich list)
Important note: it is necessary to register to access some of these resources; otherwise only the (not rich) catalogues are available

4.4   New AIs and recommendations


 5. Interconnecting ESA helio archives with Ground-Based data

5.Interconnecting ESA helio archives with Ground-Based data

The goal here is to ease the life of scientific for her/him to download together GB data related to ESA helio datasets as an option. ESA helio archives and heliophysics ground-based data can indeed be scientifically linked (SuparDarn and EISCAT network with Cluster, Solar Orbiter and DKIST, SOHO and GONG etc...). M. Roth presented his vision on how these archives could be linked in the future without GB data being stored at ESAC. As reminder, the SOHO HMI datasets alone are 10 times bigger in volume than the entire SOHO database for all the other SOHO experiments.

 6. ESA's Long Term Strategy for science archives discussion

6. ESA's Long Term Strategy for science archives discussion

Guido De Marchi illustrated the long-term strategy for the space science archives that was adopted by ESA in December 2018. He addressed how the previous strategy has evolved in order to reflect the many suggestions and comments submitted by internal stakeholders and by members of HAUS as well as of the Astronomy Archives Users Group and Planetary Science Archives Users Group, i.e. the three users group. The main pillars of the long-term strategy continue to be

1. enable maximum scientific exploitation of the data;
2. preserve data and information (including software and documentation) in the long term;
3. whenever possible merge the archives into the operations of the missions in order to facilitate the long-term preservation.

The new strategy stresses the importance of legacy missions, whose data and associated software must be dutifully identified and preserved, and of the role of the scientific community in general and the users group in particular in advising ESA on the priorities.


The presentation also covered new activities that will be implemented by the ESDC to support archives users, including a Science Exploitation and Preservations Platform (SEPP) and an Archival Research Programme (ARP) that had been already discussed with HAUS members at the previous meeting. By integrating information and processing assets from multiple domains into a single environment, SEPP will deliver advanced analysis and collaboration services. This will allow the ESDC to host high-level science products based on ESA science data and associated software and other legacy products generated by the community, for instance as part of EU funded programmes such as Horizon 2020 and Framework Programme 7. Similarly, by allowing science users to bring their code to the archives, SEPP will permit quick-look and analysis software to be run on the data before downloading them. The ARP is meant to help archive users, and in particular early career scientists, to learn how to make the best use of the archival data to pursue their research interests. Besides summer schools, the programme includes opportunities to visit ESAC or ESTEC to explore the data. More specific details of the ARP will be discussed with all Users Groups in the coming months.

  • No labels
Write a comment…