From Liisa.Juusola@fmi.fi Mon Feb 17 08:48:01 2025
Date: Mon, 17 Feb 2025 07:40:49 +0000
From: "Juusola Liisa (FMI)" <Liisa.Juusola@fmi.fi>
To: Nicolaas Gunnarsson <nico.gunnarsson@live.se>, Maria Hamrin <Maria.Hamrin@space.umu.se>
Subject: VS: VS: Advice for baseline method

Hello Nico,

are you familiar with SuperMAG data (https://supermag.jhuapl.edu/)? They have an
option for baseline removal, and they also carry out data cleaning, which might
save you a lot of time and effort if you wish to combine data from several
arrays. The majority of the data have 1 min resolution, but they also have 1 s
data.

In case you're nonetheless interested in IMAGE data, you can maybe get started
with the data stored here: 

https://ida.fairdata.fi/s/NOT_FOR_PUBLICATION_s8mkFWtFtNm5

This is an old database containing the years 1994-2018 and only Max's baseline
method has been used. A few stations from the last few years are missing. 

You can run the matlab code without field separation, but in addition to the
baseline, the combination of the grid and the epsilon value may affect the
result. The online code uses a very carefully chosen grid to avoid numerical
problems near SECS poles. The matlab code does not have this limitation. There
are other differences as well. As far as I can remember, the online code carries
out coordinate transformations numerically, whereas the matlab code performs
them analytically. If you wish to examine the effect of the baseline, you could
recalculate Audrey's plots using a quiet time interval you choose yourself at:

https://space.fmi.fi/MIRACLE/iono_2D.php

The field separation is not limited to high latitudes; you can carry it out
everywhere. However, you can only interpret the ionospheric equivalent currents
in terms of the divergence-free part of the horizontal ionospheric current
density at high latitudes. You can read more discussion on the field separation
here:

https://doi.org/10.5194/angeo-41-13-2023
https://doi.org/10.5194/angeo-41-483-2023

Kind regards,
Liisa

________________________________________________________________________________
Lähettäjä: Nicolaas Gunnarsson <nico.gunnarsson@live.se>
Lähetetty: perjantai 14. helmikuuta 2025 16.38
Vastaanottaja: Juusola Liisa (FMI) <Liisa.Juusola@fmi.fi>; Maria Hamrin
<Maria.Hamrin@space.umu.se>
Aihe: Re: VS: Advice for baseline method  

Hello again Liisa.

I just want to thank you once again for taking your time and are willing to help
me.

Most likely I will have to go through multiple years of data all around the
globe at high latitudes to investigate the differential and non-differential
equivalent currents. Meaning I will have to receive data from other databases
and not only IMAGE. I would deeply appreciate your help if you could share your
cleaned and baseline-subtracted IMAGE data for me to understand the cleaning
process for when I will have to, not only make a baseline, but also clean data
from other databases too.

The matlab code I received seems to have an option to either interpret the
magnetic field in terms of ionospheric and telluric equivalent currents, or in
ionospheric equivalent currents only. I may be misunderstanding something but
does not this imply that I could theoretically reproduce Audreys plots?

Speaking of telluric equivalent currents -
I don't fully understand exactly how it is possible to estimate the telluric
equivalent current. I am reading in "2.7.1 Separation into Internal and External
Parts, Ionospheric Multi-Spacecraft Analysis Tools" (see
https://link.springer.com/book/10.1007/978-3-030-26732-2), that it is possible
to estimate it by including the vertical magnetic disturbances. Is this due to
in the SECS method you assume that the ionospheric currents only contribute for
a divergence-free component at high magnetic latitudes?

By the way, I forgot to send a copy to Maria of this discussion last mail so I
am including her now.

Hope you have a great weekend.

Best regards,
Nico




On 2/14/25 08:44, Juusola Liisa (FMI) wrote:
      Hello Nicolaas

If you are only interested in calculating differential equivalent
currents, you could simply bypass the baseline subtraction and use the
magnetic field at epoch t1 as a baseline for t2. This will produce the
same equivalent current distribution for t2 as first subtracting a
(constant) baseline from the magnetic field data at t1 and t2, calculating
the equivalent currents, and then subtracting them.

If you want to calculate the non-differential equivalent currents, you
need a baseline. In case you are only interested in very active periods
(current densities of several hundred to thousands A/km), the baseline
selection is not so critical. You could use, for example, the quiet-time
method you mention. For more quiet periods, more care is needed with the
baseline.

What I'm myself currently using is a sliding 10-day median. I calculate
the median for the midnight of each day and then use a simple cosine curve
to interpolate over the day. The 10-day median is long enough not to
produce significant depressions during active periods but still seems to
follow the instrument drifts sufficiently. Because I'm often interested in
time derivatives, I need a baseline that does not produce large,
artificial time derivatives, such as a step-change at midnight between
baseline levels. The interpolation helps with that. Of course, the median
window could be slid over the entire day, but this is quite slow when you
have a lot of data to process.

I'd say that the main difference between Max's more complicated method and
my simple 10-day median method is that Max's method removes the diurnal
variation whereas mine does not. Not removing the diurnal variation suits
my purposes, but you may have a different need.

Implementing the baseline method is not such a big task. The most
time-consuming part is cleaning the magnetic field data from spikes,
jumps, and bad periods. I have a database of cleaned and
baseline-subtraced IMAGE data from 1994 until August 2024. For the years
1994-2014, Max's baseline method has been used, and after that, mine. I'm
planning to go back and reprocess the earlier years as well, to be
consistent, but it's slow work. How much data do you need? If you like, I
can share mine.

Also, please note that the matlab code you are using and the old online
code Audrey used are not identical. Your code, I assume, interprets the
magnetic field in terms of ionospheric and telluric equivalent currents.
The online method interprets the horizontal part of the observed magnetic
field in terms of ionospheric equivalent currents only, and uses truncated
SVD to smooth the solution.

I hope this helps!

Kind regards,
Liisa

________________________________________________________________________________
Lähettäjä: Nicolaas Gunnarsson <nico.gunnarsson@live.se>
Lähetetty: torstai 13. helmikuuta 2025 10.13
Vastaanottaja: Juusola Liisa (FMI) <Liisa.Juusola@fmi.fi>
Aihe: Advice for baseline method  
Hello Liisa.

My name is Nicolaas Gunnarsson and I am currently working on a master
thesis
for Maria Hamrin where I will investigate the ionosphere equivalent
current
from dB/dt-spikes during a non-stormy time period. I believe Maria has
been
mailing you earlier, in late december, asking about the easiest way to
compute
the differential equivalent currents.

Maria has spoken well about you and your talents about computing
equivalent currents together with the spherical elementary current systems
(SECS) and I was wondering if you could help me in my current work.

I am trying to recreate some of the plots in one of Audrey Schillings
articles called "Signatures of wedgelets over Fennoscandia during
the St Patrick’s Day Storm 2015" (https://doi.org/10.1051/swsc/2023018) by
using a MATLAB code, that I see you are one of the creators of, that
Maria provided me with. More specifically the plots in figure 3 and
figure 4.

I can somewhat recreate the differential plots in figure 4, but I am
having problem figuring out the best way to remove the data baseline to
be able
to recreate figure 3 that shows the non-differential equivalent currents.

I have found an article from M. van de Kamp where he speaks of
determining the
baseline (https://doi.org/10.5194/gi-2-289-2013). In this article he
writes
"A traditional method of removing a baseline from magne-
tometer data of a particular day is to look for a magnetically
quiet day near the day of interest, and calculate the average
value of the magnetic field of this day.".

M. van de Kamp later writes that this method provides four different
inaccuracies where the first one is:
"There may not be any day in the entire month which
is completely free from disturbances. In this case, the
“quiet” day is not really “quiet” and the data, and even
the average value, will still be affected by some distur-
bance effects." The remaining three inaccuracies also seems to be
somewhat related
to the first inaccuracy.

 From my understanding M. van de Kamp further states that
scientists use this method in some applications today mainly because of
its
simplicity even though it leads to inaccuracies. Instead of using this
method
M. van de Kamp introduces a way of determining quiet days which is
interpolated
to create a long-term baseline.

Why I am mailing you is to ask for advice about which method I should
pick.

Is it necessary take my time to create an accurate dataset from each
magnetometer stations measured data by using M. van de Kamps method? Or
can I use the
"simple and inaccurate"-method and still have a somewhat reliable
equivalent
current since I will be performing this study towards a non-stormy time?

I hope you the very best and thank you for taking the time to read this
mail.

Best regards,
Nicolaas


