Overview about MAGIC articles

[pdeiml]

This issue gives an overview about all MAGIC publication which can be found here: https://magic.mpp.mpg.de/backend/publications i will update the list during the next days.

Firstly, the articles:

Reference	relevant?	info.yaml?	yaml file?
2006ApJ...648L.105A	y	y	y
2006ApJ...642L.119A	y	y	y
2006ApJ...637L..41A	y	y	y
2006ApJ...638L.101A	y	y	y
2006ApJ...639..761A	y	y	y
2006ApJ...643L..53A	y	y	y
2006Sci...312.1771A	y	y	y
2006ApJ...641L...9A	?	n	n
2007ApJ...665L..51A	y	n	n
2007ApJ...669..862A	?	n	n
2007NIMPA.583..494A	?	n	n
2007ApJ...663..125A	?	n	n
2007A&A...474..937A	y	y	y
2007ApJ...662..892A	y	y	y
2007ApJ...667..358A	?	n	n
2007ApJ...658..245A	?	n	n
2007ApJ...664L..87A	y	y	y
2007ApJ...667L..21A	y	y	y
2007ApJ...666L..17A	y	y	y
2007ApJ...654L.119A	y	y	y
2007ApJ...669.1143A	?	n	n
2008ApJ...674.1037A	y	y	y
2008Sci...320.1752M	y	y	y
2008ApJ...685L..23A	y	y	y
2008ApJ...679..428A	?	n	n
2008NIMPA.595..200D	?	n	n
2008ApJ...681..944A	?	n	n
2008ApJ...679.1029T	y	y	y
2008PhLB..668..253M	?	n	n
2008Sci...322.1221A	y	n	n
2008ApJ...684.1351A	y	y	y
2008ApJ...675L..25A	y	n	n
2008NIMPA.588..424A	?	n	n
2008ApJ...685L..71A	?	n	n
2008NIMPA.594..407A	?	n	n
2009A&A...493..467A	y	y	y
2009ApJ...697.1299A	?	n	n
2009ApJ...691L..13D	y	y	y
2009PASJ...61.1011S	y	n	n
2009ApJ...703..169A	y	y	y
2009ApJ...705.1624A	y	y	y
2009ApJ...702..266A	y	n	n
2009Sci...325..444A	?	n	n
2009ApJ...693..303A	y	y	y
2009A&A...498...83A	?	n	n
2009APh....30..293A	?	n	n
2009ApJ...704L.129A	y	y	y
2009ApJ...692L..29A	y	y	y
2009ApJ...706L..27A	y	y	y
2010A&A...515A..76A	y	y	y
2010ApJ...723L.207A	y	y	y
2010ApJ...721..843A	?	n	n
2010ApJ...710..634A	y	y	y
2010A&A...517A...5A	?	n	n
2010A&A...519A..32A	y	y	y
2010ApJ...725.1629A	?	n	n
2010A&A...524A..77A	y	y	y
2010ApJ...710..828A	?	n	n
2011ApJ...729....2A	y	y	y
2011ApJ...735L...5A	y	n	n
2011ApJ...736..131A	y	y	y
2011ApJ...729..115A	?	n	n
2011ApJ...727..129A	y	y	y
2011ApJ...730L...8A	y	y	y
2011A&A...530A...4A	y	y	y
2011ApJ...726...58A	y	y	y
2011ApJ...742...43A	y	n	n
2011JCAP...06..035A	?	n	n
2012ApJ...746..151A	y	y	y
2012A&A...541A..99A	?	n	n
2012ApJ...746...80A	y	y	y
2012A&A...539L...2A	y	y	y
2012ApJ...754L..10A	y	y	y
2012A&A...539A.118A	y	y	y
2012A&A...544A.142A	y	y	y
2012A&A...544A..75A	y	y	y
2012A&A...544A..96A	y	y	y
2012A&A...541A..13A	y	y	y
2012A&A...542A.100A	y	y	y
2012APh....35..435A	?	n	n
2012ApJ...748...46A	y	y	y
2012A&A...540A..69A	y	?	?
2012NIMPA.692..201D	?	n	n
2013A&A...552A.112A	?	n	n
2013A&A...556A..67A	y	y	y
2013A&A...549A..23A	y	n	n
2014Sci...346.1080A	y	y	y
2014A&A...563A..91A	y	y	y
2014ApJ...786..157A	y	n	n
2014MNRAS.437.3103A	?	n	n
2014A&A...571A..96M	y	y	y
2014A&A...567A..41A	?	n	n
2014A&A...567A.135A	y	y	y
2014A&A...563A..90A	y	y	y
2014A&A...565L..12A	y	n	n
2014A&A...564A...5A	y	y	y
2014A&A...568A.109A	y	n	n
2014NIMPA.742...99D	n	n	n
2014JCAP...02..008A	y	n	n
2014A&A...567L...8A	?	n	n
2014MNRAS.440..530A	?	n	n
2014A&A...569A..46A	?	n	n
2014A&A...572A.121A	?	n	n
2015ApJ...815L..23A	y	n	n
2015A&A...576A.126A	y	y	y
2015MNRAS.450.4399A	y	y	y
2015A&A...573A..50A	y	y	y
2015JHEAp...5...30A	y	y	y
2015A&A...576A..36A	?	n	n
2015MNRAS.451..739A	y	y	y
2015MNRAS.446..217A	y	y	y
2015A&A...578A..22A	?	n	n
2015A&A...582A..67A	?	n	n
2015ApJ...812...65F	n	n	n
2015arXiv150902391P	n	n	n
2016APh....72...61A	n	n	n
2016APh....72...76A	n	n	n
2016A&A...585A.133A	y	n	n
2016A&A...591A..76A	y	y	y
2016A&A...591A.138A	?	n	n
2016ApJ...819..156B	y	y	y
2016MNRAS.459.2286A	y	y	y
2016A&A...590A..24A	y	y	y
2016A&A...593A..91A	?	n	n
2016JCAP...02..039M	?	n	n
2016MNRAS.459.3271A	?	n	n
2016A&A...591A..10A	y	y	y
2016A&A...589A..33A	y	y	y
2016A&A...595A..98A	?	?	?
2016JInst..1111009.	n	?	?

| 2017MNRAS.470.4608A | ? | n | n | | 2017A&A...603A..29A | ? | n | n | | 2017A&A...603A..31A | y | y | y | | 2017A&A...602A..98A | y | n | n | | 2017MNRAS.471.1688A | ? | n | n | | 2017arXiv170803689M | y | n | n | | 2017A&A...601A..33A | n | n | n |

These articles do not have a Reference_id: https://magic.mpp.mpg.de/backend/publication/show/361 https://magic.mpp.mpg.de/backend/publication/show/369

I propose that someone should double check the papers I marked with 'not relevant' or with the questionmark which means that I am not sure whether the paper is relevant or not. Additionally, one should check whether for the relevant papers data entry is done or not, if not, please make at least a dummy yaml file with a TODO remark in the corresponding folder. Of course, edit the table entry when you did something.

[cdeil]

@pdeiml - Thanks for doing the overview and starting the MAGIC data entry!

I'm not familiar with their papers, and have no time to look, so I won't be able to help you if you have any question which papers are relevant or not. Personally I'm only interested in Galactic sources, and would give priority to sources (if any left) that don't have a single measurement in gamma-cat yet: https://github.com/cdeil/gamma-cat-status/blob/master/cat_table_printout.csv

[pdeiml]

Ok, but as soon as I start to add any data problems may arise and then I will write you my questions in the corresponding PR :-) At some point I may will have a deeper look into the papers which I marked as 'not relevant' and if am not sure, we will have to discuss.

But don't worry, this will work in some way at some point.

[cdeil]

Just to note here what we discussed on Skype.

IMO this huge task / TODO table in this issue is not maintainable. I did the same when I added HESS data, then I moved it to lists in the todo folder, then I gave up because it's hard to always remember to update.

Suggest to instead transfer the info from the table above into info.yaml files and then generate such tables and status info on the gamma-cat webpage: https://github.com/gammapy/gamma-cat/pull/155#issuecomment-326282546

For hand-written notes like "we should have all MAGIC papers up to 2016 according to this list" or something like this, we should use some docs page. Feel free to add a section to either of these pages https://gamma-cat.readthedocs.io/changes.html https://gamma-cat.readthedocs.io/stats/index.html or make a new page status_notes.rst or something like that any time.

Suggest to do this work of getting set up in a good way first (and closing this issue) before spending time doing more MAGIC data entry.

[cdeil]

@pdeiml - Just as an example I've added a few info.yaml for MAGIC publications, as well as a file other_data_collections/MAGIC/publications.yaml where we can keep track of MAGIC papers. See 75864176 .

I'm not sure if we should try to be complete in the list of MAGIC publications. There will always be many "nice to have" papers like conference proceedings and papers with upper limits, so I think the attempt to add info.yaml for all MAGIC papers or to draw a strict line which might be relevant in gamma-cat is hard to draw and we could just skip on that and simply add info.yaml for the ones we clearly want (i.e. refereed publications with good measurements that can be entered into gamma-cat).

I'll stop with gamma-cat again now. @pdeiml - Please transfer the list above into info.yaml files.

[pdeiml]

@cdeil I will be going through the input folders and check whether the reference already have a folder/ info.yaml/ dummy yaml-file. Do you prefer big PRs with creation of e.g. all 2006 references or do you prefer one PR per reference?

[cdeil]

Please make one PR adding all info.yaml files. Also I have close to zero time to review, so please check the diff yourself before submitting the PR. The main things to check are that the reference_id is a valid ADS identifier, and that the source_id is the correct one for the given paper. I usually use https://gammapy.github.io/gamma-cat/sources.html to look up source names vs source_id in gamma-cat.

[pdeiml]

I need help with these two articles:

https://arxiv.org/abs/0706.1505 There the black hole cygnus x-1 is mentioned and I don't find this source in the defined sources in gamma-cat.

https://arxiv.org/abs/astro-ph/0602231 I am not sure, whether this paper is relevant for gamma-cat. It mentions only an 'upper limit' to the flux.

[cdeil]

@pdeiml - Please add what you can, and leave the few you're unsure about somewhere in a reminder issue. Leave both of those out for now. Then we can look at / discuss them individually, or if needed talk to someone from MAGIC about.

These are upper limit papers on sources that weren't detected. So for Cygnus X-1 we'd have to add a "detected" flag to our basic source info schema first. I.e. best to postpone this to a separate issue (or just not add it for now, focus on the meat of available measurements and add those first; that's what Gernot is doing for VERITAS).

[pdeiml]

Ok, I make here an overview about the articles I am not sure about and then we can discuss later on and close this issue as soon as EVERY article was being looked at. :-)

[pdeiml]

Overwiew about publications that need further discussion:

2006ApJ...641L...9A 2007ApJ...665L..51A 2007ApJ...669..862A 2007NIMPA.583..494A 2007ApJ...663..125A 2007ApJ...667..358A 2007ApJ...658..245A 2007ApJ...669.1143A 2008ApJ...679..428A 2008NIMPA.595..200D 2008ApJ...681..944A 2008PhLB..668..253M 2008Sci...322.1221A (I am not sure, is Crab pulsar equal to Crab nebula?) 2008ApJ...675L..25A (What is the correct source_id?) 2008NIMPA.588..424A 2008ApJ...685L..71A 2008NIMPA.594..407A 2009ApJ...697.1299A 2009PASJ...61.1011S (The source is not defined in gamma-cat, I think) 2009ApJ...702..266A (No source that matches the one in the paper) 2009A&A...498...83A 2009APh....30..293A 2010ApJ...721..843A 2010A&A...517A...5A 2010ApJ...725.1629A 2010ApJ...710..828A 2011ApJ...735L...5A (Don't know which source is meant) 2011ApJ...729..115A 2011ApJ...742...43A (Crab pulsar equal to Crab Nebula?) 2011JCAP...06..035A 2012A&A...541A..99A 2012APh....35..435A 2012A&A...540A..69A (Crab pulsar equal to Crab Nebula?) 2012NIMPA.692..201D 2013A&A...552A.112A 2013A&A...549A..23A (no matching source given in gamma-cat) 2014ApJ...786..157A (no matching source given in gamma-cat) 2014MNRAS.437.3103A 2014A&A...567A..41A 2014A&A...565L..12A (Crab pulsar equal to Crab Nebula?) 2014A&A...568A.109A (no matching source given in gamma-cat) 2014NIMPA.742...99D 2014JCAP...02..008A (no matching source given in gamma-cat) 2014A&A...567L...8A 2014MNRAS.440..530A 2014A&A...569A..46A 2014A&A...572A.121A 2015ApJ...815L..23A (no matching source given in gamma-cat) 2015A&A...576A..36A 2015A&A...578A..22A 2015A&A...582A..67A 2015ApJ...812...65F 2015arXiv150902391P 2016APh....72...61A 2016APh....72...76A 2016A&A...585A.133A (Crab pulsar equal to Crab nebula?) 2016A&A...591A.138A 2016A&A...593A..91A 2016JCAP...02..039M 2016MNRAS.459.3271A 2016A&A...595A..98A 2016JInst..1111009. 2017MNRAS.470.4608A 2017A&A...603A..29A 2017A&A...602A..98A (no matching source in gamma-cat) 2017MNRAS.471.1688A 2017A&A...601A..33A 2017arXiv170803689M (no matching source in gamma-cat)

[micheledoro]

Hi, this is Michele Doro (currently publication manager of MAGIC). I can help you with MAGIC papers (or find somebody). Note you can cross-check MAGIC Publication information with the NASA ADS archive: ui.adsabs.harvard.edu/#/public-libraries/7QbEh28fSY-6jzxej_D7mw

[micheledoro]

Hi again, I hope somebody is reading this. I discussed yesterday with people from the SSDC (ASI science data center). They told me they will contact you. They started a similar catalog some time ago: http://www.asdc.asi.it/tgevcat/. To me it looks like they have already developed the "graphical" part (table and skymap), while gamma-cat is best for the repository part.

[micheledoro]

In addition, were the HESS and VERITAS data information cross-checked before entering the gamma-cat database?

[GernotMaier]

Some of the VERITAS data has been cross checked, but not all. This is indeed important.

[micheledoro]

MAGIC is going to a) cross-check previous generate fits file, and possibly regenerate fits file, b) input the re-generate fits file in the format for gamma-cat. We discussed that we would like to define precisely (more than in the past) what and how to write. Things like energy threshold, moon status, tag for spectral fit, etc. All this needs definition.

[pdeiml]

I think, we have to find an easy and in particular reliable way to maintain the overview about all relevant papers/ data and which of them have been cross checked and which not.

I thought about writing an list like: https://github.com/gammapy/gamma-cat/blob/master/todo/todo_hess_aux.md for VERITAS and MAGIC publications to have a general overview. In my opinion the list of all relevant papers should not be hided in an issue like this one, it should be somewhere in the code itself. Unfortunately, I do not have a good solution for the "what has been cross-checked" problem.

I do not have any experience with such an project and how to maintain the overview as soon as many people start to development, hence: What do you think?

[micheledoro]

@elisaprandini will also contribute to this issue with MAGIC

[cdeil]

@micheledoro @elisaprandini - It's great that you want to contribute MAGIC data to gamma-cat.

In addition, were the HESS and VERITAS data information cross-checked before entering the gamma-cat database?

Data entry goes via "pull requests", so there is some review as data is added.

Things like energy threshold, moon status, tag for spectral fit, etc. All this needs definition.

Ideally, yes, as much information as possible would be available in the files. But there also has to be a balance between usefulness of the information for most users of gamma-cat, and time spent to define what each piece of information means and to do and review the data entry.

What we've done so far is to start simple, and keep adding fields of information as the use cases arise. We can and should add more information for a given measurement / paper, I just want to caution against the goal to be complete and get the data format schema right from the start. I think we have to improve iteratively and it's best to start small and simple.

@pdeiml and I have started to write some information with the goal to help people get started with data entry for gamma-cat: http://gamma-cat.readthedocs.io/contribute/introduction.html http://gamma-cat.readthedocs.io/contribute/input.html It's still very rough, but maybe you could read over those two pages and let us know if it's enough to get your started or what is unclear.

If you want, we could also do a short Skype call where we discuss a bit and help you get started doing a first "pull request" with data entry for one MAGIC paper / source?

[micheledoro]

@elisaprandini and I have checked all MAGIC TeVCat sources against those in gamma-cat, and found that the following published papers were not included:

• http://adsabs.harvard.edu/cgi-bin/bib_query?arXiv:1401.5646
• http://adsabs.harvard.edu/cgi-bin/bib_query?arXiv:1610.09416 they both belong to blazar 1510.

Additionally we found the following unpublished discoveries:

• S2 0109+22
• OT 081
• 1ES 2037+521
• TON 0599
• TXS 0506+056

@elisaprandini and I will try to include these one first. We will later on continue with: 1/ check whether multiple papers on sources are properly accounted for 2/ check all MAGIC entries 3/ add upper limits papers