Allow individual attachments to incoming messages to be hidden

[crowbot]

As @RichardTaylor comments often it is a particular attachment that admins would like to hide.

In order to enable this, I think we would need to:

• [ ] Add a prominence field to FoiAttachment
• [ ] Add a prominence notes field to FoiAttachment
• [ ] Either update get_attachments_for_display to consider the visibility of the attachments or (in order that we can show them or show the reason why they are hidden) leave it as it is and reference prominence and prominence notes when deciding how to display each attachment
• [ ] As with https://github.com/mysociety/alaveteli/issues/209, we need to make sure that cached versions of attachments are not accessible to users when the attachments are not.
• [ ] Decide how to display a hidden attachment when it does/doesn't have notes

[hsenag]

What's a note in this context?

We would probably normally want to have something in place of the removed attachment, e.g. giving an explanation of why it has been removed (see #423).

[crowbot]

The 'prominence notes' was the place for the explanation of why it's been hidden.

[hsenag]

Ah, all makes sense. Should have worked that out for myself!

[RichardTaylor]

Sometimes hiding isn't enough - see #2686

[petterreinholdtsen]

We have seen the need for such feature in Norway too.

[Gemmamysoc]

Belgium have also identified this need

[equivalentideas]

We've just come across this need for Right To Know as well. We'd like to replace an attachment with a redacted version to hide a person's signature who has requested it be removed.

[turukawa]

I'm more comfortable with redacting individual files and removing them entirely (rather than deleting an entire response) than I am with editing the files themselves. Editing is a problem. Redacting should be ok (and with some automated note indicating a file was removed)

[nigeljonez]

We (NZ) have faced this issue occasionally.

In the interests of transparency it'd be better to be able to hide individual attachments than full responses as useful information may still be stored in non-hidden attachments. Examples include agencies adding the requestors address(es) to cover letters but not the actual response, redaction errors, and so on.

I don't mind mucking in on this, but, taking the original task list it seems the main considerations for implementation would be...

• Do we follow the prominence options for individual messages (normal/requestor_only/hidden) or add extra?
• Do we show the filename even if the prominence is changed, if so do we make it an option to also apply the prominence to the filename?
• How do we display it to the users of the site?
• Do we need prominence for each attachment?

Answering no to the last question could be the 'neatest' solution, and would allow (from an admin view) something similar to: Tweaking the notice on the public view for when the entire message is hidden would allow (as an example):

"1 attachment is only available to the original requestor.. (login to view), 1 attachment has been hidden. Reason given: Private and confidential information given in responses. Contact us..."

Of course, if we do prominence for each attachment, then it'd mean having to enumerate each attachment (potentially with it's own icon (a document icon with an X?)) and enumerate the reason individually.

[garethrees]

@nigeljonez this would be a hugely appreciated contribution!

I think we'd want to break this in to a few PRs to help make reviewing and merging more manageable, and give us a few nice places to course-correct if we come up against any issues.

Before I dive in to this, here are a few quick answers from my POV:

Do we follow the prominence options for individual messages (normal/requestor_only/hidden) or add extra?

Yes, I think being consistent with prominence across the app would be good, so this makes sense to do.

Do we show the filename even if the prominence is changed, if so do we make it an option to also apply the prominence to the filename?

I think prominence should apply to the filename as well as the attachment content, and I think it would add too much complexity to make it worth being able to separate these out.

Do we need prominence for each attachment?

I think we should, as its often the case for us that we want to hide one of several attachments.

---

So, how would I approach this?

1. Add an admin interface for attachments
2. Add prominence and prominence_reason to FoiAttachment
3. Handle the different prominence values on the request page
4. Add the ability to manage prominence and prominence_reason through the new admin interface.

1. Add an admin interface for attachments

At the moment we only have AdminIncomingMessageController.

For this PR, I think we should add AdminFoiAttachmentsController#edit in preparation for adding editable attributes. For now, I'd just display all the attributes of FoiAttachment, and a link back to the parent IncomingMessage. Yes, we'd be using edit for essentially the behaviour of show, but we know editing attributes is coming later.

2. Add prominence and prominence_reason to FoiAttachment

Add the attributes and set sensible defaults (prominence: 'normal' and prominence_reason: nil). This will get us set up for the actual contribution of changing these.

In the PR description above we called it "prominence notes", but I think we should use the existing convention of prominence_reason for easier duck typing across the app.

Might be able to make MessageProminence more generic for this, but don't worry too much about it if not.

3. Handle the different prominence values on the request page

Now we're getting to the meat of this. In this PR I think we want to be able to change attachment prominence through the console and have the front end hide and show the attachments as appropriate.

Either update get_attachments_for_display to consider the visibility of the attachments or (in order that we can show them or show the reason why they are hidden) leave it as it is and reference prominence and prominence notes when deciding how to display each attachment

I don't have a strong opinion on this as I haven't really dug in to the code, but I think that given prominence affects front-end display, the latter approach might be best, and likely easier.

As with #209, we need to make sure that cached versions of attachments are not accessible to users when the attachments are not.

We'd also want to handle this here.

How do we display it to the users of the site?

I think the case that an attachment is hidden, we'd just show a generic attachment icon and a "filename" like "Attachment hidden", with the prominence_reason where the view / download links are. Here's a quick sketch:

<li class="attachment">
  <img class="attachment__image" alt="Attachment" src="/assets/content_type/icon_unknown.png"></a>

  <p class="attachment__name">
    This attachment has been hidden from public view
  </p>

  <p class="attachment__meta">
    Site admins have been alerted to personal data within this attachment
  </p>
</li>

We already have icon_unknown.png which I think is fine for this case. I just hacked this together through Chrome's web inspector and I think it looks okay?

Might want to have a look at how IncomingMessage prominence is handled and what we show for each state.

4. Add the ability to manage prominence and prominence_reason through the new admin interface.

Finally, once we're happy that changing prominence works as we expect, we can add the form fields to the admin interface to allow general use.

[RichardTaylor]

I had the idea of tagging requests on WhatDoTheyKnow where we might want to use this feature if it is added:

https://www.whatdotheyknow.com/search/tag:individual_attachments_to_redact/

[RichardTaylor]

Do we follow the prominence options for individual messages (normal/requestor_only/hidden) or add extra?

I think we need normal/requestor_only/hidden ..

backpage would also be useful, so we can seek to exclude a particular attachment from search engines, as a way of eg. reducing the impact of processing personal information contained within it. (Backpage currently adds headers requesting no-indexing of content).

[sallytay]

As noted on some recent inbox threads, it would have been useful to have this function to help action several GDPR RtE requests we have received. allowing for us to hide attachments rather than having to hide the whole request. Noting here for reference.

[gbp]

We need to consider what we do with the main text/html part attachment. Should setting prominence on these mean the body of the message is hidden but all other attachments remain visible?

[garethrees]

We need to consider what we do with the main text/html part attachment. Should setting prominence on these mean the body of the message is hidden but all other attachments remain visible?

Ooh, that's interesting. I'd say it's an edge case but is sometimes useful if the authority quotes a load inappropriate outgoing messages in the quoted section and the message doesn't add much value in its own right.

I think it's a case of doing the thing that feels "with the grain". Which is easier? Treating the main part as any other attachment, or giving it different mechanics?

[garethrees]

We'll also want to ensure prominence is checked in zip exports of requests.

[turukawa]

Hiding from search engines, or from users? My requests are all data-driven, and - when supported - always include a CSV or Excel file which is the critical response. I've always used WDTK as part of my data research process to ensure that these source data are public and accessible.

[garethrees]

Hiding from search engines, or from users?

Are you referring to the entire issue title, or a specific piece of he correspondence?

The general issue title refers to giving Alaveteli the moderation tools to better target hiding of material. Currently, if say, a response contains 3 attachments – with one containing a bulk accidental data breach – we'd hide the entire response (and all attachments) as a consequence. After this is complete, we'd hide only the affected attachment while we assess the breach and redact/republish a replacement.

We're not talking about reducing the discoverability of attachments (by users or search engines) as a default measure…

[turukawa]

Cool, thanks for the clarification. That's a great approach.