inbound: publish downloaded files to stream

[adamdecaf]

PayGate Version: v0.8.0

What were you trying to do? As a user of PayGate I'd like to be notified on incoming files that PayGate downloads from the ODFI's server. This could be to process NOCs, returns, or other incoming files. Currently there's no way to get notified outside of viewing logs.

What did you expect to see? A stream of events would be handy. Deciding between webhook notifications, polling an endpoint, or consuming a stream are implementation details.

[adamdecaf]

Currently I'm thinking we have PayGate support serving these files from it's odfi.storage.local.directory directory and we ignore support for referring to an ftps://, gcs://, or similar remote endpoint. Those values are not supported as storage locations currently and so adding them is out of scope for this issue. The files would be served as their NACHA formatted format from fileURL.

Events would then be published with the following JSON schema:

{
   "eventID": "<string>",
   "eventType": "InboundFile",
   "fileURL": "https://paygate:9092/inbound/<filepath>.ach"
}

[adamdecaf]

The default stream would be in-memory using the gocloud.dev/pubsub package with kafka supported as well. As the community requests we can support additional constructors.

The configuration would mirror what we have for pipeline.stream as it'll use the same values.

[dzrtc]

Have you considered using CloudEvents schema? It's CNCF. https://cloudevents.io/

[adamdecaf]

I've never heard of CloudEvents, but cool it's a CNCF project. I'll have to look into it.

An example message: https://github.com/cloudevents/spec/blob/v1.0/spec.md#example

[dzrtc]

If paygate will serve the files over HTTP, is it feasible to use content negotiation to select application/json instead of ACH? That is, a JSON-based representation of the ACH file? This would allow downstream systems to remain semi-ignorant of ACH details.

If the project doesn't have a standard representation of an ACH file as a JSON document already, then disregard.

[adamdecaf]

We have a JSON representation for files and the moov-io/ach project standardizes that. Checking the content-type would work to serve both formats.

[adamdecaf]

I setup a pull request that creates events like the following. https://github.com/moov-io/paygate/pull/532 I need to check if we are okay with this schema going forward.

{    
    "EventType": "CorrectionFile",
    "EventID": "0b0dd47d61e2a4d5740ac699fad79c72c97aaa59",
    "FileURL": "http://localhost:9092/inbound/cor-c01.ach"
}

Edit: There should probably be a way to get the JSON form of that admin url.