UidFetch not closing channel?

[glennzw]

I have the following code which works fine to search for new email by sequence numbers:

Code:

// Search for unread emails
criteria := imap.NewSearchCriteria()
criteria.WithoutFlags = []string{"\\Seen"}
seqs, err := imapclient.Search(criteria)
fmt.Println("Messages: ", seqs)

seqset := new(imap.SeqSet)
seqset.AddNum(seqs...)
section := &imap.BodySectionName{}
items := []imap.FetchItem{imap.FetchEnvelope, imap.FetchFlags, imap.FetchInternalDate, section.FetchItem()} 
messages := make(chan *imap.Message)


go func() {
	if err := imapclient.Fetch(seqset, items, messages); err != nil {
		log.Println("Error fetching emails: ", err.Error())
	}
}()

for msg := range messages {
   fmt.Printf("Message UID, SeqNun, Size: (%d, %d, %d)\n", msg.Uid, msg.SeqNum, msg.Size)
}
fmt.Println("Done")

Output:

Messages: [123 124]
Message UID, SeqNun, Size: (0, 123, 0)
Message UID, SeqNun, Size: (0, 124, 0)
Done

Now, I want to search by UID rather than sequence number, so I make two changes:

imapclient.Search to imapclient.UidSearch imapclient.Fetch to imapclient.UidFetch

The output from this small change results in the code never reaching the "Done" statement:

Output:

Messages: [131 135]
Message UID, SeqNun, Size: (131, 123, 0)
Message UID, SeqNun, Size: (135, 124, 0)
Message UID, SeqNun, Size: (131, 123, 0)

Also, strangely, it processes the first email again (UID 131).

Could anyone shed some light for me? Looks similar to this issue https://github.com/emersion/go-imap/issues/253 but there are no replies.

[glennzw]

Debugging further, it seems to be the imap.BodySectionName{} that results in the block when using the UidFetch:

// This blocks when using UidFetch, and does not return:
section := &imap.BodySectionName{}
items := []imap.FetchItem{imap.FetchEnvelope, imap.FetchFlags, imap.FetchInternalDate, section.FetchItem()} // Check this

// But this is fine, and returns
// items := []imap.FetchItem{imap.FetchEnvelope, imap.FetchFlags, imap.FetchInternalDate}

go func() {
	done <- c.UidFetch(seqset, items, messages)
}()

for msg := range messages {
	log.Println("* " + msg.Envelope.Subject)
}

Changing UidFetch to just Fetch when grabbing the body section works fine though.

Full code:

package main

import (
	"log"

	"github.com/emersion/go-imap"
	"github.com/emersion/go-imap/client"
)

func main() {
	log.Println("Connecting to server...")

	// Connect to server
	c, err := client.DialTLS("imap.gmail.com:993", nil)
	if err != nil {
		log.Fatal(err)
	}
	log.Println("Connected")

	// Don't forget to logout
	defer c.Logout()

	// Login
	if err := c.Login("username", "password"); err != nil {
		log.Fatal(err)
	}
	log.Println("Logged in")

	// Select INBOX
	c.Select("INBOX", false)

	// Search for unread emails
	criteria := imap.NewSearchCriteria()
	criteria.WithoutFlags = []string{"\\Seen"}
	seqs, err := c.UidSearch(criteria)
	//seqs2, err := imapclient.UidSearch(criteria)
	if err != nil {
		panic(err)
	}

	if len(seqs) > 0 {
		log.Println("New emails UIDS: ", seqs)
		seqset := new(imap.SeqSet)
		seqset.AddNum(seqs...)

		// This blocks, and does not return:
		section := &imap.BodySectionName{}
		items := []imap.FetchItem{imap.FetchEnvelope, imap.FetchFlags, imap.FetchInternalDate, section.FetchItem()} // Check this

		//This is fine, and returns
		//items := []imap.FetchItem{imap.FetchEnvelope, imap.FetchFlags, imap.FetchInternalDate}

		messages := make(chan *imap.Message)

		done := make(chan error, 1)
		go func() {
			done <- c.UidFetch(seqset, items, messages)
		}()

		log.Println("New messages:")

		for msg := range messages {
			log.Println("* " + msg.Envelope.Subject)
		}

		if err := <-done; err != nil {
			log.Fatal(err)
		}
	} else {
		log.Println("No new emails")
	}

	log.Println("Done!")
}

Output:

2020/04/22 12:57:41 Connecting to server...
2020/04/22 12:57:41 Connected
2020/04/22 12:57:42 Logged in
2020/04/22 12:57:43 New emails UIDS:  [131 135]
2020/04/22 12:57:43 New messages:
2020/04/22 12:57:43 * fwd: test
2020/04/22 12:57:43 * here's a test

(Never reaching the desired "Done" statement and exiting)

[glennzw]

Additional info;

• The Uid blocking condition was tested with GMail, trying now with Outlook IMAP servers and it's fine
• The Uid fetch causes a segfault if there's only one email in the GMail inbox, but not with Outlook

[emersion]

Can you obtain debug logs with Client.SetDebug(os.Stdout)?

[glennzw]

Attached are the debugging outputs from UidSearch/UidFetch vs Search/Fetch when using GMail.

non_uid_mode.txt uid_mode.txt

It looks like the logout command is missing from the Uid mode.

[glennzw]

If it's of use here's the debug output from Outlook - which succeeds for both Uid and non-Uid operations.

non_uid_mode_outlook.txt uid_mode_outlook.txt

[glennzw]

Also, here are the logs from GMail when searching/fetching via UID when there is only one unread email in the inbox - the segfault condition.

gmail_one_email_nonuid_success.txt gmail_one_email_uid_segfault.txt

The segfault seems to occur when the client, for some reason, requests the email for a second time

L45: * 125 FETCH (UID 136 FLAGS (\Seen))

[glennzw]

Any luck investigating @emersion , or anything else I can help with?

[glennzw]

I have a working hack, rather than iterating over the messages channel, we can just pop the expected number of new messages:

for i := range seqs {
	msg := <-messages
	log.Println("* " + msg.Envelope.Subject)
}

Seems I need to skip the c.Logout() otherwise the program never terminates.

Not ideal, will keep trying to figure out the root cause.

[DZariusz]

I have same issue, but above tricks doesn't work for me.

seqset := new(imap.SeqSet)
seqset.AddRange(from, to)

messages := make(chan *imap.Message)
done := make(chan error, 1)
items := []imap.FetchItem{imap.FetchEnvelope, imap.FetchRFC822}

go func() {
  logger.Debug("go func()...")
  //done <- c.Fetch(seqset, items, messages)
  done <- c.UidFetch(seqset, items, messages)

  logger.Debug("when call `c.UidFetch` code never reach here")
 // when call `c.Fetch` I got error: Error in IMAP command FETCH: Invalid messageset
}()

// its hang and never get out of `go func()`, need to use `ctrl+C` to stop it.

Client.SetDebug(os.Stdout) logs for two cases:

go-imap-case1.log go-imap-case2.log

go-imap-case2.log is not cutted at the end - it's how its logged.

# go.mod
go 1.14

require (
	github.com/emersion/go-imap v1.0.4
	github.com/emersion/go-sasl v0.0.0-20200509203442-7bfe0ed36a21
	github.com/gobuffalo/packr/v2 v2.8.0
	github.com/gorilla/mux v1.7.4
	github.com/op/go-logging v0.0.0-20160315200505-970db520ece7
	golang.org/x/net v0.0.0-20200513185701-a91f0712d120
	golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d
	google.golang.org/api v0.24.0
)

Looks like it hangs in this for and never return:

#client.go
for {
	select {
	case reply := <-replies:
		// Response handler needs to send a reply (Used for AUTHENTICATE)
		if err := c.writeReply(reply); err != nil {
			close(unregister)
			return nil, err
		}
	case <-c.loggedOut:
		// If the connection is closed (such as from an I/O error), ensure we
		// realize this and don't block waiting on a response that will never
		// come. loggedOut is a channel that closes when the reader goroutine
		// ends.
		close(unregister)
		return nil, errClosed
	case result := <-doneHandle:
		return result.status, result.err
	}
}

[glennzw]

@DZariusz It looks like you're connecting to an Apple mailserver? If so, this would mean:

GMail: ❌ Apple: ❌ Outlook: ✅

[DZariusz]

@glennzw I'm using generic mailserver on shared hosting, not Apple (don't know why Apple name is in logs... maybe because im on Mac). I tried also Gmail - no luck. I created Outlook account just to see if it will be working for me. It work-ish... c.UidFetch finalising FETCH command with success but stack in go func() as described in my previous comment. c.Fetch ends with error command but also stuck.

go-imap-outlook.com-fetch.log

[msmarino]

I may be suffering from problems related to this issue:

for testing I send and receive to the same account... tested with gmail, outlook, yahoo and VPS with postfix

Mostly, but not exclusively, the library seems to hang after Fetching an email body. full debug logs attached.

basic process:

• open connection with idle
• create sender go routine (sends continuous emails...)
• open worker connection
• on receipt of MailboxUpdate, with worker
  • search using text
  • recover subjects
  • recover mails
  • delete mails

At the end of the log the fetch does not return but sending continues...

Log refers to to a connection to the Postfix VPS

There seems to be no pattern. Sometimes running whole 2min without failures, others failing on second receipt.

TestConnection.log - Run for 1 minute

TestConnection2.log - Failed after 1st fetch

[emersion]

create sender go routine

You can't use the same imap.Client from multiple goroutines.

[msmarino]

Sorry. Lack of clearness on my behalf.

I am using 2 different connections, the idler, and the worker. Updates from the idler are acted upon by the worker.

Thank you for any and all assistance. Please don't hesitate to ask if I can give back!

[croemmich]

I'm seeing the same issue as described by @glennzw. Dug in a little bit, but can't figure out what's going on yet.

[ivy-yao]

I just solved the problem, just add imap.FetchUid to c.Fetch arguments. like:

go func() {
	done <- c.Fetch(seqset, []imap.FetchItem{imap.FetchUid, sect.FetchItem()}, messages)
	}()
for msg := range messages {
	fmt.Printf("this is uid:", msg.Uid)
}

[emersion]

Closing because this is an issue about go-imap v1. I'm now focusing on go-imap v2.

Please re-open if you can reproduce with go-imap v2.